Skip to content

merge to main pre transfer#35

Open
nickp60 wants to merge 178 commits intomainfrom
dev
Open

merge to main pre transfer#35
nickp60 wants to merge 178 commits intomainfrom
dev

Conversation

@nickp60
Copy link
Copy Markdown
Collaborator

@nickp60 nickp60 commented Mar 25, 2026

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs
  • If necessary, also make a PR on the nf-core/funcprofiler branch on the nf-core/test-datasets repository.
  • Make sure your code lints (nf-core pipelines lint).
  • Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
  • Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

nickp60 and others added 30 commits September 29, 2025 17:58
  - Remove log file from snapshot check, as it contains timestamps and
    will always fail
  - Same as last commit, remove log and biom files from snapshot check
    (they are not stable)
  - Add missing schema to allOf
  - Run 'nf-core pipelines lint --fix files_unchanged' to fix linting
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 10, 2026

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 8aa5eb9

+| ✅ 210 tests passed       |+
#| ❔   2 tests were ignored |#
!| ❗  53 tests had warnings |!
Details

❗ Test warnings:

  • readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
  • pipeline_todos - TODO string in README.md: Include a figure that guides the user through the major workflow steps. Many nf-core
  • pipeline_todos - TODO string in README.md: If applicable, make list of people who have also contributed
  • pipeline_todos - TODO string in README.md: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file.
  • pipeline_todos - TODO string in README.md: Add bibliography of tools and data used in your pipeline
  • pipeline_todos - TODO string in nextflow.config: Optionally, you can add a pipeline-specific nf-core config at https://github.com/nf-core/configs
  • pipeline_todos - TODO string in nextflow.config: Update the field with the details of the contributors to your pipeline. New with Nextflow version 24.10.0
  • pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required
  • pipeline_todos - TODO string in nextflow.config: Specify any additional parameters here
  • pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
  • pipeline_todos - TODO string in test_full.config: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA)
  • pipeline_todos - TODO string in test_full.config: Give any required params for the test so that command line flags are not needed
  • pipeline_todos - TODO string in base.config: Check the defaults for all processes
  • pipeline_todos - TODO string in base.config: Customise requirements for specific processes.
  • pipeline_todos - TODO string in main.nf: See section in main README for further information regarding finding and adding container addresses to the section below.
  • pipeline_todos - TODO string in main.nf: Named file extensions MUST be emitted for ALL output channels
  • pipeline_todos - TODO string in main.nf: A stub section should mimic the execution of the original module as best as possible
  • pipeline_todos - TODO string in main.nf: If the module doesn't use arguments ($args), you SHOULD remove:
  • pipeline_todos - TODO string in meta.yml: # Add a description of the module and list keywords
  • pipeline_todos - TODO string in meta.yml: #Add a description and other details for the software below
  • pipeline_todos - TODO string in meta.yml: ##Add a description of all of the variables used as input
  • pipeline_todos - TODO string in meta.yml: ##Add a description of all of the variables used as output
  • pipeline_todos - TODO string in main.nf.test: Once you have added the required tests, please run the following command to build this file:
  • pipeline_todos - TODO string in main.nf.test: Change the test name preferably indicating the test-data and file-format used
  • pipeline_todos - TODO string in main.nf.test: If you are created a test for a chained module
  • pipeline_todos - TODO string in main.nf.test: define inputs of the process here. Example:
  • pipeline_todos - TODO string in main.nf.test: Change the test name preferably indicating the test-data and file-format used but keep the " - stub" suffix.
  • pipeline_todos - TODO string in main.nf.test: define inputs of the process here. Example:
  • pipeline_todos - TODO string in main.nf: If in doubt look at other nf-core/modules to see how we are doing things! :)
  • pipeline_todos - TODO string in main.nf: A module file SHOULD only define input and output files as command-line parameters.
  • pipeline_todos - TODO string in main.nf: Software that can be piped together SHOULD be added to separate module files
  • pipeline_todos - TODO string in main.nf: Optional inputs are not currently supported by Nextflow. However, using an empty
  • pipeline_todos - TODO string in main.nf: See section in main README for further information regarding finding and adding container addresses to the section below.
  • pipeline_todos - TODO string in main.nf: input:Where applicable all sample-specific information e.g. "id", "single_end", "read_group"
  • pipeline_todos - TODO string in main.nf: Where applicable please provide/convert compressed files as input/output
  • pipeline_todos - TODO string in main.nf: Named file extensions MUST be emitted for ALL output channels
  • pipeline_todos - TODO string in main.nf: List additional required output channels/values here
  • pipeline_todos - TODO string in main.nf: Where possible, a command MUST be provided to obtain the version number of the software e.g. 1.10
  • pipeline_todos - TODO string in main.nf: It MUST be possible to pass additional parameters to the tool as a command-line string via the "task.ext.args" directive
  • pipeline_todos - TODO string in main.nf: If the tool supports multi-threading then you MUST provide the appropriate parameter
  • pipeline_todos - TODO string in main.nf: Please replace the example samtools command below with your module's command
  • pipeline_todos - TODO string in main.nf: Please indent the command appropriately (4 spaces!!) to help with readability ;)
  • pipeline_todos - TODO string in main.nf: A stub section should mimic the execution of the original module as best as possible
  • pipeline_todos - TODO string in main.nf: If the module doesn't use arguments ($args), you SHOULD remove:
  • pipeline_todos - TODO string in meta.yml: # Add a description of the module and list keywords
  • pipeline_todos - TODO string in meta.yml: #Add a description and other details for the software below
  • pipeline_todos - TODO string in meta.yml: ##Add a description of all of the variables used as input
  • pipeline_todos - TODO string in meta.yml: ##Add a description of all of the variables used as output
  • pipeline_todos - TODO string in main.nf.test: define inputs of the process here. Example:
  • pipeline_todos - TODO string in main.nf.test: Change the test name preferably indicating the test-data and file-format used but keep the " - stub" suffix.
  • pipeline_todos - TODO string in main.nf.test: //Add all required assertions to verify the test output.
  • local_component_structure - concatall.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - profiling.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure

❔ Tests ignored:

  • files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
  • files_unchanged - File ignored due to lint config: assets/nf-core-funcprofiler_logo_light.png

✅ Tests passed:

Run details

  • nf-core/tools version 3.5.2
  • Run at 2026-04-14 07:20:43

@miraep8
Copy link
Copy Markdown
Contributor

miraep8 commented Apr 11, 2026

Well, at least it doesn't seem to be failing due to lack of space anymore... but I am a bit flummoxed as to why those snapshots (presumably) match for the docker test but not for the singularity one 🤔

@vinisalazar
Copy link
Copy Markdown
Collaborator

@miraep8 I suspect what might be going on is that for some of these outputs, 9/10 times the tool generates a byte-per-byte identical output, but every so often a quirk happens that changes a single character or byte and changes the checksum. Perhaps we could check for file presence rather than checksum for these particular files?

  Files generated by mifaser can sometimes not be byte-by-byte identical, failing snapshot checksum assertions.

  This changes makes it so test assertions check only filename and existence. This should probably be fixed once
  mifaser is added to nf-core/modules.

  See #35 for discussion.
  Temporary solution until these modules (humann, mifaser) are added to nf-core/modules and their tests are sorted.
@vinisalazar
Copy link
Copy Markdown
Collaborator

@miraep8 you probably already knew that, but separate issue is that sometimes singularity pull is timing out > doesn't generate outputs > they are missing from the snapshot.

@nickp60
Copy link
Copy Markdown
Collaborator Author

nickp60 commented Apr 14, 2026

Hmm, I think we've got two separate issues here: singularity humann is returning empty files (with the hash d41d8cd98f00b204e9800998ecf8427e) , and it looks like we have something funny with mifaser as well. I don't think we can just check file presence absence until we understand the differences a bit better.

@nickp60
Copy link
Copy Markdown
Collaborator Author

nickp60 commented Apr 14, 2026

There are some snapshot empty files as well, which we will need to fix.

@nickp60
Copy link
Copy Markdown
Collaborator Author

nickp60 commented Apr 14, 2026

There are some snapshot empty files as well, which we will need to fix.

Oh some, maybe all, of these are stub outputs so thats expected

@vinisalazar
Copy link
Copy Markdown
Collaborator

Hmm, I think we've got two separate issues here: singularity humann is returning empty files (with the hash d41d8cd98f00b204e9800998ecf8427e) , and it looks like we have something funny with mifaser as well. I don't think we can just check file presence absence until we understand the differences a bit better.

Just to confirm, this is happening even when the HUMAnN image is pulled successfully? I thought this was only when the singularity image pull timed out.

The fact that both of the errors are happening in local modules makes me think that perhaps we should try and prioritise adding them to nf-core/modules? That way these problems can be investigated more closely, and separated from the scope of the pipeline.

@vinisalazar
Copy link
Copy Markdown
Collaborator

@nickp60 in face of the recent discussions that HUMAnN and mifaser may be generating incorrect/empty files, I may revert some of the commits I did here, if that's okay.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants