Add profile for ARM compatibility #1425

pinin4fjords · 2024-10-22T13:15:22Z

Add a config profile for ARM, referencing ARM-specific containers
Update the Trim Galore module with a new conda build with an ARM-compatible cutadapt build

PR checklist

Signed-off-by: Pablo <pablo.aledo@seqera.io>

Sw to conf

Triage arm deps

Backport support for Graviton 4 to version 3.16.1

github-actions · 2024-10-22T13:20:12Z

`nf-core pipelines lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 424137a

+| ✅ 186 tests passed       |+
#| ❔   9 tests were ignored |#
!| ❗   8 tests had warnings |!

❗ Test warnings:

files_exist - File not found: assets/multiqc_config.yml
files_exist - File not found: .github/workflows/awstest.yml
files_exist - File not found: .github/workflows/awsfulltest.yml
pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
pipeline_todos - TODO string in base.config: Check the defaults for all processes
pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline

❔ Tests ignored:

files_exist - File is ignored: conf/modules.config
nextflow_config - Config default ignored: params.ribo_database_manifest
files_unchanged - File ignored due to lint config: assets/email_template.html
files_unchanged - File ignored due to lint config: assets/email_template.txt
files_unchanged - File ignored due to lint config: .gitignore or .prettierignore
actions_ci - actions_ci
actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/rnaseq/rnaseq/.github/workflows/awstest.yml
multiqc_config - multiqc_config
modules_config - modules_config

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-rnaseq_logo_light.png
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-rnaseq_logo_light.png
files_exist - File found: docs/images/nf-core-rnaseq_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: conf/igenomes_ignored.config
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-rnaseq_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowRnaseq.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Found nf-schema plugin
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: validation.help.enabled
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable found: validation.help.beforeText
nextflow_config - Config variable found: validation.help.afterText
nextflow_config - Config variable found: validation.help.command
nextflow_config - Config variable found: validation.summary.beforeText
nextflow_config - Config variable found: validation.summary.afterText
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config variable (correctly) not found: params.max_cpus
nextflow_config - Config variable (correctly) not found: params.max_memory
nextflow_config - Config variable (correctly) not found: params.max_time
nextflow_config - Config variable (correctly) not found: params.validationFailUnrecognisedParams
nextflow_config - Config variable (correctly) not found: params.validationLenientMode
nextflow_config - Config variable (correctly) not found: params.validationSchemaIgnoreParams
nextflow_config - Config variable (correctly) not found: params.validationShowHiddenParams
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 3.17.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.hisat2_build_memory= 200.GB
nextflow_config - Config default value correct: params.gtf_extra_attributes= gene_name
nextflow_config - Config default value correct: params.gtf_group_features= gene_id
nextflow_config - Config default value correct: params.featurecounts_group_type= gene_biotype
nextflow_config - Config default value correct: params.featurecounts_feature_type= exon
nextflow_config - Config default value correct: params.igenomes_base= s3://ngi-igenomes/igenomes/
nextflow_config - Config default value correct: params.trimmer= trimgalore
nextflow_config - Config default value correct: params.min_trimmed_reads= 10000
nextflow_config - Config default value correct: params.umitools_extract_method= string
nextflow_config - Config default value correct: params.umitools_grouping_method= directional
nextflow_config - Config default value correct: params.aligner= star_salmon
nextflow_config - Config default value correct: params.pseudo_aligner_kmer_size= 31
nextflow_config - Config default value correct: params.min_mapped_reads= 5.0
nextflow_config - Config default value correct: params.kallisto_quant_fraglen= 200
nextflow_config - Config default value correct: params.kallisto_quant_fraglen_sd= 200
nextflow_config - Config default value correct: params.stranded_threshold= 0.8
nextflow_config - Config default value correct: params.unstranded_threshold= 0.1
nextflow_config - Config default value correct: params.deseq2_vst= true
nextflow_config - Config default value correct: params.rseqc_modules= bam_stat,inner_distance,infer_experiment,junction_annotation,junction_saturation,read_distribution,read_duplication
nextflow_config - Config default value correct: params.bracken_precision= S
nextflow_config - Config default value correct: params.skip_bbsplit= true
nextflow_config - Config default value correct: params.skip_preseq= true
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
nextflow_config - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/7f1614baeb0ddf66e60be78c3d9fa55440465ac8/
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-rnaseq_logo_light.png matches the template
files_unchanged - docs/images/nf-core-rnaseq_logo_light.png matches the template
files_unchanged - docs/images/nf-core-rnaseq_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
readme - README Nextflow minimum version badge matched config. Badge: 24.04.2, Config: 24.04.2
readme - README Zenodo placeholder was replaced with DOI.
plugin_includes - No wrong validation plugin imports have been found
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (0 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: template_version_comment.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: cloud_tests_small.yml
actions_schema_validation - Workflow validation passed: cloud_tests_full.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 3.0.2

Run details

nf-core/tools version 3.0.2
Run at 2024-10-23 12:56:09

maxulysse

Looking good. How did you generate the config file?
Nextflow inspect?
Do we test for this anywhere?

docs/usage.md

conf/arm.config

Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>

drpatelh · 2024-10-23T10:27:02Z

docs/usage.md

+    -profile docker,arm
+```
+
+This will use ARM-compatible containers, and apply a small number of overrides to Conda definitions to support ARM operation.


Suggested change

This will use ARM-compatible containers, and apply a small number of overrides to Conda definitions to support ARM operation.

This will use ARM-compatible containers, and apply a small number of overrides to Conda definitions to support ARM operations.

docs/usage.md

drpatelh · 2024-10-23T10:28:14Z

docs/usage.md

+:::warning
+Please note that the ARM profile is experimental. It is expected to function correctly in all cases unless explicitly indicated otherwise—currently, exceptions include the use of the hisat2 aligner and contaminant screening via kraken2. However, because testing is presently conducted manually, we cannot guarantee its reliability.
+:::



Suggested change

:::warning

Please note that the ARM profile is experimental. It is expected to function correctly in all cases unless explicitly indicated otherwise—currently, exceptions include the use of the hisat2 aligner and contaminant screening via kraken2. However, because testing is presently conducted manually, we cannot guarantee its reliability.

:::

Maybe we just link to the new section we created in the arm profile above? Means we don't need to repeat the text.

drpatelh

LGTM! 🚀

conf/arm.config

…rm_3.16.1

pinin4fjords · 2024-10-23T12:52:54Z

Thanks @drpatelh !

pabloaledo and others added 23 commits October 17, 2024 09:26

appy changes for ARM

d59ecf7

Signed-off-by: Pablo <pablo.aledo@seqera.io>

Merge branch 'arm_3.16.1' into dev

991637c

Fix up config

4bdae48

Reset environment ymls

c9d616b

Reset main.nfs with software changes only

9f640ab

Link sw configs to arm profile

00cff4c

Add process block

f281e0b

note pins

f5208dd

Remove conda overrides that were having no impact

ea8b9e8

Use different RSEQC pin

db02d99

update BEDGRAPHTOBIGWIG pin

4a8ed2b

Fix annotation

b86a6f9

Don't think we need the RSEQC pin

1b96eb8

correct ucsc

4e5d7a3

Merge pull request #1 from pinin4fjords/sw_to_conf

bbd3452

Sw to conf

Triage arm deps

902367e

Non-igenomes STAR can use the latest star

328d02d

remove rogue comment

2f92bfc

update arm conf

0a563dc

Cut out module redundant software overrides

0be2d6a

Merge pull request #2 from pinin4fjords/triage_arm

bba6b92

Triage arm deps

Merge pull request #1414 from pabloaledo/dev

56f1a47

Backport support for Graviton 4 to version 3.16.1

Merge branch 'dev' into arm_3.16.1

a5e831f

pinin4fjords mentioned this pull request Oct 22, 2024

Backport support for Graviton 4 to version 3.16.1 #1414

Merged

11 tasks

pinin4fjords added 3 commits October 22, 2024 17:42

Add frozen ARM builds

ed026be

Container directives for conda overrides

a8d514f

Temporary Trimgalore override, singularity fixes for rsem processes

8e14711

maxulysse reviewed Oct 22, 2024

View reviewed changes

update docs

161cf95

drpatelh reviewed Oct 23, 2024

View reviewed changes

docs/usage.md Outdated Show resolved Hide resolved

drpatelh reviewed Oct 23, 2024

View reviewed changes

conf/arm.config Outdated Show resolved Hide resolved

drpatelh reviewed Oct 23, 2024

View reviewed changes

conf/arm.config Outdated Show resolved Hide resolved

drpatelh reviewed Oct 23, 2024

View reviewed changes

conf/arm.config Outdated Show resolved Hide resolved

drpatelh reviewed Oct 23, 2024

View reviewed changes

conf/arm.config Outdated Show resolved Hide resolved

drpatelh reviewed Oct 23, 2024

View reviewed changes

conf/arm.config Outdated Show resolved Hide resolved

drpatelh reviewed Oct 23, 2024

View reviewed changes

conf/arm.config Show resolved Hide resolved

pinin4fjords and others added 6 commits October 23, 2024 10:38

Apply suggestions from code review

d3580a4

Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>

Apply suggestions from code review

32f7d32

Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>

Add notes for kraken2/ braken

7df162f

Update docs

642d2a8

prettier

d9e534d

lint fix

d4fcb56

drpatelh reviewed Oct 23, 2024

View reviewed changes

docs/usage.md Outdated Show resolved Hide resolved

drpatelh reviewed Oct 23, 2024

View reviewed changes

pinin4fjords and others added 2 commits October 23, 2024 10:36

[skip ci] Reorg ARM bits

896dd74

Update arm.config

314ea85

drpatelh approved these changes Oct 23, 2024

View reviewed changes

pditommaso reviewed Oct 23, 2024

View reviewed changes

conf/arm.config Show resolved Hide resolved

pinin4fjords and others added 5 commits October 23, 2024 11:59

Fix syntax error

251481f

Merge branch 'arm_3.16.1' of github.com:nf-core/rnaseq into arm_3.16.1

cde5fef

revert trimgalore tweak to appease linter

f59132a

Update pipeline-level tests

40f05c5

Merge branch 'arm_3.16.1' of https://github.com/nf-core/rnaseq into a…

2156f5d

…rm_3.16.1

pinin4fjords added 2 commits October 23, 2024 13:54

Update CHANGELOG

79370d3

Merge branch 'arm_3.16.1' of github.com:nf-core/rnaseq into arm_3.16.1

424137a

pinin4fjords merged commit fb057c9 into dev Oct 23, 2024
32 checks passed

pinin4fjords deleted the arm_3.16.1 branch October 23, 2024 13:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add profile for ARM compatibility #1425

Add profile for ARM compatibility #1425

pinin4fjords commented Oct 22, 2024 •

edited

Loading

github-actions bot commented Oct 22, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

maxulysse left a comment

drpatelh Oct 23, 2024

drpatelh Oct 23, 2024

drpatelh Oct 23, 2024

drpatelh left a comment

pinin4fjords commented Oct 23, 2024

	This will use ARM-compatible containers, and apply a small number of overrides to Conda definitions to support ARM operation.
	This will use ARM-compatible containers, and apply a small number of overrides to Conda definitions to support ARM operations.

	:::warning
	Please note that the ARM profile is experimental. It is expected to function correctly in all cases unless explicitly indicated otherwise—currently, exceptions include the use of the hisat2 aligner and contaminant screening via kraken2. However, because testing is presently conducted manually, we cannot guarantee its reliability.
	:::

Add profile for ARM compatibility #1425

Add profile for ARM compatibility #1425

Conversation

pinin4fjords commented Oct 22, 2024 • edited Loading

PR checklist

github-actions bot commented Oct 22, 2024 • edited Loading

nf-core pipelines lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

maxulysse left a comment

Choose a reason for hiding this comment

drpatelh Oct 23, 2024

Choose a reason for hiding this comment

drpatelh Oct 23, 2024

Choose a reason for hiding this comment

drpatelh Oct 23, 2024

Choose a reason for hiding this comment

drpatelh left a comment

Choose a reason for hiding this comment

pinin4fjords commented Oct 23, 2024

pinin4fjords commented Oct 22, 2024 •

edited

Loading

github-actions bot commented Oct 22, 2024 •

edited

Loading

`nf-core pipelines lint` overall result: Passed ✅ ⚠️