diff --git a/docs/_static/report-resources-min.png b/docs/_static/report-resources-min.png index 10fecfb45b..60436f2699 100644 Binary files a/docs/_static/report-resources-min.png and b/docs/_static/report-resources-min.png differ diff --git a/docs/_static/report-summary-min.png b/docs/_static/report-summary-min.png index 9e804c161d..de50d8bbda 100644 Binary files a/docs/_static/report-summary-min.png and b/docs/_static/report-summary-min.png differ diff --git a/docs/_static/report-tasks-min.png b/docs/_static/report-tasks-min.png index 7b25c2c3c8..c376b44c9d 100644 Binary files a/docs/_static/report-tasks-min.png and b/docs/_static/report-tasks-min.png differ diff --git a/docs/_static/timeline-min.png b/docs/_static/timeline-min.png index 347f5a8ceb..a0f177ccd2 100644 Binary files a/docs/_static/timeline-min.png and b/docs/_static/timeline-min.png differ diff --git a/docs/reference/cli.md b/docs/reference/cli.md index bf700de6bf..3861cac92a 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -1011,6 +1011,133 @@ The `log` command is used to query the execution metadata associated with pipeli `-f, -fields` : Comma-separated list of fields to include in the printed log. Use the `-l` option to see the list of available fields. +: Available fields: + + `task_id` + : Task ID. + + `hash` + : Task hash code. + + `native_id` + : Task ID given by the underlying execution system e.g. POSIX process PID when executed locally, job ID when executed by a grid engine, etc. + + `process` + : Nextflow process name. + + `tag` + : User provided identifier associated with this task. + + `name` + : Task name. + + `status` + : Task status. Options: `NEW`, `SUBMITTED`, `RUNNING`, `COMPLETED`, `FAILED`, and `ABORTED`. + + `exit` + : POSIX process exit status. + + `module` + : Environment module used to run the task. + + `container` + : Docker image name used to execute the task. + + `cpus` + : The CPUs number request for the task execution. + + `time` + : The time request for the task execution + + `disk` + : The disk space request for the task execution. + + `memory` + : The memory request for the task execution. + + `attempt` + : Attempt at which the task completed. + + `submit` + : Timestamp when the task has been submitted. + + `start` + : Timestamp when the task execution has started. + + `complete` + : Timestamp when task execution has completed. + + `duration` + : Time elapsed to complete since the submission. + + `realtime` + : Task execution time i.e. delta between completion and start timestamp. + + `queue` + : The queue that the executor attempted to run the process on. + + `%cpu` + : Percentage of CPU used by the process. + + `%mem` + : Percentage of memory used by the process. + + `rss` + : Real memory (resident set) size of the process. Equivalent to `ps -o rss` . + + `vmem` + : Virtual memory size of the process. Equivalent to `ps -o vsize` . + + `peak_rss` + : Peak of real memory. Data is read from field `VmHWM` in `/proc/$pid/status` file. + + `peak_vmem` + : Peak of virtual memory. Data is read from field `VmPeak` in `/proc/$pid/status` file. + + `rchar` + : Number of bytes the process read, using any read-like system call from files, pipes, tty, etc. Data is read from `/proc/$pid/io`. + + `wchar` + : Number of bytes the process wrote, using any write-like system call. Data is read from `/proc/$pid/io`. + + `syscr` + : Number of read-like system call invocations that the process performed. Data is read from `/proc/$pid/io`. + + `syscw` + : Number of write-like system call invocations that the process performed. Data is read from `/proc/$pid/io`. + + `read_bytes` + : Number of bytes the process directly read from disk. Data is read from `/proc/$pid/io`. + + `write_bytes` + : Number of bytes the process originally dirtied in the page-cache (assuming they will go to disk later). Data is read from `/proc/$pid/io`. + + `vol_ctxt` + : Number of voluntary context switches. Data is read from field `voluntary_ctxt_switches` in `/proc/$pid/status` file. + + `inv_ctxt` + : Number of involuntary context switches. Data is read from field `nonvoluntary_ctxt_switches` in `/proc/$pid/status` file. + + `env` + : The variables defined in task execution environment. + + `workdir` + : The directory path where the task was executed. + + `script` + : The task command script. + + `scratch` + : The value of the process `scratch` directive. + + `error_action` + : The action applied on error for task failure. + + `hostname` + : The host on which the task was executed. Supported only for the Kubernetes executor yet. Activate with `k8s.fetchNodeName = true` in the Nextflow config file. + + `cpu_model` + : The name of the CPU model used to execute the task. This data is read from`/proc/cpuinfo`. `-F, -filter` : Filter log entries by a custom expression, e.g. `process =~ /hello.*/ && status == 'COMPLETED'`. @@ -1344,10 +1471,32 @@ The `run` command is used to execute a local pipeline script or remote pipeline : Use the specified Conda environment package or file (must end with `.yml` or `.yaml`) `-with-dag` (`dag-.html`) -: Create pipeline DAG file. -: :::{versionchanged} 23.10.0 - The default format was changed from `dot` to `html`. - ::: +: Create pipeline DAG file. Default format: `html`. +: Supported file formats: + + `dot` + : Graphviz [DOT](http://www.graphviz.org/content/dot-language) file. + + `gexf` + : Graph Exchange XML file (Gephi). + + `html` + : HTML file with Mermaid diagram. + + `mmd` + : Mermaid diagram. + + `pdf` + : *Requires [Graphviz](http://www.graphviz.org).* + : Graphviz PDF file. + + `png` + : *Requires [Graphviz](http://www.graphviz.org)* + : Graphviz PNG file. + + `svg` + : *Requires [Graphviz](http://www.graphviz.org)* + : Graphviz SVG file. `-with-docker` : Enable process execution in a Docker container. diff --git a/docs/reports.md b/docs/reports.md index f5fc2b2020..b557cb6c8e 100644 --- a/docs/reports.md +++ b/docs/reports.md @@ -2,68 +2,88 @@ # Reports +Nextflow provides several reporting mechanisms to monitor, debug, and analyze pipeline executions: + +- [**Execution log**](#execution-log): Query past pipeline runs without re-executing them +- [**Execution report**](#execution-report): Generate an HTML report with resource usage metrics and task details +- [**Trace file**](#trace-file): Create a file with detailed task metrics for custom analysis +- [**Execution timeline**](#execution-timeline): Display a visual timeline showing task execution and scheduling +- [**Workflow diagram**](#workflow-diagram): Produce a graph visualization of pipeline structure + (execution-log)= ## Execution log -The `nextflow log` command shows information about executed pipelines in the current folder: +The `nextflow log` command helps you debug pipelines and inspect execution metadata. You can query by run name or session ID, and the command provides multiple options to filter and format the output. ```bash -nextflow log [options] +nextflow log [options] ``` -:::{note} -Both the [execution report](#execution-report) and the [trace file](#trace-file) must be specified when the pipeline is first called. By contrast, the `log` option is useful after a pipeline has already run and is available for every executed pipeline. -::: - -By default, `log` prints the list of executed pipelines: +By default, `log` prints a list of pipeline runs: ```console $ nextflow log TIMESTAMP RUN NAME SESSION ID COMMAND -2016-08-01 11:44:51 grave_poincare 18cbe2d3-d1b7-4030-8df4-ae6c42abaa9c nextflow run hello -2016-08-01 11:44:55 small_goldstine 18cbe2d3-d1b7-4030-8df4-ae6c42abaa9c nextflow run hello -resume -2016-08-01 11:45:09 goofy_kilby 0a1f1589-bd0e-4cfc-b688-34a03810735e nextflow run rnatoy -with-docker +2025-12-09 11:43:18 1s naughty_heisenberg OK 2ce0b0e294 bae65ec6-a2b9-49bf-b63e-2fec91945e48 nextflow run hello +2025-12-09 11:43:28 931ms thirsty_swanson OK 2ce0b0e294 bae65ec6-a2b9-49bf-b63e-2fec91945e48 nextflow run hello -resume +2025-12-09 11:43:37 8.5s goofy_kilby OK ca20a6dfd2 d0a60572-076e-451a-b10e-16059ed77e36 nextflow run rnaseq-nf -profile conda ``` -Specifying a run name or session id prints tasks executed by that pipeline run: +You can specify a run name or session ID prints tasks executed by that pipeline run: ```console $ nextflow log goofy_kilby -/Users/../work/0b/be0d1c4b6fd6c778d509caa3565b64 -/Users/../work/ec/3100e79e21c28a12ec2204304c1081 -/Users/../work/7d/eb4d4471d04cec3c69523aab599fd4 -/Users/../work/8f/d5a26b17b40374d37338ccfe967a30 -/Users/../work/94/dfdfb63d5816c9c65889ae34511b32 +/Users/user/workspace/work/e1/fc2d06782f8263476426c576888033 +/Users/user/workspace/work/72/f0542cfa81ad6abeaf1adf02e5cc2b +/Users/user/workspace/work/83/009346a3958358bb704e996d935c7a +/Users/user/workspace/work/84/15e4cd16df1db6f9bc0e22cc05316d ``` +:::{note} +The `nextflow log` command works retroactively and does not require advance configuration. In contrast, execution reports, trace files, and timelines must be enabled with their respective flags (`-with-report`, `-with-trace`, `-with-timeline`) when running the pipeline. +::: + ### Customizing fields -By default, only the task execution paths are printed. A custom list of fields to print can be provided via the `-f` (`-fields`) option. For example: +The `-f` (`-fields`) option specifies a custom list of log fields to include in a printed log. + +```bash +nextflow log -f [fields] +``` + +You can set a custom list of fields to focus on specific information. For example, `hash`, `name`, `exit`, and `status`: ```console $ nextflow log goofy_kilby -f hash,name,exit,status -0b/be0d1c buildIndex (ggal_1_48850000_49020000.Ggal71.500bpflank) 0 COMPLETED -ec/3100e7 mapping (ggal_gut) 0 COMPLETED -7d/eb4d44 mapping (ggal_liver) 0 COMPLETED -8f/d5a26b makeTranscript (ggal_liver) 0 COMPLETED -94/dfdfb6 makeTranscript (ggal_gut) 0 COMPLETED +e1/fc2d06 RNASEQ:INDEX (ggal_1_48850000_49020000) 0 COMPLETED +72/f0542c RNASEQ:FASTQC (ggal_gut) 0 COMPLETED +83/009346 RNASEQ:QUANT (ggal_gut) 0 COMPLETED +84/15e4cd MULTIQC 0 COMPLETED ``` -The fields accepted by the `-f` options are the ones in the {ref}`trace report`, as well as: script, stdout, stderr, env. List available fields using the `-l` (`-list-fields`) option. +Optional fields include `script`, `stdout`, `stderr`, `env`, and fields accessible from the {ref}`trace report`. See {ref}`cli-log` for a full list of available fields or run the `log` command with the `-l` option. -The `script` field is useful for examining script commands run in each task: +:::{tip} +Use the `script` field to examine script commands run in each task: ```console $ nextflow log goofy_kilby -f name,status,script -align_genome COMPLETED - bowtie --index /data/genome input.fastq > output +RNASEQ:FASTQC (ggal_gut) COMPLETED + fastqc.sh "ggal_gut" "ggal_gut_1.fq ggal_gut_2.fq" ... ``` +::: ### Templates -The `-t` option allows a template (string or file) to be specified. This makes it possible to create complex custom reports in any text-based format. For example, you could save this Markdown snippet to a file: +The `-t` option specifies a template (string or file) to generate custom reports in any text-based format. + +```bash +nextflow log -t