Skip to content

Commit

Permalink
2024.09
Browse files Browse the repository at this point in the history
  • Loading branch information
antgonza committed Sep 12, 2024
1 parent 913a31f commit 2441099
Show file tree
Hide file tree
Showing 12 changed files with 112 additions and 14 deletions.
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,20 @@
# Qiita changelog

Version 2024.09
---------------

Deployed on September 23rd, 2024

* Added update_resource_allocation_redis and companion code, so resource allocations summaries are available for review. Thank you @Gossty!
* Now is possible to have default workflows with only one step.
* `qiita_client.update_job_step` now accepts an ignore_error optional parameter. Thank you @charles-cowart!
* Initial changes in `qiita_client` to have more accurate variable names: `QIITA_SERVER_CERT` -> `QIITA_ROOTCA_CERT`. Thank you @charles-cowart!
* Added `get_artifact_html_summary` to `qiita_client` to retrieve the summary file of an artifact.
* Re-added github actions to `https://github.com/qiita-spots/qiita_client`.
* `Woltka v0.1.4, paired-end` superseded `Woltka v0.1.4` in `qp-woltka`; [more information](https://qiita.ucsd.edu/static/doc/html/processingdata/woltka_pairedend.html). Thank you to @qiyunzhu for the benchmarks!
* Other general fixes, like [#3424](https://github.com/qiita-spots/qiita/pull/3424), [#3425](https://github.com/qiita-spots/qiita/pull/3425).


Version 2024.07
---------------

Expand Down
2 changes: 1 addition & 1 deletion qiita_core/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,4 @@
# The full license is in the file LICENSE, distributed with this software.
# -----------------------------------------------------------------------------

__version__ = "2024.02"
__version__ = "2024.09"
2 changes: 1 addition & 1 deletion qiita_db/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
from . import user
from . import processing_job

__version__ = "2024.02"
__version__ = "2024.09"

__all__ = ["analysis", "artifact", "archive", "base", "commands",
"environment_manager", "exceptions", "investigation", "logger",
Expand Down
2 changes: 1 addition & 1 deletion qiita_pet/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,4 @@
# The full license is in the file LICENSE, distributed with this software.
# -----------------------------------------------------------------------------

__version__ = "2024.02"
__version__ = "2024.09"
2 changes: 1 addition & 1 deletion qiita_pet/handlers/api_proxy/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@
from .user import (user_jobs_get_req)
from .util import check_access, check_fp

__version__ = "2024.02"
__version__ = "2024.09"

__all__ = ['prep_template_summary_get_req', 'data_types_get_req',
'study_get_req', 'sample_template_filepaths_get_req',
Expand Down
7 changes: 7 additions & 0 deletions qiita_pet/support_files/doc/source/processingdata/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -202,3 +202,10 @@ Closed-Reference OTU Picking
* **Sortmerna e_value** :ref:`[10]<reference10>` (required): Maximum e-value when clustering (local sequence alignment tool for filtering, mapping, and OTU picking) can expect to see by chance when searching a database
* **Sortmerna max-pos** :ref:`[10]<reference10>` (required): Maximum number of positions per seed to store in the indexed database
* **Threads** (required): Number of threads to use per job

Woltka / bowtie2 paired-end
---------------------------

.. toctree::

woltka_pairedend.rst
Original file line number Diff line number Diff line change
Expand Up @@ -71,17 +71,22 @@ References:
Shotgun sequencing
------------------

Qiita currently has one active shotgun metagenomics data analysis pipeline: a per sample
Qiita currently has one active shotgun metagenomics data analysis pipeline: a per sample, paired-end
bowtie2 alignment step with Woltka classification using either the WoLr2 (default) or RS210 databases.
Below you will find more information about each of these options.

.. note::
The bowtie2 settings are maximum and minimum mismatch penalties (mp=[1,1]), a
penalty for ambiguities (np=1; default), read and reference gap open- and
The bowtie2 settings are set for interleaved processing with a maximum and minimum mismatch
penalties (mp=[1,1]), a penalty for ambiguities (np=1; default), read and reference gap open- and
extend penalties (rdg=[0,1], rfg=[0,1]), a minimum alignment score for an
alignment to be considered valid (score-min=[L,0,-0.05]), a defined number of
distinct, valid alignments (k=16), and the suppression of SAM records for
unaligned reads, as well as SAM headers (no-unal, no-hd).
unaligned reads, as well as SAM headers (no-unal, no-hd), and using end-to-end alignments
before using the multiseed heuristic (no-exact-upfront, no-1mm-upfront). More information visit:

.. toctree::

woltka_pairedend.rst

The current workflow is as follows:

Expand Down Expand Up @@ -110,10 +115,9 @@ For more information about the versions in this plugin, visit:

qp-fastp-minimap2.rst

Note that the command produces up to 6 output artifacts based on the aligner and database selected:
Note that the command produces up to 5 output artifacts based on the aligner and database selected:

- Alignment Profile: contains the raw alignment file and the no rank classification BIOM table
- Per genome Predictions: contains the per genome level predictions BIOM table
- Per genome Predictions: contains the raw alignment file and the per genome level predictions BIOM table
- Per gene Predictions: Only WoLr2, contains the per gene level predictions BIOM table
- KEGG Pathways: Only WoLr2, contains the functional profile
- KEGG Ontology (KO): Only WoLr2, contains the functional profile
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
Wolka and Bowtie2 using Read Pairing Schemes
============================================

Benchmarks created by Qiyun Zhu (@qiyunzhu) on Aug 1, 2024.

Summary
-------

Here I tested alternative read pairing schemes in the analysis of shotgun metagenomic sequencing data. Sequencing reads were aligned against a reference microbial genome database as unpaired or paired, with or without singleton and/or discordant alignments suppressed. A series of synthetic datasets were used in the analysis.

The results reveal that treating reads as paired is always advantageous over unpaired. Suppressing singleton alignments further increases the accuracy of results, despite at the cost of lower mapping rate. Suppressing discordant alignments has no obvious impact on the result. Regardless of accuracy, the downstream community ecology analyses are not obviously impacted by the choice of parameters.

Therefore, I recommend the general adoption of paired alignments as a standard procedure. I also endorse suppressing singleton and discordant alignments, but note the favor of further tests on whether they may reduce sensitivity with complex communities.

Alignment parameters
--------------------

Sequencing data were aligned using Bowtie2 v2.5.1 in the “very sensitive” mode against the WoL2 database. They were treated as either unpaired or paired-end:

- SE: Reads are treated as unpaired (Bowtie2 input: -U merged.fq)
- PE: Reads are treated as paired (Bowtie2 input: -1 fwd.fq, -2 rev.fq)

Under the paired mode, certain flags were applied:

- ND: Discordant alignments are suppressed (Bowtie2 flag: --no-discordant). For example, if a pair of alignments are not pointing toward each other or are too far apart from each other in the reference genome, then both alignments are discarded. See the Bowtie2 manual for a discussion.
- NM: Singleton alignments are suppressed (Bowtie2 flag: --no-mixed). For example, if a read was aligned but its mate was not, then the alignment was discarded.
- NDM: Both flags were applied.

Extra parameters tested:
- PE.NU: flags `--no-exact-upfront --no-1mm-upfront`.

Resulting alignment files (SAM format) were processed by Woltka v0.1.6 using default parameters to generate OGU tables.

Synthetic data
--------------

Five synthetic datasets were generated with 25 samples each consisting of randomly selected WoL2 genomes. CAMISIM was executed to simulate 500 Mbp of 150 bp paired-end Illumina sequencing reads (appr. 3.3 million reads) per sample. The five datasets have different taxon count and distribution patterns. The result of one of the five datasets is displayed below. It consists of 400 taxa (more than others) and therefore is presumably the most realistic. However, all five results largely shared the same pattern.

The results of the five Bowtie2 parameter sets were compared using nine metrics:

Three metrics that only rely on each result.

- Mapping rate (%)
- Number of taxa
- Entropy (i.e., Shannon index, but without subsampling)

Six metrics that rely on comparing each result against the ground truth (higher is better):

- Presence/absence-based:
- Precision (fraction of discovered taxa that are true)
- Recall (sensitivity) (fraction of true taxa that are discovered)
- F1 score (combination of precision and recall)
- Abundance-based:

- Spearman correlation coefficient
- Bray-Curtis similarity *
- Weighted UniFrac similarity *

* Note: Bray-Curtis and weighted UniFrac similarities were calculated after subsampling to a constant sum of taxon frequencies per sample.

.. figure:: woltka_synthetic.png
:align: center


The results revealed:

#. PE outperforms SE in all metrics. Most importantly, it reduces false positive rate (higher precision) while retaining mapping rate. Meanwhile, the sensitivity (recall) of identifying true taxa is not obviously compromised (note the y-axis scale).
#. Suppressing singleton alignments (no mixing; NM) further significantly reduces false positive rate (higher precision), while not obviously reducing sensitivity (recall). However, the mapping rate is also significantly reduced.
#. Suppressing discordant alignments (no discordance; ND) seem to have little to no effect on the outcome. In fact, most profiles are identical with or without ND (but a few are not).
#. PE.NU the two additional parameters had minimum effect on the result and make the alignment step faster. This may suggest that the additional parameters are safe to use.

Therefore, I would recommend adopting paired alignment in preference to unpaired alignment. I may suggest no mixing as it has improved accuracy, but the potential adverse effect of lower mapping rate may be further explored before making a compelling recommendation. Although not having a visible effect, no discordance may be added for logical coherency.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion qiita_ware/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,4 @@
# The full license is in the file LICENSE, distributed with this software.
# -----------------------------------------------------------------------------

__version__ = "2024.02"
__version__ = "2024.09"
2 changes: 1 addition & 1 deletion scripts/qiita-recover-jobs
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ def _submit_jobs(jids_to_recover, recover_type):
st = int(ceil(SLEEP_TIME/CHANCES))
len_jids_to_recover = len(jids_to_recover)
for i, j in enumerate(jids_to_recover):
print('recovering %s: %d/%d' % (recover_type, len_jids_to_recover, i))
print(f'recovering {j} {recover_type}: {len_jids_to_recover}/{i}')
job = ProcessingJob(j)
job._set_status('in_construction')
job.submit()
Expand Down
2 changes: 1 addition & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
from setuptools import setup
from glob import glob

__version__ = "2024.02"
__version__ = "2024.09"


classes = """
Expand Down

0 comments on commit 2441099

Please sign in to comment.