Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Really Efficient" mode #20

Merged
merged 13 commits into from
Jul 16, 2023
Merged

"Really Efficient" mode #20

merged 13 commits into from
Jul 16, 2023

Conversation

jennydaman
Copy link
Contributor

This PR introduces a mechanism for calling px-repack once at the end of the series, instead of for each DICOM file. As a result, performance is greatly improved.

Design Overview

pypx/README.rst

Lines 267 to 296 in a6ebedb

6. "Really Efficient" Mode
**************************
Really Efficient using px-REcount to call px-REpack with the help of REdis.
Typically, a `px-repack` process is called per received DICOM file.
``px-find --reallyEfficient --then receive`` does one `px-repack`
per series (i.e. one directory of DICOM files). This dramatically
improves performance See benchmarks_.
"Really efficient" (RE) mode requires a redis server.
In RE mode, ``px-find`` records the number of DICOM files per series (``NumberOfSeriesRelatedInstances``) to redis.
``storescp`` is configured to run ``px-recount`` on each instance instead of ``px-repack``.
``px-recount`` does nothing until the entire series is received.
Finally, ``px-recount`` calls ``px-repack`` once on the entire series.
``px-recount`` is implemented in Rust to reduce its footprint when being
called once per instance.
RE Mode Usage
=============
To use RE mode, install ``pypx`` in RE mode (``pip install 'pypx[re]'``)
and run ``storescp`` as
.. code-block:: bash
storescp --fork -od /tmp/data -pm -sp -xcr 'px-recount --xcrdir \"#p\" --xcrfile \"#f\" --verbosity 0 --logdir /home/dicom/log --datadir /home/dicom/data --cleanup' 11113

A Rust project, px-recount, is added to this repo. Its functionality is described here:

px-recount has the same command-line options as px-repack.
Its intended usage is as a DICOM instance handler called by storescp:
storescp --fork -od /tmp/data -pm -sp -xcr 'px-recount --xcrdir \"#p\" --xcrfile \"#f\" --verbosity 0 --logdir /home/dicom/log --datadir /home/dicom/data --cleanup' 11113
px-recount runs px-repack if the received DICOM file given by --xcrdir
and --xcrfile is the last of its series. The number of files of a
series received so far are counted by px-repack and saved to a Redis
database.
The algorithm:
0. On input of a DICOM file
1. Read DICOM tags to get SeriesInstanceUID
2. Get NumberOfSeriesRelatedInstances and fileCounter from Redis
3. If this file is the last of its series, run px-repack.
Else, write fileCounter+1 to Redis.
ENVIRONMENT VARIABLES
PYPX_REDIS_URL Redis connection URL (example: redis://redis-server.svc.local:6379)
NOTES
- It is expected that px-find writes NumberOfSeriesRelatedInstances for
the series before px-recount runs. This is typically achieved by running
px-find --reallyEfficient --then receive ...
- Generally, px-recount is fail-safe. It will call px-repack in the event
of any redis-related exceptions.

@jennydaman jennydaman merged commit ee60518 into master Jul 16, 2023
3 checks passed
@jennydaman jennydaman deleted the really-efficient branch July 16, 2023 21:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant