Update documentation

Author: actions-user

_images/dpp_centroid_binary.png

@@ -1,7 +1,7 @@
 (configuration)=
 # Configuration
 
-The configuration file uses the [TOML](https://toml.io) format. There are many options that have defaults, sometimes even sensible ones. In general, if an entire section is missing, the operation will be excluded. Note that sections for the configuration (e.g., `[calibration]`, `[collapsing]`) can be in any order, although keeping them in the same order as the pipeline execution may be clearer.
+The configuration file uses the [TOML](https://toml.io) format. There are many options that have defaults, sometimes even sensible ones. In general, if an entire section is missing, the operation will be excluded. Note that sections for the configuration (e.g., `[calibration]`, `[polarimetry]`) can be in any order, although keeping them in the same order as the pipeline execution may be clearer.
 
 We use [semantic versioning](https://semver.org/), so there are certain guarantees about the backwards compatibility of the pipeline, which means you don't have to have the exact same version of `vampires_dpp` as in the configuration- merely a version that is compatible.
 
@@ -29,6 +29,14 @@ We notate where the pipeline ends up saving data files with the "💾" emoji.
 .. autoclass:: vampires_dpp.pipeline.config.TargetConfig
 ```
 
+### File combination
+```{margin} 💾 File Output
+‎
+```
+```{eval-rst}
+.. autoclass:: vampires_dpp.pipeline.config.CombineConfig
+```
+
 ### Calibration
 ```{margin} 💾 File Output
 ‎
@@ -45,13 +53,29 @@ We notate where the pipeline ends up saving data files with the "💾" emoji.
 .. autoclass:: vampires_dpp.pipeline.config.AnalysisConfig
 ```
 
+### Frame selection
+```{margin} 💾 File Output
+‎
+```
+```{eval-rst}
+.. autoclass:: vampires_dpp.pipeline.config.FrameSelectConfig
+```
+
+### Frame alignment
+```{margin} 💾 File Output
+‎
+```
+```{eval-rst}
+.. autoclass:: vampires_dpp.pipeline.config.AlignmentConfig
+```
 
-### Frame-collapsing
+
+### Cube coadding
 ```{margin} 💾 File Output
 ‎
 ```
 ```{eval-rst}
-.. autoclass:: vampires_dpp.pipeline.config.CollapseConfig
+.. autoclass:: vampires_dpp.pipeline.config.CoaddConfig
 ```
 
 ### Spectrophotometric Calibration
@@ -68,13 +92,3 @@ We notate where the pipeline ends up saving data files with the "💾" emoji.
 .. autoclass:: vampires_dpp.pipeline.config.PolarimetryConfig
 ```
 
-
-## Examples
-
-### PDI
-
-```{eval-python}
-
-from vampires_dpp.pipeline.templates import VAMPIRES_PDI
-VAMPIRES_PDI.to_toml()
-```

_images/dpp_centroid_coro.png

@@ -31,4 +31,12 @@ The time it takes to open a file, write to disk, and close it will add a lot to
 
 3. Use multi-processing
 
-Using more processes should improve some parts of the pipeline, but don't expect multiplicative increases in speed since most operations are limited by the storage IO speed.
+Using more processes should improve some parts of the pipeline, but don't expect multiplicative increases in speed since most operations are limited by the storage IO speed. 
+
+```{admonition} Tip: multiprocessing with numpy
+:class: tip
+Make sure when you're multiprocessing to set the correct environment variables in numpy
+
+    OMP_NUM_THREADS=1 dpp run -j <num_proc> <config> <input_filenames>
+    OMP_NUM_THREADS=1 dpp run -j 4 20230101_ABAur_vampires.toml norm/*.fits
+```

_images/dpp_centroid_planet.png

@@ -14,6 +14,16 @@ We strive to make the VAMPIRES DPP as automated as possible- the following setup
 The pipeline uses Python multi-processing to help speed-up reductions. You will likely be limited by your computer's available memory (RAM)- if you spawn too many processes and load too much data in parallel you can run out of memory. Commands with multi-processing enabled have the `-j` command line flag, which will default to a single process `-j1` by default.
 
 When possible, we indicate a command that has multi-processing enabled with the "🚀" emoji. We do not multi-process by default (to avoid awkward out-of-memory errors) so you must *opt in* to all commands using the `-j` flag.
+
+In general, multiprocessing does seem to help when not limited by file I/O (e.g., slow hard disk drive), so give it a try!
+```
+
+```{admonition} Tip: multiprocessing and numpy
+:class: tip
+
+Internal numpy routines can cause multiprocessing to become _much_ slower. In this case, make sure to set the [appropriate environment variables](https://numpy.org/doc/stable/reference/global_state.html#number-of-threads-used-for-linear-algebra).
+
+    export OMP_NUM_THREADS=1
 ```
 
 ```{admonition} Warning: Large data volume
@@ -68,6 +78,13 @@ There are a few recognized data formats the processing pipeline recognizes. Depe
 Any EMCCD VAMPIRES format data needs normalized- at minimum it will removed the corrupted detector readout frame and empty frames.
 ```
 
+### CMOS format
+
+Any data taken after the June 2023 upgrades has the same format regardless if data is downloaded from the archive or from the SCExAO computers directly. It is assumed that any FLC deinterleaving has been done beforehand, which is expected to be done by the support astronomer.
+
+No further action is required for the CMOS format- `dpp norm` can be skipped.
+
+
 ### EMCCD formats
 
 These formats are used for VAMPIRES data prior to the June 2023 upgrades. They come in two distinct formats
@@ -95,15 +112,21 @@ dpp norm [-j 1] [-d] -o normed 750-50_em300_00100ms_512x512/*.fits
 Not all data needs deinterleaved- calibration files (darks, flats, pinholes, skies, etc.) typically do not need deinterleaved. If you do not plan to do polarimetry (e.g., speckle imaging, ADI-only) you can skip deinterleaving entirely, effectively averaging together data between the two FLC states. If you prefer to model each state separately then deinterleave away.
 ```
 
-### CMOS format
+## Quick look and filter
 
-Any data taken after the June 2023 upgrades has the same format regardless if data is downloaded from the archive or from the SCExAO computers directly. It is assumed that any FLC deinterleaving has been done before hand, which is expected to be done by the support astronomer.
+Before running files through the pipeline, it is recommended to inspect your raw data and discard errant cubes and cubes with poor seeing. Doing this ahead of time saves on processing time and avoids errors.
 
-No further action is required for the CMOS format- `dpp norm` can be skipped.
+```
+dpp select <input_files>/*.fits
+```
 
-## Quick look and filter
+```{image} dpp_select.mov
+```
 
-Before running files through the pipeline, it is recommended to inspect your raw data and discard errant cubes and cubes with poor seeing. Doing this ahead of time saves on processing time and avoids errors.
+After this, you can use the filelist as your input, for example
+```
+dpp run my_config.toml $(< filelist_select.txt)
+```
 
 ## Create calibration files
 
@@ -140,7 +163,6 @@ To allow efficient data analysis we only measure the PSF centroid and statistics
 We recommend running the centroid step as it greatly reduces the chances for errors in image registration. In addition, for coronagraphic data you *must* run the centroid step to specify which satellite spots to use for analysis. The centroid step is not *strictly* required for non-coronagraphic data, though- if you do not provide it the center of the frame will be used as the starting estimate, and your analysis window size can be made larger to accommadate misalignment. Nonetheless, we highly recommend this step.
 ```
 
-
 ### Automated mode
 
 If the terminal you are connected to has working graphics (i.e., X-forwarding for SSH connections, not inside a tmux sesssion) and you have a copy of matplotlib installed you can run
@@ -151,6 +173,20 @@ If the terminal you are connected to has working graphics (i.e., X-forwarding fo
 dpp centroid [-j 1] 20230101_ABAur.toml 750-50_em300_00100ms_512x512/*.fits
 ```
 
+**Coronagraphic**
+```{image} dpp_centroid_coro.png
+:width: 80%
+```
+
+**Non-coronagraphic/Binaries**
+```{image} dpp_centroid_binary.png
+:width: 80%
+```
+
+**Planets**
+```{image} dpp_centroid_planet.png
+:width: 80%
+```
 ### Manual mode
 
 If you do not have a graphics-ready terminal, or you want to skip the interactive process and use your own methods, we provide a manual method for entering centroids. You will be prompted for the centroid of each PSF- enter the centroid as an `x, y` comma-separated tuple.

_images/dpp_select.mov

@@ -48,7 +48,7 @@ div.sphinxsidebarwrapper {
 
 div.sphinxsidebar {
     float: left;
-    width: 270px;
+    width: 230px;
     margin-left: -100%;
     font-size: 90%;
     word-wrap: break-word;

_sources/configuration.md

@@ -0,0 +1,5 @@
+/*!
+ * Font Awesome Free 6.5.2 by @fontawesome - https://fontawesome.com
+ * License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License)
+ * Copyright 2024 Fonticons, Inc.
+ */

_sources/faq.md

@@ -2,30 +2,23 @@
   AUTO-GENERATED from webpack.config.js, do **NOT** edit by hand.
   These are re-used in layout.html
 -->
-{# Load FontAwesome icons #}
-{% macro head_pre_icons() %}
-  <link href="{{ pathto('_static/vendor/fontawesome/6.5.2/css/all.min.css', 1) }}?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
-  <link rel="preload" as="font" type="font/woff2" crossorigin href="{{ pathto('_static/vendor/fontawesome/6.5.2/webfonts/fa-solid-900.woff2', 1) }}" />
-<link rel="preload" as="font" type="font/woff2" crossorigin href="{{ pathto('_static/vendor/fontawesome/6.5.2/webfonts/fa-brands-400.woff2', 1) }}" />
-<link rel="preload" as="font" type="font/woff2" crossorigin href="{{ pathto('_static/vendor/fontawesome/6.5.2/webfonts/fa-regular-400.woff2', 1) }}" />
-{% endmacro %}
 
 {% macro head_pre_assets() %}
   <!-- Loaded before other Sphinx assets -->
-  <link href="{{ pathto('_static/styles/theme.css', 1) }}?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
-<link href="{{ pathto('_static/styles/bootstrap.css', 1) }}?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
-<link href="{{ pathto('_static/styles/pydata-sphinx-theme.css', 1) }}?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
+  <link href="{{ pathto('_static/styles/theme.css', 1) }}?digest=8878045cc6db502f8baf" rel="stylesheet" />
+<link href="{{ pathto('_static/styles/pydata-sphinx-theme.css', 1) }}?digest=8878045cc6db502f8baf" rel="stylesheet" />
 {% endmacro %}
 
 {% macro head_js_preload() %}
+  <!-- So that users can add custom icons -->
+  <script src="{{ pathto('_static/scripts/fontawesome.js', 1) }}?digest=8878045cc6db502f8baf"></script>
   <!-- Pre-loaded scripts that we'll load fully later -->
-  <link rel="preload" as="script" href="{{ pathto('_static/scripts/bootstrap.js', 1) }}?digest=dfe6caa3a7d634c4db9b" />
-<link rel="preload" as="script" href="{{ pathto('_static/scripts/pydata-sphinx-theme.js', 1) }}?digest=dfe6caa3a7d634c4db9b" />
-  <script src="{{ pathto('_static/vendor/fontawesome/6.5.2/js/all.min.js', 1) }}?digest=dfe6caa3a7d634c4db9b"></script>
+  <link rel="preload" as="script" href="{{ pathto('_static/scripts/bootstrap.js', 1) }}?digest=8878045cc6db502f8baf" />
+<link rel="preload" as="script" href="{{ pathto('_static/scripts/pydata-sphinx-theme.js', 1) }}?digest=8878045cc6db502f8baf" />
 {% endmacro %}
 
 {% macro body_post() %}
   <!-- Scripts loaded after <body> so the DOM is not blocked -->
-  <script src="{{ pathto('_static/scripts/bootstrap.js', 1) }}?digest=dfe6caa3a7d634c4db9b"></script>
-<script src="{{ pathto('_static/scripts/pydata-sphinx-theme.js', 1) }}?digest=dfe6caa3a7d634c4db9b"></script>
+  <script defer src="{{ pathto('_static/scripts/bootstrap.js', 1) }}?digest=8878045cc6db502f8baf"></script>
+<script defer src="{{ pathto('_static/scripts/pydata-sphinx-theme.js', 1) }}?digest=8878045cc6db502f8baf"></script>
 {% endmacro %}