Update max concurrent omicron job (#177)

areeda · Joseph Areeda · web-flow · commit af50aad24fb8 · 2023-12-12T11:21:51.000-08:00
* Do not cross metric day boundaries. * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <joseph.areeda@ligo.org> * Check point merge (#147) * Do not cross metric day boundaries. * add log file arg delete empty directories when done * Tweak remove empty dir removal * Tweak remove empty dir removal again * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <joseph.areeda@ligo.org> * rebase agaist last approved PR * rebase against last approved PR * rebase against last approved PR again, fix flake8 * Fix a bug in remove empty directories. Co-authored-by: Joseph Areeda <joseph.areeda@ligo.org> * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <joseph.areeda@ligo.org> * minor doc changes * Fix a bug where an xml.gz file could get compressed again in merge-with-gaps * Fix a double gzip of ligolw files (#151) * Do not cross metric day boundaries. * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <joseph.areeda@ligo.org> * Check point merge (#147) * Do not cross metric day boundaries. * add log file arg delete empty directories when done * Tweak remove empty dir removal * Tweak remove empty dir removal again * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <joseph.areeda@ligo.org> * rebase agaist last approved PR * rebase against last approved PR * rebase against last approved PR again, fix flake8 * Fix a bug in remove empty directories. Co-authored-by: Joseph Areeda <joseph.areeda@ligo.org> * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <joseph.areeda@ligo.org> * minor doc changes * Fix a bug where an xml.gz file could get compressed again in merge-with-gaps * Implement a periodic vacate to address permanent D-state (uninterupptible wait) causing jobs to fail to complete * Always create a log file. If not specified put one in the output directory * Fix a problem with periodic vacate. * Up the periodic vacate time to 3 hrs * Found a job killing typo * Add time limits to post processing also * Don't save segments.txt file if no sgments founds because we don't know if it's an issue of not finding them or a valid not analyzable state. * disable periodic vacate to demo the problem. * Fix reported version in some utilities. Only update segments.txt if omicron is actually run. * Clarify relative imports. and add details to a few log messages * Resolve flake8 issues --------- Co-authored-by: Joseph Areeda <joseph.areeda@ligo.org> * Resolve flake8 issues * Update log format to use human readble date/time instead of gps tweak logging to better underst guardian channel usage- * remove old setup.cfg * Work vonpytest failures. The remaining errors are the result of omicron segfaults if environment variable not set * missing blank line, from flake8 * Up the default max-concurrent to 64 in hopes of much higher throughput. And deal with it properly instead of my paren/child hack. --------- Co-authored-by: Joseph Areeda <joseph.areeda@ligo.org>
diff --git a/omicron/cli/process.py b/omicron/cli/process.py
@@ -279,8 +279,8 @@ def create_parser():
                        help='With no immediately previous run, or one that was long ago this is the max time of an '
                             'online job. Default: %(default)d')
     # max concurrent omicron jobs
-    procg.add_argument('--max-concurrent', default=10, type=int,
-                       help='Max omicron jobs at one time [%(default)s]')
+    procg.add_argument('--max-concurrent', default=64, type=int,
+                       help='Max omicron jobs run at one time [%(default)s]')
     procg.add_argument(
         '-x',
         '--exclude-channel',
@@ -1246,24 +1246,8 @@ def main(args=None):
                         print('\n'.join(operations), file=f)
                     if newdag:
                         script.chmod(0o755)
-    parent_jobs = list()
-    child_jobs = list()
-    maxcon = args.max_concurrent
-    for j in omicron_nodes:
-        if len(parent_jobs) < maxcon:
-            parent_jobs.append(j)
-        elif len(child_jobs) < maxcon:
-            child_jobs.append(j)
-        else:
-            for pj in parent_jobs:
-                for cj in child_jobs:
-                    cj.add_parent(pj)
-            parent_jobs = child_jobs
-            child_jobs = [j]
-    if len(child_jobs) > 0 and len(parent_jobs) > 0:
-        for pj in parent_jobs:
-            for cj in child_jobs:
-                cj.add_parent(pj)
+
+    dag.add_maxjobs_category('omicron', args.max_concurrent)
 
     # set 'strict' option for Omicron
     # this is done after the nodes are written so that 'strict' is last in
