Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Max lookback issue #178

Merged
merged 58 commits into from
Sep 25, 2024
Merged
Changes from 1 commit
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
73ceb29
Do not cross metric day boundaries.
Sep 9, 2022
3798f20
Merge day boundary (#146)
areeda Sep 27, 2022
ef93d47
Check point merge (#147)
areeda Oct 16, 2022
ead6747
Merge day boundary (#146)
areeda Sep 27, 2022
6b5403b
minor doc changes
Nov 22, 2022
978cc76
Fix a bug where an xml.gz file could get compressed again in merge-wi…
Nov 28, 2022
da36cb4
Fix a double gzip of ligolw files (#151)
areeda Mar 28, 2023
3a2052f
Resolve flake8 issues
Apr 5, 2023
9c9f35f
Merge remote-tracking branch 'upstream/master'
May 10, 2023
d7eb601
Update log format to use human readble date/time instead of gps
May 30, 2023
9d06142
Merge remote-tracking branch 'upstream/master'
Jun 29, 2023
4f0ab08
remove old setup.cfg
Jun 30, 2023
59840cc
Merge remote-tracking branch 'upstream/master'
Oct 25, 2023
e57e810
Work vonpytest failures. The remaining errors are the result of omicr…
Oct 26, 2023
9d175ed
missing blank line, from flake8
Oct 26, 2023
6167626
Merge remote-tracking branch 'upstream/master'
Oct 28, 2023
cc9348e
v2.0.8 merged
areeda Dec 4, 2023
a7748e9
Fix a problem wit hmax-online-lookback not working properly in all pa…
areeda Dec 13, 2023
ec6f2ea
Fix logging problems from different gps time objects
areeda Dec 14, 2023
c901fcf
Better logging why online effort did not run
areeda Dec 14, 2023
d5ac9d0
Up default lookback window to 40 min
areeda Dec 16, 2023
8954f5a
Up default maximum lookback window to 60 min. Better logging of why w…
areeda Dec 18, 2023
5795c6e
Fix flake8 and more logging updates
areeda Dec 19, 2023
1cb0501
Merge branch 'gwpy:master' into max-lookback
areeda Dec 19, 2023
280929e
Fix flake8 and more logging updates
areeda Dec 19, 2023
dfe1cc4
Merge remote-tracking branch 'origin/max-lookback' into max-lookback
areeda Dec 19, 2023
7ad765c
More logging updates
areeda Dec 23, 2023
a00d121
More logging updates, paths through could cause error
areeda Dec 23, 2023
62c0e15
Trap and print errors from main()
areeda Dec 26, 2023
5cebbe5
fix dag submission command
areeda Mar 28, 2024
d80249f
add smart postscript to allow retries befor ignoring errors
areeda Apr 9, 2024
19c475d
tst version of scitokens and smart post script
areeda Jun 11, 2024
0e9fd39
tst version of scitokens and smart post script
areeda Jun 11, 2024
ea43ef1
add arg to specify auth type (x5099, vault or apissuer)
areeda Jun 30, 2024
b0cdb94
memopry units in the wrong place
areeda Jun 30, 2024
349a007
memory units in the wrong place, condor_run
areeda Jul 15, 2024
13ff4f4
flake8 nit picked
areeda Jul 15, 2024
7cb01d6
Again try to get periodic_release and periodic_remove correct
areeda Jul 17, 2024
5316b73
Sort console scripts
areeda Jul 17, 2024
728e3f9
Typo in periodic_remove
areeda Aug 3, 2024
7404b8d
Better error message when programs not available
areeda Aug 9, 2024
129143b
implement cona run for all jobs in dag
areeda Aug 10, 2024
7525c29
conda run complications with cvmfs
areeda Aug 11, 2024
23e072e
archive.py deals with renamed trigger files
areeda Aug 15, 2024
a5702b4
archive.py deals with renamed trigger files take 2
areeda Aug 15, 2024
c4136ce
condor run needed in all scripts.
areeda Aug 16, 2024
814bc34
minor logging changes
areeda Aug 16, 2024
ea0c51a
working on archive issues
areeda Aug 19, 2024
da27c79
working on archive issues, keep "temporary" files to help debugging
areeda Aug 31, 2024
1a61ded
more logging
areeda Aug 31, 2024
5bb41f6
Default max lookback changed to 30min. more logging tweaks
areeda Sep 9, 2024
1ea5734
Add omicron_utils to insta;; requirements
areeda Sep 9, 2024
327f289
Work on build and test workflow error.
areeda Sep 9, 2024
e114911
Still working on build and test workflow error. Remove Python 3.9 fro…
areeda Sep 9, 2024
2da8df9
Set loglevel for OmicronConfig to Critical so --version command is clean
areeda Sep 14, 2024
691c6eb
Resolve all conversations
areeda Sep 25, 2024
d5a9c0a
try to deal with github error
areeda Sep 25, 2024
978b837
try to deal with github error
areeda Sep 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix logging problems from different gps time objects
areeda committed Dec 14, 2023
commit ec6f2ea00a8e5217f669672964e043da77bd0a42
15 changes: 9 additions & 6 deletions omicron/cli/process.py
Original file line number Diff line number Diff line change
@@ -476,7 +476,7 @@

logger.debug("Command line args:")
for arg in vars(args):
logger.debug(f' {arg} = {str(getattr(args, arg))}')

Check warning on line 479 in omicron/cli/process.py

Codecov / codecov/patch

omicron/cli/process.py#L479

Added line #L479 was not covered by tests

# validate command line arguments
if args.ifo is None:
@@ -506,6 +506,8 @@
ifo = args.ifo
group = args.group
online = args.gps is None
if online:
logger.info('Online process. gps start, end determined automatically')

Check warning on line 510 in omicron/cli/process.py

Codecov / codecov/patch

omicron/cli/process.py#L509-L510

Added lines #L509 - L510 were not covered by tests

# format file-tag as underscore-delimited upper-case string
filetag = args.file_tag
@@ -648,12 +650,12 @@
pass

if statechannel:
logger.debug(f"State channel {statechannel}")
if statebits == 'guardian':
logger.debug(f"State bits {statebits}")

Check warning on line 655 in omicron/cli/process.py

Codecov / codecov/patch

omicron/cli/process.py#L653-L655

Added lines #L653 - L655 were not covered by tests
else:
logger.debug("State bits = %s" % ', '.join(map(str, statebits)))
logger.debug(f"State frametype {stateft}")

Check warning on line 658 in omicron/cli/process.py

Codecov / codecov/patch

omicron/cli/process.py#L657-L658

Added lines #L657 - L658 were not covered by tests

# parse padding for state segments
if statechannel or stateflag:
@@ -681,7 +683,7 @@
# -- set directories ------------------------------------------------------

rundir.mkdir(exist_ok=True, parents=True)
logger.info(f"Using run directory: {rundir}")

Check warning on line 686 in omicron/cli/process.py

Codecov / codecov/patch

omicron/cli/process.py#L686

Added line #L686 was not covered by tests

cachedir = rundir / "cache"
condir = rundir / "condor"
@@ -730,22 +732,22 @@

segfile = str(rundir / "segments.txt")
keepfiles.append(segfile)
max_lookback = args.max_online_lookback

Check warning on line 735 in omicron/cli/process.py

Codecov / codecov/patch

omicron/cli/process.py#L735

Added line #L735 was not covered by tests

if newdag and online:
# get limit of available data (allowing for padding)
end = data.get_latest_data_gps(ifo, frametype) - padding
now = tconvert()
earliest_online = now - max_lookback

Check warning on line 741 in omicron/cli/process.py

Codecov / codecov/patch

omicron/cli/process.py#L740-L741

Added lines #L740 - L741 were not covered by tests
try: # start from where we got to last time
last_run_segment = segments.get_last_run_segment(segfile)
start = last_run_segment[1]
if start < earliest_online:
logger.warning(f'Segments.txt produced a start time for this run before max-lookback {max_lookback}'
f'Found {gps_to_hr(start)} earliest is {gps_to_hr(earliest_online)}')
logger.warning(f'Segments.txt produced a start time for this run before max-lookback {max_lookback}\n'

Check warning on line 746 in omicron/cli/process.py

Codecov / codecov/patch

omicron/cli/process.py#L743-L746

Added lines #L743 - L746 were not covered by tests
f' Found {gps_to_hr(start)} earliest is {gps_to_hr(earliest_online)}')
start = earliest_online

Check warning on line 748 in omicron/cli/process.py

Codecov / codecov/patch

omicron/cli/process.py#L748

Added line #L748 was not covered by tests
else:
logger.debug(f"Online segment record recovered: {gps_to_hr(last_run_segment[0])} - "

Check warning on line 750 in omicron/cli/process.py

Codecov / codecov/patch

omicron/cli/process.py#L750

Added line #L750 was not covered by tests
f"{gps_to_hr(last_run_segment[1])}")

except IOError: # otherwise start with a sensible amount of data
@@ -754,17 +756,17 @@
"%s seconds" % chunkdur)
start = end - chunkdur + padding
else: # process the last requested seconds (arbitrarily)
logger.debug(f"No online segment record, starting with {max_lookback} seconds ago, "

Check warning on line 759 in omicron/cli/process.py

Codecov / codecov/patch

omicron/cli/process.py#L759

Added line #L759 was not covered by tests
f"{gps_to_hr(earliest_online)}")
start = earliest_online

Check warning on line 761 in omicron/cli/process.py

Codecov / codecov/patch

omicron/cli/process.py#L761

Added line #L761 was not covered by tests

elif online:
start, end = segments.get_last_run_segment(segfile)
if end - start > max_lookback:

Check warning on line 765 in omicron/cli/process.py

Codecov / codecov/patch

omicron/cli/process.py#L765

Added line #L765 was not covered by tests

start = end - max_lookback

Check warning on line 767 in omicron/cli/process.py

Codecov / codecov/patch

omicron/cli/process.py#L767

Added line #L767 was not covered by tests
else:
logger.debug(f"Online segment record recovered: {gps_to_hr(start)} - {gps_to_hr(end)}")

Check warning on line 769 in omicron/cli/process.py

Codecov / codecov/patch

omicron/cli/process.py#L769

Added line #L769 was not covered by tests

else:
start, end = args.gps
@@ -776,7 +778,7 @@
dataend = end + padding
dataduration = dataend - datastart

logger.info(f'Processing segment determined as: {gps_to_hr(datastart)} - {gps_to_hr(dataend)}')

Check warning on line 781 in omicron/cli/process.py

Codecov / codecov/patch

omicron/cli/process.py#L781

Added line #L781 was not covered by tests
dur_str = '{} {}'.format(int(dataduration / 86400) if dataduration > 86400 else '',
time.strftime('%H:%M:%S', time.gmtime(int(dataduration))))
logger.info(f"Duration = {dataduration} - {dur_str}")
@@ -806,11 +808,12 @@
if (online and statechannel) or (statechannel and not stateflag) or (
statechannel and args.no_segdb):
logger.info(f'Finding segments for relevant state... from:{datastart} length: {dataduration}s')
logger.debug(f'For segment finding: online: {online}, statechannel: {statechannel}, '

Check warning on line 811 in omicron/cli/process.py

Codecov / codecov/patch

omicron/cli/process.py#L811

Added line #L811 was not covered by tests
f'stateflag: {stateflag} args.no_segdb: {args.no_segdb}')
seg_qry_strt = time.time()
if statebits == "guardian": # use guardian
logger.debug(f'Using guardian for {statechannel}: {datastart}-{dataend}: {(dataend-datastart)} seconds')
logger.debug(f'Using guardian for {statechannel}: {gps_to_hr(datastart)}-{gps_to_hr(dataend)}:'

Check warning on line 815 in omicron/cli/process.py

Codecov / codecov/patch

omicron/cli/process.py#L815

Added line #L815 was not covered by tests
f' {(dataend-datastart)} seconds')
segs = segments.get_guardian_segments(
statechannel,
stateft,
@@ -853,7 +856,7 @@
if len(segs):
logger.info("State/frame segments recovered as")
for seg in segs:
logger.info(" %d %d [%d]" % (seg[0], seg[1], abs(seg)))
logger.info(f" {gps_to_hr(seg[0])} {gps_to_hr(seg[1])} [{abs(seg)}]")

Check warning on line 859 in omicron/cli/process.py

Codecov / codecov/patch

omicron/cli/process.py#L859

Added line #L859 was not covered by tests
logger.info("Duration = %d seconds" % abs(segs))

# if running online, we want to avoid processing up to the extent of
@@ -901,7 +904,7 @@
step = chunkdur - overlap
segs[-1] = type(segs[-1])(lastseg[0], t)
dataend = segs[-1][1]
logger.info("This analysis will now run to %d" % dataend)
logger.info(f"This analysis will now run to {gps_to_hr(dataend)}")

Check warning on line 907 in omicron/cli/process.py

Codecov / codecov/patch

omicron/cli/process.py#L907

Added line #L907 was not covered by tests

# recalculate the processing segment
dataspan = type(segs)([segments.Segment(datastart, dataend)])
@@ -996,7 +999,7 @@

logger.info("This will output triggers for")
for seg in trigsegs:
logger.info(f" {seg[0]:d} {seg[1]:d} {abs(seg):d}")
logger.info(f" {gps_to_hr(seg[0])} {gps_to_hr(seg[1])} {abs(seg)}")

Check warning on line 1002 in omicron/cli/process.py

Codecov / codecov/patch

omicron/cli/process.py#L1002

Added line #L1002 was not covered by tests
logger.info(f"Duration = {abs(trigsegs):d} seconds")

# -- config omicron config directory --------------------------------------

Unchanged files with check annotations Beta

os.remove(path)
else:
ret = True
logger.debug(f'valid_file: {ret} {path.name} ({ntrig}), took {time.time() - vf_strt:.2f}')

Check warning on line 184 in omicron/cli/merge_with_gaps.py

Codecov / codecov/patch

omicron/cli/merge_with_gaps.py#L184

Added line #L184 was not covered by tests
return ret
if __name__ == "__main__":
main()
if logger:
logger.info(f'Run time: {(time.time() - prog_start):.1f} seconds')

Check warning on line 865 in omicron/cli/status.py

Codecov / codecov/patch

omicron/cli/status.py#L865

Added line #L865 was not covered by tests
if on_gaps != 'ignore':
seglist = SegmentList(map(file_segment, cache)).coalesce()
missing = (SegmentList([Segment(start, end)]) - seglist).coalesce()
msg = "Missing frames:\n{}".format('\n'.join(map(lambda s: f'[{s[0]}, {s[1]}) -> {s[1] - s[0]}s', missing)))

Check warning on line 193 in omicron/data.py

Codecov / codecov/patch

omicron/data.py#L193

Added line #L193 was not covered by tests
if missing and on_gaps == 'warn':
warnings.warn(msg)
elif missing:
if version is None:
try:
version = utils.get_omicron_version()
except (KeyError, RuntimeError):

Check warning on line 68 in omicron/parameters.py

Codecov / codecov/patch

omicron/parameters.py#L68

Added line #L68 was not covered by tests
version = utils.OmicronVersion(const.OMICRON_VERSION)
self.version = version
self._set_defaults()
@param LIGOTimeGPS | int | float gps: time to consider
@return str: hr string eg: "1386527433 (12/13/23 18:30:16)"
"""
dt = from_gps(int(gps))
dt_str = dt.strftime('%x %X')
ret = f'{int(gps)} ({dt_str})'
return ret

Check warning on line 133 in omicron/utils.py

Codecov / codecov/patch

omicron/utils.py#L130-L133

Added lines #L130 - L133 were not covered by tests