SAT demod mapmaker updates #1044
Conversation
obsfiledb time on large file-systems is non-negligible; this cuts time in half (10 -> 5s on NERSC test case).
This is activated through SOTODLIB_TOD_CACHE envvar.
|
@chervias |
|
The actual important time is the ctime on the obsdb, which is when the obs actually started. In the atomic database we have both the obs_id and the path to the maps, so we can match obs and maps with the atomic database, right? I would say we can leave as it is now. |
|
I merged from the branch in #1037 to load obs faster. Please look at that PR for instructions, since an env variable has to be set up to use |
…right after we write the maps
|
The PR is ready for review. Instead of using a separate script for appending to the atomic map DB, now everything is done in the same mapmaker as the maps are being written. Otherwise we have completed the other issues that is PR was meant to solve |
| obs_id: Mapped[str] = mapped_column(primary_key=True) | ||
| telescope: Mapped[str] = mapped_column(primary_key=True) | ||
| freq_channel: Mapped[str] = mapped_column(primary_key=True) | ||
| wafer: Mapped[str] = mapped_column(primary_key=True) | ||
| ctime: Mapped[int] = mapped_column(primary_key=True) | ||
| split_label: Mapped[str] = mapped_column(primary_key=True) |
There was a problem hiding this comment.
Is your aim here to have these be a composite primary key in the database?
There was a problem hiding this comment.
Also why use ctime instead of a datetime object?
There was a problem hiding this comment.
yes, these are the things that combined define a unique atomic map. I thought a primary key was something that uniquely defines a row. Therefore, I made all these primary keys. ctime is just our tradition, we define observations by their ctime.
There was a problem hiding this comment.
Yes, that's correct. You can certainly do this. One other option is to have a unique ID as the primary key, but if you are certain that there will never be rows in the database with a duplicate value for these combined items then it is fine.
There was a problem hiding this comment.
Ok. If I try to add another row to the database with the same set of these primary keys, it will error, right? That should never happen because these same keys are the ones used to check for existing maps at the beginning of the mapmaker. That way you don't re-map and existing map.
| info = mapmaking.AtomicInfo( | ||
| obslist[0][0], | ||
| obs_infos[obslist[0][3]].telescope, | ||
| band, | ||
| detset, | ||
| int(t)) |
There was a problem hiding this comment.
I would recommend using kwargs here instead of positional
There was a problem hiding this comment.
Could you elaborate a bit? I do not understand what you mean. Thanks
There was a problem hiding this comment.
I would have written:
info = mapmaking.AtomicInfo(
obs_id=obslist[0][0],
telescope=obs_infos[obslist[0][3]].telescope,
...
)
This allows you to change the order of parameters, etc, later without impacting this code in potentially invisible ways.
| info.split_label = split_label | ||
| info.split_detail = '' | ||
| info.prefix_path = str(prefix + '_%s' % split_label) | ||
| info.elevation = obs_infos[obslist[0][3]].el_center | ||
| info.azimuth =obs_infos[obslist[0][3]].az_center | ||
| info.pwv = float(pwv_atomic) | ||
| info_list.append(info) |
There was a problem hiding this comment.
Why not just do this in the initializer?
There was a problem hiding this comment.
I guess I could do that in the init, but I reserved that for the primary keys that are mandatory, and then keep adding the optional things.
… data. Maps are not created and not added to the atomic map database
Updates to SAT mapmaking scripts including:
Add site_pipeline script to make atomic maps dbWrite to atomic maps db from the mapmakermake_atomic_filterbin_map.pyto avoiding runningget_obstwice