Merge tables

[magland]

Hi, I'm a bit confused about merge tables. I see this tutorial
https://github.com/LorenFrankLab/spyglass/blob/master/notebooks/03_Merge_Tables.ipynb

but I'm having a hard time wrapping my head around how they are used. Wondering if someone could provide a conceptual overview beyond what's in that notebook.

[edeno]

Merge tables represent the end processing point of a given way of processing the data in our pipelines. They were conceived to allow us to build new ways of processing things without having to drop or migrate the old tables (we are thinking of these as versions). They allow data to be processed in different ways but the end output interface is the same so that downstream pipelines can access the data.

[magland]

Thanks @edeno that helps.

[magland]

How would I for example fetch all of the LFPOutputs for a given nwb_file_name. (LFPOutput & {'nwb_file_name': '...'}).fetch() doesn't work because LFPOutput table only has merge_id key.

[edeno]

You would do:

 LFPOutput.merge_get_part(restriction={'nwb_file_name': '...'}, join_master=True)

[magland]

Thanks. I'm trying to wrap my head around join_master. The docs say

join_master: bool
            Join part with Merge master to show source field. Default False.

could you explain a bit more?

[CBroz1]

The merge tables navigate the awkwardness of a sparse master table by showing dj.Union views into one or more parts, possibly joined with the master table. The join_master in this case adds the master's source field to the view. get_part was designed for operations on the parts themselves, and will error by default if not sufficiently restricted

To get a list back, you can bypass this restriction, but, to see all entries associated with a given restriction, merge_view might be better. It concatenates the parts and fills in None or 0 where necessary

[CBroz1]

The above describes looking at the data. merge_fetch will actually pass back the data like a native datajoint fetch

[magland]

That's helpful, thanks @CBroz1