Merge branch 'master' into merge_fetch_nwb_fix

CHANGELOG.md

@@ -6,16 +6,47 @@
 
 <!-- Running draft to be removed immediately prior to release. -->
 
+```python
+import datajoint as dj
+from spyglass.linearization.v1.main import TrackGraph
+
+TrackGraph.alter()  # Add edge map parameter
+dj.FreeTable(dj.conn(), "common_session.session_group").drop()
+```
+
 ### Infrastructure
 
-- Disable populate transaction protection for long-populating tables #1066
+- Disable populate transaction protection for long-populating tables #1066,
+    #1108
+- Add docstrings to all public methods #1076
+- Update DataJoint to 0.14.2 #1081
 - Allow restriction based on parent keys in `Merge.fetch_nwb()` #1086
 
 ### Pipelines
 
+- Common
+
+    - Drop `SessionGroup` table #1106
+
 - Decoding
+
     - Fix edge case errors in spike time loading #1083
 
+- Linearization
+
+    - Add edge_map parameter to LinearizedPositionV1 #1091
+
+- Position
+
+    - Fix video directory bug in `DLCPoseEstimationSelection` #1103
+    - Restore #973, allow DLC without position tracking #1100
+    - Minor fix to `DLCCentroid` make function order #1112
+
+- Spike Sorting
+
+    - Fix bug in `get_group_by_shank` #1096
+    - Fix bug in `_compute_metric` #1099
+
 ## [0.5.3] (August 27, 2024)
 
 ### Infrastructure
@@ -103,6 +134,7 @@
     - Set `sparse` parameter to waveform extraction step in `spikesorting.v1`
         #1039
     - Efficiency improvement to `v0.Curation.insert_curation` #1072
+    - Add pytests for `spikesorting.v1` #1078
 
 ## [0.5.2] (April 22, 2024)
 

CITATION.cff

@@ -166,5 +166,5 @@ keywords:
   - spike sorting
   - kachery
 license: MIT
-version: 0.5.2
+version: 0.5.3
 date-released: '2024-04-22'

docs/mkdocs.yml

@@ -74,7 +74,6 @@ nav:
     - FigURL: Features/FigURL.md
     - Merge Tables: Features/Merge.md
     - Export: Features/Export.md
-    - Session Groups: Features/SessionGroups.md
     - Centralized Code: Features/Mixin.md
   - For Developers:
     - Overview: ForDevelopers/index.md

docs/src/Features/Mixin.md

@@ -5,8 +5,9 @@ functionalities that have been added to DataJoint tables. This includes...
 
 - Fetching NWB files
 - Long-distance restrictions.
-- Delete functionality, including permission checks and part/master pairs
+- Permission checks on delete
 - Export logging. See [export doc](./Export.md) for more information.
+- Miscellaneous helper functions
 
 To add this functionality to your own tables, simply inherit from the mixin:
 
@@ -102,12 +103,7 @@ my_table << upstream_restriction >> downstream_restriction
 When providing a restriction of the parent, use 'up' direction. When providing a
 restriction of the child, use 'down' direction.
 
-## Delete Functionality
-
-The mixin overrides the default `delete` function to provide two additional
-features.
-
-### Permission Checks
+## Delete Permission Checks
 
 By default, DataJoint is unable to set delete permissions on a per-table basis.
 If a user is able to delete entries in a given table, she can delete entries in
@@ -127,66 +123,76 @@ curcumvent the default permission checks by adding themselves to the relevant
 team or removing the mixin from the class declaration. However, it provides a
 reasonable level of security for the average user.
 
-### Master/Part Pairs
+Because parts of this process rely on caching, this process will be faster if
+you assign the instanced table to a variable.
 
-By default, DataJoint has protections in place to prevent deletion of a part
-entry without deleting the corresponding master. This is useful for enforcing
-the custom of adding/removing all parts of a master at once and avoids orphaned
-masters, or null entry masters without matching data.
+```python
+# Slower
+YourTable().delete()
+YourTable().delete()
 
-For [Merge tables](./Merge.md), this is a significant problem. If a user wants
-to delete all entries associated with a given session, she must find all part
-table entries, including Merge tables, and delete them in the correct order. The
-mixin provides a function, `delete_downstream_parts`, to handle this, which is
-run by default when calling `delete`.
+# Faster
+nwbfile = YourTable()
+nwbfile.delete()
+nwbfile.delete()
+```
 
-`delete_downstream_parts`, also aliased as `ddp`, identifies all part tables
-with foreign key references downstream of where it is called. If `dry_run=True`,
-it will return a list of entries that would be deleted, otherwise it will delete
-them.
+<details><summary>Deprecated delete feature</summary>
 
-Importantly, `delete_downstream_parts` cannot properly interact with tables that
-have not been imported into the current namespace. If you are having trouble
-with part deletion errors, import the offending table and rerun the function
-with `reload_cache=True`.
+Previous versions of Spyglass also deleted masters of parts with foreign key
+references. This functionality has been migrated to DataJoint in version 0.14.2
+via the `force_masters` delete argument. This argument is `True` by default in
+Spyglass tables.
 
-```python
-import datajoint as dj
-from spyglass.common import Nwbfile
+</details>
 
-restricted_nwbfile = Nwbfile() & "nwb_file_name LIKE 'Name%'"
+## Populate Calls
 
-vanilla_dj_table = dj.FreeTable(dj.conn(), Nwbfile.full_table_name)
-vanilla_dj_table.delete()
-# DataJointError("Attempt to delete part table MyMerge.Part before ... ")
+The mixin also overrides the default `populate` function to provide additional
+functionality for non-daemon process pools and disabling transaction protection.
 
-restricted_nwbfile.delete()
-# [WARNING] Spyglass: No part deletes found w/ Nwbfile ...
-# OR
-# ValueError("Please import MyMerge and try again.")
+### Non-Daemon Process Pools
 
-from spyglass.example import MyMerge
+To allow the `make` function to spawn a new process pool, the mixin overrides
+the default `populate` function for tables with `_parallel_make` set to `True`.
+See [issue #1000](https://github.com/LorenFrankLab/spyglass/issues/1000) and
+[PR #1001](https://github.com/LorenFrankLab/spyglass/pull/1001) for more
+information.
 
-restricted_nwbfile.delete_downstream_parts(reload_cache=True, dry_run=False)
-```
+### Disable Transaction Protection
+
+By default, DataJoint wraps the `populate` function in a transaction to ensure
+data integrity (see
+[Transactions](https://docs.datajoint.io/python/definition/05-Transactions.html)).
 
-Because each table keeps a cache of downstream merge tables, it is important to
-reload the cache if the table has been imported after the cache was created.
-Speed gains can also be achieved by avoiding re-instancing the table each time.
+This can cause issues when populating large tables if another user attempts to
+declare/modify a table while the transaction is open (see
+[issue #1030](https://github.com/LorenFrankLab/spyglass/issues/1030) and
+[DataJoint issue #1170](https://github.com/datajoint/datajoint-python/issues/1170)).
 
-```python
-# Slow
-from spyglass.common import Nwbfile
+Tables with `_use_transaction` set to `False` will not be wrapped in a
+transaction when calling `populate`. Transaction protection is replaced by a
+hash of upstream data to ensure no changes are made to the table during the
+unprotected populate. The additional time required to hash the data is a
+trade-off for already time-consuming populates, but avoids blocking other users.
 
-(Nwbfile() & "nwb_file_name LIKE 'Name%'").ddp(dry_run=False)
-(Nwbfile() & "nwb_file_name LIKE 'Other%'").ddp(dry_run=False)
+## Miscellaneous Helper functions
 
-# Faster
-from spyglass.common import Nwbfile
+`file_like` allows you to restrict a table using a substring of a file name.
+This is equivalent to the following:
 
-nwbfile = Nwbfile()
-(nwbfile & "nwb_file_name LIKE 'Name%'").ddp(dry_run=False)
-(nwbfile & "nwb_file_name LIKE 'Other%'").ddp(dry_run=False)
+```python
+MyTable().file_like("eg")
+MyTable() & ('nwb_file_name LIKE "%eg%" OR analysis_file_name LIKE "%eg%"')
+```
+
+`find_insert_fail` is a helper function to find the cause of an `IntegrityError`
+when inserting into a table. This checks parent tables for required keys.
+
+```python
+my_key = {"key": "value"}
+MyTable().insert1(my_key)  # Raises IntegrityError
+MyTable().find_insert_fail(my_key)  # Shows the parent(s) missing the key
 ```
 
 ## Populate Calls

docs/src/Features/index.md

@@ -9,4 +9,3 @@ Spyglass.
 - [Mixin](./Mixin.md) - Spyglass-specific functionalities to DataJoint tables,
     including fetching NWB files, long-distance restrictions, and permission
     checks on delete operations.
-- [Session Groups](./SessionGroups.md) - How to operate on sets of sessions.

examples/cli_examples/readme.md

@@ -109,13 +109,6 @@ spyglass insert-spike-sorter-parameters parameters.yaml
 spyglass list-spike-sorter-parameters
 ```
 
-## Create a session group
-
-A session group is a collection of sessions that can be viewed via spyglassview.
-
-See [session_groups.md](../../docs/session_groups.md) and
-[create_session_group.py](./create_session_group.py).
-
 ## Create spyglass view
 
 ```bash

notebooks/01_Concepts.ipynb

@@ -71,10 +71,7 @@
     "```python\n",
     "my_key = dict(value=key)  # whatever you're inserting\n",
     "MyTable.insert1(my_key)  # error here\n",
-    "parents = MyTable.parents(as_objects=True)  # get the parents as FreeTables\n",
-    "for parent in parents:  # iterate through the parents, with only relevant fields\n",
-    "    parent_key = {k: v for k, v in my_key.items() if k in parent.heading.names}\n",
-    "    print(parent & parent_key)  # restricted parent\n",
+    "parents = MyTable().find_insert_fail(my_key)\n",
     "```\n",
     "\n",
     "If any of the printed tables are empty, you know you need to insert into that\n",

notebooks/py_scripts/01_Concepts.py

@@ -60,10 +60,7 @@
 # ```python
 # my_key = dict(value=key)  # whatever you're inserting
 # MyTable.insert1(my_key)  # error here
-# parents = MyTable.parents(as_objects=True)  # get the parents as FreeTables
-# for parent in parents:  # iterate through the parents, with only relevant fields
-#     parent_key = {k: v for k, v in my_key.items() if k in parent.heading.names}
-#     print(parent & parent_key)  # restricted parent
+# parents = MyTable().find_insert_fail(my_key)
 # ```
 #
 # If any of the printed tables are empty, you know you need to insert into that

pyproject.toml

@@ -39,7 +39,7 @@ dependencies = [
     "black[jupyter]",
     "bottleneck",
     "dask",
-    "datajoint>=0.13.6",
+    "datajoint>=0.14.2",
     # "ghostipy", # removed from list bc M1 users need to install pyfftw first
     "hdmf>=3.4.6",
     "ipympl",
@@ -165,7 +165,8 @@ omit = [ # which submodules have no tests
     # "*/position/*",
     "*/ripple/*",
     "*/sharing/*",
-    "*/spikesorting/*",
+    "*/spikesorting/v0/*",
+    # "*/spikesorting/*",
     # "*/utils/*",
     "settings.py",
 ]

src/spyglass/cli/cli.py

@@ -447,16 +447,6 @@ def list_spike_sortings(nwb_file_name: str):
     print(results)
 
 
-@click.command(help="Create spyglass view of a session group.")
-@click.argument("session_group_name")
-def create_spyglass_view(session_group_name: str):
-    import spyglass.common as sgc
-
-    F = sgc.SessionGroup.create_spyglass_view(session_group_name)
-    url = F.url(label=session_group_name)
-    print(url)
-
-
 cli.add_command(insert_session)
 cli.add_command(list_sessions)
 cli.add_command(insert_lab_team)
@@ -480,4 +470,3 @@ def create_spyglass_view(session_group_name: str):
 cli.add_command(list_spike_sorter_parameters)
 cli.add_command(run_spike_sorting)
 cli.add_command(list_spike_sortings)
-cli.add_command(create_spyglass_view)

src/spyglass/common/__init__.py

@@ -56,7 +56,7 @@
 )
 from spyglass.common.common_region import BrainRegion
 from spyglass.common.common_sensors import SensorData
-from spyglass.common.common_session import Session, SessionGroup
+from spyglass.common.common_session import Session
 from spyglass.common.common_subject import Subject
 from spyglass.common.common_task import Task, TaskEpoch
 from spyglass.common.populate_all_common import populate_all_common