Dandi export fixes

[samuelbray32]

Description

Fixes for the Dandi export process

• Solves Dandi export of nwb w/ external files #1092
  • Copy files with external paths (e.g. videos) rather than symlink when compiling dandi set (to prevent editing original)
  • Allow dandi.organize to edit the copied nwb to resolve external path issues
• Solves external_link nwb objects
  • Copied raw file (session_.nwb) contains external links for e-series data (not permitted by dandi)
  • During export, trace links to source file, upload that instead. Ensures all raw data uploaded for Dandi
  • When fetching nwb from Dandi, if entry is not in DandiPath check for non-copied file name. If so, stream that file
• Other improvements
  • speed up translation of names to dandi path in translate_name_to_dandi by using lookup of the object id when available
  • Minor fixes in update_analysis_for_dandi_standards()

Development notes

Todo:

• Handle Raw nwb files

  • Original (.nwb)
    • not captured by export since never directly accessed (Add to export list)
    • large file with external filepaths. Ideally would not copy and edit while preventing Dandi export of nwb w/ external files #1092
  • Copy (_.nwb)
    • included in the accessed files list in testing
    • copied to solve Dandi export of nwb w/ external files #1092
      • This breaks the relative filepath for linked nwb objects (e.g. e-series, analog)
      • copying the original nwb into the same folder would resolve but expensive~
    • NEW ERROR
      • cannot be uploaded to DANDI to to external links to nwb objects
      • Will instead need to upload the original nwb file

• Alternative (?)

  • Upload the original nwb but not the copy (_.nwb) file
    • Update This is required by DANDI not allowing external links
  • make new copy nwb file when necessary in the docker image from the original on Dandi

Checklist:

• N This PR should be accompanied by a release: (yes/no/unsure)
• NA If release, I have updated the CITATION.cff
• N This PR makes edits to table definitions: (yes/no)
• NA If table edits, I have included an alter snippet for release notes.
• NA If this PR makes changes to position, I ran the relevant tests locally.
• I have updated the CHANGELOG.md with PR number and description.
• NA I have added/edited docs/notebooks to reflect the changes

[edeno]

Just a couple of minor things.

[CBroz1]

I have some formatting/efficiency comments. If these fixes can fit in before your vacation, great. If not, we can merge what we have and I'll do a clean-up pass later

[CBroz1]

It seems like you're adding items to file_paths, just to remove them again with the index, is that right? I think I would structure this as on overwrite, rather than a gather + remove

keep_links = set() 
for file in file_paths:
    if not (links := get_linked(...)):
         keep_files.add(file)
    logger.warning(...)
file_paths = {'file_path': link for link in keep_links}

unique_dicts is slow. Adding things to a set enforces uniqueness as you go

[samuelbray32]

Fair, I 'll put in a cleaner method along these lines

[edeno]

Looks good on my end (pending changelog conflicts)