Barbara/airflow mapper

[barbarahui]

@amywieliczka @bibliotechy Can you take a look at this draft of this mapper dag and give feedback on the approach? Any and all feedback very welcome!

Some specific questions/comments:

1. It seems like parameters can only be accessed inside a task run, meaning that params needs to be unpacked both in fetch_pages() and then again in map_page() (or passed from one task to another) which seems clunky. Chad, are you aware of a different approach to this?
2. Amy, can you verify that this is the format of the payload we're expecting to get from Registry? (lines 57-61)
3. Do we have any collections that have > 1024 pages of vernacular metadata? I'm guessing that some of the Nuxeo complex object collections exceed that. Assuming so, I'll need to figure out how to handle this case. (See my comments on lines 69-72). We can of course increase the value to > 1024, but that seems like a recipe for overloading our MWAA instance.

[bibliotechy]

@barbarahui @amywieliczka My biggest outstanding question is around how we structure the full workflow, from fetching the vernacular to publishing the updated collection to stage calisphere.

If each "part" of the workflow (fetch, map, content fetch, etc) is its own dag, how will we build a single workflow where an operator can trigger a dag with a single collection ID and have the entire workflow execute?

[amywieliczka]

@bibliotechy agreed - I think a harvest_collection dag makes sense, which is comprised of the full pipeline. Though also, I don't think what's done here poses a major problem, and for development iteration purposes, it might be nice to have an isolated component up until the point that it is considered done?

My understanding with the taskflow API is that you can treat tasks like functions for use in multiple dags, and I think even import them between different dag definition python modules. I don't think that's particularly ideal, though.

[barbarahui]

@bibliotechy If we put all of the tasks in a single DAG so that it's possible to run a harvest from start to finish, how do we allow users to run just a subset of tasks, e.g. if somebody wants to just rerun the mapper? I guess we add a parameter that allows the user to specify which task(s) to run, and then exit the task immediately if not included in the list of tasks to run? Is that the right model?