Does Hamilton support datasets that are split across multiple files, specifically non-tabular data?

[mtillman14]

I'm a Ph.D. student in biomechanics doing lots of data analysis. My datasets consist of data from multiple Subjects that each have multiple Trials. Each Trial's binary data file lives in the Subject/Trial folder. Unlike tabular data where a column (a variable) is often processed all at once (from one or multiple CSV files), which seems to be how Hamilton functions (couldn't find this in the docs) in my dataset that is not possible. In my dataset, each "row" is one Trial, which contains multiple timeseries variables (for example, gait speed over time, or even multiple timeseries such as multiple force sensors' readings over time). Therefore each Trial must be processed separately.

Is this structure of dataset & analysis supported by Hamilton? This is such a cool tool and I really hope to be able to use it.

[skrawcz]

Thanks for the questions @mtillman14 .

In short Hamilton is data type agnostic. So I'm sure there's a way to get what you want. But to help you some questions:

1. Can you share a sketch of your data so that we're on the same page please? An example of the structure would be great.
2. What would you like to do with the data? What types of transformations? Do you have any current code now that you could share to give us a sense of that?

[skrawcz]

We generally use the marker data dict directly, there's no benefit for us in converting to tabular form.

Really? Oh okay. I assume this is "small" data then? Tabular data just means you can make better use of underlying data structures/types, e.g. pyarrow, to do faster computation - e.g. you can then also leverage their implementations, e.g. smoothing via pandas API, etc.

For example, the first thing we generally do is smooth the marker data. For that, we just iterate over each marker name's key, apply the smoothing, and save the smoothed data in another dict.

Okay so you basically for each operation create a new dictionary? Sure that works.

Then a sketch of code would be something like this -- assuming you do everything to every marker and treat that data set as a "whole" if you get what I mean:

def raw_marker_data(subject: str, trial: int) -> dict:
    # load marker data for subject trial
    return # some dict

def smoothed_data(raw_marker_data: dict, some_value: int) -> dict:
   # code to smooth ; some_value is some value for smoothing
   return # new dict

def next_transform(smoothed_data: dict) -> dict:
  # code for next transform
  return # new dict

would then create something like this:

Then when you come to run things, you'd just request what "outputs" you want and Hamilton would only walk that part of the graph.

[mtillman14]

This all looks sensible for pipeline dependency management within one Subject & Trial, I'm just not sure how it's implemented when running the code across multiple Subjects & Trials? If I have 10 Subjects each with 10 Trials (100 Trials total), how would Hamilton help manage that?

One of my biggest needs for pipeline management is for example when execution is interrupted during a particular Trial (e.g. Subject 2 Trial 5), and after fixing whatever issue caused the interruption I want to resume processing from that same Trial (Subject 2 Trial 5) onward. Ideally, my management tool would automagically know to resume from Subject 2 Trial 5, instead of restarting at Subject 1 Trial 1.

Does Hamilton support this? Thank you!

[skrawcz]

This all looks sensible for pipeline dependency management within one Subject & Trial, I'm just not sure how it's implemented when running the code across multiple Subjects & Trials? If I have 10 Subjects each with 10 Trials (100 Trials total), how would Hamilton help manage that?

You could either do the double for loop you had, and delegate the inside to Hamilton. Or you use a construct like Parallelizable/Collect.

One of my biggest needs for pipeline management is for example when execution is interrupted during a particular Trial (e.g. Subject 2 Trial 5), and after fixing whatever issue caused the interruption I want to resume processing from that same Trial (Subject 2 Trial 5) onward. Ideally, my management tool would automagically know to resume from Subject 2 Trial 5, instead of restarting at Subject 1 Trial 1.

So Hamilton doesn't contain a database. You would need to use something to manage what was run, e.g. after each successful run mark it somewhere in the filesystem. You can then check that say in the Parallelizable function to know whether to retry it or not. This used in conjunction with the new caching functionality would be a lightweight way to achieve what you want; the caching would only rerun parts that would have changed.

The Hamilton UI would be useful in tracking / seeing runs and executions. But we haven't exposed querying that for a purpose such as this.

CC @zilto who can provide some more thoughts here next week.

[mtillman14]

Thank you, this is great info! My takeaway so far is that the best way to use Hamilton in my workflow would be to treat each unique Subject & Trial combination as its own individual "run" in the Hamilton UI? Is there a way to have something like "nested runs" so that I could see that the last 100 "runs" were all part of 1 meta-run across all 100 Trials? Maybe even have summary stats for the meta-run?

[skrawcz]

So with Parallelizable/Collect you could group a bunch of trials into a single nested run but it would be seen as one big run. Note if any failures occurred, it would stop the whole run.

Alternatively you can "tag" runs -- so you could then filter to them all in the UI. e.g. tag the subject and/or trial, the for a given subject see their trials. This would give you some visibility here -- but we could definitely add more functionality here.

Office hours are on Tuesdays 9:30am PT if you wanted to drop by -- link in the slack.