Role of concurrency in order of script execution

[ritab-snow]

I have two python scripts that need to be run in sequence.

version: 2
models:
  - name: tfm_trip_distance
    description: "trip distance"
    meta:
      owner: "@Ritab"
      fal:
        scripts:
        - python_transform/tfm_getzipcode.py  
        - python_transform/tfm_aggr.py
  - name: tfm_trip_aggr
    description: "trip distance aggr"

If I run the setup with concurrency : 200, both the scripts starts running in parallel. However if I run then with concurrency :1 then they run in sequence.

Is that an expected behaviour ?
The documentation says that the order of script in the schema.yml will always be run in sequence. Is that not true ?

[chamini2]

Hey, @ritab-snow . You have a good point that we do not really run them in sequence. I will check out the documentation we have about this to fix it.

The way I would fix this is by making a new file python_transform/tfm_getzipcode_aggr.py (or some other name) with:

import python_transform.tfm_getzipcode as getzipcode
import python_transform.tfm_aggr as aggr

# Make the appropriate calls from here to each script
# getzipcode.<function>
# aggr.<function>

and change the schema.yml to be:

version: 2
models:
  - name: tfm_trip_distance
    description: "trip distance"
    meta:
      owner: "@Ritab"
      fal:
        scripts:
        - python_transform/tfm_getzipcode_aggr.py  
  - name: tfm_trip_aggr
    description: "trip distance aggr"

That way it is a single script running the 2 necessary things in orer.