You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
To improve performance a potentially useful change to the underlying code is to add dask-geopandas compatibility (which is an extension of dask). In the case of Midway, this would involve using the dask-slurm scheduler.
The current recommended approach for parallelization is as follows:
Bash script that contains a sbatch template that submits 1 job per node for each country (up to 10 countries at once) and have the script serially process each GADM file and chunk the rows of the GADM file in parallel across the 28 cores
The text was updated successfully, but these errors were encountered:
To improve performance a potentially useful change to the underlying code is to add dask-geopandas compatibility (which is an extension of dask). In the case of Midway, this would involve using the dask-slurm scheduler.
https://jobqueue.dask.org/en/latest/examples.html#slurm-deployments
The current recommended approach for parallelization is as follows:
Bash script that contains a sbatch template that submits 1 job per node for each country (up to 10 countries at once) and have the script serially process each GADM file and chunk the rows of the GADM file in parallel across the 28 cores
The text was updated successfully, but these errors were encountered: