diff --git a/README.md b/README.md index 485576e..f990105 100644 --- a/README.md +++ b/README.md @@ -27,6 +27,7 @@ Comprehensive benchmarking of protein-ligand structure generation methods - [Tutorials](#tutorials) - [How to prepare PoseBench data](#how-to-prepare-posebench-data) - [Available inference methods](#available-inference-methods) +- [How to run a sweep of benchmarking experiments](#how-to-run-a-sweep-of-benchmarking-experiments) - [How to run inference with individual methods](#how-to-run-inference-with-individual-methods) - [How to run inference with a method ensemble](#how-to-run-inference-with-a-method-ensemble) - [How to create comparative plots of inference results](#how-to-create-comparative-plots-of-inference-results) @@ -325,6 +326,26 @@ python3 posebench/data/components/protein_apo_to_holo_alignment.py dataset=astex +## How to run a sweep of benchmarking experiments + +
+ +Build inference scripts for one's desired sweep + +```bash +python3 scripts/build_inference_script.py sweep=true export_hpc_headers=true +``` + +Submit the inference scripts for job scheduling + +```bash +sbatch scripts/inference/*_inference_*.sh +``` + +**NOTE**: See the config file `configs/scripts/build_inference_script.yaml` for more details. + +
+ ## How to run inference with individual methods
@@ -337,7 +358,7 @@ Prepare CSV input files python3 posebench/data/diffdock_input_preparation.py dataset=posebusters_benchmark python3 posebench/data/diffdock_input_preparation.py dataset=astex_diverse python3 posebench/data/diffdock_input_preparation.py dataset=dockgen -python3 posebench/data/diffdock_input_preparation.py dataset=casp15 input_data_dir="$PWD"/data/casp15_set/targets input_protein_structure_dir="$PWD"/data/casp15_set/casp15_holo_aligned_predicted_structures +python3 posebench/data/diffdock_input_preparation.py dataset=casp15 input_data_dir=data/casp15_set/targets input_protein_structure_dir=data/casp15_set/casp15_holo_aligned_predicted_structures ``` Run inference on each dataset @@ -356,16 +377,14 @@ python3 posebench/models/diffdock_inference.py dataset=casp15 batch_size=1 repea Relax the generated ligand structures inside of their respective protein pockets ```bash -python3 posebench/models/inference_relaxation.py method=diffdock dataset=posebusters_benchmark remove_initial_protein_hydrogens=true assign_partial_charges_manually=true num_processes=1 repeat_index=1 +python3 posebench/models/inference_relaxation.py method=diffdock dataset=posebusters_benchmark remove_initial_protein_hydrogens=true assign_partial_charges_manually=true repeat_index=1 ... -python3 posebench/models/inference_relaxation.py method=diffdock dataset=astex_diverse remove_initial_protein_hydrogens=true assign_partial_charges_manually=true num_processes=1 repeat_index=1 +python3 posebench/models/inference_relaxation.py method=diffdock dataset=astex_diverse remove_initial_protein_hydrogens=true assign_partial_charges_manually=true repeat_index=1 ... -python3 posebench/models/inference_relaxation.py method=diffdock dataset=dockgen remove_initial_protein_hydrogens=true assign_partial_charges_manually=true num_processes=1 repeat_index=1 +python3 posebench/models/inference_relaxation.py method=diffdock dataset=dockgen remove_initial_protein_hydrogens=true assign_partial_charges_manually=true repeat_index=1 ... ``` -**NOTE**: Increase `num_processes` according to your available CPU/GPU resources to improve throughput - Analyze inference results for each dataset ```bash @@ -414,16 +433,14 @@ python3 posebench/models/fabind_inference.py dataset=dockgen repeat_index=1 Relax the generated ligand structures inside of their respective protein pockets ```bash -python3 posebench/models/inference_relaxation.py method=fabind dataset=posebusters_benchmark remove_initial_protein_hydrogens=true assign_partial_charges_manually=true num_processes=1 repeat_index=1 +python3 posebench/models/inference_relaxation.py method=fabind dataset=posebusters_benchmark remove_initial_protein_hydrogens=true assign_partial_charges_manually=true repeat_index=1 ... -python3 posebench/models/inference_relaxation.py method=fabind dataset=astex_diverse remove_initial_protein_hydrogens=true assign_partial_charges_manually=true num_processes=1 repeat_index=1 +python3 posebench/models/inference_relaxation.py method=fabind dataset=astex_diverse remove_initial_protein_hydrogens=true assign_partial_charges_manually=true repeat_index=1 ... -python3 posebench/models/inference_relaxation.py method=fabind dataset=dockgen remove_initial_protein_hydrogens=true assign_partial_charges_manually=true num_processes=1 repeat_index=1 +python3 posebench/models/inference_relaxation.py method=fabind dataset=dockgen remove_initial_protein_hydrogens=true assign_partial_charges_manually=true repeat_index=1 ... ``` -**NOTE**: Increase `num_processes` according to your available CPU/GPU resources to improve throughput - Analyze inference results for each dataset ```bash @@ -443,7 +460,7 @@ Prepare CSV input files python3 posebench/data/dynamicbind_input_preparation.py dataset=posebusters_benchmark python3 posebench/data/dynamicbind_input_preparation.py dataset=astex_diverse python3 posebench/data/dynamicbind_input_preparation.py dataset=dockgen -python3 posebench/data/dynamicbind_input_preparation.py dataset=casp15 input_data_dir="$PWD"/data/casp15_set/targets +python3 posebench/data/dynamicbind_input_preparation.py dataset=casp15 input_data_dir=data/casp15_set/targets ``` Run inference on each dataset @@ -455,23 +472,21 @@ python3 posebench/models/dynamicbind_inference.py dataset=astex_diverse repeat_i ... python3 posebench/models/dynamicbind_inference.py dataset=dockgen repeat_index=1 ... -python3 posebench/models/dynamicbind_inference.py dataset=casp15 batch_size=1 input_data_dir="$PWD"/data/casp15_set/casp15_holo_aligned_predicted_structures repeat_index=1 +python3 posebench/models/dynamicbind_inference.py dataset=casp15 batch_size=1 input_data_dir=data/casp15_set/casp15_holo_aligned_predicted_structures repeat_index=1 ... ``` Relax the generated ligand structures inside of their respective protein pockets ```bash -python3 posebench/models/inference_relaxation.py method=dynamicbind dataset=posebusters_benchmark remove_initial_protein_hydrogens=true assign_partial_charges_manually=true num_processes=1 repeat_index=1 +python3 posebench/models/inference_relaxation.py method=dynamicbind dataset=posebusters_benchmark remove_initial_protein_hydrogens=true assign_partial_charges_manually=true repeat_index=1 ... -python3 posebench/models/inference_relaxation.py method=dynamicbind dataset=astex_diverse remove_initial_protein_hydrogens=true assign_partial_charges_manually=true num_processes=1 repeat_index=1 +python3 posebench/models/inference_relaxation.py method=dynamicbind dataset=astex_diverse remove_initial_protein_hydrogens=true assign_partial_charges_manually=true repeat_index=1 ... -python3 posebench/models/inference_relaxation.py method=dynamicbind dataset=dockgen remove_initial_protein_hydrogens=true assign_partial_charges_manually=true num_processes=1 repeat_index=1 +python3 posebench/models/inference_relaxation.py method=dynamicbind dataset=dockgen remove_initial_protein_hydrogens=true assign_partial_charges_manually=true repeat_index=1 ... ``` -**NOTE**: Increase `num_processes` according to your available CPU/GPU resources to improve throughput - Analyze inference results for each dataset ```bash @@ -504,7 +519,7 @@ Prepare CSV input files python3 posebench/data/neuralplexer_input_preparation.py dataset=posebusters_benchmark python3 posebench/data/neuralplexer_input_preparation.py dataset=astex_diverse python3 posebench/data/neuralplexer_input_preparation.py dataset=dockgen -python3 posebench/data/neuralplexer_input_preparation.py dataset=casp15 input_data_dir="$PWD"/data/casp15_set/targets input_receptor_structure_dir="$PWD"/data/casp15_set/casp15_holo_aligned_predicted_structures +python3 posebench/data/neuralplexer_input_preparation.py dataset=casp15 input_data_dir=data/casp15_set/targets input_receptor_structure_dir=data/casp15_set/casp15_holo_aligned_predicted_structures ``` Run inference on each dataset @@ -516,23 +531,21 @@ python3 posebench/models/neuralplexer_inference.py dataset=astex_diverse repeat_ ... python3 posebench/models/neuralplexer_inference.py dataset=dockgen repeat_index=1 ... -python3 posebench/models/neuralplexer_inference.py dataset=casp15 repeat_index=1 +python3 posebench/models/neuralplexer_inference.py dataset=casp15 chunk_size=4 repeat_index=1 ... ``` Relax the generated ligand structures inside of their respective protein pockets ```bash -python3 posebench/models/inference_relaxation.py method=neuralplexer dataset=posebusters_benchmark num_processes=1 remove_initial_protein_hydrogens=true assign_partial_charges_manually=true cache_files=false repeat_index=1 +python3 posebench/models/inference_relaxation.py method=neuralplexer dataset=posebusters_benchmark remove_initial_protein_hydrogens=true assign_partial_charges_manually=true repeat_index=1 ... -python3 posebench/models/inference_relaxation.py method=neuralplexer dataset=astex_diverse num_processes=1 remove_initial_protein_hydrogens=true assign_partial_charges_manually=true cache_files=false repeat_index=1 +python3 posebench/models/inference_relaxation.py method=neuralplexer dataset=astex_diverse remove_initial_protein_hydrogens=true assign_partial_charges_manually=true repeat_index=1 ... -python3 posebench/models/inference_relaxation.py method=neuralplexer dataset=dockgen num_processes=1 remove_initial_protein_hydrogens=true assign_partial_charges_manually=true cache_files=false repeat_index=1 +python3 posebench/models/inference_relaxation.py method=neuralplexer dataset=dockgen remove_initial_protein_hydrogens=true assign_partial_charges_manually=true repeat_index=1 ... ``` -**NOTE**: Increase `num_processes` according to your available CPU/GPU resources to improve throughput - Align predicted protein-ligand structures to ground-truth complex structures ```bash @@ -576,7 +589,7 @@ Prepare CSV input files python3 posebench/data/rfaa_input_preparation.py dataset=posebusters_benchmark python3 posebench/data/rfaa_input_preparation.py dataset=astex_diverse python3 posebench/data/rfaa_input_preparation.py dataset=dockgen -python3 posebench/data/rfaa_input_preparation.py dataset=casp15 input_data_dir="$PWD"/data/casp15_set/targets +python3 posebench/data/rfaa_input_preparation.py dataset=casp15 input_data_dir=data/casp15_set/targets ``` Run inference on each dataset @@ -602,13 +615,11 @@ python3 posebench/data/rfaa_output_extraction.py dataset=casp15 Relax the generated ligand structures inside of their respective protein pockets ```bash -python3 posebench/models/inference_relaxation.py method=rfaa dataset=posebusters_benchmark num_processes=1 remove_initial_protein_hydrogens=true -python3 posebench/models/inference_relaxation.py method=rfaa dataset=astex_diverse num_processes=1 remove_initial_protein_hydrogens=true -python3 posebench/models/inference_relaxation.py method=rfaa dataset=dockgen num_processes=1 remove_initial_protein_hydrogens=true +python3 posebench/models/inference_relaxation.py method=rfaa dataset=posebusters_benchmark remove_initial_protein_hydrogens=true +python3 posebench/models/inference_relaxation.py method=rfaa dataset=astex_diverse remove_initial_protein_hydrogens=true +python3 posebench/models/inference_relaxation.py method=rfaa dataset=dockgen remove_initial_protein_hydrogens=true ``` -**NOTE**: Increase `num_processes` according to your available CPU/GPU resources to improve throughput - Align predicted protein-ligand structures to ground-truth complex structures ```bash @@ -634,7 +645,7 @@ python3 posebench/models/ensemble_generation.py ensemble_methods=\[rfaa\] input_ # NOTE: the suffixes for both `output_dir` and `ensemble_benchmarking_repeat_index` should be modified to e.g., 2, 3, ... ... # now score the CASP15-compliant submissions using the official CASP scoring pipeline -python3 posebench/analysis/inference_analysis_casp.py method=rfaa dataset=casp15 targets="[T1124, T1127v2, T1146, T1152, T1158v1, T1158v2, T1158v3, T1158v4, T1186, T1187, T1188]" repeat_index=1 +python3 posebench/analysis/inference_analysis_casp.py method=rfaa dataset=casp15 targets='[T1124, T1127v2, T1146, T1152, T1158v1, T1158v2, T1158v3, T1158v4, T1186, T1187, T1188]' repeat_index=1 ... ``` @@ -678,16 +689,14 @@ mkdir -p forks/Vina/inference/vina_diffdock_casp15_outputs_1 && cp -r data/test_ Relax the generated ligand structures inside of their respective protein pockets ```bash -python3 posebench/models/inference_relaxation.py method=vina vina_binding_site_method=diffdock dataset=posebusters_benchmark remove_initial_protein_hydrogens=true assign_partial_charges_manually=true num_processes=1 repeat_index=1 +python3 posebench/models/inference_relaxation.py method=vina vina_binding_site_method=diffdock dataset=posebusters_benchmark remove_initial_protein_hydrogens=true assign_partial_charges_manually=true repeat_index=1 ... -python3 posebench/models/inference_relaxation.py method=vina vina_binding_site_method=diffdock dataset=astex_diverse remove_initial_protein_hydrogens=true assign_partial_charges_manually=true num_processes=1 repeat_index=1 +python3 posebench/models/inference_relaxation.py method=vina vina_binding_site_method=diffdock dataset=astex_diverse remove_initial_protein_hydrogens=true assign_partial_charges_manually=true repeat_index=1 ... -python3 posebench/models/inference_relaxation.py method=vina vina_binding_site_method=diffdock dataset=dockgen remove_initial_protein_hydrogens=true assign_partial_charges_manually=true num_processes=1 repeat_index=1 +python3 posebench/models/inference_relaxation.py method=vina vina_binding_site_method=diffdock dataset=dockgen remove_initial_protein_hydrogens=true assign_partial_charges_manually=true repeat_index=1 ... ``` -**NOTE**: Increase `num_processes` according to your available CPU/GPU resources to improve throughput - Analyze inference results for each dataset ```bash @@ -719,20 +728,21 @@ Gather all template ligands generated by `TULIP` via its dedicated [GitHub repos ```bash python3 posebench/data/tulip_output_extraction.py dataset=posebusters_benchmark python3 posebench/data/tulip_output_extraction.py dataset=astex_diverse +python3 posebench/data/tulip_output_extraction.py dataset=dockgen python3 posebench/data/tulip_output_extraction.py dataset=casp15 ``` Relax the generated ligand structures inside of their respective protein pockets ```bash -python3 posebench/models/inference_relaxation.py method=tulip dataset=posebusters_benchmark remove_initial_protein_hydrogens=true assign_partial_charges_manually=true num_processes=1 +python3 posebench/models/inference_relaxation.py method=tulip dataset=posebusters_benchmark remove_initial_protein_hydrogens=true assign_partial_charges_manually=true +... +python3 posebench/models/inference_relaxation.py method=tulip dataset=astex_diverse remove_initial_protein_hydrogens=true assign_partial_charges_manually=true ... -python3 posebench/models/inference_relaxation.py method=tulip dataset=astex_diverse remove_initial_protein_hydrogens=true assign_partial_charges_manually=true num_processes=1 +python3 posebench/models/inference_relaxation.py method=tulip dataset=dockgen remove_initial_protein_hydrogens=true assign_partial_charges_manually=true ... ``` -**NOTE**: Increase `num_processes` according to your available CPU/GPU resources to improve throughput - Analyze inference results for each dataset ```bash @@ -740,6 +750,8 @@ python3 posebench/analysis/inference_analysis.py method=tulip dataset=posebuster ... python3 posebench/analysis/inference_analysis.py method=tulip dataset=astex_diverse ... +python3 posebench/analysis/inference_analysis.py method=tulip dataset=dockgen +... ``` Analyze inference results for the CASP15 dataset @@ -789,8 +801,8 @@ python3 posebench/models/ensemble_generation.py input_csv_filepath=data/test_cas python3 posebench/models/ensemble_generation.py input_csv_filepath=data/test_cases/astex_diverse/ensemble_inputs.csv output_dir=data/test_cases/astex_diverse/top_consensus_ensemble_predictions_1 max_method_predictions=40 export_top_n=1 export_file_format=null skip_existing=true relax_method_ligands_post_ranking=true resume=true cuda_device_index=0 ensemble_methods='[diffdock, dynamicbind, neuralplexer, rfaa, tulip, vina]' ensemble_benchmarking=true ensemble_benchmarking_dataset=astex_diverse ensemble_ranking_method=consensus ensemble_benchmarking_repeat_index=1 ... # benchmark using the DockGen dataset e.g., after generating 40 complexes per target with each method -python3 posebench/models/ensemble_generation.py input_csv_filepath=data/test_cases/dockgen/ensemble_inputs.csv output_dir=data/test_cases/dockgen/top_consensus_ensemble_predictions_1 max_method_predictions=40 export_top_n=1 export_file_format=null skip_existing=true relax_method_ligands_post_ranking=false resume=true cuda_device_index=0 ensemble_methods='[diffdock, dynamicbind, neuralplexer, rfaa, vina]' ensemble_benchmarking=true ensemble_benchmarking_dataset=dockgen ensemble_ranking_method=consensus ensemble_benchmarking_repeat_index=1 -python3 posebench/models/ensemble_generation.py input_csv_filepath=data/test_cases/dockgen/ensemble_inputs.csv output_dir=data/test_cases/dockgen/top_consensus_ensemble_predictions_1 max_method_predictions=40 export_top_n=1 export_file_format=null skip_existing=true relax_method_ligands_post_ranking=true resume=true cuda_device_index=0 ensemble_methods='[diffdock, dynamicbind, neuralplexer, rfaa, vina]' ensemble_benchmarking=true ensemble_benchmarking_dataset=dockgen ensemble_ranking_method=consensus ensemble_benchmarking_repeat_index=1 +python3 posebench/models/ensemble_generation.py input_csv_filepath=data/test_cases/dockgen/ensemble_inputs.csv output_dir=data/test_cases/dockgen/top_consensus_ensemble_predictions_1 max_method_predictions=40 export_top_n=1 export_file_format=null skip_existing=true relax_method_ligands_post_ranking=false resume=true cuda_device_index=0 ensemble_methods='[diffdock, dynamicbind, neuralplexer, rfaa, tulip, vina]' ensemble_benchmarking=true ensemble_benchmarking_dataset=dockgen ensemble_ranking_method=consensus ensemble_benchmarking_repeat_index=1 +python3 posebench/models/ensemble_generation.py input_csv_filepath=data/test_cases/dockgen/ensemble_inputs.csv output_dir=data/test_cases/dockgen/top_consensus_ensemble_predictions_1 max_method_predictions=40 export_top_n=1 export_file_format=null skip_existing=true relax_method_ligands_post_ranking=true resume=true cuda_device_index=0 ensemble_methods='[diffdock, dynamicbind, neuralplexer, rfaa, tulip, vina]' ensemble_benchmarking=true ensemble_benchmarking_dataset=dockgen ensemble_ranking_method=consensus ensemble_benchmarking_repeat_index=1 ... # benchmark using the CASP15 dataset e.g., after generating 40 complexes per target with each method python3 posebench/models/ensemble_generation.py input_csv_filepath=data/test_cases/casp15/ensemble_inputs.csv output_dir=data/test_cases/casp15/top_consensus_ensemble_predictions_1 combine_casp_output_files=true max_method_predictions=40 export_top_n=5 export_file_format=casp15 skip_existing=true relax_method_ligands_post_ranking=false resume=true cuda_device_index=0 ensemble_methods='[diffdock, dynamicbind, neuralplexer, rfaa, tulip, vina]' ensemble_benchmarking=true ensemble_benchmarking_dataset=casp15 ensemble_ranking_method=consensus ensemble_benchmarking_repeat_index=1 @@ -806,7 +818,7 @@ python3 posebench/analysis/inference_analysis.py method=ensemble dataset=astex_d python3 posebench/analysis/inference_analysis.py method=ensemble dataset=dockgen repeat_index=1 ... # analyze benchmarking results for the CASP15 dataset -python3 posebench/analysis/inference_analysis_casp.py method=ensemble dataset=casp15 ensemble_ranking_method=consensus repeat_index=1 +python3 posebench/analysis/inference_analysis_casp.py method=ensemble dataset=casp15 repeat_index=1 ... ``` diff --git a/docs/source/acknowledgements.rst b/docs/source/acknowledgements.rst index af2b6aa..6f5faa7 100644 --- a/docs/source/acknowledgements.rst +++ b/docs/source/acknowledgements.rst @@ -2,5 +2,5 @@ Acknowledgements ================ .. mdinclude:: ../../README.md - :start-line: 875 - :end-line: 892 + :start-line: 887 + :end-line: 904 diff --git a/docs/source/available_methods.rst b/docs/source/available_methods.rst index 0ebb58d..5441165 100644 --- a/docs/source/available_methods.rst +++ b/docs/source/available_methods.rst @@ -2,8 +2,8 @@ Available inference methods ================ .. mdinclude:: ../../README.md - :start-line: 286 - :end-line: 323 + :start-line: 287 + :end-line: 324 .. note:: Have a new method to add? Please let us know by creating a pull request. We would be happy to work with you to integrate new methodology into this benchmark! diff --git a/docs/source/bonus.rst b/docs/source/bonus.rst index e0896f8..73c3257 100644 --- a/docs/source/bonus.rst +++ b/docs/source/bonus.rst @@ -2,8 +2,8 @@ Bonus ================ .. mdinclude:: ../../README.md - :start-line: 910 - :end-line: 912 + :start-line: 922 + :end-line: 924 .. image:: ./_static/WorkBench.jpeg :alt: My brain after building PoseBench diff --git a/docs/source/citing_this_work.rst b/docs/source/citing_this_work.rst index cca6931..26cb2e1 100644 --- a/docs/source/citing_this_work.rst +++ b/docs/source/citing_this_work.rst @@ -2,5 +2,5 @@ Citing this work ================ .. mdinclude:: ../../README.md - :start-line: 894 - :end-line: 906 + :start-line: 906 + :end-line: 918 diff --git a/docs/source/comparative_plots.rst b/docs/source/comparative_plots.rst index c4cd431..58fb04a 100644 --- a/docs/source/comparative_plots.rst +++ b/docs/source/comparative_plots.rst @@ -2,5 +2,5 @@ How to create comparative plots of inference results ================ .. mdinclude:: ../../README.md - :start-line: 822 - :end-line: 831 + :start-line: 834 + :end-line: 843 diff --git a/docs/source/data_preparation.rst b/docs/source/data_preparation.rst index 10eecd2..bc54df2 100644 --- a/docs/source/data_preparation.rst +++ b/docs/source/data_preparation.rst @@ -2,5 +2,5 @@ How to prepare `PoseBench` data ================ .. mdinclude:: ../../README.md - :start-line: 147 - :end-line: 280 + :start-line: 148 + :end-line: 281 diff --git a/docs/source/ensemble_inference.rst b/docs/source/ensemble_inference.rst index 11f20e4..fde71c3 100644 --- a/docs/source/ensemble_inference.rst +++ b/docs/source/ensemble_inference.rst @@ -2,8 +2,8 @@ How to run inference with a method ensemble ================ .. mdinclude:: ../../README.md - :start-line: 763 - :end-line: 814 + :start-line: 775 + :end-line: 828 .. note:: In addition to having `consensus` as an available value for `ensemble_ranking_method`, one can also set `ensemble_ranking_method=ff` to have the method ensemble's top-ranked predictions selected using the criterion of "minimum (molecular dynamics) force field energy" (albeit while incurring a very large runtime complexity). diff --git a/docs/source/for_developers.rst b/docs/source/for_developers.rst index ecc12d3..195e22c 100644 --- a/docs/source/for_developers.rst +++ b/docs/source/for_developers.rst @@ -2,5 +2,5 @@ For developers ================ .. mdinclude:: ../../README.md - :start-line: 837 - :end-line: 871 + :start-line: 849 + :end-line: 883 diff --git a/docs/source/index.rst b/docs/source/index.rst index 874b012..ca81fad 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -29,6 +29,7 @@ Welcome to PoseBench's documentation! tutorials data_preparation available_methods + sweep_inference method_inference ensemble_inference comparative_plots diff --git a/docs/source/installation.rst b/docs/source/installation.rst index 3c00cf9..d0786be 100644 --- a/docs/source/installation.rst +++ b/docs/source/installation.rst @@ -2,5 +2,5 @@ Installation ================ .. mdinclude:: ../../README.md - :start-line: 41 - :end-line: 129 + :start-line: 42 + :end-line: 130 diff --git a/docs/source/method_inference.rst b/docs/source/method_inference.rst index 2bfcb18..09a268e 100644 --- a/docs/source/method_inference.rst +++ b/docs/source/method_inference.rst @@ -2,5 +2,5 @@ How to run inference with individual methods ================ .. mdinclude:: ../../README.md - :start-line: 331 - :end-line: 757 + :start-line: 352 + :end-line: 769 diff --git a/docs/source/sweep_inference.rst b/docs/source/sweep_inference.rst new file mode 100644 index 0000000..3924214 --- /dev/null +++ b/docs/source/sweep_inference.rst @@ -0,0 +1,6 @@ +How to run a sweep of benchmarking experiments +================ + +.. mdinclude:: ../../README.md + :start-line: 332 + :end-line: 346 diff --git a/docs/source/tutorials.rst b/docs/source/tutorials.rst index cdaab80..090eb38 100644 --- a/docs/source/tutorials.rst +++ b/docs/source/tutorials.rst @@ -2,5 +2,5 @@ Tutorials ================ .. mdinclude:: ../../README.md - :start-line: 135 - :end-line: 141 + :start-line: 136 + :end-line: 142