Running Data Preparation Code

[infusion-zero-edit]

Once all the .pdb files downloaded and then the post process is run it says "{file path} could not be parsed: Muscle could not be found in the path". Can you please help resolve this issue.

[kxz18]

Hi, did you run scripts/setup.sh first to create the right environment?

[infusion-zero-edit]

Yes have run that the conda env with name mean is created using env.yml file

[kxz18]

Could you please provide the traceback of the error?

[infusion-zero-edit]

100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7152/7152 [04:19<00:00, 27.55it/s]
number of downloaded entries: 6989
writing PDB files to summaries/pdb
6989it [00:04, 1498.85it/s]
post processing
0%| | 0/6989 [00:00<?, ?it/s]/home/ec2-user/SageMaker/MEAN/summaries/pdb/7up9.pdb could not be parsed: Muscle was not found and there are suspected missing residues. Numbering could be incorrect
renumbering failed for 7up9. scheme imgt
/home/ec2-user/SageMaker/MEAN/summaries/pdb/7upb.pdb could not be parsed: Muscle could not be found in the path
/home/ec2-user/SageMaker/MEAN/summaries/pdb/7uop.pdb could not be parsed: Muscle was not found and there are suspected missing residues. Numbering could be incorrect
/home/ec2-user/SageMaker/MEAN/summaries/pdb/7upd.pdb could not be parsed: Muscle was not found and there are suspected missing residues. Numbering could be incorrect
renumbering failed for 7upb. scheme imgt
renumbering failed for 7uop. scheme imgt
0%| | 1/6989 [00:03<5:56:56, 3.06s/it]renumbering failed for 7upd. scheme imgt
0%| | 4/6989 [00:03<1:11:04, 1.64it/s]/home/ec2-user/SageMaker/MEAN/summaries/pdb/7ys6.pdb could not be parsed: Muscle was not found and there are suspected missing residues. Numbering could be incorrect
renumbering failed for 7ys6. scheme imgt
/home/ec2-user/SageMaker/MEAN/summaries/pdb/8d7e.pdb could not be parsed: Muscle could not be found in the path
/home/ec2-user/SageMaker/MEAN/summaries/pdb/7upk.pdb could not be parsed: Muscle was not found and there are suspected missing residues. Numbering could be incorrect
renumbering failed for 8d7e. scheme imgt
renumbering failed for 7upk. scheme imgt
0%| | 6/6989 [00:05<1:42:17, 1.14it/s]/home/ec2-user/SageMaker/MEAN/summaries/pdb/8cxr.pdb could not be parsed: Muscle could not be found in the path
renumbering failed for 8cxr. scheme imgt
0%|▏ | 7/6989 [00:06<1:45:40, 1.10it/s]/home/ec2-user/SageMaker/MEAN/summaries/pdb/8f8v.pdb could not be parsed: Muscle could not be found in the path
renumbering failed for 8f8v. scheme imgt
/home/ec2-user/SageMaker/MEAN/summaries/pdb/8dk6.pdb could not be parsed: Muscle was not found and there are suspected missing residues. Numbering could be incorrect
renumbering failed for 8dk6. scheme imgt
0%|▏ | 9/6989 [00:07<1:14:53, 1.55it/s]/home/ec2-user/SageMaker/MEAN/summaries/pdb/8f0l.pdb could not be parsed: Muscle was not found and there are suspected missing residues. Numbering could be incorrect
renumbering failed for 8f0l. scheme imgt
0%|▏ | 10/6989 [00:07<1:18:43, 1.48it/s]/home/ec2-user/SageMaker/MEAN/summaries/pdb/8f8x.pdb could not be parsed: Muscle could not be found in the path
renumbering failed for 8f8x. scheme imgt
/home/ec2-user/SageMaker/MEAN/summaries/pdb/8f8w.pdb could not be parsed: Muscle could not be found in the path
renumbering failed for 8f8w. scheme imgt
0%|▏ | 12/6989 [00:09<1:19:22, 1.46it/s]/home/ec2-user/SageMaker/MEAN/summaries/pdb/8fmz.pdb could not be parsed: Muscle was not found and there are suspected missing residues. Numbering could be incorrect
renumbering failed for 8fmz. scheme imgt
0%|▎ | 14/6989 [00:09<54:04, 2.15it/s]/home/ec2-user/SageMaker/MEAN/summaries/pdb/8fn0.pdb could not be parsed: Muscle was not found and there are suspected missing residues. Numbering could be incorrect
renumbering failed for 8fn0. scheme imgt
0%|▎ | 15/6989 [00:10<59:28, 1.95it/s]/home/ec2-user/SageMaker/MEAN/summaries/pdb/8fn1.pdb could not be parsed: Muscle could not be found in the path
renumbering failed for 8fn1. scheme imgt
0%|▎ | 16/6989 [00:10<1:06:44, 1.74it/s]/home/ec2-user/SageMaker/MEAN/summaries/pdb/8il3.pdb could not be parsed: Muscle could not be found in the path
renumbering failed for 8il3. scheme imgt
/home/ec2-user/SageMaker/MEAN/summaries/pdb/6fe4.pdb could not be parsed: Muscle could not be found in the path
renumbering failed for 6fe4. scheme imgt
/home/ec2-user/SageMaker/MEAN/summaries/pdb/8i2g.pdb could not be parsed: Muscle was not found and there are suspected missing residues. Numbering could be incorrect
/home/ec2-user/SageMaker/MEAN/summaries/pdb/7jmo.pdb could not be parsed: Muscle could not be found in the path
renumbering failed for 7jmo. scheme imgt
renumbering failed for 8i2g. scheme imgt
0%|▎ | 17/6989 [00:12<1:37:07, 1.20it/s]/home/ec2-user/SageMaker/MEAN/summaries/pdb/6ch9.pdb could not be parsed: Muscle was not found and there are suspected missing residues. Numbering could be incorrect
renumbering failed for 6ch9. scheme imgt
/home/ec2-user/SageMaker/MEAN/summaries/pdb/4o51.pdb could not be parsed: Muscle could not be found in the path
renumbering failed for 4o51. scheme imgt
/home/ec2-user/SageMaker/MEAN/summaries/pdb/7sgf.pdb could not be parsed: Muscle was not found and there are suspected missing residues. Numbering could be incorrect
renumbering failed for 7sgf. scheme imgt
0%|▎ | 21/6989 [00:16<1:42:39, 1.13it/s]/home/ec2-user/SageMaker/MEAN/summaries/pdb/7mtb.pdb could not be parsed: Muscle was not found and there are suspected missing residues. Numbering could be incorrect
renumbering failed for 7mtb. scheme imgt
0%|▍ | 24/6989 [00:17<1:14:34, 1.56it/s]/home/ec2-user/SageMaker/MEAN/summaries/pdb/5l21.pdb could not be parsed: Muscle was not found and there are suspected missing residues. Numbering could be incorrect
renumbering failed for 5l21. scheme imgt
/home/ec2-user/SageMaker/MEAN/summaries/pdb/7orb.pdb could not be parsed: Muscle could not be found in the path
renumbering failed for 7orb. scheme imgt
/home/ec2-user/SageMaker/MEAN/summaries/pdb/7n0h.pdb could not be parsed: Muscle was not found and there are suspected missing residues. Numbering could be incorrect
renumbering failed for 7n0h. scheme imgt
0%|▍ | 25/6989 [00:20<1:58:15, 1.02s/it]/home/ec2-user/SageMaker/MEAN/summaries/pdb/7q1u.pdb could not be parsed: Muscle was not found and there are suspected missing residues. Numbering could be incorrect
renumbering failed for 7q1u. scheme imgt
/home/ec2-user/SageMaker/MEAN/summaries/pdb/6cw2.pdb could not be parsed: Muscle could not be found in the path
renumbering failed for 6cw2. scheme imgt
/home/ec2-user/SageMaker/MEAN/summaries/pdb/5w08.pdb could not be parsed: Muscle could not be found in the path
renumbering failed for 5w08. scheme imgt
0%|▍ | 27/6989 [00:23<2:13:34, 1.15s/it]/home/ec2-user/SageMaker/MEAN/summaries/pdb/5i1e.pdb could not be parsed: Muscle could not be found in the path
/home/ec2-user/SageMaker/MEAN/summaries/pdb/7a17.pdb could not be parsed: Muscle was not found and there are suspected missing residues. Numbering could be incorrect
renumbering failed for 5i1e. scheme imgt
renumbering failed for 7a17. scheme imgt
/home/ec2-user/SageMaker/MEAN/summaries/pdb/7lg6.pdb could not be parsed: Muscle was not found and there are suspected missing residues. Numbering could be incorrect
renumbering failed for 7lg6. scheme imgt
0%|▌ | 30/6989 [00:25<1:53:19, 1.02it/s]/home/ec2-user/SageMaker/MEAN/summaries/pdb/3eo0.pdb could not be parsed: Muscle could not be found in the path
renumbering failed for 3eo0. scheme imgt
/home/ec2-user/SageMaker/MEAN/summaries/pdb/6p67.pdb could not be parsed: Muscle could not be found in the path
renumbering failed for 6p67. scheme imgt
/home/ec2-user/SageMaker/MEAN/summaries/pdb/6njm.pdb could not be parsed: Muscle was not found and there are suspected missing residues. Numbering could be incorrect
renumbering failed for 6njm. scheme imgt
0%|▌ | 32/6989 [00:28<2:20:19, 1.21s/it]/home/ec2-user/SageMaker/MEAN/summaries/pdb/6x02.pdb could not be parsed: Muscle was not found and there are suspected missing residues. Numbering could be incorrect
renumbering failed for 6x02. scheme imgt
/home/ec2-user/SageMaker/MEAN/summaries/pdb/7t6s.pdb could not be parsed: Muscle was not found and there are suspected missing residues. Numbering could be incorrect
renumbering failed for 7t6s. scheme imgt
/home/ec2-user/SageMaker/MEAN/summaries/pdb/4ffv.pdb could not be parsed: Muscle could not be found in the path
renumbering failed for 4ffv. scheme imgt
/home/ec2-user/SageMaker/MEAN/summaries/pdb/6zdh.pdb could not be parsed: Muscle was not found and there are suspected missing residues. Numbering could be incorrect
renumbering failed for 6zdh. scheme imgt
1%|▋ | 37/6989 [00:33<1:59:12, 1.03s/it]/home/ec2-user/SageMaker/MEAN/summaries/pdb/5cjs.pdb could not be parsed: Muscle could not be found in the path
renumbering failed for 5cjs. scheme imgt
/home/ec2-user/SageMaker/MEAN/summaries/pdb/7wwj.pdb could not be parsed: Muscle was not found and there are suspected missing residues. Numbering could be incorrect
renumbering failed for 7wwj. scheme imgt
/home/ec2-user/SageMaker/MEAN/summaries/pdb/7c

[infusion-zero-edit]

after successfully downloading that, in post process step it is giving the same error for all of the entries

[kxz18]

Did you pass --pre_numbered when running download.py? It looks that the post process is renumbering the pdbs, which is unnecessary because SAbDab has already renumbered them. Also, I think this error might be related to the anarci packages used by the renumbering program, so installing anarci with conda might solve the problem.

[infusion-zero-edit]

parsing pdb failed for 7tas. reason: Not ['CYS'] at 23
parsing pdb failed for 7e14. reason: Not ['CYS'] at 23
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6989/6989 [17:54<00:00, 6.50it/s]
number of valid entries: 47
Only has heavy_chain, 3, examples: ['7omn', '5yax', '7ooi']
Only has heavy_chain, antigen_chains, 6, examples: ['6i8g', '4c57', '6i8h', '4c58', '4c59']
Only has heavy_chain, light_chain, 9, examples: ['6rlm', '6wyr', '4zd3', '5it2', '5ihz']
Only has heavy_chain, light_chain, antigen_chains, 29, examples: ['7kxk', '5ifj', '6xp6', '7klh', '7uz4']

HAve installed anarci package, but now there is some other error occuring for most of the entries like for example "parsing pdb failed for 7e14. reason: Not ['CYS'] at 23". Out of 7152 it gives only 47 valid entries which is not correct. Let me know what i am missing here

[infusion-zero-edit]

I am passing --pre_numbered in the download script.

[infusion-zero-edit]

Out of 7152, there are 6989 which are successfully downloaded. out if which only 47 entries it is showing valid which is incorrect

[kxz18]

Could you provide the full command you run with download.py? I'm trying to reproduce this error.

[infusion-zero-edit]

I am running this script:

bash scripts/prepare_data_kfold.sh summaries/sabdab_summary.tsv all_structures/imgt

where the following commands are used to download

#!/bin/bash
PROJ_FOLDER=$(cd "$(dirname "$0")";cd ..;pwd)
echo "Locate project at ${PROJ_FOLDER}"

add python path

export PYTHONPATH=$PYTHONPATH:${PROJ_FOLDER}
cd ${PROJ_FOLDER}

if [ $# != 2 ]; then
echo "Usage: bash $0 /path/to/summary /directory/to/pdb"
exit 1;
fi

path to summary file of SAbDab

SUMMARY=$1

path to structure folder of SAbDab (our paper use IMGT numbering)

PDB_DIR=$2

extract data directory

DATA_DIR=$(dirname ${SUMMARY})
echo "Summary file at ${SUMMARY}. PDB folder at ${PDB_DIR}. Data working directory at ${DATA_DIR}"

ALL=${DATA_DIR}/sabdab_all.json

download data

python -m data.download
--summary ${SUMMARY}
--pdb_dir ${PDB_DIR}
--fout ${ALL}
--type sabdab
--numbering imgt
--pre_numbered
--n_cpu 4

[kxz18]

Hi, I have run this command but didn't encounter this problem. However, I noticed that the number of entries is different from which was provided in this repo, and that the downloading process took 4 times longer than my run. So maybe I need further information. May I ask how you got the summary file and the corresponding pdb database?

[infusion-zero-edit]

As written in your github repo "We have provided the summary data used in our paper from SAbDab, RAbD, SKEMPI_V2 in the summaries folder, please download all structure data from the download page of SAbDab. Since the SAbDab is updating on a weekly basis, you may also download the newest summary file from its official website. The following instructions will suppose the structure data renumbered by imgt is located at the folder all_structures/imgt."

So i downloaded the latest file from SabDab

[kxz18]

Emmm, this is a bit weird, because if the correct database is provided and --pre_numbered is passed, then the post process shouldn't enter the renumber branch (line 246 in download.py). Is it because the most of the pdb data is downloaded from pdb server instead of from the SAbDab provided locally?

[kxz18]

I just downloaded the newest version of SAbDab, and the first part of logs is as follows:

PDB file already renumbered with scheme imgt
downloading raw files
11%|███████▍ | 753/7152 [00:00<00:05, 1230.75it/s]
6TNP not found in > ../tmp_mean_data/all_structures/imgt/, try fetching from remote server
fetched
100%|██████████████████████████████████████████████████████████████████████| > 7152/7152 [00:03<00:00, 1939.55it/s]
number of downloaded entries: 7152

It seems that only 6TNP is not included in the pdb database of SAbDab, which is not consistent with your logs. Did you provide the correct path to the pdb database in ${PDB_DIR}?