GraphCNN and CPU memory usage

[MLPharmaTron]

Hi AMPL staff.

I am currently using a GraphCNN and CPU usage is massive. @2.5% of the final intended training data, CPU usage has approached 150GB.

Questions:

1. I am intending to work with ultra large libraries. Is this something AMPL is equipped to handle? I can't find other publications which have used or validated it for 100M++ datasets.
2. Can we use our own precomputed features? It seems the dictionary requires us to put in a featuriser that is pre-approved in AMPL. I am wanting to use my own encoder to create my own tensors for training. I am unable to figure out if this is possible on AMPLs workflow.

Thanks.

[stewarthe6]

Hello,

1. I don't think AMPL is suitable for datasets that large. The entire dataset is held in memory using a NumpyDataset from DeepChem. We used to use a DiskDataset, which could stream data in, but we never have datasets large enough to need it and using a NumpyDataset is faster.
2. Yes, you can. There are a few limitations though and it is an advanced user feature. The features need to be a vector and saved in a csv file. You'll have to edit descriptor_sets_sources_by_descr_type.csv and add a new row for your encoding. Then you can call AMPL with 'descriptor_type' with the new name. Since AMPL won't know how to calculate these descriptors, you'll have to place the pre-computed features somewhere AMPL can find it. Here's an example.

data.csv
scaled_descriptors/
    data_with_example_encoding_descriptors.csv

[MLPharmaTron]

Thank you. Following on, I have two questions.

1. Has DiskDataset is deprecated as a function?
2. It seems active learning is not a component built into AMPL, correct? It does seem using the wrapper in conjunction with other libraries such as moDAL is not really AMPL friendly?

[stewarthe6]

1. We're not supporting that anymore. I will record that you requested it.
2. I'm not familiar with moDAL and what it needs. AMPL does a prediction uncertainty for some model types that you can use for uncertainty based sampling.

[MLPharmaTron]

Thanks Stewart.

Any idea what is the likely outcome and status on the DiskDataset?

Also I've tried to address this problem using ECFP4s instead of graphs, and while I saw a 10% decrease in memory, and a 4x training speed, I noticed that training failed after 17 epochs, signified by a drop in r2 vs epoch. I suspect this is because ECFP4s are binary vectors and explosion problems occur, which graphs smoothing out can avoid (unless you have other insights). AMPL isn't really geared to troubleshoot this within its framework is it?

If so, it seems AMPL is really not something that can train in the 100M, let alone 1B range is it?

[stewarthe6]

There aren't any plans to implement the DiskDataset. I don't think it will be able to scale to the 100M or 1B range.

You can try experimenting with batch size, network size, weight decay, or learning rate. I don't think we have any tools for you to directly inspect the weights or deltas.

[MLPharmaTron]

Thank you Stewart