Honors Research In Chemistry 2020-2021 : Application of Machine Learning in Chemical Identification. (CNN)
Research Paper Sem 1:
Application of Machine Learning in Effective Identification of Chemical Agents
Rex Liu
Abstract In this report, we examine the effectiveness of machine learning in identifying a chemical substance. To simulate chemical detection at a much smaller scale, a training set of 1000 images of sodium chloride and calcium chloride were fit into a Convolutional Neural Network (CNN) and trained to classify each chemical. The training accuracies (accuracy achieved from testing against training dataset) and validation accuracies (accuracy achieved from testing against new data) of the neural network were then evaluated. While the neural network was able to achieve a 100% training accuracy, the validation accuracy was roughly 94.5%, a classification performance existing tools like WISER fail to match. An accurate chemical identifier would allow emergency care respondents to provide the most effective care to patients during a Mass Casualty Incident (MCI) involving a chemical culprit as they would be able to act on crucial information such as possible chemical symptoms and properties.
Introduction In a world where the evolution of technology radically transforms our society, the field of medical sciences is also seeing innumerable advances. However, when it comes to ensuring patient’s safety, addressing their needs, and providing emergency care, the current healthcare system’s performance is subpar. Improving the current health care system is a subject of great interest and debate in our society. One obvious approach in achieving a more organized emergency aid system is by eliminating the existing inefficiencies. This can be accomplished by integrating the technological advances already present in the medical sciences into emergency aid systems. One such branch of emergency aid that could really benefit from the integration of technology to stream-line patient care would be decision support for emergency care providers during a mass casualty incident (MCI). Over the past 10 years, the chances of injury for any given person caused by MCIs were twice as high (incidence rate of 0.2% 1) as compared to a lethal disease like tuberculosis (incidence rate of 0.1% 2), one of the major causes of death in the United States. Since MCIs require rapid operations under resource limited conditions, studying the existing inefficiencies during an MCI event could really help us identify and propose improvements to our emergency care system. Most emergency care personnel were required to focus on the treatment of patients during the incident rather than collecting detailed patient information to identify the culprit chemical substance. In other words, rapid patient treatment was given a natural priority over data collection. While putting an emphasis on rapid treatment is certainly more important, without accurate information and data, emergency health care personnel may not be providing the best aid possible, or even be providing any aid at all. As a result, little advances have been made improving the treatment of chemical MCI events. To help the healthcare workforce in their emergency decision making, better understanding of the information and technological needs must be integrated. The National Library of Medicine has introduced a limited clinical decision support software named WISER (Wireless Information System for Emergency Responders) to aid in the process of emergency care. The system, available to all, allows emergency responders to identify a list of possible chemical substances based on its properties and observed patient symptoms. It then guides first-responders through a series of questions related to signs and symptoms that leads to a probabilistic diagnosis of a few syndromes. Although WISER makes great progress in helping emergency decision making, its application in real life was not as effective as anticipated, with a high chance of identification failure of roughly 15% (3). This was mainly caused by a great abundance of chemicals having repetitive properties, WISER’s intolerance of erroneous data exacerbated during an MCI, and WISER’s giant database containing some totally unrelated chemicals. An experiment done by the University of South Carolina has identified that using WISER, the prediction accuracy could drop to as low as 25.6% if the patient data and chemical properties were perturbed, or inputted incorrectly, by just 15%; and that even if only 5% of the data was mistakenly inputted, the prediction accuracy is still dropped to an average of 64.9%:
Figure 1: Chemical Identification prediction accuracy results using WISER obtained by the University of South Carolina 3
In this report, the use of machine learning as a potential solution to identify chemical substances during an MCI event was explored. Instead of filtering out possible chemicals based on their properties like WISER, a neural network is able to detect patterns between the properties of each chemical that humans cannot perceive and instantly output a prediction based on its past experiences to identify the chemical in question. Thus, using a machine learning solution eliminates any chance of data perturbation caused by human input, a major problem programs like WISER face as mentioned previously. To train the machine learning model for chemical identification, datasets of images containing each chemical were fit into the Convolutional Neural Network (CNN). Convolutional Neural Networks are deep learning algorithms that take in images as inputs and assigns importance (learnable weights and biases) to various aspects/objects in an image in order to identify and extract certain features. These features are highlighted to the neural network while training using convolutional filters, filters that produce feature maps while sliding over the input images. By minimizing the size of these kernels/filters in the convolutional layers, the neural network focuses on each individual chemical grain in an image, allowing it to better differentiate between chemicals with similar properties.
Figure 2: Sample CNN sequence used for image classification: convolutional layers highlight features in an image while pooling layers reduce image size after each convolution
Using convolutions to automatically highlight certain properties of a chemical alleviates the need for emergency providers to enter specific data in WISER themselves, and thus significantly reduces the stress and error rate involved in chemical identification. However, using many convolutional layers will quickly rack up the size of the training data, and the amount of computational power needed to process the data during identification would be too much for a mobile device to handle. Thus, pooling layers are implemented to reduce the spatial size of the convolved feature. Pooling layers work by returning the maximum value from a portion of the image covered by the kernel. Doing so not only decreases the size of the images through dimensionality reduction, it is also useful in extracting dominant features and acting as a noise suppressant after a convolution. Thus, even though the complexities in the images were sacrificed, the important features were actually enhanced at the cost of much lower computational power. To simulate chemical detection at a much smaller scale, images of two chemicals with nearly identical physical properties, calcium chloride and sodium chloride, are fed into a machine learning model trained to identify each chemical when a completely new image is presented. If an emergency care personnel was to be presented with the same two chemicals, it would be impossible for them to differentiate between the two using WISER as all the program could do is to filter through all the chemicals and output those that fit the properties specified by them. Unfortunately, all this does is to tell the emergency provider that the chemical was either calcium chloride or sodium chloride, without offering a specific nor accurate answer. The goal of this machine learning solution is to allow emergency providers to simply arrive at an MCI event, open up their phone camera, and to instantly know the culprit chemical substance, the possible symptoms and signs associated with it, and how to provide the most effective emergency care to the patients in need. This would allow emergency care personnel to both focus on the treatment of patients and also to gather important data by simplifying the process of chemical identification.
Method To begin training a machine learning model, first, a training data set was created for the two chemicals, sodium chloride (Choneye Pure Chemicals variant) and calcium chloride (Showa variant). The chemicals were poured into a weighing boat with dimensions of 9cm by 9cm by 3cm and filled up halfway. Images of the chemicals were then taken using a smart phone camera to simulate the mobile devices that emergency care respondents may carry. Roughly 1000 photos of each chemical were taken as a training phase and another 200 were taken for validating the accuracy of the neural network. The photos were then converted to 1000 by 2000-pixel images to lower the complexity of the images and help reduce training time. Since this training dataset is balanced (same number of calcium chloride and sodium chloride images), there is no need of passing a class weight to the model; however, to make sure that the dataset is sufficiently randomized so that the order in which the images are presented to the neural network does not affect the training process, the dataset was shuffled thoroughly to mix up images of both chemicals. With a complete training dataset, the neural network was now ready for training. To kick off training, the filters and parameters within each convolutional layer was initialized with random values. In this experiment, four 2D Convolutional layers were used along with two Max 2x2 Pooling layers. The neural network then goes through forward propagation steps (Convolution, Rectify Linear, and pooling operations) during the training phase to calculate the total error at the output layer. Using backpropagation steps, the gradients of error was calculated with respect to the weights in the network. This gradient descent was then used to update all the filter values and parameters within each convolutional layer to optimize the neural network by minimizing output error. This process of making predictions based on the filter values and parameters within the convolutional layers, calculating the error resulting from the predictions, then going back and optimizing the values and parameters in each convolutional layer, essentially constitutes the whole training process of the neural network. Over a series of 1000 epochs (the number of times the neural network goes through the aforementioned optimization process), the ML model was able to reach roughly equivalent training and validation accuracies. This neural network was trained using PyTorch on weights and biases' (WandB) interface using a NVIDIA GP106GL [Quadro P2000] running the most up to date Ubuntu Linux OS. The metrics logged by WandB during training was then used to graph the validating accuracy against the training accuracy to evaluate the overall prediction accuracy of the ML model in classifying the two chemicals.
Figure 3: Sample forward propagation steps used in image classification
Sample Images of calcium chloride used in the training dataset: https://drive.google.com/drive/folders/1_vyS9Jtws_VJyhH6YTzA5ltZxp_A5s6l?usp=sharing'
Figure 4: Appearance of an image of Calcium Chloride used in the training Dataset
Sample Images of sodium chloride used in the training dataset: https://drive.google.com/drive/folders/1uErozhmhQF2PweaXTbQIF0g8-ar8KDiY?usp=sharing
Figure 5: Appearance of an image of Sodium Chloride used in the training Dataset
The code used in this experiment can be found in the GitHub repository linked below: https://github.com/Rexypad/Machine-Learning_Chemical_Identification_RIC
Results & Discussion Testing results of the trained Machine Learning Model are shown in Figure 5 below. In this graph, the steps on the x-axis refers to the number of epochs ran to train the neural network, and the y-axis represents the prediction accuracies associated with each type of training. As expected, the prediction accuracies gradually increased with each epoch eventually reaching the highest possible prediction accuracy with its specific convolutional layers. The graph of Chem-ID refers to the prediction accuracy when testing the final neural network against the training dataset. Thus, in a well-trained neural network, this prediction accuracy should be almost flawless as the variables in each convolutional layer were optimized for this specific training dataset of images. This is reflected in our trained neural network, with a 99.5% prediction accuracy when classifying the chemicals in the training dataset. However, the more practical type of prediction accuracy that ultimately determines the effectiveness of the model is the accuracy associated with the test loop (with-tl). This graph represents the accuracy when testing against a dataset of completely new images that the neural network has never seen before during training nor during validation. This prediction accuracy more accurately represents how effective the neural network would be in predicting an actual chemical agent during an MCI. As seen in Figure 5, the prediction accuracy with-tl was roughly 94.5%, slightly lower than the prediction accuracy from testing against the training dataset. ning dataset.
Figure 6: Training metrics logged by Wand B displaying results of training session
The results for using a machine learning model in chemical identification highlights the significant robustness of a more sophisticated approach compared to outdated systems like WISER. Machine learning and the use of CNNs show promise compared to WISER for quickly and accurately identifying a culprit chemical during a chemical MCI. WISER's intolerance of erroneous and imperfect data, a condition that is likely to occur during the chaos and confusion that occurs during an MCI, greatly decreases its prediction accuracy. Furthermore, WISER operates with a luxury of reporting a potentially long list of unrelated chemicals that share a common list of properties recorded by emergency personnel. Presenting a long list of unrelated chemicals may also provide additional confusion during an emergency like an MCI. The clear advantage in using machine learning is that it sidesteps the need for human involvement in identifying a chemical agent, which not only improves the accuracy of data collection, but also saves valuable time for emergency care personnel to treat and tend to patients. This boost of robustness provided by machine learning is achieved through the use of a trained neural network's ability to take in more information than an emergency personnel can record and process it much faster and more accurately. Machine Learning tools employed during chemical MCIs could dramatically reduce the amount of information collected from patients resulting in increased accuracy, precision, and efficiency in identifying the chemical. Although this experiment only explored the differentiation of two chemicals, more chemicals can be easily added and the neural network trained to accommodate them, with little to no loss in prediction accuracy.
Future Work Future investigations will focus on further improving the application value of the machine learning model through a variety of different ways. One such improvement would be expanding the variety of chemicals in the CNN. It is possible that including more chemicals would require a completely new set of convolutions in my neural network; however, since the model was already capable of detecting the difference between chemicals similar in appearance as shown in our experiment, it is likely that the 2D convolutions currently used in the CNN are suitable enough for differentiating between even more chemicals. Although machine learning can be a highly effective solution to chemical identification, this experiment has only tested the neural network using a machine with high computational power. To improve the practical use of this model, it must be deployed in a way easily accessible to emergency care personnel. Since most emergency providers carry a mobile device with them, one way to provide easy access to this tool is by deploying the neural network as an application through the integration of tensorflow-lite, a machine learning tool for application-based neural networks. This would allow emergency providers to simply arrive at an MCI event, open up their phone camera, and to instantly know the culprit chemical substance, the possible symptoms and signs associated with it, and how to provide the most effective emergency care to the patients in need. However, shifting the convolutional neural network for usage on a phone may require a different set of convolutions. AlexNet, the convolutions used in this experiment is more suited for image recognition but isn't as optimized for streaming solutions as MobileNet is. Since using a phone for chemical identification may be more similar to a stream than just an image, it may be more desirable to use MobileNet convolutions for the CNN instead. Another possible improvement that would bring this project one step closer for deployment in an actual MCI is to evaluate the accuracy of the neural network under more realistic conditions. Since it may be hard for an emergency care personnel to have to grab a sample of the chemical agent during an MCI, place it in a weighing boat, then use the neural network to identify the chemical agent; it is much more effective to simply train the neural network based on a variety of different conditions, so that the neural network may identify the chemical agent no matter what the background may be. An object detection machine learning model could be integrated to detect the chemical agent within an image and run the classification neural network on just the cropped image. However, this process is much easier said than done, as it would be incredibly hard to generate the training dataset for this model. Not only will the neural network have to account for different backgrounds, it will also need to tell the difference between different chemical manufacturers, and possible chemical reactions that may have occurred prior to the investigation. Finally, even though the prediction accuracy of this machine learning model was a respectable 94.5%, during an MCI, where tens and maybe hundreds of lives are at stake, there is little leeway for mistakes to be made. An inaccurate prediction could be the difference between a flammable or inflammable; toxic or nontoxic; hazardous or nonhazardous chemical agent and knowing for sure what the chemical agent may be is crucial under such circumstances. Thus, any improvements to the accuracy in identifying these chemicals are much appreciated. One such way to do so at the expense of more time spent and perhaps more equipment cost is the use of Scanning Electron Microscope (SEM) images of chemicals to train the machine learning model rather than images straight from the camera. Since all chemicals have different structures, by training with SEM images, it would be much easier for the neural network to identify each chemical based on their structure. This could potentially boost the prediction accuracy of the model to somewhere close to 99%, a much more satisfying prediction accuracy emergency care personnel can trust. However, this also means that a mobile SEM must be present with the emergency care personnel during an MCI for the neural network to function properly, something fairly unlikely to implement on a large scale unless all ambulances are equipped with an SEM.
Acknowledgements Special Thanks to Mr. Alex Dezieck, Mr. Jude Clapper, Mr. Matt Fagen, Dr. Allan Bayntun, Jimmy Su, and Generous Donations to Taipei American School
References
- Incidence and Mortality Rates of Disasters and Mass Casualty Incidents Schenk E, Wijetunge G, Mann NC, Lerner EB, Longthorne A, Dawson D. Prehosp Emerg Care. 2014 Jul-Sep; 18(3):408-16. Epub 2014 Mar 27.
- Incidence of tuberculosis (per 100,000 people) | Data "Incidence Of Tuberculosis (Per 100,000 People) | Data". Data.Worldbank.Org, 2020, https://data.worldbank.org/indicator/SH.TBS.INCD. Accessed 12 Oct 2020.
- Nicholas Boltin, Homayoun Valafar. "An AI Model For Rapid And Accurate Identification Of Chemical Agents In Mass Casualty Incidents". HIMS 2016 : Proceedings Of The 2016 International Conference On Health Informatics And Medical Systems. HIMS (Conference) (2016 : Las Vegas, Nev.), vol 2016, 2016, p. 169., https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6191201/. Accessed 12 Oct 2020.
- (PDF) Fruit recognition from images using deep learning "(PDF) Fruit Recognition From Images Using Deep Learning". Researchgate, 2020, https://www.researchgate.net/publication/321475443_Fruit_recognition_from_images_using_deep_learning. Accessed 27 Nov 2020.