diff --git a/.vs/DeepSynthDefectDetector/v17/.suo b/.vs/DeepSynthDefectDetector/v17/.suo new file mode 100644 index 0000000..5d67c49 Binary files /dev/null and b/.vs/DeepSynthDefectDetector/v17/.suo differ diff --git a/.vs/ProjectSettings.json b/.vs/ProjectSettings.json new file mode 100644 index 0000000..f8b4888 --- /dev/null +++ b/.vs/ProjectSettings.json @@ -0,0 +1,3 @@ +{ + "CurrentProjectSetting": null +} \ No newline at end of file diff --git a/.vs/slnx.sqlite b/.vs/slnx.sqlite new file mode 100644 index 0000000..5b6e8c5 Binary files /dev/null and b/.vs/slnx.sqlite differ diff --git a/Deep Learning/__pycache__/pycococreatortools.cpython-38.pyc b/Deep Learning/__pycache__/pycococreatortools.cpython-38.pyc deleted file mode 100644 index 93e10e6..0000000 Binary files a/Deep Learning/__pycache__/pycococreatortools.cpython-38.pyc and /dev/null differ diff --git a/README.md b/README.md index 04d954f..e402cc7 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,14 @@ # Synthetic Image Assisted Deep Learning Framework for Detecting Defects During Composite Sheet Layup -The repository consists of a codebase and dataset for work presented at ASME's IDETC-CIE Conference, 2022 +The repository consists of the codebase and dataset for work presented at ASME's IDETC-CIE Conference, 2022 The link to the entire annotated dataset can be found here: https://drive.google.com/drive/folders/191EgISHWs6FH6FCKOSBTKoLQjo29bWh0?usp=sharing +In case there are any issues accessing the dataset kindly reach out via email to: manyar@usc.edu or guptask@usc.edu + +The codebase consists of two subfolders: +## Synthetic Image Generation +The Synthetic Image Generation module contains the Texture File of Carbon Fiber, Procedure to Generate your own texture and also talks about generation of synthetic images +The code, blender file and procedure to generate synthetic images can be found [here](https://github.com/RROS-Lab/DeepSynthDefectDetector/SyntheticImageGenerator) + +## ResNest-based Image Segmentation Model +The image segmentation module is based on the ResNest Framework +The code and implementation details are discussed [here](https://github.com/RROS-Lab/DeepSynthDefectDetector/ResNeSTImageSegmentation) diff --git a/Deep Learning/README.md b/ResNeSTImageSegmentation/README.md similarity index 100% rename from Deep Learning/README.md rename to ResNeSTImageSegmentation/README.md diff --git a/Deep Learning/checkpoints/PUT_CHECKPOINTS_IN_THIS_FOLDER b/ResNeSTImageSegmentation/checkpoints/PUT_CHECKPOINTS_IN_THIS_FOLDER similarity index 100% rename from Deep Learning/checkpoints/PUT_CHECKPOINTS_IN_THIS_FOLDER rename to ResNeSTImageSegmentation/checkpoints/PUT_CHECKPOINTS_IN_THIS_FOLDER diff --git a/Deep Learning/configs/mask_rcnn_r50_2s_scaled.py b/ResNeSTImageSegmentation/configs/mask_rcnn_r50_2s_scaled.py similarity index 100% rename from Deep Learning/configs/mask_rcnn_r50_2s_scaled.py rename to ResNeSTImageSegmentation/configs/mask_rcnn_r50_2s_scaled.py diff --git a/Deep Learning/dataset/annotations/real_defect_dev.json b/ResNeSTImageSegmentation/dataset/annotations/real_defect_dev.json similarity index 100% rename from Deep Learning/dataset/annotations/real_defect_dev.json rename to ResNeSTImageSegmentation/dataset/annotations/real_defect_dev.json diff --git a/Deep Learning/dataset/annotations/real_defect_test.json b/ResNeSTImageSegmentation/dataset/annotations/real_defect_test.json similarity index 100% rename from Deep Learning/dataset/annotations/real_defect_test.json rename to ResNeSTImageSegmentation/dataset/annotations/real_defect_test.json diff --git a/Deep Learning/dataset/annotations/real_defect_train.json b/ResNeSTImageSegmentation/dataset/annotations/real_defect_train.json similarity index 100% rename from Deep Learning/dataset/annotations/real_defect_train.json rename to ResNeSTImageSegmentation/dataset/annotations/real_defect_train.json diff --git a/Deep Learning/dataset/annotations/real_defect_train_scaled.json b/ResNeSTImageSegmentation/dataset/annotations/real_defect_train_scaled.json similarity index 100% rename from Deep Learning/dataset/annotations/real_defect_train_scaled.json rename to ResNeSTImageSegmentation/dataset/annotations/real_defect_train_scaled.json diff --git a/Deep Learning/dataset/annotations/real_predefect_dev.json b/ResNeSTImageSegmentation/dataset/annotations/real_predefect_dev.json similarity index 100% rename from Deep Learning/dataset/annotations/real_predefect_dev.json rename to ResNeSTImageSegmentation/dataset/annotations/real_predefect_dev.json diff --git a/Deep Learning/dataset/annotations/real_predefect_test.json b/ResNeSTImageSegmentation/dataset/annotations/real_predefect_test.json similarity index 100% rename from Deep Learning/dataset/annotations/real_predefect_test.json rename to ResNeSTImageSegmentation/dataset/annotations/real_predefect_test.json diff --git a/Deep Learning/dataset/annotations/real_predefect_train.json b/ResNeSTImageSegmentation/dataset/annotations/real_predefect_train.json similarity index 100% rename from Deep Learning/dataset/annotations/real_predefect_train.json rename to ResNeSTImageSegmentation/dataset/annotations/real_predefect_train.json diff --git a/Deep Learning/dataset/annotations/real_predefect_train_scaled.json b/ResNeSTImageSegmentation/dataset/annotations/real_predefect_train_scaled.json similarity index 100% rename from Deep Learning/dataset/annotations/real_predefect_train_scaled.json rename to ResNeSTImageSegmentation/dataset/annotations/real_predefect_train_scaled.json diff --git a/Deep Learning/dataset/annotations/real_undefective_dev.json b/ResNeSTImageSegmentation/dataset/annotations/real_undefective_dev.json similarity index 100% rename from Deep Learning/dataset/annotations/real_undefective_dev.json rename to ResNeSTImageSegmentation/dataset/annotations/real_undefective_dev.json diff --git a/Deep Learning/dataset/annotations/real_undefective_test.json b/ResNeSTImageSegmentation/dataset/annotations/real_undefective_test.json similarity index 100% rename from Deep Learning/dataset/annotations/real_undefective_test.json rename to ResNeSTImageSegmentation/dataset/annotations/real_undefective_test.json diff --git a/Deep Learning/dataset/annotations/real_undefective_train.json b/ResNeSTImageSegmentation/dataset/annotations/real_undefective_train.json similarity index 100% rename from Deep Learning/dataset/annotations/real_undefective_train.json rename to ResNeSTImageSegmentation/dataset/annotations/real_undefective_train.json diff --git a/Deep Learning/dataset/annotations/real_undefective_train_scaled.json b/ResNeSTImageSegmentation/dataset/annotations/real_undefective_train_scaled.json similarity index 100% rename from Deep Learning/dataset/annotations/real_undefective_train_scaled.json rename to ResNeSTImageSegmentation/dataset/annotations/real_undefective_train_scaled.json diff --git a/Deep Learning/dataset/images/PUT_ALL_IMAGES_IN_THIS_FOLDER b/ResNeSTImageSegmentation/dataset/images/PUT_ALL_IMAGES_IN_THIS_FOLDER similarity index 100% rename from Deep Learning/dataset/images/PUT_ALL_IMAGES_IN_THIS_FOLDER rename to ResNeSTImageSegmentation/dataset/images/PUT_ALL_IMAGES_IN_THIS_FOLDER diff --git a/Deep Learning/pycococreatortools.py b/ResNeSTImageSegmentation/pycococreatortools.py similarity index 100% rename from Deep Learning/pycococreatortools.py rename to ResNeSTImageSegmentation/pycococreatortools.py diff --git a/Deep Learning/test.py b/ResNeSTImageSegmentation/test.py similarity index 100% rename from Deep Learning/test.py rename to ResNeSTImageSegmentation/test.py diff --git a/Deep Learning/train.py b/ResNeSTImageSegmentation/train.py similarity index 100% rename from Deep Learning/train.py rename to ResNeSTImageSegmentation/train.py diff --git a/SyntheticImageGeneration/Carbonfiber.m b/SyntheticImageGeneration/Carbonfiber.m new file mode 100644 index 0000000..4de301a --- /dev/null +++ b/SyntheticImageGeneration/Carbonfiber.m @@ -0,0 +1,326 @@ +%% This file creates all the maps for the 8x8 cf. This file is reconfigured to take into account the actual measurements + + + +%% Creating the B+W matte + +clear;clc; close all; + + + +x = 1000; + +y = 1000; + + + +matte = zeros(x,y); + + + +v = fix(y/8); + +z = fix(x/6.76); % From Seed. ± 0.4 + + + +% Locate rectangles + +for i = 1:4 + + rectLocation{i} = [1+2*v*(i-1),1+(i-1)*x/4]; + +end + +% Hard code in the four other rectangles + + + +rectLocation{5} = [1+v,1+5*x/8]; + +rectLocation{6} = [1+3*v,1+7*x/8]; + +rectLocation{7} = [1+5*v,1+x/8]; + +rectLocation{8} = [1+7*v,1+3*x/8]; + + + +% Draw rectangles + +for j = 1:length(rectLocation) + + for i = 1:z + + a = fix(rectLocation{j}(1)); + + b = fix(rectLocation{j}(2)); + + matte(a:v+a,i+b-1) = ones(1,v+1); + + end + +end + + + +matte(:,1:size(matte,2)-999) = matte(:,1:size(matte,2)-999) + matte(:,1000:end); + +matte = matte(1:1000,1:1000); + +%figure();imshow(matte); + + + +%% Creating the horizontal bump map + + + +horiz = zeros(x,y); + + + +for j = 1:length(rectLocation) + + max = fix((y-z)/2); + + for i = 1:max + + a = fix(rectLocation{j}(1)); + + b = fix(rectLocation{j}(2)); + + horiz(a:v+a-1,i+b+z) = i/max; + + horiz(a:v+a-1,i+b+z+max) = (max-i)/max; + + end + +end + + + +horiz(:,1:size(horiz,2)-999) = horiz(:,1:size(horiz,2)-999) + horiz(:,1000:end); + +horiz = horiz(1:1000,1:1000); + + + +%figure();imshow(horiz); + + + +%% Creating the vertical bump map + + + +vert = zeros(x,y); + + + +for j = 1:length(rectLocation) + + for i = 1:v/2 + + a = fix(rectLocation{j}(1)); + + b = fix(rectLocation{j}(2)); + + vert(a+i,b:z+b) = i/v*2; + + vert(a+fix(v/2)+i,b:z+b) = (v/2-i)/(v/2); + + end + +end + + + +vert(:,1:size(vert,2)-999) = vert(:,1:size(vert,2)-999) + vert(:,1000:end); + +vert = vert(1:1000,1:1000); + + + +%figure();imshow(vert+horiz); + + + + + +%% Creating the horizontal stitchings + + + +horizStitchings = zeros(x,y); + + + +stitchingsThickness = fix(y/17); % Seed. 17 ± 1 + + + +for j = 1:length(rectLocation) + + max = fix((y-z)/2); + + for i = 1:max + + a = fix(rectLocation{j}(1)); + + b = fix(rectLocation{j}(2)); + + horizStitchings(a:stitchingsThickness+a-1,i+b+z) = 1; + + horizStitchings(a:stitchingsThickness+a-1,i+b+z+max) = 1; + + end + +end + + + +horizStitchings(:,1:size(horizStitchings,2)-999) = horizStitchings(:,1:size(horizStitchings,2)-999) + horizStitchings(:,1000:end); + +horizStitchings = horizStitchings(1:1000,1:1000); + +%figure();imshow(horizStitchings); + + + +%figure();imshow(horiz); + +horizStitchingsBump = zeros(x,y); + + + +for j = 1:length(rectLocation) + + max = fix((y-z)/2); + + for i = 1:max + + a = fix(rectLocation{j}(1)); + + b = fix(rectLocation{j}(2)); + + horizStitchingsBump(a:stitchingsThickness+a-1,i+b+z) = i/(y/2-z); + + horizStitchingsBump(a:stitchingsThickness+a-1,i+b+z+max) = (y/2-z-i)/(y/2-z); + + end + +end + + + +horizStitchingsBump(:,1:size(horizStitchingsBump,2)-999) = horizStitchingsBump(:,1:size(horizStitchingsBump,2)-999) + horizStitchingsBump(:,1000:end); + +horizStitchingsBump = horizStitchingsBump(1:1000,1:1000); + + +%{ +figure();imshow(horizStitchingsBump); +%} + + +%% Create the vertical stitchings + + + +vertStitchings = zeros(x,y); + + + +j = 6; + +for i = 1:v + + a = fix(rectLocation{j}(1)); + + b = fix(rectLocation{j}(2)); + + vertStitchings(i+a,b:stitchingsThickness+b-1) = 1; + +end + + +%{ +figure();imshow(vertStitchings); +%} + + +%% Create the vertical stitchings Bump Map + + + +vertStitchingsBump = zeros(x,y); + +vertStitchingsBump2 = zeros(x,y); + + + +j = 6; + +max = fix(v/2); + +for i = 1:max + + a = fix(rectLocation{j}(1)); + + b = fix(rectLocation{j}(2)); + + vertStitchingsBump(i+a,b:stitchingsThickness+b-1) = i/(v/2); + + vertStitchingsBump(i+a+max,b:stitchingsThickness+b-1) = (max-i)/(v/2); + +end + + + +max = stitchingsThickness/2; + + + +for i = 1:max + + a = fix(rectLocation{j}(1)); + + b = fix(rectLocation{j}(2)); + + vertStitchingsBump2(a:a+v,b+i) = i/max; + + vertStitchingsBump2(a:a+v,b+i+max) = (max-i)/max; + +end + + +%{ +figure();imshow(vertStitchingsBump); + +figure();imshow(vertStitchingsBump2); +%} + + +%% Write out images + + +% Create destination folder, you can change this to whatever you like +destinationFolder = 'D:/DeepSynthImageFolder/'; +if ~exist(destinationFolder, 'dir') + mkdir(destinationFolder); +end +imwrite(matte, strcat(destinationFolder,'matte.png')); + +imwrite(vert+horiz, strcat(destinationFolder,'vert+horiz.png')); + +imwrite(horizStitchings, strcat(destinationFolder,'horizStitchings.png')); + +imwrite(horizStitchingsBump, strcat(destinationFolder,'horizStitchingsBump.png')); + +imwrite(vertStitchings, strcat(destinationFolder,'vertStitchings.png')); + +imwrite(vertStitchingsBump, strcat(destinationFolder,'vertStitchingsBump.png')); + +imwrite(vertStitchingsBump2, strcat(destinationFolder,'vertStitchingsBump2.png')); + diff --git a/SyntheticImageGeneration/Carbonfiber_Renderer.blend b/SyntheticImageGeneration/Carbonfiber_Renderer.blend new file mode 100644 index 0000000..f06bf0b Binary files /dev/null and b/SyntheticImageGeneration/Carbonfiber_Renderer.blend differ diff --git a/SyntheticImageGeneration/README.md b/SyntheticImageGeneration/README.md new file mode 100644 index 0000000..8a2f965 --- /dev/null +++ b/SyntheticImageGeneration/README.md @@ -0,0 +1,14 @@ +# Generation of Synthetic Images for Composite Prepreg Sheets + +## How to Create an HDRI (High Dynamic Range Imaging) +1. First you will need to purchase a shiny metal sphere at least 2 inches in diameter [link](https://www.amazon.com/dp/B01KCB5UWO?ref_=cm_sw_r_cp_ud_dp_04AWNNNZG4YY9894WPQ2). Place this chrome sphere within the cell where the composite layup task will be performed +2. Set up a camera on a tripod approximately 20 feet away from the sphere +3. From this spot, take photographs of the sphere on every exposure setting that the camera allows +4. Repeat the same process, this time taking photos from a different angle, about 90˚ away from the initial position. Make sure to take all these photos at the same time of day that you plan to be doing carbon fiber layups; changes in ambient light will affect the accuracy of the HDRI +5. Use an imaging software like Photoshop software to combine all images into an HDRI + +## How to use the blend File to generate PhotoRealistic CGI Images of the Carbon Fiber +1. Open the blend file in blender software [link](https://www.blender.org/) +2. On line 5, replace the “2” in the parentheses with the number of meshes you want to render +3. On line 8, replace the text inside the quotes with the pathname of the .stl’s in your folder of .stl files. The files should be named serially, such as: mesh1.stl, mesh2.stl, etc +4. When you press the “▶” button at the top of the scripting panel, the program will render and export the files. \ No newline at end of file