TBDLab-UI

Modification of code from https://github.com/flawlesscorzo/TBD-Research

GETTING STARTED

To edit the app1.mlapp file and to make sure everything runs correctly, you need

• MATLAB App Designer (for editing app1.mlapp)
• MATLAB Computer Vision Toolbox (for running plot_trajectories.m)
• MATLAB Image Processing Toolbox (typically gets installed when you install Computer Vision Toolbox, also for running plot_trajectories.m)
• MATLAB Compiler (for deploying app1.mlapp into a web app)
• Must have plot_trajectories.m and draw_sq.m in the MATLAB folder
• Must have any files that you intend to upload (i.e. videos with trajectories drawn, raw videos, or trajectory files) in the MATLAB folder

UPLOADING FILES

• NEW - We can now upload a video file parameter through the terminal instead of using the upload button to upload a video. Type into terminal (in the same directory as 'app1.mlapp') app1('path to video')

There are three buttons in the top left corner of the window that are for uploading files.

• Upload Video - This button is for uploading the video with the plotted trajectories drawn onto it. This is the video that the user will see and transform the trajectories of. Files with type .mp4 and .avi are accepted.
• Upload Raw - This button is for uploading the raw video so that after a transformation is performed, the video can be redrawn with the new trajectories using plot_trajectories.m. Files with type .mp4 and .avi are accepted.
• Upload Traj. - This button is for uploading the trajectory files associated with the video uploaded via "Upload Video." This .mat file should have a variable called "traj_starts", which is a cell array containing the starting frames for all of the trajectories (each column corresponds to a trajectory, and for each column there should be one row containing the start frame), and a variable called "trajectories", which is a cell array that contains the (x, y) location of each of the trajectories for each frame that they are being tracked (each column corresponds to a trajectory, and for each column there should be one row containing a Fx2 double array, where the first column represents the x coordinate of the trajectory and the second column corresponds to the y coordinate of the trajectory. F is the number of frames for which the trajectory should be drawn. The coordinates are with respect to the pixel dimensions of the video).

VIDEO PLAYER

Under where the video is displayed, there are some buttons and a slider that are used to pause, play, replay, or go to a particular frame in the video.

• Slider - Directly under the displayed video, there is a slider which goes from 1 to F, where F is the number of frames in the video. The user is allowed to click on the slider and move it to whichever frame they want to view.
• Rewind - Pressing this button will change the current frame to the frame right before it. If the current frame is 1, then pressing Rewind will not have any effect. The position of the slider also changes to reflect the new frame value.
• Forward - Pressing this button will change the current frame to the frame right after it. If the current frame is the final frame in the video, then pressing Forward will not have an effect. The position of the slider also changes to reflect the new frame value.
• Frame # To See - The user can go to a specific frame in the video by typing the frame number into this textbox and hitting the Enter button. The frame value must be valid (between 1 and F, where F is the number of frames, and an integer value).
• Play - Pressing this button will play the video from the current frame onward. While the video is playing, the button will say Pause, and pressing the button will cause the video to stop at whichever frame the button was pressed on. Once the end of the video is reached, the button will say Play Again, which will play the video from the first frame. This button can also be toggled by pressing the spacebar.
• Current Frame and Previous Frame - These textboxes display the current frame of the video and the previous frame of the video, respectively. These boxes cannot be edited by the user.

BUTTONS FOR SELECTING/MODIFYING TRAJECTORIES

To the left and to the right of the displayed video, there are some buttons for selecting and/or modifying the trajectories displayed in the video. Something to note is that after clicking Break, Join, Delete, or Relabel (and in the case of Relabel, after the relabeling has been completed), the user should wait for plot_trajectories.m to finish running (as the trajectories are being redrawn post-modification). They will know it is done when File Explorer opens, prompting the user to upload a file with type .mp4 or .avi. Users should select the video called detected_trajectories.avi, which is the video that results from running plot_trajectories.m.

For any of these buttons to work, you must have the video with trajectories, raw video, and trajectory files uploaded.

• Select - This button is for selecting trajectories using ginput. After clicking this button, the user will be able to hover over the video and select a point in the video. A yellow circle will be drawn around the selected point. The x and y coordinates of the selected point will be displayed in the x and y textboxes above the video, respectively (the coordinate axes are with respect to the pixel dimensions of the video, and the axis values are displayed alongside the video). Additionally, after the user selects a point, the trajectory nearest to that point will be recorded in the textbox labeled "Trajectory 1" (if the value in the "Trajectory 1" textbox is zero or the value in the "Trajectory 2" textbox is nonzero) or in the textbox labeled "Trajectory 2" (if the previous conditions do not apply). These textboxes are also editable by the user, so the user can manually input a trajectory. To exit selection mode, right click anywhere on the video.
• Clear - This button sets the value in both the "Trajectory 1" textbox and "Trajectory 2" textbox as 0.
• Break - This button divides the selected trajectory (which is determined by the trajectory number inside of the "Trajectory 1" textbox) at the current frame into two separate trajectories. The first trajectory begins where the original trajectory began and ends at the current frame, and the second begins at the current frame and ends where the original trajectory ended. After the Break button is pressed, a new video is generated using plot_trajectories.m (as described above) and a new file called break_{selected trajectory}_{current frame}.mat is created. This file contains the new "traj_starts" and "trajectories" variables after the transformation. The value inside of the "Trajectory 1" textbox should be a valid trajectory number.
• Join - This button connects the two selected trajectories (which is determined by the values inside the "Trajectory 1" and "Trajectory 2" textboxes). The resulting trajectory begins where "Trajectory 1" started and ends where "Trajectory 2" ends. After the Join button is pressed, a new video is generated using plot_trajectories.m (as described above) and a new file called join_{trajectory 1}_{trajectory 2}_{current frame}.mat is created. This file contains the new "traj_starts" and "trajectories" variables after the transformation. The values inside of both the "Trajectory 1" textbox and the "Trajectory 2" textbox should be valid trajectory numbers.
• Delete - This button deletes the selected trajectory (which is determined by the trajectory number inside of the "Trajectory 1" textbox) from the current frame onwards. If the current frame is the beginning of the trajectory, then the entire trajectory will be deleted. After the Delete button is pressed, a new video is generated using plot_trajectories.m (as described above) and a new file called delete_{selected trajectory}_{current frame}.mat is created. This file contains the new "traj_starts" and "trajectories" variables after the transformation. The value inside of the "Trajectory 1" textbox should be a valid trajectory number.
• Relabel - This button allows the user to draw a new trajectory starting at the current frame. After pressing the button, the user will get a ginput prompt to select a point on the screen. After selecting a point, the user must right click, which will make the video jump ahead 60 frames so that the user can select the next point in the trajectory. This continues until there are no more frames or the user manually stops the relabeling. To stop relabeling, the user must press the Stop button (which is only visible after Relabel is pressed) and then right click on the video to end the ginput prompting. After relabeling has been stopped, the points are joined together into a single solid trajectory using linear interpolation, a new video is generated using plot_trajectories.m (as described above), and a new file called relabel.mat is created. This file contains the new "traj_starts" and "trajectories" variables after the transformation.
• Other - The textbox labeled "Other" is for submitting any errors seen in the video that cannot be fixed with the above transformations. The user will describe the error in the textbox and then press the Submit button, which will record their message along with the current frame and the currently selected trajectory (i.e. the trajectory corresponding to the number in the "Trajectory 1" textbox) as a row in a cell array called "errors". Once the user is done with all of their messages, they must hit the Done button to export the cell array of messages to a file called other_errors.mat.

ENDING THE SESSION

Once the user is done viewing the video, applying transformations, and submitting messages, they should click the Done button. This button will close the app and create two files: final_trajectories.mat, which has the "traj_starts" and "trajectories" variables after all of the transformations have been applied, and other_errors.mat, which contains the cell array of all the error messages the user has submitted through the Other textbox.