ChromaR is an experimental R toolkit for analyzing and exploring chromatic storytelling in movies or any other video source. What does this actually mean? You can check it reading this two articles:
- Getting StartedExploring chromatic storytelling in movies with R: Introduction
- Exploring chromatic storytelling in movies with R: the ChromaR package
This documentation assumes that the reader is already familiar with R. If not so, a gentle introduction to R can be found here.
Examples of framelines
- Getting Started
- Additional Features
- Generate your own dataset
- Built With
- Authors
- Acknowledgements
- License
The ChromaR package is hosted on GitHub and must be therefore installed through Devtools:
> install.packages("devtools")
> library(devtools)
> install_github("detsutut/chroma", subdir="chromaR")
> library(chromaR)ChromaR exposes two ready-to-go databases you can use for practice purposes: Matrix and Miyazaki. These two lists collect the movie datasets for the Matrix trilogy and Miyazaki's (partial) filmography. Let's look at the latter, focusing then on "Princess Mononoke":
> miyazaki = chromaR::miyazaki
> miyazaki_grouped = lapply(miyazaki, function(movie){groupframes(movie,seconds=5)})
> getSummary(miyazaki_grouped)
R G B lum duration RGB title frameId
1 95.43707 84.02420 76.22149 86.58976 7020 #5F544C 1984 - Nausicaa of the Valley of the Wind 1
2 70.16424 72.72785 70.92556 71.76052 7470 #464846 1986 - Laputa Castle in the Sky 2
3 75.58941 82.22896 74.08047 79.34076 5185 #4B524A 1988 - My Neighbor Totoro 3
4 85.45816 93.22114 88.66284 90.39083 5595 #555D58 1992 - Porco Rosso 4
5 58.40236 65.24263 61.49390 62.77819 7995 #3A413D 1997 - Princess Mononoke 5
6 77.02667 87.91691 91.37884 85.03065 6180 #4D575B 1998 - Kiki's Delivery Service 6
7 95.17101 83.02897 68.00188 85.01860 7470 #5F5344 2001 - Spirited Away 7
8 81.94425 77.87252 67.05723 77.90436 7145 #514D43 2005 - Howl's Moving Castle 8
9 105.48353 112.59336 96.48504 108.68850 6045 #697060 2008 - Ponyo on the Cliff by the Sea 9
10 92.98617 97.15226 80.36500 94.05584 7585 #5C6150 2013 - The Wind Rises 10First, we applied the groupframesfunction to each movie in the collection, then we displayed a summary of it through getSummary, which shows the average RGB triplet for each movie along with other details.
Moving to the single movie, let's look into Princess Mononoke:
> mononoke = miyazaki[[5]]
> attr(mononoke, "title")
"1997 - Princess Mononoke"
> mononoke[sample(nrow(mononoke),5),]
frameId R G B hex lum
99055 112.930 125.730 95.100 #707D5F 118.52070
157672 62.832 54.164 57.341 #3E3639 57.11387
80875 66.538 77.007 84.124 #424D54 74.64917
19967 56.969 106.630 111.510 #386A6F 92.26850
159740 44.664 49.680 60.627 #2C313C 49.37937Here we see 5 random rows from the Mononoke dataset. Each row represents a single frame of the movie. For each frame, we collect the average R, G and B value, its Hex string equivalent and the average luminance. This is all we need to plot the framelines.
In order to draw a nice plot for Mononoke, we have to merge the frames together first.
> mononoke_grouped = groupframes(mononoke, seconds = 10)
> plotFrameline(mononoke_grouped, verbose = 1)
Mononoke
Here we use a 10 seconds merging window, but you can try to set bigger values to get thicker bars. If the seconds argument is not provided, a merging time window will be calculated automatically.
Changing the merging window width changes the appearance of the plot
When we merge frames together through groupframes, the time window we pick with the seconds argument will affect the aesthetic outcome of our plot. In fact, depending on this parameter, the resulting frameline could be sharp or very blurred.
A 5-seconds window is usually fine for most of the movies, but you can visually tune it using the 'plotTimeWindows' function.
> nausicaa = chromaR::miyazaki[[1]]
> plotTimeWindows(nausicaa)
The algorithm suggests two different cutoffs: a soft one (sharp frameline, most of the color information is preserved) and a hard one (blurried frameline, defined by the time constant of the exponential decay)
…or you can simply pick up your favourite window size by visual inspection
A chromatic study on entire seasons or movie franchises can be focused on many different features. Summaries about luminance, hue and saturation can be retrieved using the getSummary function.
> ldr = getFrames()
> ldr.grouped = lapply(ldr,function(x){groupframes(x,seconds = 5)})
> ldr.summary = getSummary(ldr.grouped)
> plotTilesSummary(ldr.summary,
mode = "lum", #lum = luminance, h = hue, s = saturation
verbose = 1,
title = "Love Death & Robots",
subtitle = "Brightness")
Netflix's Love, Death & Robots brightness summary
Netflix's Love, Death & Robots summary for hue and saturation
ChromaR exposes some useful function to inspect color channels' trends over time.
The temperature function measures and plot the distance of the hue of each frame from pure red.
> ldr_episode1 = ldr.grouped[[1]]
> ldr_episode2 = ldr.grouped[[2]]
> temperature(ldr_episode1)
> temperature(ldr_episode2)
Temperature comparison between two episodes
Similar conclusions may be drawn inspecting the hue channel trend through the plotChannel function, which returns the derivative trend in addition to the channel plot.
> plotChannel(ldr_episode1, channel = "h", npoly = 3)
Hue channel trend and its derivative
ChromaR also provides an utility to track color transitions from a frame to the following one. The colorCircle function generates a chord diagram (a graphical method of displaying the inter-relationships between data) where each frame is arranged radially around a circle, with the color transitions between frames drawn as arcs connecting the source with the endpoint.
> colorCircle(ldr_episode1, extra = TRUE) # extra = TRUE adds more color shades
Chord diagram of “Zima Blue” (left) and “Sucker of Souls” (right)
The extractFramePalette function allows to extract color palettes from image sources using unsupervised machine learning. This feature runs the k-means++ algorithm to extract palette_dim main colors from the input image and uses this palette to reconstruct the original image.
> extractFramePalette(imgPath, palette_dim = 10, max_res =780, graphics = TRUE)
Different sizes of color palettes automatically extracted from a famous Breaking Bad frame
If you want to explore your own clips, you must convert your video source to a collection of averaged pixels first. In order to do this, you need to run the following Matlab script.
[names,path] = uigetfile({'*.mp4;*.m4v;*.3gp;*.mov;*.wmv;*.avi;*.webm;*.mkv;*.flv;*.mpeg;*.mpg',...
'Video files (*.mp4,*.avi,*.mkv,...)';'*.*', 'All Files (*.*)'},...
'Frames Extractor - select video files','MultiSelect','on');
if iscell(names)==0
name = names;
tot = 1;
else
tot = length(names);
end
for i = 1:tot
if iscell(names)~=0
name = names{i};
end
waittitle = strcat(name,' selected (',num2str(i),' out of',' ',num2str(tot),')');
disp(waittitle);
path_input = strcat(path,name);
path_output = strcat(path,name,'.csv');
mov = VideoReader(path_input);
nof = floor(mov.duration*mov.FrameRate);
palette = zeros(nof,3);
k=0;
waitmsg = strcat(num2str(k),'/',num2str(nof));
f = waitbar(0,waitmsg,'Name',waittitle);
for k = 3 : length(palette)
if hasFrame(mov)==0
break;
end
frame = readFrame(mov);
rgb = mean(reshape(frame,[],size(frame,3)));
palette(k,1)=rgb(1);
palette(k,2)=rgb(2);
palette(k,3)=rgb(3);
waitmsg = strcat(num2str(k),'/',num2str(nof));
waitbar(k/nof,f,waitmsg);
end
close(f)
csvwrite(path_output,palette)
endThis means that you must have Matlab installed on your local machine. The script above inspects the video source frame by frame. Each frame is a Height×Width×3 tensor. Since we are mostly interested in exploring the color trend over the entire clip rather than focusing on the palette of a single frame at this point, the only information we need to extract here is the average color (i.e RGB triplet) of each frame.
Thanks to thecolorsofmotion and moviebarcode for the inspiration.
This project is licensed under the GNU GPLv3 License - see the LICENSE.md file for details