diff --git a/docs/404.html b/docs/404.html
new file mode 100644
index 0000000..56d5ee3
--- /dev/null
+++ b/docs/404.html
@@ -0,0 +1 @@
+
Not Found
\ No newline at end of file
diff --git a/docs/404/index.html b/docs/404/index.html
new file mode 100644
index 0000000..5b11932
--- /dev/null
+++ b/docs/404/index.html
@@ -0,0 +1 @@
+
{`Impression is a family of algorithms for image simplification, segmentation and vectorization. Try the `}\n {`Demo Web App`}\n {`.`}
\n \n
{`Premise`}
\n
{`The fundamental information result from visual perception is shape. For example, we would describe ‘there is an orange on a basket’ or more conceptually ‘there is a circle above a triangle’.`}
\n
{`If we regard simplification as a process of information reduction, we could imagine information being taken away from an image shape by shape, starting from the least important shapes. At the same time, we can also simplify each shape in the image to make it ‘less fractal’.`}
\n
{`By controlling this reduction process, we would be able to control the amount of visual information in an image in a quantitative manner.`}
\n
{`Under this understanding, segmentation is an extreme degree of simplification, while vectorization is a slight degree of simplification.`}
\n
{`The Clustering Algorithm`}
\n
{`The `}\n {`clustering algorithm`}\n {` is hierarchical in nature and the idea is best described by the following verse:`}
\n
\n\n
{`On an unclaimed land, there are villages. Villages ally with each other to form tribes. Tribes conquer each other and form Kingdoms. Kingdoms clashes with each other and form empires. Empires breakdown and finally the entire earth becomes one union.`}
\n\n
\n
{`Stage 1: clustering`}
\n
{`Clustering is a process to connect pixels of similar colors into clusters. Each cluster have a unique position and shape. There is well known algorithm to perform clustering and Impression’s implementation is essentially same as Kopelman Algorithm.`}
\n
{`Stage 2: hierarchical clustering`}
\n
{`Hierarchical clustering is a process to build an image tree from the clusters of stage 1. Impression builds a binary tree bottom up. In preparation, clusters are sorted by their size. Starting from the smallest, in each iteration, a cluster is merged into the closest cluster. There are different formulas for ‘close’, but it generally involves comparing the average colors between neighboring clusters.`}
\n
{`Stage 3: tree walking`}
\n
{`In image vectorization as in `}\n {`VTracer`}\n {`, we would walk the tree from top to bottom, trace each layer, and stack the vector paths on the output canvas. As if a painter paints on an empty canvas, he would lay down background first and then overlay objects atop, and finally add on details.`}
\n
{`Simplification`}
\n
{`In image simplification, there are multiple dimensions of information which can be controlled quantitatively:`}
\n \n\n
{`Shape details: proportional to how many points we use to outline each shape`}
\n\n\n
{`Fidelity: proportional to how many nodes we retain from the image tree`}
\n\n\n
{`Color levels: proportional to how densely we sample the image tree layers`}
\n\n \n
{`Shape Simplification`}
\n
{`Impression currently chose the Ramer-Douglas-Peucker algorithm for shape simplification. While the original algorithm is designed for open curves, Impression adapted it for closed shapes. We cut a closed path into 4 sections, simplify each, and stitch them back together after path simplification. We choose the north most, east most, south most and west most points to cut a given path. Effectively the simplest possible shape is a diamond with four points. However in practice, it is not desirable to have shapes with sharp corners, and so we would smooth them with a 4-point scheme as described in `}\n {`VTracer`}\n {`.`}
\n \n
{`Fidelity`}
\n
{`It has to be understood that statistically there are exponentially more nodes as we decrease the cluster size. There is one root node known as the image background, but as many leaf nodes as there are number of pixels. As such, fidelity is actually a cutoff which we discard all clusters smaller than a desired size. When this cutoff threshold is low, we are removing salt and pepper noise but in a true color sense. As we increase this cutoff, structures are being discarded. Further increasing this cutoff, the image would become more abstract, and in the end only one solid background will be left.`}
\n
{`It is important to note that this background color is not the mean pixel color of the image, but is the average color of the largest cluster, which conceptually is the ‘base tone’ of the given image.`}
\n \n
{`Color Levels`}
\n
{`More color levels mean finer gradient. Color levels set to 256 means utilizing the full color precision of RGB888. Setting color levels to 32 meaning a cluster has to have at least a color difference of 8 in order to be considered a separate cluster with the cluster on the upper level. In effect, tuning down the color levels would create a retro 8-bit color look. While the number of colors in the palette is limited, the colors are still 24-bit and thus are faithful to the original image.`}
\n \n
{`Segmentation`}
\n
{`Segmentation`}\n {` follows stage 3 of the algorithm. Similar clusters are grouped together by the `}\n {`disjoint sets`}\n {` algorithm, where each set of the result represents a solid color patch. For each cluster, its neighbours are being considered. If the color difference is smaller than a defined deviation, they will be put into the same union. To prevent grouping too greedily, after the first union, subsequent union would require stricter thresholds. To output segmentation result, each set would be rendered by the average color of all its constituting clusters.`}
\n
{`After this stage, there would still be unwanted patches in the output. The output would further be re-clustered by Stage 1 to 3 described before, and an `}\n {`aggregation`}\n {` pass be applied afterwards. It would prune away smaller patches by merging into the closest (in terms of color) cluster if deviation would allow.`}
\n
{`Finally, to output result, each aggregate would be rendered by its original cluster color.`}
\n \n
{`The Implementation`}
\n
{`The described algorithms are implemented as reusable components under `}\n {`VisionMagic`}\n {`. Together with a processing pipeline, they are designed to support real-time and interactive applications. Users can easily organize processing stages and add additional processing passes as needed.`}
\n
{`The `}\n {`Processor`}\n {` trait is the interface that future additions of algorithms will conform to. It also allows us to support video streaming applications in the future.`}
\n
{`Impressionism`}
\n
{`The above methodology is designed to imitate visual perception in humans. It is thus no surprising that resulting images exhibit characteristics similar to paintings drawn by Artists. As the name suggests, it is heavy inspired by a painting technique that we called ‘Impressionism’ for the meaning it convey during a brief period of art history.`}
\n \n\n ;\n}\n;\nMDXContent.isMDXComponent = true;\n "],"sourceRoot":""}
\ No newline at end of file
diff --git a/docs/component---src-index-mdx-0934d2e2f2b0fd64a218.js b/docs/component---src-index-mdx-0934d2e2f2b0fd64a218.js
new file mode 100644
index 0000000..11f71f1
--- /dev/null
+++ b/docs/component---src-index-mdx-0934d2e2f2b0fd64a218.js
@@ -0,0 +1,2 @@
+(window.webpackJsonp=window.webpackJsonp||[]).push([[5],{"7A3R":function(e,t,i){"use strict";i.r(t),i.d(t,"_frontmatter",(function(){return s})),i.d(t,"default",(function(){return b}));var a=i("Fcif"),n=i("+I+c"),o=(i("mXGw"),i("/FXl")),c=i("TjRS"),r=i("Kh4r"),s=(i("aD51"),{});void 0!==s&&s&&s===Object(s)&&Object.isExtensible(s)&&!s.hasOwnProperty("__filemeta")&&Object.defineProperty(s,"__filemeta",{configurable:!0,value:{name:"_frontmatter",filename:"src/index.mdx"}});var l=c.a;function b(e){var t=e.components,i=Object(n.a)(e,["components"]);return Object(o.b)(l,Object(a.a)({},i,{components:t,mdxType:"MDXLayout"}),Object(o.b)(r.a,{mdxType:"SquareLogo"}),Object(o.b)("h1",{id:"vision-cortex---semantic-computer-vision"},"Vision Cortex - Semantic Computer Vision"),Object(o.b)("p",null,"The goal of Semantic Computer Vision is to allow computers to understand the content of images and graphics as intended and perceived by humans, and construct a high level representation of such information."),Object(o.b)("p",null,"This technology can be embodied in different applications:"),Object(o.b)("h2",{id:"symbolic-barcode"},"Symbolic Barcode"),Object(o.b)("img",{style:{width:"100%"},src:"public/images/symcode-banner.jpg"}),Object(o.b)("p",null,"We developed ",Object(o.b)("a",{href:"/symcode-docs",parentName:"p"},"SymCode"),", a 2D barcode designed to be both human-readable and machine-readable."),Object(o.b)("h2",{id:"image-simplification"},"Image Simplification"),Object(o.b)("img",{style:{width:"100%"},src:"public/images/visionmagic-preview.png"}),Object(o.b)("p",null,"We developed ",Object(o.b)("a",{href:"/impression-docs",parentName:"p"},"Impression"),", a family of algorithms for image simplification and segmentation. It allows us to control the amount of visual information in an image in a quantitative manner."),Object(o.b)("h2",{id:"image-vectorization"},"Image Vectorization"),Object(o.b)("img",{style:{width:"100%"},src:"public/images/vtracer-demo.svg"}),Object(o.b)("p",null,"We developed ",Object(o.b)("a",{href:"/vtracer-docs",parentName:"p"},"VTracer"),", a utility to convert raster images (like jpg & png) into vector graphics (svg). Our graphics pipeline is able to process high resolution scans and photographs and trace the content to output compact vector files."),Object(o.b)("h2",{id:"optical-character-recognition"},"Optical Character Recognition"),Object(o.b)("img",{style:{width:"100%"},src:"public/images/forest-layering.svg"}),Object(o.b)("p",null,"We are developing a new Optical Character Recognition (OCR) engine from the ground up specifically for pictorial languages like Chinese, Korean and Japanese (CKJ) to encompass wider character sets and font variations."))}void 0!==b&&b&&b===Object(b)&&Object.isExtensible(b)&&!b.hasOwnProperty("__filemeta")&&Object.defineProperty(b,"__filemeta",{configurable:!0,value:{name:"MDXContent",filename:"src/index.mdx"}}),b.isMDXComponent=!0}}]);
+//# sourceMappingURL=component---src-index-mdx-0934d2e2f2b0fd64a218.js.map
\ No newline at end of file
diff --git a/docs/component---src-index-mdx-0934d2e2f2b0fd64a218.js.map b/docs/component---src-index-mdx-0934d2e2f2b0fd64a218.js.map
new file mode 100644
index 0000000..10ef550
--- /dev/null
+++ b/docs/component---src-index-mdx-0934d2e2f2b0fd64a218.js.map
@@ -0,0 +1 @@
+{"version":3,"sources":["webpack:///../src/index.mdx"],"names":["_frontmatter","MDXLayout","DefaultLayout","MDXContent","components","props","mdxType","style","width","src","isMDXComponent"],"mappings":"qQAOaA,G,UAAe,S,+LAE5B,IAKMC,EAAYC,IACH,SAASC,EAAT,GAGZ,IAFDC,EAEC,EAFDA,WACGC,EACF,8BACD,OAAO,YAACJ,EAAD,eAAeI,EAAf,CAAsBD,WAAYA,EAAYE,QAAQ,cAC3D,YAAC,IAAD,CAAYA,QAAQ,eACpB,iBAAQ,CACN,GAAM,4CADR,4CAGA,wOACA,mFACA,iBAAQ,CACN,GAAM,oBADR,oBAGA,mBAAKC,MAAO,CACVC,MAAO,QACNC,IAAK,qCACR,qCACE,gBAAO,CACL,KAAQ,gBACR,WAAc,KAFhB,WADF,2EAMA,iBAAQ,CACN,GAAM,wBADR,wBAGA,mBAAKF,MAAO,CACVC,MAAO,QACNC,IAAK,0CACR,qCACE,gBAAO,CACL,KAAQ,mBACR,WAAc,KAFhB,cADF,sKAMA,iBAAQ,CACN,GAAM,uBADR,uBAGA,mBAAKF,MAAO,CACVC,MAAO,QACNC,IAAK,mCACR,qCACE,gBAAO,CACL,KAAQ,gBACR,WAAc,KAFhB,WADF,0NAMA,iBAAQ,CACN,GAAM,iCADR,iCAGA,mBAAKF,MAAO,CACVC,MAAO,QACNC,IAAK,sCACR,wP,6LAKJN,EAAWO,gBAAiB","file":"component---src-index-mdx-0934d2e2f2b0fd64a218.js","sourcesContent":["import * as React from 'react'\n /* @jsx mdx */\nimport { mdx } from '@mdx-js/react';\n/* @jsx mdx */\n\nimport DefaultLayout from \"/home/chris/visioncortex.github.io/node_modules/gatsby-theme-docz/src/base/Layout.js\";\nimport { SquareLogo } from './components/SquareLogo';\nexport const _frontmatter = {};\n\nconst makeShortcode = name => function MDXDefaultShortcode(props) {\n console.warn(\"Component \" + name + \" was not imported, exported, or provided by MDXProvider as global scope\");\n return ;\n};\n\nconst MDXLayout = DefaultLayout;\nexport default function MDXContent({\n components,\n ...props\n}) {\n return \n \n
{`Vision Cortex - Semantic Computer Vision`}
\n
{`The goal of Semantic Computer Vision is to allow computers to understand the content of images and graphics as intended and perceived by humans, and construct a high level representation of such information.`}
\n
{`This technology can be embodied in different applications:`}
\n
{`Symbolic Barcode`}
\n \n
{`We developed `}\n {`SymCode`}\n {`, a 2D barcode designed to be both human-readable and machine-readable.`}
\n
{`Image Simplification`}
\n \n
{`We developed `}\n {`Impression`}\n {`, a family of algorithms for image simplification and segmentation. It allows us to control the amount of visual information in an image in a quantitative manner.`}
\n
{`Image Vectorization`}
\n \n
{`We developed `}\n {`VTracer`}\n {`, a utility to convert raster images (like jpg & png) into vector graphics (svg). Our graphics pipeline is able to process high resolution scans and photographs and trace the content to output compact vector files.`}
\n
{`Optical Character Recognition`}
\n \n
{`We are developing a new Optical Character Recognition (OCR) engine from the ground up specifically for pictorial languages like Chinese, Korean and Japanese (CKJ) to encompass wider character sets and font variations.`}
\n\n ;\n}\n;\nMDXContent.isMDXComponent = true;\n "],"sourceRoot":""}
\ No newline at end of file
diff --git a/docs/component---src-pages-404-js-9c134c3c1761f19419a4.js b/docs/component---src-pages-404-js-9c134c3c1761f19419a4.js
new file mode 100644
index 0000000..716e634
--- /dev/null
+++ b/docs/component---src-pages-404-js-9c134c3c1761f19419a4.js
@@ -0,0 +1,2 @@
+(window.webpackJsonp=window.webpackJsonp||[]).push([[6],{w2l6:function(t,e,n){"use strict";n.r(e);n("mXGw");var i=n("aD51");e.default=function(){return Object(i.c)("div",{style:{display:"flex",alignItems:"center",justifyContent:"center",height:"100vh",width:"100vw",fontSize:32}},"Not Found")}}}]);
+//# sourceMappingURL=component---src-pages-404-js-9c134c3c1761f19419a4.js.map
\ No newline at end of file
diff --git a/docs/component---src-pages-404-js-9c134c3c1761f19419a4.js.map b/docs/component---src-pages-404-js-9c134c3c1761f19419a4.js.map
new file mode 100644
index 0000000..971aff7
--- /dev/null
+++ b/docs/component---src-pages-404-js-9c134c3c1761f19419a4.js.map
@@ -0,0 +1 @@
+{"version":3,"sources":["webpack:///./src/pages/404.js"],"names":["style","display","alignItems","justifyContent","height","width","fontSize"],"mappings":"2FAAA,iCAeA,UAbiB,WAUf,OAAO,mBAAKA,MATE,CACZC,QADY,OAEZC,WAFY,SAGZC,eAHY,SAIZC,OAJY,QAKZC,MALY,QAMZC,SAAU,KAGZ","file":"component---src-pages-404-js-9c134c3c1761f19419a4.js","sourcesContent":["import * as React from 'react'\n\nconst NotFound = () => {\n const style = {\n display: 'flex',\n alignItems: 'center',\n justifyContent: 'center',\n height: '100vh',\n width: '100vw',\n fontSize: 32,\n }\n\n return
Not Found
\n}\n\nexport default NotFound\n"],"sourceRoot":""}
\ No newline at end of file
diff --git a/docs/component---src-reversi-mdx-891984c4e973d2621598.js b/docs/component---src-reversi-mdx-891984c4e973d2621598.js
new file mode 100644
index 0000000..e71622f
--- /dev/null
+++ b/docs/component---src-reversi-mdx-891984c4e973d2621598.js
@@ -0,0 +1,2 @@
+(window.webpackJsonp=window.webpackJsonp||[]).push([[7],{V3A2:function(e,t,n){"use strict";n.r(t),n.d(t,"_frontmatter",(function(){return c})),n.d(t,"default",(function(){return s}));var a=n("Fcif"),o=n("+I+c"),r=(n("mXGw"),n("/FXl")),b=n("TjRS"),i=n("zmid"),c=(n("aD51"),{});void 0!==c&&c&&c===Object(c)&&Object.isExtensible(c)&&!c.hasOwnProperty("__filemeta")&&Object.defineProperty(c,"__filemeta",{configurable:!0,value:{name:"_frontmatter",filename:"src/reversi.mdx"}});var l=b.a;function s(e){var t=e.components,n=Object(o.a)(e,["components"]);return Object(r.b)(l,Object(a.a)({},n,{components:t,mdxType:"MDXLayout"}),Object(r.b)("h1",{id:"the-reversi-code"},"The Reversi Code"),Object(r.b)("p",null,"Researcher: ",Object(r.b)("a",{href:"//github.com/tyt2y3",parentName:"p"},"Chris Tsang")," | Published: 2021-04-01"),Object(r.b)("hr",null),Object(r.b)("p",null,"All we know about the barcode that somehow resembles a Reversi game."),Object(r.b)(i.a,{text:"The first Amazon Go store, Downtown Seattle",src:"symcode/The_first_Amazon_Go_store,_Downtown_Seattle_(49005185111).jpg",mdxType:"Figure"}),Object(r.b)("blockquote",null,Object(r.b)("p",{parentName:"blockquote"},"Photo by GoToVan via ",Object(r.b)("a",{href:"https://commons.wikimedia.org/wiki/File:The_first_Amazon_Go_store,_Downtown_Seattle_(49005185111).jpg",parentName:"p"},"Wikimedia Commons"))),Object(r.b)("p",null,"Disclaimer: I have never been to an Amazon Go store myself. All samples are collected from the internet, and I merely compiled my observations as below. I never received any hints from anyone related to Amazon, so my design and analysis of the Reversi Code is 'clean room'."),Object(r.b)("pre",null,Object(r.b)("code",{parentName:"pre"},"057\n⚫🔷⚪⚫⚫⚫\n⚪⚪⚫⚫⚫⚫\n⚫⚪⚫⚫🔷⚫\n⚪🔷⚫⚪⚪⚫\n")),Object(r.b)("pre",null,Object(r.b)("code",{parentName:"pre"},"206\n⚫⚪⚫⚪⚫⚪\n⚫⚫⚪🔷⚪⚫\n⚫⚫⚪⚫⚪⚫\n🔷⚫⚪⚪⚫🔷\n")),Object(r.b)("pre",null,Object(r.b)("code",{parentName:"pre"},"268\n⚫⚫⚫⚫⚫⚫\n🔷🔷⚪⚫🔷⚫\n⚫⚪⚫⚪⚪⚫\n⚪⚫⚪⚪⚫⚫\n")),Object(r.b)("pre",null,Object(r.b)("code",{parentName:"pre"},"322\n⚫🔷⚪⚪⚫⚫\n⚪⚪⚫⚪⚫⚫\n⚫⚪🔷⚪⚫🔷\n⚪⚫⚫⚪⚫⚫\n")),Object(r.b)("pre",null,Object(r.b)("code",{parentName:"pre"},"508\n⚫⚪⚫⚪⚫⚪\n⚫⚫⚫⚫⚫⚫\n⚫⚫🔷🔷⚪⚪\n⚫🔷⚪⚪⚫⚫\n")),Object(r.b)("pre",null,Object(r.b)("code",{parentName:"pre"},"520\n⚫⚫⚫⚪⚫⚫\n⚪⚫⚫⚪⚪⚫\n⚫⚫⚫🔷⚫⚫\n⚪🔷⚪⚫⚪🔷\n")),Object(r.b)("pre",null,Object(r.b)("code",{parentName:"pre"},"628\n⚫⚫⚪⚫⚫⚫\n⚪⚪⚫⚫⚪⚫\n🔷⚪⚫⚪⚫🔷\n⚪⚫🔷⚫⚫⚫\n")),Object(r.b)("pre",null,Object(r.b)("code",{parentName:"pre"},"974\n🔷⚫⚫⚫🔷⚫\n⚪⚪⚪⚫⚫⚫\n⚫⚪⚫⚫⚪⚫\n⚪⚫⚪⚪⚪🔷\n")),Object(r.b)("pre",null,Object(r.b)("code",{parentName:"pre"},"906\n🔷⚫⚫⚪⚫⚫\n⚪⚫⚪⚪⚫⚫\n⚫⚫⚪⚪⚪⚫\n⚪🔷⚫⚫⚪🔷\n")),Object(r.b)("pre",null,Object(r.b)("code",{parentName:"pre"},"445\n🔷⚫⚪⚪⚫⚫\n⚫⚪⚪⚪⚫⚫\n⚫⚫⚪⚪⚫⚫\n⚪⚫🔷⚫⚪🔷\n")),Object(r.b)("p",null,"Observations:"),Object(r.b)("ol",null,Object(r.b)("li",{parentName:"ol"},Object(r.b)("p",{parentName:"li"},"The matrix is 6x4.")),Object(r.b)("li",{parentName:"ol"},Object(r.b)("p",{parentName:"li"},"There are exactly three diamonds, which can be placed anywhere on the matrix.")),Object(r.b)("li",{parentName:"ol"},Object(r.b)("p",{parentName:"li"},"15 to 17 of them are occupied by circles or diamonds, meaning there are 12 to 14 circles.")),Object(r.b)("li",{parentName:"ol"},Object(r.b)("p",{parentName:"li"},"If we number the modules from top left to bottom right, as in"),Object(r.b)("pre",{parentName:"li"},Object(r.b)("code",{parentName:"pre"},"0 1 2 3 4 5\n6 7 8 9 10 11\n12 13 14 15 16 17\n18 19 20 21 22 23\n")),Object(r.b)("p",{parentName:"li"},"The 0th, 4th, 11th, 12th, 19th and 23rd module are always occupied. My guess is they act as a timing pattern.")),Object(r.b)("li",{parentName:"ol"},Object(r.b)("p",{parentName:"li"},"Either 5th or 18th module is occupied, but never both, my guess is they act as an orientation marker (otherwise the code will be rotational symmetric with itself).")),Object(r.b)("li",{parentName:"ol"},Object(r.b)("p",{parentName:"li"},"The remaining 16 positions can freely toggle. Due to #3, 8 to 10 of them must be occupied.")),Object(r.b)("li",{parentName:"ol"},Object(r.b)("p",{parentName:"li"},"There are (16 choose 8) + (16 choose 9) + (16 choose 10) = 32318 combinations.")),Object(r.b)("li",{parentName:"ol"},Object(r.b)("p",{parentName:"li"},"The diamonds can be put on any occupied position, so it multiplies an extra (17 choose 3) + (16 choose 3) + (15 choose 3) = 1695 combinations.")),Object(r.b)("li",{parentName:"ol"},Object(r.b)("p",{parentName:"li"},"It results in 54779010 combinations, just above 2^25 = 33554432\nWhich is reasonable enough, as it is a 6x4 matrix with a little extra freedom.")),Object(r.b)("li",{parentName:"ol"},Object(r.b)("p",{parentName:"li"},"If we set aside 5 bits as redundancy, then the Reversi Code can encode 20 bits payload. This is a reasonable guess in the sense that 20% redundancy provides good error detection capability.")),Object(r.b)("li",{parentName:"ol"},Object(r.b)("p",{parentName:"li"},"This is merely a combinatorial analysis representing the theoretic capacity of the barcode. Even given answers of the payload, it is extremely hard to reverse engineer the complete encoding scheme.")),Object(r.b)("li",{parentName:"ol"},Object(r.b)("p",{parentName:"li"},"There is a decimal number above every barcode on each label. From the layout of the label, we believe that it ranges from 000 to 999. It may or may not directly corresponds to the numeric value of the barcode."))),Object(r.b)("p",null,"Now, we can engineer a barcode that looks exactly the same, but encode data in a different way."))}void 0!==s&&s&&s===Object(s)&&Object.isExtensible(s)&&!s.hasOwnProperty("__filemeta")&&Object.defineProperty(s,"__filemeta",{configurable:!0,value:{name:"MDXContent",filename:"src/reversi.mdx"}}),s.isMDXComponent=!0}}]);
+//# sourceMappingURL=component---src-reversi-mdx-891984c4e973d2621598.js.map
\ No newline at end of file
diff --git a/docs/component---src-reversi-mdx-891984c4e973d2621598.js.map b/docs/component---src-reversi-mdx-891984c4e973d2621598.js.map
new file mode 100644
index 0000000..f0c8e70
--- /dev/null
+++ b/docs/component---src-reversi-mdx-891984c4e973d2621598.js.map
@@ -0,0 +1 @@
+{"version":3,"sources":["webpack:///../src/reversi.mdx"],"names":["_frontmatter","MDXLayout","DefaultLayout","MDXContent","components","props","mdxType","text","src","isMDXComponent"],"mappings":"mQAOaA,G,UAAe,S,iMAE5B,IAKMC,EAAYC,IACH,SAASC,EAAT,GAGZ,IAFDC,EAEC,EAFDA,WACGC,EACF,8BACD,OAAO,YAACJ,EAAD,eAAeI,EAAf,CAAsBD,WAAYA,EAAYE,QAAQ,cAC3D,iBAAQ,CACN,GAAM,oBADR,oBAGA,oCACE,gBAAO,CACL,KAAQ,sBACR,WAAc,KAFhB,eADF,4BAMA,uBACA,6FACA,YAAC,IAAD,CAAQC,KAAK,8CAA8CC,IAAI,wEAAwEF,QAAQ,WAC/I,8BAEE,gBAAO,CACL,WAAc,cADhB,wBAGE,gBAAO,CACL,KAAQ,wGACR,WAAc,KAFhB,uBAMJ,0SACA,uBAAK,mBAAU,CACX,WAAc,OADb,6CAQL,uBAAK,mBAAU,CACX,WAAc,OADb,6CAQL,uBAAK,mBAAU,CACX,WAAc,OADb,6CAQL,uBAAK,mBAAU,CACX,WAAc,OADb,6CAQL,uBAAK,mBAAU,CACX,WAAc,OADb,6CAQL,uBAAK,mBAAU,CACX,WAAc,OADb,6CAQL,uBAAK,mBAAU,CACX,WAAc,OADb,6CAQL,uBAAK,mBAAU,CACX,WAAc,OADb,6CAQL,uBAAK,mBAAU,CACX,WAAc,OADb,6CAQL,uBAAK,mBAAU,CACX,WAAc,OADb,6CAQL,sCACA,sBAEE,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,uBAOF,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,kFAOF,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,8FAOF,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,iEAKA,kBAAS,CACP,WAAc,MACb,mBAAU,CACT,WAAc,OADf,gFASH,gBAAO,CACL,WAAc,MADhB,kHAOF,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,wKAOF,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,+FAOF,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,mFAOF,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,mJAOF,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,oJAQF,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,kMAOF,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,0MAOF,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,uNAOJ,8H,+LAKJH,EAAWM,gBAAiB","file":"component---src-reversi-mdx-891984c4e973d2621598.js","sourcesContent":["import * as React from 'react'\n /* @jsx mdx */\nimport { mdx } from '@mdx-js/react';\n/* @jsx mdx */\n\nimport DefaultLayout from \"/home/chris/visioncortex.github.io/node_modules/gatsby-theme-docz/src/base/Layout.js\";\nimport { Figure } from './components/Figure';\nexport const _frontmatter = {};\n\nconst makeShortcode = name => function MDXDefaultShortcode(props) {\n console.warn(\"Component \" + name + \" was not imported, exported, or provided by MDXProvider as global scope\");\n return ;\n};\n\nconst MDXLayout = DefaultLayout;\nexport default function MDXContent({\n components,\n ...props\n}) {\n return \n
{`Disclaimer: I have never been to an Amazon Go store myself. All samples are collected from the internet, and I merely compiled my observations as below. I never received any hints from anyone related to Amazon, so my design and analysis of the Reversi Code is 'clean room'.`}
\n
{`057\n⚫🔷⚪⚫⚫⚫\n⚪⚪⚫⚫⚫⚫\n⚫⚪⚫⚫🔷⚫\n⚪🔷⚫⚪⚪⚫\n`}
\n
{`206\n⚫⚪⚫⚪⚫⚪\n⚫⚫⚪🔷⚪⚫\n⚫⚫⚪⚫⚪⚫\n🔷⚫⚪⚪⚫🔷\n`}
\n
{`268\n⚫⚫⚫⚫⚫⚫\n🔷🔷⚪⚫🔷⚫\n⚫⚪⚫⚪⚪⚫\n⚪⚫⚪⚪⚫⚫\n`}
\n
{`322\n⚫🔷⚪⚪⚫⚫\n⚪⚪⚫⚪⚫⚫\n⚫⚪🔷⚪⚫🔷\n⚪⚫⚫⚪⚫⚫\n`}
\n
{`508\n⚫⚪⚫⚪⚫⚪\n⚫⚫⚫⚫⚫⚫\n⚫⚫🔷🔷⚪⚪\n⚫🔷⚪⚪⚫⚫\n`}
\n
{`520\n⚫⚫⚫⚪⚫⚫\n⚪⚫⚫⚪⚪⚫\n⚫⚫⚫🔷⚫⚫\n⚪🔷⚪⚫⚪🔷\n`}
\n
{`628\n⚫⚫⚪⚫⚫⚫\n⚪⚪⚫⚫⚪⚫\n🔷⚪⚫⚪⚫🔷\n⚪⚫🔷⚫⚫⚫\n`}
\n
{`974\n🔷⚫⚫⚫🔷⚫\n⚪⚪⚪⚫⚫⚫\n⚫⚪⚫⚫⚪⚫\n⚪⚫⚪⚪⚪🔷\n`}
\n
{`906\n🔷⚫⚫⚪⚫⚫\n⚪⚫⚪⚪⚫⚫\n⚫⚫⚪⚪⚪⚫\n⚪🔷⚫⚫⚪🔷\n`}
\n
{`445\n🔷⚫⚪⚪⚫⚫\n⚫⚪⚪⚪⚫⚫\n⚫⚫⚪⚪⚫⚫\n⚪⚫🔷⚫⚪🔷\n`}
\n
{`Observations:`}
\n \n\n
\n\n
{`The matrix is 6x4.`}
\n\n
\n\n\n
\n\n
{`There are exactly three diamonds, which can be placed anywhere on the matrix.`}
\n\n
\n\n\n
\n\n
{`15 to 17 of them are occupied by circles or diamonds, meaning there are 12 to 14 circles.`}
\n\n
\n\n\n
\n\n
{`If we number the modules from top left to bottom right, as in`}
{`The 0th, 4th, 11th, 12th, 19th and 23rd module are always occupied. My guess is they act as a timing pattern.`}
\n\n
\n\n\n
\n\n
{`Either 5th or 18th module is occupied, but never both, my guess is they act as an orientation marker (otherwise the code will be rotational symmetric with itself).`}
\n\n
\n\n\n
\n\n
{`The remaining 16 positions can freely toggle. Due to #3, 8 to 10 of them must be occupied.`}
{`The diamonds can be put on any occupied position, so it multiplies an extra (17 choose 3) + (16 choose 3) + (15 choose 3) = 1695 combinations.`}
\n\n
\n\n\n
\n\n
{`It results in 54779010 combinations, just above 2^25 = 33554432\nWhich is reasonable enough, as it is a 6x4 matrix with a little extra freedom.`}
\n\n
\n\n\n
\n\n
{`If we set aside 5 bits as redundancy, then the Reversi Code can encode 20 bits payload. This is a reasonable guess in the sense that 20% redundancy provides good error detection capability.`}
\n\n
\n\n\n
\n\n
{`This is merely a combinatorial analysis representing the theoretic capacity of the barcode. Even given answers of the payload, it is extremely hard to reverse engineer the complete encoding scheme.`}
\n\n
\n\n\n
\n\n
{`There is a decimal number above every barcode on each label. From the layout of the label, we believe that it ranges from 000 to 999. It may or may not directly corresponds to the numeric value of the barcode.`}
\n\n
\n\n \n
{`Now, we can engineer a barcode that looks exactly the same, but encode data in a different way.`}
\n\n ;\n}\n;\nMDXContent.isMDXComponent = true;\n "],"sourceRoot":""}
\ No newline at end of file
diff --git a/docs/component---src-shapesense-mdx-a28963d66c7dc0995df0.js b/docs/component---src-shapesense-mdx-a28963d66c7dc0995df0.js
new file mode 100644
index 0000000..9e8d93e
--- /dev/null
+++ b/docs/component---src-shapesense-mdx-a28963d66c7dc0995df0.js
@@ -0,0 +1,2 @@
+(window.webpackJsonp=window.webpackJsonp||[]).push([[8],{PJ5t:function(e,t,n){"use strict";n.r(t),n.d(t,"_frontmatter",(function(){return l})),n.d(t,"default",(function(){return h}));var i=n("Fcif"),a=n("+I+c"),o=(n("mXGw"),n("/FXl")),s=n("TjRS"),r=n("zmid"),l=(n("aD51"),{});void 0!==l&&l&&l===Object(l)&&Object.isExtensible(l)&&!l.hasOwnProperty("__filemeta")&&Object.defineProperty(l,"__filemeta",{configurable:!0,value:{name:"_frontmatter",filename:"src/shapesense.mdx"}});var c=s.a;function h(e){var t=e.components,n=Object(a.a)(e,["components"]);return Object(o.b)(c,Object(i.a)({},n,{components:t,mdxType:"MDXLayout"}),Object(o.b)("h1",{id:"shapesense---shape-completion-by-curve-stitching"},"ShapeSense - Shape Completion by Curve Stitching"),Object(o.b)("p",null,"Researcher: ",Object(o.b)("a",{href:"//github.com/shpun817",parentName:"p"},"Sanford Pun")," | Supervisor: ",Object(o.b)("a",{href:"//github.com/tyt2y3",parentName:"p"},"Chris Tsang")," | Published: 2022-01-02"),Object(o.b)("hr",null),Object(o.b)("p",null,"Repository: ",Object(o.b)("a",{href:"https://github.com/visioncortex/ShapeSense",parentName:"p"},"visioncortex/ShapeSense")),Object(o.b)("p",null,"This project aims at recovering (completing) the ",Object(o.b)("strong",{parentName:"p"},"structure"),' of a shape after a portion of it is erased. In other words, given a shape with arbitrarily "discontinued parts", assuming that we have ',Object(o.b)("strong",{parentName:"p"},"no prior knowledge")," about the shape, how can we ",Object(o.b)("strong",{parentName:"p"},"reasonably connect the endpoints")," at the discontinuities?"),Object(o.b)(r.a,{src:"shape-sense/hole_showcase.png",text:"A shape with a hole. There are 6 endpoints in this case.",alt:"Hole showcase",mdxType:"Figure"}),Object(o.b)("p",null,"In the demo images shown throughout this documentation, the following coloring scheme is used: Black pixels denote the background. Red pixels denote the rasterized ",Object(o.b)("em",{parentName:"p"},"shape"),". White pixels denote the ",Object(o.b)("em",{parentName:"p"},"hole"),'. The (imaginary) intersection of the shape and the hole is the aforementioned "discontinued parts", which is what we try to recover.'),Object(o.b)(r.a,{src:"shape-sense/recovered_shape.png",text:"An example of completed shape.",alt:"Recovered shape showcase.",mdxType:"Figure"}),Object(o.b)("p",null,"Blue pixels will be used to denote the outline of the recovered parts of the shape."),Object(o.b)("p",null,"The whole process of shape completion involves intrapolating the missing outline and then filling the pixels in the hole with appropriate colors."),Object(o.b)("h2",{id:"simple-shape-completion"},"Simple Shape Completion"),Object(o.b)("p",null,"Let's begin the experiment with simple shapes, like an ellipse."),Object(o.b)(r.a,{src:"shape-sense/simple/ellipse_groundtruth_and_with_hole.png",text:"The first image is the ground truth (for reference only). The second image is the input to the ShapeCompletion pipeline.",alt:"An ellipse ground truth and another one with a hole.",mdxType:"Figure"}),Object(o.b)("h3",{id:"path-preprocessing"},"Path Preprocessing"),Object(o.b)("p",null,"The first stage of the pipeline is to obtain and process the paths (curves) representing the existing outline of the shape."),Object(o.b)("p",null,Object(o.b)("a",{href:"//github.com/visioncortex/visioncortex",parentName:"p"},"Vision Cortex's core library")," provides the necessary utilities to extract raw paths from an image."),Object(o.b)(r.a,{src:"shape-sense/simple/ellipse_preprocessed_path.png",text:"Yellow pixels denote the identified outline of the shape after simplification.",alt:"Path segments after preprocessing.",mdxType:"Figure"}),Object(o.b)("h3",{id:"tail-tangent-approximation"},"Tail Tangent Approximation"),Object(o.b)("p",null,"The next step is to (divide, if needed, and) extract the two curves from the two endpoints; smoothing is performed to better approximate the tangents near the endpoints (",Object(o.b)("em",{parentName:"p"},"tails")," of the whole curve). After this step, we will obtain two tangents (2-D direction vectors), one at each tail. We will call these tangents ",Object(o.b)("em",{parentName:"p"},"tail tangents"),"."),Object(o.b)(r.a,{src:"shape-sense/simple/tail_tangent_approx.png",text:"The naive approach is to simply take A as the tail tangent, but better (more practical/useful) approximations may be obtained by taking more subsequent segments into account (e.g. B, C, D).",alt:"Tail tangent approximation results.",mdxType:"Figure"}),Object(o.b)("p",null,"A number of factors determine the accuracy and robustness of tail tangent approximation. Our implementation supports configurations like how many points to consider from the tails, how long should the segments being considered accumulate to, and how the weights for each segment should change towards the tails."),Object(o.b)("h3",{id:"intrapolation"},"Intrapolation"),Object(o.b)("blockquote",null,Object(o.b)("p",{parentName:"blockquote"},Object(o.b)("strong",null,'Why "',Object(o.b)("b",null,"Intra"),'polation"?'),Object(o.b)("br",null),"\nIf we considered the existing outline of the shape as separate curves at each endpoint, we would be doing ",Object(o.b)("i",null,Object(o.b)("b",null,"inter"),"polation")," ",Object(o.b)("b",null,"between")," curves. However, in this project, we are focusing on curves (existing + missing) that form an outline of a single shape, so we argue that we are doing ",Object(o.b)("i",null,Object(o.b)("b",null,"intra"),"polation")," ",Object(o.b)("b",null,"within")," a curve.")),Object(o.b)("p",null,"With the two endpoints and their corresponding tail tangents, we can calculate for the missing part in different scenarios. The type of curves used in this project is cubic ",Object(o.b)("a",{href:"//en.wikipedia.org/wiki/B%C3%A9zier_curve",parentName:"p"},"Bézier curves"),". To specify such a curve, four points are required. An important property to note about this type of curves is that ",Object(o.b)("em",{parentName:"p"},"the curve constructed is always fully contained in the quadrilateral defined by the four points"),"."),Object(o.b)("p",null,"The first scenario is when the two tail tangents point to the same side with respect to the line connecting the two endpoints (we call this line the ",Object(o.b)("em",{parentName:"p"},"base"),")."),Object(o.b)(r.a,{src:"shape-sense/simple/intrapolate_same_side.png",text:"Both tail tangents at A and B point to the same side of the red, dashed line (the base).",alt:"Intrapolation when both tail tangents point to the same side",mdxType:"Figure"}),Object(o.b)("div",null,Object(o.b)("p",null,"To construct the curve between A and B, we need to identify two ",Object(o.b)("em",{parentName:"p"},"control points")," (C",Object(o.b)("sub",null,"A")," and C",Object(o.b)("sub",null,"B"),") between them. In our approach, we started with what we think is intuitive and made tweaks to resolve some issues in practice; this is what we end up with:")),Object(o.b)("br",null),Object(o.b)("div",null,Object(o.b)("p",null,"First, find the intersection of the two lines starting at A and B along the corresponding tail tangent. The mid-points between A/B and the intersection are then set to be C",Object(o.b)("sub",null,"A")," and C",Object(o.b)("sub",null,"B"),". If the intersection is too far away (i.e. the two lines are close to, if not exactly, parallel), we simply use a point on each line as the control points (e.g. translate A/B along their tail tangent by a factor of base length). Either way, if either C",Object(o.b)("sub",null,"A")," or C",Object(o.b)("sub",null,"B")," end up lying outside the hole region, we ",Object(o.b)("em",{parentName:"p"},"retract")," it by pushing it towards the endpoint, until it reaches the hole region.")),Object(o.b)("blockquote",null,Object(o.b)("p",{parentName:"blockquote"},Object(o.b)("strong",null,"What if the intersection was in the other direction?"),Object(o.b)("br",null),"\n",Object(o.b)(r.a,{src:"shape-sense/simple/intrapolation_bent_outwards.png",text:"If the line originating from A and B intersect in the negative direction (as shown above), we simply correct them by bending them inwards to be perpendicular with the base.",alt:"Tail tangents bent outwards; pulled back to be perpendicular to the base.",mdxType:"Figure"}))),Object(o.b)("p",null,"Another scenario is when the two tail tangents point to different sides of the base, as below."),Object(o.b)(r.a,{src:"shape-sense/simple/intrapolate_diff_sides.png",text:"A simple dot product operation can be used to detect such a scenario.",alt:"Intrapolation with tail tangents pointing to different sides of the base",mdxType:"Figure"}),Object(o.b)("p",null,"In this case, any intersections detected are meaningless because they must lie outside the hole region. Instead, we divide the curve into two halves and intrapolate two subcurves from each endpoint to the mid-point of the base as shown above."),Object(o.b)("p",null,"The last possible scenario is trivial to handle: when the lines are coincident, simply connect the endpoints with a straight line."),Object(o.b)(r.a,{src:"shape-sense/simple/intrapolate_coincidence.png",alt:"Intrapolation becomes connecting endpoints with a straight line in coincidence",mdxType:"Figure"}),Object(o.b)("p",null,"The case of our simple ellipse falls into the first scenario. The intrapolated outline is shown as follows:"),Object(o.b)(r.a,{src:"shape-sense/simple/ellipse_intrapolated.png",alt:"Full ellipse outline after intrapolation.",mdxType:"Figure"}),Object(o.b)("h3",{id:"color-filling"},"Color filling"),Object(o.b)("p",null,"To fill the hole with appropriate colors, we define three element types: ",Object(o.b)("em",{parentName:"p"},"Blank"),", ",Object(o.b)("em",{parentName:"p"},"Structure"),", and ",Object(o.b)("em",{parentName:"p"},"Texture"),". Note that in this project, it is restricted that only one shape is processed at once, and actual texture recovery is not performed - we're only interested in knowing which parts of the hole are Blank, Structure, or Texture."),Object(o.b)("table",null,Object(o.b)("thead",{parentName:"table"},Object(o.b)("tr",{parentName:"thead"},Object(o.b)("th",{align:"left",parentName:"tr"},"Element"),Object(o.b)("th",{align:"left",parentName:"tr"},"Description"))),Object(o.b)("tbody",{parentName:"table"},Object(o.b)("tr",{parentName:"tbody"},Object(o.b)("td",{align:"left",parentName:"tr"},"Blank"),Object(o.b)("td",{align:"left",parentName:"tr"},"Background pixels. (Black in our demo)")),Object(o.b)("tr",{parentName:"tbody"},Object(o.b)("td",{align:"left",parentName:"tr"},"Structure"),Object(o.b)("td",{align:"left",parentName:"tr"},"Outline of the shape; The intrapolated curve(s) obtained above is rasterized and drawn onto the hole. (Blue in our demo)")),Object(o.b)("tr",{parentName:"tbody"},Object(o.b)("td",{align:"left",parentName:"tr"},"Texture"),Object(o.b)("td",{align:"left",parentName:"tr"},"Solid part of the shape; To be filled in this section. (Red in our demo)")))),Object(o.b)("p",null,"The Structure elements divide the hole into several subregions. Each of these subregions contains wholly either Blank or Texture elements."),Object(o.b)(r.a,{src:"shape-sense/simple/filling_subregions.png",alt:"Hole of ellipse divided into two subregions",mdxType:"Figure"}),Object(o.b)("p",null,"In our example, the hole is divided by the intrapolated curve into two subregions. In order to guess whether to fill the subregions with Blank or Texture elements, we check if the number of Blank elements outside the hole boundary exceeds a certain tolerance."),Object(o.b)("p",null,"For the bottom subregion, the pixels right outside the bottom boundary are (almost) all red (Texture), therefore this subregion is classified as Texture and filled with Texture elements."),Object(o.b)("p",null,"For the top subregion, the pixels outside the left, top, and right sides of the boundary are considered. All of those pixels are background (Blank), so this subregion is classified as Blank."),Object(o.b)("p",null,"After filling, the shape of the ellipse is completed, as follows:"),Object(o.b)(r.a,{src:"shape-sense/simple/ellipse_complete.png",alt:"Complete ellipse",mdxType:"Figure"}),Object(o.b)("p",null,"If we move the hole around, shape completion yields the following results:"),Object(o.b)(r.a,{src:"shape-sense/simple/ellipse_diff_holes.png",alt:"Complete ellipses with different holes",mdxType:"Figure"}),Object(o.b)("h2",{id:"complex-shape-completion"},"Complex Shape Completion"),Object(o.b)("p",null,"The process of shape completion shown above has been rather straightforward because there is a strong assumption - the hole cuts the shape at exactly 2 endpoints only. Consider the following case:"),Object(o.b)(r.a,{src:"shape-sense/complex/ellipse_hole_across.png",text:"At a glance, we can tell how the endpoints should be grouped - A with B, and C with D, but how can we model the problem to match the endpoints such that the result of color filling always makes sense?",alt:"Ellipse with a long hole cutting its boundary at 4 endpoints",mdxType:"Figure"}),Object(o.b)("h3",{id:"endpoint-matching"},"Endpoint Matching"),Object(o.b)("h4",{id:"failed-attempt-local-proximity"},"Failed attempt: Local Proximity"),Object(o.b)("p",null,"An intuitive approach might be by endpoint proximity in a greedy manner. If we simply connect each endpoint to its nearest neighbor, the correct matching is found for the above case. However, this approach ceases to work for the following case:"),Object(o.b)(r.a,{src:"shape-sense/complex/ellipse_local_proximity_counterexample.png",text:"The correct matching seems to be A with B and C with D, but the top two endpoints are the closest.",alt:"Tall hole over ellipse.",mdxType:"Figure"}),Object(o.b)("h4",{id:"avoiding-intersections"},"Avoiding Intersections"),Object(o.b)("p",null,"Problematic matchings are the ones that lead to intersecting curves. If intersection occurs, the resulting shape deforms and there may be subregions that are surrounded by others, leading to problems in color filling. Therefore, the key of endpoint matching lies in ",Object(o.b)("strong",{parentName:"p"},"avoiding intersections"),"."),Object(o.b)("p",null,"Before intrapolation, some intersecting curves can already be identified by looking at endpoint connections that intersect."),Object(o.b)("p",null,"Imagine we have 4 endpoints A, B, C, and D. If the line segment AB intersects with CD, then the curve intrapolated from A to B must intersect with that from C to D."),Object(o.b)(r.a,{src:"shape-sense/complex/line_intersect_implies_curve_intersect.png",alt:"Line intersection among endpoints implies intersection of intrapolated curves",mdxType:"Figure"}),Object(o.b)("p",null,"Therefore, the first step to avoiding intersecting curves is to filter out matchings that contain intersecting lines."),Object(o.b)("p",null,"This ",Object(o.b)("a",{href:"//prase.cz/kalva/putnam/psoln/psol794.html",parentName:"p"},"webpage")," shows that minimizing the total length of endpoint connections is equivalent to finding a matching with no intersecting connections. Hence the problem is reduced to a ",Object(o.b)("a",{href:"//core.ac.uk/download/pdf/82212931.pdf",parentName:"p"},"Euclidean Bipartite Matching Problem"),", i.e. optimizing the global weights over matchings. The ",Object(o.b)("a",{href:"//en.wikipedia.org/wiki/Hungarian_algorithm",parentName:"p"},"Hungarian algorithm")," is used to solve such a problem."),Object(o.b)("p",null,"The rest of the intersecting curves have to be caught and filtered out after intrapolation has taken place. Bézier curve intersection can be detected by a recursive method called ",Object(o.b)("a",{href:"//en.wikipedia.org/wiki/De_Casteljau%27s_algorithm",parentName:"p"},"De Casteljau's (Bézier Clipping) algorithm"),", which is implemented in ",Object(o.b)("a",{href:"//crates.io/crates/flo_curves",parentName:"p"},"flo_curves"),", the Bézier curve library we use."),Object(o.b)("h2",{id:"robustness"},"Robustness"),Object(o.b)("p",null,"Throughout the process of development, we encountered many edge cases where the pipeline would break down when faced with certain geometrical configurations. For example, an endpoint detected at the corner of the hole may or may not be useful, because it may be an edge entering the hole at a corner, or it may just be the hole touching an edge with its corner."),Object(o.b)("p",null,"As much as we hope to correctly handle every single situation (if it's possible in the first place), we figured, from an engineering perspective, that we want an effective way to secure maximum overall robustness with a reasonable amount of effort."),Object(o.b)("p",null,"From experimentation, we observed that most of the pain-inducing geometrical configurations were deformed once we move the hole by just 1 pixel. Therefore, we decided that once the pipeline breaks down, it should try to ",Object(o.b)("strong",{parentName:"p"},"recover by expanding the hole by 1 pixel in each direction"),". If it successfully produces something in one of these attempts, that result is used."),Object(o.b)("hr",null),Object(o.b)(r.a,{src:"shape-sense/animated_demo.gif",alt:"Animated Demo.",mdxType:"Figure"}),Object(o.b)("p",null,"As shown above, the performance of our implementation is stable for the most part. Occasionally, the pipeline (arguably) incorrectly handles the cases when tail tangents are (nearly) coincident to the hole boundaries."),Object(o.b)(r.a,{src:"shape-sense/incorrect.png",alt:"Arguably incorrect case.",mdxType:"Figure"}),Object(o.b)("h2",{id:"discussion"},"Discussion"),Object(o.b)("p",null,"The direct application of this algorithm is symbolic recognition, where a part of a symbol might be obscured artificially or accidentally."),Object(o.b)("p",null,"It can also be used in image repairing provided we are able to construct a high level shape structures of a natural photograph."),Object(o.b)("p",null,"As only the local information around the unknown region is used, the above shape reconstruction operation is entropy neutral, where no new information is introduced and no priori knowledge is assumed."),Object(o.b)("p",null,"This is a key factor, consider that if we have priori knowledge or simply are making wild guesses to what the original symbol might be, it completely changes the problem or actually defies the reason of shape completion."),Object(o.b)("p",null,"Under the symbolic recognition framework we propose (will be detailed in another article), recognition is a macroscopic statistical measurement that would greatly benefit from missing pieces filled in."))}void 0!==h&&h&&h===Object(h)&&Object.isExtensible(h)&&!h.hasOwnProperty("__filemeta")&&Object.defineProperty(h,"__filemeta",{configurable:!0,value:{name:"MDXContent",filename:"src/shapesense.mdx"}}),h.isMDXComponent=!0}}]);
+//# sourceMappingURL=component---src-shapesense-mdx-a28963d66c7dc0995df0.js.map
\ No newline at end of file
diff --git a/docs/component---src-shapesense-mdx-a28963d66c7dc0995df0.js.map b/docs/component---src-shapesense-mdx-a28963d66c7dc0995df0.js.map
new file mode 100644
index 0000000..f26c562
--- /dev/null
+++ b/docs/component---src-shapesense-mdx-a28963d66c7dc0995df0.js.map
@@ -0,0 +1 @@
+{"version":3,"sources":["webpack:///../src/shapesense.mdx"],"names":["_frontmatter","MDXLayout","DefaultLayout","MDXContent","components","props","mdxType","src","text","alt","isMDXComponent"],"mappings":"mQAOaA,G,UAAe,S,oMAE5B,IAKMC,EAAYC,IACH,SAASC,EAAT,GAGZ,IAFDC,EAEC,EAFDA,WACGC,EACF,8BACD,OAAO,YAACJ,EAAD,eAAeI,EAAf,CAAsBD,WAAYA,EAAYE,QAAQ,cAC3D,iBAAQ,CACN,GAAM,oDADR,oDAGA,oCACE,gBAAO,CACL,KAAQ,wBACR,WAAc,KAFhB,eADF,kBAME,gBAAO,CACL,KAAQ,sBACR,WAAc,KAFhB,eANF,4BAWA,uBACA,oCACE,gBAAO,CACL,KAAQ,6CACR,WAAc,KAFhB,4BAIF,yEACE,qBAAY,CACV,WAAc,KADhB,aADF,2IAKE,qBAAY,CACV,WAAc,KADhB,sBALF,gCASE,qBAAY,CACV,WAAc,KADhB,oCATF,4BAaA,YAAC,IAAD,CAAQC,IAAI,gCAAgCC,KAAK,2DAA2DC,IAAI,gBAAgBH,QAAQ,WACxI,4LACE,iBAAQ,CACN,WAAc,KADhB,SADF,6BAKE,iBAAQ,CACN,WAAc,KADhB,QALF,yIASA,YAAC,IAAD,CAAQC,IAAI,kCAAkCC,KAAK,iCAAiCC,IAAI,4BAA4BH,QAAQ,WAC5H,4GACA,0KACA,iBAAQ,CACN,GAAM,2BADR,2BAGA,wFACA,YAAC,IAAD,CAAQC,IAAI,2DAA2DC,KAAK,2HAA2HC,IAAI,uDAAuDH,QAAQ,WAC1Q,iBAAQ,CACN,GAAM,sBADR,sBAGA,oJACA,qBAAG,gBAAO,CACN,KAAQ,yCACR,WAAc,KAFf,gCAAH,yEAKA,YAAC,IAAD,CAAQC,IAAI,mDAAmDC,KAAK,iFAAiFC,IAAI,qCAAqCH,QAAQ,WACtM,iBAAQ,CACN,GAAM,8BADR,8BAGA,kMACE,iBAAQ,CACN,WAAc,KADhB,SADF,6IAKE,iBAAQ,CACN,WAAc,KADhB,iBALF,KASA,YAAC,IAAD,CAAQC,IAAI,6CAA6CC,KAAK,gMAAgMC,IAAI,sCAAsCH,QAAQ,WAChT,gVACA,iBAAQ,CACN,GAAM,iBADR,iBAGA,8BAEE,gBAAO,CACL,WAAc,cACb,kCACC,8BADD,cAGD,uBALF,+GAQE,qBAAG,8BAAH,YARF,IAWE,gCAXF,2JAaE,qBAAG,8BAAH,YAbF,IAgBE,+BAhBF,cAoBF,qMACE,gBAAO,CACL,KAAQ,4CACR,WAAc,KAFhB,iBADF,wHAME,iBAAQ,CACN,WAAc,KADhB,mGANF,KAUA,6KACE,iBAAQ,CACN,WAAc,KADhB,QADF,MAKA,YAAC,IAAD,CAAQC,IAAI,+CAA+CC,KAAK,2FAA2FC,IAAI,+DAA+DH,QAAQ,WACtO,uBACE,wFACE,iBAAQ,CACN,WAAc,KADhB,kBADF,MAKE,4BALF,SAOE,4BAPF,iKAUF,uBACA,uBACE,oMACE,4BADF,SAGE,4BAHF,gQAKE,4BALF,QAOE,4BAPF,6CASE,iBAAQ,CACN,WAAc,KADhB,WATF,8EAcF,8BAEE,gBAAO,CACL,WAAc,cACb,kFACD,uBAHF,KAME,YAAC,IAAD,CAAQC,IAAI,qDAAqDC,KAAI,+KAAkLC,IAAI,4EAA4EH,QAAQ,aAGnV,uHACA,YAAC,IAAD,CAAQC,IAAI,gDAAgDC,KAAK,wEAAwEC,IAAI,2EAA2EH,QAAQ,WAChO,2QACA,2JACA,YAAC,IAAD,CAAQC,IAAI,iDAAiDE,IAAI,iFAAiFH,QAAQ,WAC1J,oIACA,YAAC,IAAD,CAAQC,IAAI,8CAA8CE,IAAI,4CAA4CH,QAAQ,WAClH,iBAAQ,CACN,GAAM,iBADR,iBAGA,iGACE,iBAAQ,CACN,WAAc,KADhB,SADF,KAKE,iBAAQ,CACN,WAAc,KADhB,aALF,SASE,iBAAQ,CACN,WAAc,KADhB,WATF,qOAaA,yBAEE,oBAAW,CACT,WAAc,SAGd,iBAAQ,CACN,WAAc,SAGd,iBAAQ,CACN,MAAS,OACT,WAAc,MAFhB,WAMA,iBAAQ,CACN,MAAS,OACT,WAAc,MAFhB,iBAUJ,oBAAW,CACT,WAAc,SAGd,iBAAQ,CACN,WAAc,SAGd,iBAAQ,CACN,MAAS,OACT,WAAc,MAFhB,SAMA,iBAAQ,CACN,MAAS,OACT,WAAc,MAFhB,2CAQF,iBAAQ,CACN,WAAc,SAGd,iBAAQ,CACN,MAAS,OACT,WAAc,MAFhB,aAMA,iBAAQ,CACN,MAAS,OACT,WAAc,MAFhB,6HAQF,iBAAQ,CACN,WAAc,SAGd,iBAAQ,CACN,MAAS,OACT,WAAc,MAFhB,WAMA,iBAAQ,CACN,MAAS,OACT,WAAc,MAFhB,+EAUN,mKACA,YAAC,IAAD,CAAQC,IAAI,4CAA4CE,IAAI,8CAA8CH,QAAQ,WAClH,4RACA,mNACA,uNACA,0FACA,YAAC,IAAD,CAAQC,IAAI,0CAA0CE,IAAI,mBAAmBH,QAAQ,WACrF,mGACA,YAAC,IAAD,CAAQC,IAAI,4CAA4CE,IAAI,yCAAyCH,QAAQ,WAC7G,iBAAQ,CACN,GAAM,4BADR,4BAGA,6NACA,YAAC,IAAD,CAAQC,IAAI,8CAA8CC,KAAK,2MAA2MC,IAAI,+DAA+DH,QAAQ,WACrV,iBAAQ,CACN,GAAM,qBADR,qBAGA,iBAAQ,CACN,GAAM,kCADR,mCAGA,6QACA,YAAC,IAAD,CAAQC,IAAI,iEAAiEC,KAAK,qGAAqGC,IAAI,0BAA0BH,QAAQ,WAC7N,iBAAQ,CACN,GAAM,0BADR,0BAGA,kSACE,qBAAY,CACV,WAAc,KADhB,0BADF,KAKA,oJACA,6LACA,YAAC,IAAD,CAAQC,IAAI,iEAAiEE,IAAI,gFAAgFH,QAAQ,WACzK,8IACA,6BACE,gBAAO,CACL,KAAQ,6CACR,WAAc,KAFhB,WADF,2KAME,gBAAO,CACL,KAAQ,yCACR,WAAc,KAFhB,wCANF,4DAWE,gBAAO,CACL,KAAQ,8CACR,WAAc,KAFhB,uBAXF,qCAgBA,2MACE,gBAAO,CACL,KAAQ,qDACR,WAAc,KAFhB,8CADF,6BAME,gBAAO,CACL,KAAQ,gCACR,WAAc,KAFhB,cANF,sCAWA,iBAAQ,CACN,GAAM,cADR,cAGA,kYACA,iRACA,oPACE,qBAAY,CACV,WAAc,KADhB,8DADF,0FAKA,uBACA,YAAC,IAAD,CAAQC,IAAI,gCAAgCE,IAAI,iBAAiBH,QAAQ,WACzE,kPACA,YAAC,IAAD,CAAQC,IAAI,4BAA4BE,IAAI,2BAA2BH,QAAQ,WAC/E,iBAAQ,CACN,GAAM,cADR,cAGA,mKACA,wJACA,iOACA,qPACA,wO,kMAKJH,EAAWO,gBAAiB","file":"component---src-shapesense-mdx-a28963d66c7dc0995df0.js","sourcesContent":["import * as React from 'react'\n /* @jsx mdx */\nimport { mdx } from '@mdx-js/react';\n/* @jsx mdx */\n\nimport DefaultLayout from \"/home/chris/visioncortex.github.io/node_modules/gatsby-theme-docz/src/base/Layout.js\";\nimport { Figure } from './components/Figure';\nexport const _frontmatter = {};\n\nconst makeShortcode = name => function MDXDefaultShortcode(props) {\n console.warn(\"Component \" + name + \" was not imported, exported, or provided by MDXProvider as global scope\");\n return ;\n};\n\nconst MDXLayout = DefaultLayout;\nexport default function MDXContent({\n components,\n ...props\n}) {\n return \n
{`ShapeSense - Shape Completion by Curve Stitching`}
{`This project aims at recovering (completing) the `}\n {`structure`}\n {` of a shape after a portion of it is erased. In other words, given a shape with arbitrarily \"discontinued parts\", assuming that we have `}\n {`no prior knowledge`}\n {` about the shape, how can we `}\n {`reasonably connect the endpoints`}\n {` at the discontinuities?`}
\n \n
{`In the demo images shown throughout this documentation, the following coloring scheme is used: Black pixels denote the background. Red pixels denote the rasterized `}\n {`shape`}\n {`. White pixels denote the `}\n {`hole`}\n {`. The (imaginary) intersection of the shape and the hole is the aforementioned \"discontinued parts\", which is what we try to recover.`}
\n \n
{`Blue pixels will be used to denote the outline of the recovered parts of the shape.`}
\n
{`The whole process of shape completion involves intrapolating the missing outline and then filling the pixels in the hole with appropriate colors.`}
\n
{`Simple Shape Completion`}
\n
{`Let's begin the experiment with simple shapes, like an ellipse.`}
\n \n
{`Path Preprocessing`}
\n
{`The first stage of the pipeline is to obtain and process the paths (curves) representing the existing outline of the shape.`}
{`The next step is to (divide, if needed, and) extract the two curves from the two endpoints; smoothing is performed to better approximate the tangents near the endpoints (`}\n {`tails`}\n {` of the whole curve). After this step, we will obtain two tangents (2-D direction vectors), one at each tail. We will call these tangents `}\n {`tail tangents`}\n {`.`}
\n \n
{`A number of factors determine the accuracy and robustness of tail tangent approximation. Our implementation supports configurations like how many points to consider from the tails, how long should the segments being considered accumulate to, and how the weights for each segment should change towards the tails.`}
\n
{`Intrapolation`}
\n
\n\n
{`Why \"`}\n {`Intra`}\n {`polation\"?`}\n \n {`\nIf we considered the existing outline of the shape as separate curves at each endpoint, we would be doing `}\n {`inter`}\n {`polation`}\n {` `}\n {`between`}\n {` curves. However, in this project, we are focusing on curves (existing + missing) that form an outline of a single shape, so we argue that we are doing `}\n {`intra`}\n {`polation`}\n {` `}\n {`within`}\n {` a curve.`}
\n\n
\n
{`With the two endpoints and their corresponding tail tangents, we can calculate for the missing part in different scenarios. The type of curves used in this project is cubic `}\n {`Bézier curves`}\n {`. To specify such a curve, four points are required. An important property to note about this type of curves is that `}\n {`the curve constructed is always fully contained in the quadrilateral defined by the four points`}\n {`.`}
\n
{`The first scenario is when the two tail tangents point to the same side with respect to the line connecting the two endpoints (we call this line the `}\n {`base`}\n {`).`}
\n \n
\n
{`To construct the curve between A and B, we need to identify two `}\n {`control points`}\n {` (C`}\n {`A`}\n {` and C`}\n {`B`}\n {`) between them. In our approach, we started with what we think is intuitive and made tweaks to resolve some issues in practice; this is what we end up with:`}
\n
\n \n
\n
{`First, find the intersection of the two lines starting at A and B along the corresponding tail tangent. The mid-points between A/B and the intersection are then set to be C`}\n {`A`}\n {` and C`}\n {`B`}\n {`. If the intersection is too far away (i.e. the two lines are close to, if not exactly, parallel), we simply use a point on each line as the control points (e.g. translate A/B along their tail tangent by a factor of base length). Either way, if either C`}\n {`A`}\n {` or C`}\n {`B`}\n {` end up lying outside the hole region, we `}\n {`retract`}\n {` it by pushing it towards the endpoint, until it reaches the hole region.`}
\n
\n
\n\n
{`What if the intersection was in the other direction?`}\n \n {`\n`}\n
\n\n
\n
{`Another scenario is when the two tail tangents point to different sides of the base, as below.`}
\n \n
{`In this case, any intersections detected are meaningless because they must lie outside the hole region. Instead, we divide the curve into two halves and intrapolate two subcurves from each endpoint to the mid-point of the base as shown above.`}
\n
{`The last possible scenario is trivial to handle: when the lines are coincident, simply connect the endpoints with a straight line.`}
\n \n
{`The case of our simple ellipse falls into the first scenario. The intrapolated outline is shown as follows:`}
\n \n
{`Color filling`}
\n
{`To fill the hole with appropriate colors, we define three element types: `}\n {`Blank`}\n {`, `}\n {`Structure`}\n {`, and `}\n {`Texture`}\n {`. Note that in this project, it is restricted that only one shape is processed at once, and actual texture recovery is not performed - we're only interested in knowing which parts of the hole are Blank, Structure, or Texture.`}
\n
\n\n \n\n
\n\n
{`Element`}
\n\n\n
{`Description`}
\n\n
\n\n \n\n\n \n\n
\n\n
{`Blank`}
\n\n\n
{`Background pixels. (Black in our demo)`}
\n\n
\n\n\n
\n\n
{`Structure`}
\n\n\n
{`Outline of the shape; The intrapolated curve(s) obtained above is rasterized and drawn onto the hole. (Blue in our demo)`}
\n\n
\n\n\n
\n\n
{`Texture`}
\n\n\n
{`Solid part of the shape; To be filled in this section. (Red in our demo)`}
\n\n
\n\n \n\n
\n
{`The Structure elements divide the hole into several subregions. Each of these subregions contains wholly either Blank or Texture elements.`}
\n \n
{`In our example, the hole is divided by the intrapolated curve into two subregions. In order to guess whether to fill the subregions with Blank or Texture elements, we check if the number of Blank elements outside the hole boundary exceeds a certain tolerance.`}
\n
{`For the bottom subregion, the pixels right outside the bottom boundary are (almost) all red (Texture), therefore this subregion is classified as Texture and filled with Texture elements.`}
\n
{`For the top subregion, the pixels outside the left, top, and right sides of the boundary are considered. All of those pixels are background (Blank), so this subregion is classified as Blank.`}
\n
{`After filling, the shape of the ellipse is completed, as follows:`}
\n \n
{`If we move the hole around, shape completion yields the following results:`}
\n \n
{`Complex Shape Completion`}
\n
{`The process of shape completion shown above has been rather straightforward because there is a strong assumption - the hole cuts the shape at exactly 2 endpoints only. Consider the following case:`}
\n \n
{`Endpoint Matching`}
\n
{`Failed attempt: Local Proximity`}
\n
{`An intuitive approach might be by endpoint proximity in a greedy manner. If we simply connect each endpoint to its nearest neighbor, the correct matching is found for the above case. However, this approach ceases to work for the following case:`}
\n \n
{`Avoiding Intersections`}
\n
{`Problematic matchings are the ones that lead to intersecting curves. If intersection occurs, the resulting shape deforms and there may be subregions that are surrounded by others, leading to problems in color filling. Therefore, the key of endpoint matching lies in `}\n {`avoiding intersections`}\n {`.`}
\n
{`Before intrapolation, some intersecting curves can already be identified by looking at endpoint connections that intersect.`}
\n
{`Imagine we have 4 endpoints A, B, C, and D. If the line segment AB intersects with CD, then the curve intrapolated from A to B must intersect with that from C to D.`}
\n \n
{`Therefore, the first step to avoiding intersecting curves is to filter out matchings that contain intersecting lines.`}
\n
{`This `}\n {`webpage`}\n {` shows that minimizing the total length of endpoint connections is equivalent to finding a matching with no intersecting connections. Hence the problem is reduced to a `}\n {`Euclidean Bipartite Matching Problem`}\n {`, i.e. optimizing the global weights over matchings. The `}\n {`Hungarian algorithm`}\n {` is used to solve such a problem.`}
\n
{`The rest of the intersecting curves have to be caught and filtered out after intrapolation has taken place. Bézier curve intersection can be detected by a recursive method called `}\n {`De Casteljau's (Bézier Clipping) algorithm`}\n {`, which is implemented in `}\n {`flo_curves`}\n {`, the Bézier curve library we use.`}
\n
{`Robustness`}
\n
{`Throughout the process of development, we encountered many edge cases where the pipeline would break down when faced with certain geometrical configurations. For example, an endpoint detected at the corner of the hole may or may not be useful, because it may be an edge entering the hole at a corner, or it may just be the hole touching an edge with its corner.`}
\n
{`As much as we hope to correctly handle every single situation (if it's possible in the first place), we figured, from an engineering perspective, that we want an effective way to secure maximum overall robustness with a reasonable amount of effort.`}
\n
{`From experimentation, we observed that most of the pain-inducing geometrical configurations were deformed once we move the hole by just 1 pixel. Therefore, we decided that once the pipeline breaks down, it should try to `}\n {`recover by expanding the hole by 1 pixel in each direction`}\n {`. If it successfully produces something in one of these attempts, that result is used.`}
\n \n \n
{`As shown above, the performance of our implementation is stable for the most part. Occasionally, the pipeline (arguably) incorrectly handles the cases when tail tangents are (nearly) coincident to the hole boundaries.`}
\n \n
{`Discussion`}
\n
{`The direct application of this algorithm is symbolic recognition, where a part of a symbol might be obscured artificially or accidentally.`}
\n
{`It can also be used in image repairing provided we are able to construct a high level shape structures of a natural photograph.`}
\n
{`As only the local information around the unknown region is used, the above shape reconstruction operation is entropy neutral, where no new information is introduced and no priori knowledge is assumed.`}
\n
{`This is a key factor, consider that if we have priori knowledge or simply are making wild guesses to what the original symbol might be, it completely changes the problem or actually defies the reason of shape completion.`}
\n
{`Under the symbolic recognition framework we propose (will be detailed in another article), recognition is a macroscopic statistical measurement that would greatly benefit from missing pieces filled in.`}
\n\n ;\n}\n;\nMDXContent.isMDXComponent = true;\n "],"sourceRoot":""}
\ No newline at end of file
diff --git a/docs/component---src-symcode-mdx-4c86476283db67665ee4.js b/docs/component---src-symcode-mdx-4c86476283db67665ee4.js
new file mode 100644
index 0000000..7384a39
--- /dev/null
+++ b/docs/component---src-symcode-mdx-4c86476283db67665ee4.js
@@ -0,0 +1,2 @@
+(window.webpackJsonp=window.webpackJsonp||[]).push([[9],{AJ3O:function(e,t,a){"use strict";a.r(t),a.d(t,"_frontmatter",(function(){return c})),a.d(t,"default",(function(){return l}));var n=a("Fcif"),o=a("+I+c"),i=(a("mXGw"),a("/FXl")),s=a("TjRS"),r=a("zmid"),c=(a("aD51"),{});void 0!==c&&c&&c===Object(c)&&Object.isExtensible(c)&&!c.hasOwnProperty("__filemeta")&&Object.defineProperty(c,"__filemeta",{configurable:!0,value:{name:"_frontmatter",filename:"src/symcode.mdx"}});var b=s.a;function l(e){var t=e.components,a=Object(o.a)(e,["components"]);return Object(i.b)(b,Object(n.a)({},a,{components:t,mdxType:"MDXLayout"}),Object(i.b)("h1",{id:"symcode-the-symbolic-barcode-for-humans-and-machines"},"SymCode: The Symbolic Barcode for Humans and Machines"),Object(i.b)("p",null,"Researchers: ",Object(i.b)("a",{href:"//github.com/tyt2y3",parentName:"p"},"Chris Tsang")," and ",Object(i.b)("a",{href:"//github.com/shpun817",parentName:"p"},"Sanford Pun")," | Published: 2021-04-03"),Object(i.b)("hr",null),Object(i.b)("p",null,"This is our story of the design and implementation of a symbolic barcode system.\nTry the Demo Web App on ",Object(i.b)("a",{href:"https://symcode.visioncortex.org/",target:"_blank"},"symcode.visioncortex.org"),"."),Object(i.b)("iframe",{src:"https://player.vimeo.com/video/529877814?muted=1",style:{width:"100%",height:"80vh",maxHeight:"75vw"},frameBorder:"0",allow:"autoplay; fullscreen",allowFullScreen:!0}),Object(i.b)("h2",{id:"forewords"},"Forewords"),Object(i.b)("p",null,"If you are lucky enough to live in a place where an Amazon Go store is nearby, you definitely should visit it. It is packed full of esoteric technology that enthusiast like us get super intrigued. One item that caught our eyes was the labels on the salad and sandwiches in the ready-made food section."),Object(i.b)(r.a,{text:"Photo within an Amazon Go store",src:"symcode/Amazon Go Seattle - crop.jpg",mdxType:"Figure"}),Object(i.b)("blockquote",null,Object(i.b)("p",{parentName:"blockquote"},"Photo by Sikander Iqbal via ",Object(i.b)("a",{href:"https://commons.wikimedia.org/wiki/File:Amazon_Go_-_Seattle_(20180804111213).jpg",parentName:"p"},"Wikimedia Commons"))),Object(i.b)("p",null,"There printed a matrix of cute little circles and squares! It must be a barcode of some sort. Our attempt to decipher the '",Object(i.b)("a",{href:"/reversi-docs",parentName:"p"},"Amazon Reversi Code"),"' was futile, but this is the start of the story."),Object(i.b)("h2",{id:"background"},"Background"),Object(i.b)("p",null,"A bit of context for the unfamiliar reader, the name Barcode is referring to the labels we still find in virtually every packaged products today. Barcode encode data by varying the widths and spacings of the vertical bars, which are to be scanned by a laser beam crossing horizontally. With the advent of image sensors, 2D barcodes were developed, where QR code and Aztec Code are some of the most well-known."),Object(i.b)("p",null,"In general, 2D barcode scanner works as follows:"),Object(i.b)("ol",null,Object(i.b)("li",{parentName:"ol"},Object(i.b)("p",{parentName:"li"},"Locate the finders. These finders act as a trigger to tell the scanner that 'here is a barcode!'. The finders have to be easily detectable yet not interfering with the data modules.")),Object(i.b)("li",{parentName:"ol"},Object(i.b)("p",{parentName:"li"},"Attempt to scan and read metadata surrounding the finder candidates. The metadata usually includes a format code and timing patterns.")),Object(i.b)("li",{parentName:"ol"},Object(i.b)("p",{parentName:"li"},"If the previous step succeeds, the scanner would sample the entire barcode to construct a raw bit string. A checksum operation is done to detect errors.")),Object(i.b)("li",{parentName:"ol"},Object(i.b)("p",{parentName:"li"},"If the previous step succeeds, the payload is extract from the raw bits, which usually involves some bit masking and shuffling."))),Object(i.b)("p",null,"This framework has aged well and they are quite robust and low-cost to implement in embedded systems. However, they are meant to be read by machines and not by our naked eyes. They are not designed for aesthetics and are impossible for humans to comprehend."),Object(i.b)("h2",{id:"goals-and-constraints"},"Goals and Constraints"),Object(i.b)("p",null,"Our goal is to design a barcode system that are both human-readable and machine-readable. We now turn our attention to human readability by studying human languages."),Object(i.b)("p",null,"Machines run on bits, so we can say the alphabet consists of 0 and 1. In human languages, we have a larger set of alphabets. In English, we have 26 distinct lowercase letters. In Korean, characters are constructed by composing 2 to 6 elements from a set of 40 distinct Jamo."),Object(i.b)("p",null,"There is a direct tradeoff between information density and visual ambiguity. If the symbol set is too large, humans would have difficulty in remembering all of them. In addition, there may be some visually similar symbols that are hard to disambiguate. If the symbol set is too small, the same message has to be encoded with more symbols, which again, humans often have a hard time in processing long strings."),Object(i.b)("p",null,"We determined that a symbol set in the range of 16 to 64 symbols is a good balance."),Object(i.b)("h2",{id:"design-of-symbols"},"Design of symbols"),Object(i.b)("p",null,"What makes good symbols?"),Object(i.b)("ol",null,Object(i.b)("li",{parentName:"ol"},Object(i.b)("p",{parentName:"li"},"Prominence"),Object(i.b)("p",{parentName:"li"},"First, the symbols have to stand out from the natural world, to manifest that they are created deliberately to convey a message but not a result of some natural phenomenon.")),Object(i.b)("li",{parentName:"ol"},Object(i.b)("p",{parentName:"li"},"Repeatable"),Object(i.b)("p",{parentName:"li"},"Symbols are constructed with specific tools and processes that can be taught to other people. The meaning of a symbol should remain the same when reproduced, in which variation is tolerated.")),Object(i.b)("li",{parentName:"ol"},Object(i.b)("p",{parentName:"li"},"Distinctive"),Object(i.b)("p",{parentName:"li"},"Symbols within a set should not be similar with each other and have distinctive features allowing the reader to resolve ambiguity between symbols.")),Object(i.b)("li",{parentName:"ol"},Object(i.b)("p",{parentName:"li"},"Aesthetics"),Object(i.b)("p",{parentName:"li"},"Finally, good symbols should observe the aesthetics rules of the human eye, including being anti-symmetric, rotational symmetric, balanced and consistent. As a pointer, the ",Object(i.b)("a",{href:"https://en.wikipedia.org/wiki/Gestalt_psychology",parentName:"p"},"Gestalt Principles")," are fantastic rules of thumb."))),Object(i.b)("p",null,"With the above rules in mind, we designed a minimalistic symbol set. Each symbol is composed of multiple triangles, the basic geometric primitive. Each symbol is symmetric or anti-symmetric in overall, but exhibits asymmetry internally. They are like Tangram, in which a human child can easily reproduce the symbols by assembly some triangular pieces together."),Object(i.b)("p",null,"The next section would quantitatively analyze and justify this design methodology."),Object(i.b)("h2",{id:"trace-of-shapes"},"Trace of shapes"),Object(i.b)("p",null,"The naive way to match a shape against a symbol set is to perform a linear search, XOR it with every symbol and then chooses the one with the lowest delta. It works in principle, but is inefficient. Ideally, an algorithm can condense a complex shape into a feature vector, which we can lookup in the feature space of the symbol set for the nearest neighbour and arrive to a match."),Object(i.b)("p",null,"Instead of using an array of real numbers, we devised that an array of bits are sufficient to capture the essence of symbols, and from now on we refer this bit string as 'trace'."),Object(i.b)("h3",{id:"the-english-alphabet"},"The English alphabet"),Object(i.b)("p",null,"Now, let us take a closer look at the lowercase English alphabet set to illustrate this idea."),Object(i.b)(r.a,{text:"English alphabets",src:"symcode/english alphabet.svg",height:"200px",mdxType:"Figure"}),Object(i.b)("p",null,"First off, we can classify the 26 alphabets as either tall or short, giving:"),Object(i.b)("pre",null,Object(i.b)("code",{parentName:"pre"},"Tall: b d f g h j k l p q t y\nShort: a c e i m n o r s u v w x z\n")),Object(i.b)("p",null,"Next, we can divide the letter into two halfs horizontally and compare their weights:"),Object(i.b)("pre",null,Object(i.b)("code",{parentName:"pre"},"Left > right: b c f h k p r t y\nLeft < right: a d g q\nLeft ~ right: e i j l m n o s u v w x z\n")),Object(i.b)("p",null,"Then, we can divide the letter into two halfs vertically and compare their weights:"),Object(i.b)("pre",null,Object(i.b)("code",{parentName:"pre"},"Up > down: f m n p q r y\nUp < down: b d h k u\nUp ~ down: a c e g i j l o s t v w x z\n")),Object(i.b)("p",null,"At this point, we had the following:"),Object(i.b)("pre",null,Object(i.b)("code",{parentName:"pre"},"a: SR=\nb: TLD\nc: SL=\nd: TRD\ne: S==\nf: TLU\ng: TR=\nh: TLD\ni: S==\nj: T==\nk: TLD\nl: T==\nm: S=U\nn: S=U\no: S==\np: TLU\nq: TRU\nr: SLU\ns: S==\nt: TLD\nu: S=D\nv: S==\nw: S==\nx: S==\ny: TLU\nz: S==\n")),Object(i.b)("p",null,"Group by trace:"),Object(i.b)("pre",null,Object(i.b)("code",{parentName:"pre"},"SR=: a\nTLD: b h k\nSL=: c\nTRD: d\nS==: e i o s v w x z\nTL=: t\nTR=: g\nT==: j l \nS=U: m n\nTLU: f p y\nTRU: q\nSLU: r\nS=D: u\n")),Object(i.b)("p",null,"Which is a surprisingly good classifier using only three comparisons. We can do more trinary comparisons on smaller partitions to further differentiate the collisions, but our investigation on English alphabets ends here for the scope of this article."),Object(i.b)("h3",{id:"symcode"},"SymCode"),Object(i.b)("p",null,"Our trace for SymCode symbols follows a similar scheme. We define the symbol traces starting from the three comparisons defined in the previous section."),Object(i.b)("p",null,"The pixel count in each of the four quadrants of a symbol (in order of top-left, top-right, bottom-left, bottom-right) are denoted by non-negative quantities ",Object(i.b)("inlineCode",{parentName:"p"},"a"),",",Object(i.b)("inlineCode",{parentName:"p"},"b"),",",Object(i.b)("inlineCode",{parentName:"p"},"c"),",",Object(i.b)("inlineCode",{parentName:"p"},"d")," respectively, the three comparisons are defined as follows:"),Object(i.b)(r.a,{text:"Grid quadrants",src:"symcode/grid abcd.svg",height:"200px",mdxType:"Figure"}),Object(i.b)("ul",null,Object(i.b)("li",{parentName:"ul"},"Vertical comparison = ",Object(i.b)("inlineCode",{parentName:"li"},"(a+b)")," vs ",Object(i.b)("inlineCode",{parentName:"li"},"(c+d)")),Object(i.b)("li",{parentName:"ul"},"Horizontal comparison = ",Object(i.b)("inlineCode",{parentName:"li"},"(a+c)")," vs ",Object(i.b)("inlineCode",{parentName:"li"},"(b+d)")),Object(i.b)("li",{parentName:"ul"},"Diagonal comparison = ",Object(i.b)("inlineCode",{parentName:"li"},"(a+d)")," vs ",Object(i.b)("inlineCode",{parentName:"li"},"(b+c)"))),Object(i.b)("p",null,"Below are two examples:"),Object(i.b)(r.a,{text:"LongRR symbol image",src:"symcode/LongRR.png",mdxType:"Figure"}),Object(i.b)("ul",null,Object(i.b)("li",{parentName:"ul"},"Vertical comparison = Top ~ Bottom"),Object(i.b)("li",{parentName:"ul"},"Horizontal comparison = Left > Right"),Object(i.b)("li",{parentName:"ul"},"Diagonal comparison = Backslash ~ Slash")),Object(i.b)(r.a,{text:"SmallDoubleLR symbol image",src:"symcode/SmallDoubleLR.png",mdxType:"Figure"}),Object(i.b)("ul",null,Object(i.b)("li",{parentName:"ul"},"Vertical comparison = Top ~ Bottom"),Object(i.b)("li",{parentName:"ul"},"Horizontal comparison = Left ~ Right"),Object(i.b)("li",{parentName:"ul"},"Diagonal comparison = Backslash > Slash")),Object(i.b)("p",null,"Each comparison has ",Object(i.b)("strong",{parentName:"p"},"3 possible outcomes (","<",", ",">",", ~)"),". For simplicity, we assign ",Object(i.b)("strong",{parentName:"p"},"2 bits")," to encode each comparison. Therefore, this naive implementation uses ",Object(i.b)("strong",{parentName:"p"},"3 * 2 = 6 bits")," to store each trace."),Object(i.b)("p",null,"The above worked well enough for sets of 16 symbols, but it was found inadequate for 32 symbols. Acute32 requires more comparisons for traces."),Object(i.b)("h2",{id:"trace-of-symcode"},"Trace of SymCode"),Object(i.b)("p",null,"The following figure is Acute32's alphabet set."),Object(i.b)(r.a,{text:"Acute32 alphabet set",src:"symcode/alphabet2.png",width:"480px",mdxType:"Figure"}),Object(i.b)("p",null,"Apart from the three basic comparisons explained in the previous section (",Object(i.b)("em",{parentName:"p"},"V,H,D"),"), we also compare every pair of the four quadrants (each quadrant is compared to every other quadrant exactly once), requiring ",Object(i.b)("strong",{parentName:"p"},"an extra of ",Object(i.b)("em",{parentName:"strong"},"4 choose 2")," = 6 comparisons")," (",Object(i.b)("em",{parentName:"p"},"ab, cd, ac"),", ...). The last comparison ",Object(i.b)("em",{parentName:"p"},"efgh")," is explained in the details below."),Object(i.b)(r.a,{text:"Acute32 trace counts",src:"symcode/trace_counts_balanced.png",width:"600px",mdxType:"Figure"}),Object(i.b)("details",null,Object(i.b)("h3",{id:"side-note-efgh"},"Side note: ",Object(i.b)("inlineCode",{parentName:"h3"},"efgh")),Object(i.b)("p",null,"It is important to note that not any arbitrary extra comparisons are effective. The rule of thumb is each extra comparison should introduce new information than the existing ones, making them ",Object(i.b)("strong",{parentName:"p"},"(at least partially) independent")," of each other. In general, comparisons that use ",Object(i.b)("strong",{parentName:"p"},"different numbers of blocks")," should be independent. For example, in the previous section all comparisons used 2 blocks vs 2 blocks, so the extra ones in this section, which use 1 block vs 1 block, are all (partially) independent of the previous ones. This is because as more blocks are being considered at once, the scope of analysis becomes irreversibly broader - just like how you cannot retrieve neither ",Object(i.b)("em",{parentName:"p"},"x")," nor ",Object(i.b)("em",{parentName:"p"},"y")," from the sum ",Object(i.b)("em",{parentName:"p"},"x+y"),"."),Object(i.b)("p",null,"Adding the extra ones results in a total of 3 + 6 = 9 comparisons. Using 2 bits to encode each, we are using 9 * 2 = ",Object(i.b)("strong",{parentName:"p"},"18 bits")," to store each trace in Acute32."),Object(i.b)("p",null,'Denote a comparison operation by "U vs V". The vertical, horizontal, and diagonal comparisons become "Top vs Bottom", "Left vs Right", and "Backslash vs Slash" respectively. The rest of the comparisons become "a vs b", "c vs d", and so on. We set the bits as follows:'),Object(i.b)("h3",{id:"for-each-comparison-u-vs-v"},"For each comparison U vs V"),Object(i.b)("ul",null,Object(i.b)("li",{parentName:"ul"},'11 means "U ',"~",' V"'),Object(i.b)("li",{parentName:"ul"},'10 means "U ',">",' V"'),Object(i.b)("li",{parentName:"ul"},'01 means "U ',"<",' V"'),Object(i.b)("li",{parentName:"ul"},"00 is invalid")),Object(i.b)("p",null,"The trace of all 1's, which happens when all four quadrants of the symbol share (approximately) the same number of dots, is shared by about one-third of the entire Acute32. This makes approximately one-third of the searches scan through 12 symbol images, which is 4 times that of most of the rest, which only scan through 3 symbol images."),Object(i.b)("p",null,"Our solution here is one more comparison."),Object(i.b)(r.a,{text:"efgh (top/bottom/left/right) grid",src:"symcode/efgh_grid.png",mdxType:"Figure"}),Object(i.b)("p",null,"We further partition the grid of each symbol image so that there are 4x4 = 16 small blocks, and denote the top/bottom/left/right blocks along the edge by ",Object(i.b)("em",{parentName:"p"},"e,f,g,h"),' respectively, as illustrated in the figure above. Next, we define an "',Object(i.b)("em",{parentName:"p"},"ef/gh"),' comparison" which compares ',Object(i.b)("em",{parentName:"p"},"e+f")," to ",Object(i.b)("em",{parentName:"p"},"g+h"),"."),Object(i.b)("p",null,"Now we have a more balanced distribution.")),Object(i.b)("h3",{id:"advantages-of-traces"},"Advantages of traces"),Object(i.b)("p",null,"The trace partitions the symbol set into smaller, mutually-exclusive subsets. This enhances robustness against noisy inputs."),Object(i.b)("p",null,"As searching through traces is much more efficient than comparing with each symbol individually, traces also help with performance."),Object(i.b)("p",null,"Most importantly, it provides us a guidance in designing the symbols to maximize differences within the symbol set."),Object(i.b)("h2",{id:"scanning-pipeline"},"Scanning Pipeline"),Object(i.b)("p",null,"The pipeline of SymCode scanner consists of 4 stages: ",Object(i.b)("inlineCode",{parentName:"p"},"finder"),", ",Object(i.b)("inlineCode",{parentName:"p"},"fitter"),", ",Object(i.b)("inlineCode",{parentName:"p"},"recognizer"),", ",Object(i.b)("inlineCode",{parentName:"p"},"decoder"),"."),Object(i.b)("p",null,"Each stage below begins with a more general description of the processing stage (general across any SymCode), followed by explanation of the implementation of Acute32."),Object(i.b)("h3",{id:"stage-1-locating-finder-candidates"},"Stage 1: Locating Finder Candidates"),Object(i.b)("p",null,"We first binarize the input color image using an adaptive thresholding strategy, shapes are then extracted from a binary image by clustering."),Object(i.b)("p",null,"The goal of this stage is to find the positions of all finder candidates in the frame."),Object(i.b)("p",null,"A minimum of 4 feature points is needed because at least ",Object(i.b)("strong",{parentName:"p"},"4 point correspondences")," are required to fit a ",Object(i.b)("strong",{parentName:"p"},"perspective transform"),"."),Object(i.b)("h3",{id:"acute32"},"Acute32"),Object(i.b)("p",null,"Acute32 uses circles as finders because they are distinct from the heavily cornered symbol set. The 4 finders are arranged in a Y shaped manner to break the rotational symmetry with itself."),Object(i.b)("p",null,"The advantage of using a circle is that, in general, it always transforms into an ellipse under any perspective distortion, making it relatively easy to detect."),Object(i.b)("p",null,"The disadvantage is the ",Object(i.b)("strong",{parentName:"p"},"lack of corners in circles"),". It is too easy to detect false positives because there are indeed many circles in real life."),Object(i.b)("h3",{id:"stage-2-fitting-perspective-transform"},"Stage 2: Fitting Perspective Transform"),Object(i.b)("p",null,'We define the "',Object(i.b)("strong",{parentName:"p"},"image space"),'" as the space of pixels on the ',Object(i.b)("strong",{parentName:"p"},"input frame"),', and the "',Object(i.b)("strong",{parentName:"p"},"object space"),'" as the space of pixels on the ',Object(i.b)("strong",{parentName:"p"},"code image")," (whose boundary is predefined). An image space is simply the input frame image. An object space can either be generated using a code instance, or by rectifying the corresponding image space."),Object(i.b)(r.a,{text:"Left: image space; Right: object space",src:"symcode/image_object_space_example.png",width:"800px",mdxType:"Figure"}),Object(i.b)("p",null,"In essence, this stage chooses the correct perspective transform to be used in the next stage."),Object(i.b)("p",null,"Each perspective transform ",Object(i.b)("strong",{parentName:"p"},"converts the image space into ",Object(i.b)("em",{parentName:"strong"},"an")," object space")," (but not necessarily the correct one) and ",Object(i.b)("strong",{parentName:"p"},"is defined by (at least) 4 point pairs")," (source and destination points), where each pair consists of ",Object(i.b)("strong",{parentName:"p"},"a point in the image space")," and ",Object(i.b)("strong",{parentName:"p"},"the other one in the object space"),"."),Object(i.b)("p",null,"Since we have obtained a list of finder candidates from the previous stage, we can extract ",Object(i.b)("strong",{parentName:"p"},Object(i.b)("em",{parentName:"strong"},"n")," feature points in the image space")," from them. Matching the 4-permutations of them to the ",Object(i.b)("strong",{parentName:"p"},"4 predefined feature points in the object space")," gives us at most ",Object(i.b)("em",{parentName:"p"},"n permute 4 = k")," perspective transforms."),Object(i.b)("p",null,"It is inefficient to accept each transformation and generate ",Object(i.b)("em",{parentName:"p"},"k")," object spaces to proceed to next stage. Therefore, we need to design a method to evaluate each transform and chooses which ones to proceed. The simplest way is to ",Object(i.b)("strong",{parentName:"p"},"define some extra feature points in the object space as ",Object(i.b)("em",{parentName:"strong"},"check points")),", which are ",Object(i.b)("strong",{parentName:"p"},"re-projected to the image space"),", and ",Object(i.b)("strong",{parentName:"p"},"check if the feature exists there"),", and reject the candidate otherwise."),Object(i.b)("h3",{id:"acute32-1"},"Acute32"),Object(i.b)("p",null,"The re-projection method mentioned above could not work for us because circles do not provide extra feature points for us to verify."),Object(i.b)("p",null,"Instead, we take each of the ",Object(i.b)("em",{parentName:"p"},"k")," perspective transforms and calculates an error value."),Object(i.b)("p",null,"We define 4 ",Object(i.b)("strong",{parentName:"p"},"object check points")," as the top of the 4 circle finders in the object space. Re-projecting these 4 points to the image space, we obtain the ",Object(i.b)("strong",{parentName:"p"},"image check points")," (",Object(i.b)("em",{parentName:"p"},"i1")," to ",Object(i.b)("em",{parentName:"p"},"i4"),"). Furthermore, we denote the ",Object(i.b)("strong",{parentName:"p"},"centres of the circle finders")," in the ",Object(i.b)("strong",{parentName:"p"},"image space"),", in the same order as the previously defined points, by ",Object(i.b)("em",{parentName:"p"},"c1")," to ",Object(i.b)("em",{parentName:"p"},"c4"),"."),Object(i.b)("p",null,"Our metric of evaluation relies on ",Object(i.b)("strong",{parentName:"p"},"magnitude and direction")," of the difference vector. 4 vectors interest us here: the vectors from the centres of finders to the image check points, and we denote them by ",Object(i.b)("em",{parentName:"p"},"v1")," to ",Object(i.b)("em",{parentName:"p"},"v4")," respectively."),Object(i.b)(r.a,{text:"Transform evaluation points and vectors: the 4 green vectors are v1 to v4",src:"symcode/transform_evaluation.png",mdxType:"Figure"}),Object(i.b)("p",null,"We observed that given a correct transform, the 4 green vectors are consistently pointing in the same direction, with their magnitudes less uniform but still very similar. We found it effective to choose the transform with the ",Object(i.b)("strong",{parentName:"p"},"minimum variance")," among ",Object(i.b)("em",{parentName:"p"},"v1")," to ",Object(i.b)("em",{parentName:"p"},"v4"),"."),Object(i.b)("h3",{id:"stage-3-recognizing-symbols"},"Stage 3: Recognizing Symbols"),Object(i.b)("p",null,"In the previous stage, we have obtained a perspective transform which converts between the image and object spaces. Next, we're going to rectify the input image into a code image, and recognize the symbols on it."),Object(i.b)("h4",{id:"rectify-the-image-space"},"Rectify the image space"),Object(i.b)("p",null,"Once we have a transform that we believe is correct, the object space can be obtained by applying it on the image space. We sample the image with bilinear interpolation."),Object(i.b)("h4",{id:"recognition"},"Recognition"),Object(i.b)(r.a,{text:"Rectified image (code image in the correct object space)",src:"symcode/object_space_example.png",mdxType:"Figure"}),Object(i.b)("p",null,"Assuming the transform is correct, the coordinates of the symbols on the code image should be close to the ",Object(i.b)("em",{parentName:"p"},"anchors")," we defined in the object space."),Object(i.b)(r.a,{text:"Recognition process marked by bounding boxes",src:"symcode/recognition_demo.png",mdxType:"Figure"}),Object(i.b)("p",null,"The bounding boxes in blue are all clusters found on the code image. The boxes in red are the grouped clusters used to recognize each symbol."),Object(i.b)("p",null,"Once we have the images, we can ",Object(i.b)("strong",{parentName:"p"},"evaluate their traces")," and compare them with the ones in our symbol library, obtaining ",Object(i.b)("strong",{parentName:"p"},"a small number of candidates")," for each symbol image. Each of these candidates is compared to the symbol image and the one with the ",Object(i.b)("strong",{parentName:"p"},"lowest delta")," is the final recognized symbol."),Object(i.b)("p",null,"Each symbol is mapped to a unique bit string, and they are concatenated into a longer bit string."),Object(i.b)("h3",{id:"stage-4-decoding-the-symcode"},"Stage 4: Decoding the SymCode"),Object(i.b)("p",null,"This stage performs error detection (possibly correction) and extract the payload."),Object(i.b)("h3",{id:"acute32-2"},"Acute32"),Object(i.b)("p",null,"As there are 32 symbols in the set of Acute32, each symbol can encode 5 bits. There are 5 data symbols and 4 finders in the 3x3 configuration. It can easily be extended to a 6x4 to carry more data. For now, each ",Object(i.b)("inlineCode",{parentName:"p"},"SymCode")," instance encodes ",Object(i.b)("em",{parentName:"p"},"25")," raw bits, where ",Object(i.b)("em",{parentName:"p"},"20")," bits are payload and ",Object(i.b)("em",{parentName:"p"},"5")," bits are CRC checksum."),Object(i.b)("h2",{id:"conclusion"},"Conclusion"),Object(i.b)("p",null,"We developed a theoretic as well as a programming framework for designing and implementing symbolic barcodes. The programming library is available as ",Object(i.b)("a",{href:"https://github.com/visioncortex/symcode",parentName:"p"},Object(i.b)("inlineCode",{parentName:"a"},"symcode")),". For those who simply want to integrate SymCode into your application, please refer to the wasm distribution ",Object(i.b)("a",{href:"https://github.com/visioncortex/acute32",parentName:"p"},Object(i.b)("inlineCode",{parentName:"a"},"acute32"))," which is ready to use in any browser based Javascript projects."),Object(i.b)("h2",{id:"about-vision-cortex"},"About Vision Cortex"),Object(i.b)("p",null,"This story is a manifest of the philosophy behind Vision Cortex, and a glimpse of the ongoing research and development by the Vision Cortex Research Group."),Object(i.b)("p",null,"If you enjoyed the reading and appreciate our work, consider starring (on ",Object(i.b)("a",{href:"https://github.com/visioncortex/visioncortex",parentName:"p"},"GitHub"),"), discuss (on ",Object(i.b)("a",{href:"https://www.reddit.com/r/rust/comments/mjpp84/a_symbolic_2d_barcode_in_rust/",parentName:"p"},"Reddit")," / ",Object(i.b)("a",{href:"https://github.com/visioncortex/visioncortex/discussions",parentName:"p"},"GitHub"),"), share (on ",Object(i.b)("a",{href:"https://twitter.com/visioncortex",parentName:"p"},"Twitter"),") and collaborate."))}void 0!==l&&l&&l===Object(l)&&Object.isExtensible(l)&&!l.hasOwnProperty("__filemeta")&&Object.defineProperty(l,"__filemeta",{configurable:!0,value:{name:"MDXContent",filename:"src/symcode.mdx"}}),l.isMDXComponent=!0}}]);
+//# sourceMappingURL=component---src-symcode-mdx-4c86476283db67665ee4.js.map
\ No newline at end of file
diff --git a/docs/component---src-symcode-mdx-4c86476283db67665ee4.js.map b/docs/component---src-symcode-mdx-4c86476283db67665ee4.js.map
new file mode 100644
index 0000000..c1f36d8
--- /dev/null
+++ b/docs/component---src-symcode-mdx-4c86476283db67665ee4.js.map
@@ -0,0 +1 @@
+{"version":3,"sources":["webpack:///../src/symcode.mdx"],"names":["_frontmatter","MDXLayout","DefaultLayout","MDXContent","components","props","mdxType","href","target","src","style","frameBorder","allow","allowFullScreen","text","height","width","isMDXComponent"],"mappings":"mQAOaA,G,UAAe,S,iMAE5B,IAKMC,EAAYC,IACH,SAASC,EAAT,GAGZ,IAFDC,EAEC,EAFDA,WACGC,EACF,8BACD,OAAO,YAACJ,EAAD,eAAeI,EAAf,CAAsBD,WAAYA,EAAYE,QAAQ,cAC3D,iBAAQ,CACN,GAAM,wDADR,yDAGA,qCACE,gBAAO,CACL,KAAQ,sBACR,WAAc,KAFhB,eADF,QAME,gBAAO,CACL,KAAQ,wBACR,WAAc,KAFhB,eANF,4BAWA,uBACA,kIAEE,iBAAGC,KAAK,oCAAoCC,OAAO,UAAnD,4BAFF,KAIA,sBAAQC,IAAI,mDAAmDC,MAAO,CACpE,MAAS,OACT,OAAU,OACV,UAAa,QACZC,YAAY,IAAIC,MAAM,uBAAuBC,iBAAe,IAC/D,iBAAQ,CACN,GAAM,aADR,aAGA,sUACA,YAAC,IAAD,CAAQC,KAAK,kCAAkCL,IAAI,uCAAuCH,QAAQ,WAClG,8BAEE,gBAAO,CACL,WAAc,cADhB,+BAGE,gBAAO,CACL,KAAQ,mFACR,WAAc,KAFhB,uBAMJ,mJACE,gBAAO,CACL,KAAQ,gBACR,WAAc,KAFhB,uBADF,qDAMA,iBAAQ,CACN,GAAM,cADR,cAGA,kbACA,yEACA,sBAEE,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,0LAOF,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,0IAOF,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,6JAOF,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,qIAOJ,0RACA,iBAAQ,CACN,GAAM,yBADR,yBAGA,8LACA,2SACA,kbACA,4GACA,iBAAQ,CACN,GAAM,qBADR,qBAGA,iDACA,sBAEE,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,cAKA,gBAAO,CACL,WAAc,MADhB,iLAOF,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,cAKA,gBAAO,CACL,WAAc,MADhB,mMAOF,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,eAKA,gBAAO,CACL,WAAc,MADhB,uJAOF,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,cAKA,gBAAO,CACL,WAAc,MADhB,gLAGE,gBAAO,CACL,KAAQ,mDACR,WAAc,KAFhB,sBAHF,oCAYJ,iYACA,2GACA,iBAAQ,CACN,GAAM,mBADR,mBAGA,qZACA,2MACA,iBAAQ,CACN,GAAM,wBADR,wBAGA,sHACA,YAAC,IAAD,CAAQQ,KAAK,oBAAoBL,IAAI,+BAA+BM,OAAO,QAAQT,QAAQ,WAC3F,qGACA,uBAAK,mBAAU,CACX,WAAc,OADb,wEAKL,8GACA,uBAAK,mBAAU,CACX,WAAc,OADb,sGAML,4GACA,uBAAK,mBAAU,CACX,WAAc,OADb,6FAML,6DACA,uBAAK,mBAAU,CACX,WAAc,OADb,qNA6BL,wCACA,uBAAK,mBAAU,CACX,WAAc,OADb,wIAgBL,oRACA,iBAAQ,CACN,GAAM,WADR,WAGA,iLACA,sLACE,yBAAgB,CACd,WAAc,KADhB,KADF,IAKE,yBAAgB,CACd,WAAc,KADhB,KALF,IASE,yBAAgB,CACd,WAAc,KADhB,KATF,IAaE,yBAAgB,CACd,WAAc,KADhB,KAbF,gEAiBA,YAAC,IAAD,CAAQQ,KAAK,iBAAiBL,IAAI,wBAAwBM,OAAO,QAAQT,QAAQ,WACjF,sBAEE,iBAAQ,CACN,WAAc,MADhB,yBAGE,yBAAgB,CACd,WAAc,MADhB,SAHF,OAOE,yBAAgB,CACd,WAAc,MADhB,UAKF,iBAAQ,CACN,WAAc,MADhB,2BAGE,yBAAgB,CACd,WAAc,MADhB,SAHF,OAOE,yBAAgB,CACd,WAAc,MADhB,UAKF,iBAAQ,CACN,WAAc,MADhB,yBAGE,yBAAgB,CACd,WAAc,MADhB,SAHF,OAOE,yBAAgB,CACd,WAAc,MADhB,WAKJ,gDACA,YAAC,IAAD,CAAQQ,KAAK,sBAAsBL,IAAI,qBAAqBH,QAAQ,WACpE,sBAEE,iBAAQ,CACN,WAAc,MADhB,sCAKA,iBAAQ,CACN,WAAc,MADhB,wCAKA,iBAAQ,CACN,WAAc,MADhB,4CAKF,YAAC,IAAD,CAAQQ,KAAK,6BAA6BL,IAAI,4BAA4BH,QAAQ,WAClF,sBAEE,iBAAQ,CACN,WAAc,MADhB,sCAKA,iBAAQ,CACN,WAAc,MADhB,wCAKA,iBAAQ,CACN,WAAc,MADhB,4CAKF,4CACE,qBAAY,CACV,WAAc,KADhB,6CADF,+BASE,qBAAY,CACV,WAAc,KADhB,UATF,yEAaE,qBAAY,CACV,WAAc,KADhB,kBAbF,yBAiBA,uKACA,iBAAQ,CACN,GAAM,oBADR,oBAGA,wEACA,YAAC,IAAD,CAAQQ,KAAK,uBAAuBL,IAAI,wBAAwBO,MAAM,QAAQV,QAAQ,WACtF,kGACE,iBAAQ,CACN,WAAc,KADhB,SADF,mIAKE,qBAAY,CACV,WAAc,KADhB,eAGE,iBAAQ,CACN,WAAc,UADhB,cAHF,oBALF,KAaE,iBAAQ,CACN,WAAc,KADhB,cAbF,+BAiBE,iBAAQ,CACN,WAAc,KADhB,QAjBF,uCAqBA,YAAC,IAAD,CAAQQ,KAAK,uBAAuBL,IAAI,oCAAoCO,MAAM,QAAQV,QAAQ,WAClG,2BACE,iBAAQ,CACN,GAAM,kBADR,cAGE,yBAAgB,CACd,WAAc,MADhB,SAGF,wNACE,qBAAY,CACV,WAAc,KADhB,oCADF,oDAKE,qBAAY,CACV,WAAc,KADhB,+BALF,8XASE,iBAAQ,CACN,WAAc,KADhB,KATF,QAaE,iBAAQ,CACN,WAAc,KADhB,KAbF,iBAiBE,iBAAQ,CACN,WAAc,KADhB,OAjBF,KAqBA,6IACE,qBAAY,CACV,WAAc,KADhB,WADF,oCAKA,oSACA,iBAAQ,CACN,GAAM,8BADR,8BAGA,sBAEE,iBAAQ,CACN,WAAc,MADhB,0BAOA,iBAAQ,CACN,WAAc,MADhB,0BAOA,iBAAQ,CACN,WAAc,MADhB,0BAOA,iBAAQ,CACN,WAAc,MADhB,kBAKF,2WACA,kEACA,YAAC,IAAD,CAAQQ,KAAK,oCAAoCL,IAAI,wBAAwBH,QAAQ,WACrF,kLACE,iBAAQ,CACN,WAAc,KADhB,WADF,0EAKE,iBAAQ,CACN,WAAc,KADhB,SALF,+BASE,iBAAQ,CACN,WAAc,KADhB,OATF,OAaE,iBAAQ,CACN,WAAc,KADhB,OAbF,KAiBA,mEAEF,iBAAQ,CACN,GAAM,wBADR,wBAGA,qJACA,4JACA,4IACA,iBAAQ,CACN,GAAM,qBADR,qBAGA,8EACE,yBAAgB,CACd,WAAc,KADhB,UADF,KAKE,yBAAgB,CACd,WAAc,KADhB,UALF,KASE,yBAAgB,CACd,WAAc,KADhB,cATF,KAaE,yBAAgB,CACd,WAAc,KADhB,WAbF,KAiBA,gMACA,iBAAQ,CACN,GAAM,sCADR,uCAGA,sKACA,+GACA,iFACE,qBAAY,CACV,WAAc,KADhB,2BADF,0BAKE,qBAAY,CACV,WAAc,KADhB,yBALF,KASA,iBAAQ,CACN,GAAM,WADR,WAGA,sNACA,yLACA,gDACE,qBAAY,CACV,WAAc,KADhB,8BADF,kGAKA,iBAAQ,CACN,GAAM,yCADR,0CAGA,uCACE,qBAAY,CACV,WAAc,KADhB,eADF,mCAKE,qBAAY,CACV,WAAc,KADhB,eALF,cASE,qBAAY,CACV,WAAc,KADhB,gBATF,mCAaE,qBAAY,CACV,WAAc,KADhB,cAbF,mMAiBA,YAAC,IAAD,CAAQQ,KAAK,yCAAyCL,IAAI,yCAAyCO,MAAM,QAAQV,QAAQ,WACzH,uHACA,mDACE,qBAAY,CACV,WAAc,KADhB,iCAGE,iBAAQ,CACN,WAAc,UADhB,MAHF,iBADF,8CASE,qBAAY,CACV,WAAc,KADhB,0CATF,iEAaE,qBAAY,CACV,WAAc,KADhB,8BAbF,QAiBE,qBAAY,CACV,WAAc,KADhB,qCAjBF,KAqBA,mHACE,qBAAY,CACV,WAAc,KACb,iBAAQ,CACP,WAAc,UADf,KAFH,sCADF,0DAQE,qBAAY,CACV,WAAc,KADhB,mDARF,qBAYE,iBAAQ,CACN,WAAc,KADhB,mBAZF,4BAgBA,qFACE,iBAAQ,CACN,WAAc,KADhB,KADF,uKAKE,qBAAY,CACV,WAAc,KADhB,2DAGE,iBAAQ,CACN,WAAc,UADhB,iBARJ,eAYE,qBAAY,CACV,WAAc,KADhB,mCAZF,SAgBE,qBAAY,CACV,WAAc,KADhB,qCAhBF,yCAoBA,iBAAQ,CACN,GAAM,aADR,WAGA,6JACA,qDACE,iBAAQ,CACN,WAAc,KADhB,KADF,0DAKA,oCACE,qBAAY,CACV,WAAc,KADhB,uBADF,2HAKE,qBAAY,CACV,WAAc,KADhB,sBALF,KASE,iBAAQ,CACN,WAAc,KADhB,MATF,OAaE,iBAAQ,CACN,WAAc,KADhB,MAbF,iCAiBE,qBAAY,CACV,WAAc,KADhB,iCAjBF,WAqBE,qBAAY,CACV,WAAc,KADhB,eArBF,4DAyBE,iBAAQ,CACN,WAAc,KADhB,MAzBF,OA6BE,iBAAQ,CACN,WAAc,KADhB,MA7BF,KAiCA,2DACE,qBAAY,CACV,WAAc,KADhB,2BADF,mJAKE,iBAAQ,CACN,WAAc,KADhB,MALF,OASE,iBAAQ,CACN,WAAc,KADhB,MATF,kBAaA,YAAC,IAAD,CAAQQ,KAAK,4EAA4EL,IAAI,mCAAmCH,QAAQ,WACxI,2PACE,qBAAY,CACV,WAAc,KADhB,oBADF,UAKE,iBAAQ,CACN,WAAc,KADhB,MALF,OASE,iBAAQ,CACN,WAAc,KADhB,MATF,KAaA,iBAAQ,CACN,GAAM,+BADR,gCAGA,6OACA,iBAAQ,CACN,GAAM,2BADR,2BAGA,kMACA,iBAAQ,CACN,GAAM,eADR,eAGA,YAAC,IAAD,CAAQQ,KAAK,2DAA2DL,IAAI,mCAAmCH,QAAQ,WACvH,mIACE,iBAAQ,CACN,WAAc,KADhB,WADF,oCAKA,YAAC,IAAD,CAAQQ,KAAK,+CAA+CL,IAAI,+BAA+BH,QAAQ,WACvG,sKACA,wDACE,qBAAY,CACV,WAAc,KADhB,yBADF,oEAKE,qBAAY,CACV,WAAc,KADhB,gCALF,yGASE,qBAAY,CACV,WAAc,KADhB,gBATF,oCAaA,0HACA,iBAAQ,CACN,GAAM,gCADR,iCAGA,2GACA,iBAAQ,CACN,GAAM,aADR,WAGA,4OACE,yBAAgB,CACd,WAAc,KADhB,WADF,qBAKE,iBAAQ,CACN,WAAc,KADhB,MALF,oBASE,iBAAQ,CACN,WAAc,KADhB,MATF,yBAaE,iBAAQ,CACN,WAAc,KADhB,KAbF,2BAiBA,iBAAQ,CACN,GAAM,cADR,cAGA,8KACE,gBAAO,CACL,KAAQ,0CACR,WAAc,KACb,yBAAgB,CACf,WAAc,KADf,YAJL,iHAQE,gBAAO,CACL,KAAQ,0CACR,WAAc,KACb,yBAAgB,CACf,WAAc,KADf,YAXL,oEAeA,iBAAQ,CACN,GAAM,uBADR,uBAGA,oLACA,kGACE,gBAAO,CACL,KAAQ,+CACR,WAAc,KAFhB,UADF,kBAME,gBAAO,CACL,KAAQ,+EACR,WAAc,KAFhB,UANF,MAWE,gBAAO,CACL,KAAQ,2DACR,WAAc,KAFhB,UAXF,gBAgBE,gBAAO,CACL,KAAQ,mCACR,WAAc,KAFhB,WAhBF,4B,+LAyBJH,EAAWc,gBAAiB","file":"component---src-symcode-mdx-4c86476283db67665ee4.js","sourcesContent":["import * as React from 'react'\n /* @jsx mdx */\nimport { mdx } from '@mdx-js/react';\n/* @jsx mdx */\n\nimport DefaultLayout from \"/home/chris/visioncortex.github.io/node_modules/gatsby-theme-docz/src/base/Layout.js\";\nimport { Figure } from './components/Figure';\nexport const _frontmatter = {};\n\nconst makeShortcode = name => function MDXDefaultShortcode(props) {\n console.warn(\"Component \" + name + \" was not imported, exported, or provided by MDXProvider as global scope\");\n return ;\n};\n\nconst MDXLayout = DefaultLayout;\nexport default function MDXContent({\n components,\n ...props\n}) {\n return \n
{`SymCode: The Symbolic Barcode for Humans and Machines`}
{`This is our story of the design and implementation of a symbolic barcode system.\nTry the Demo Web App on `}\n {`symcode.visioncortex.org`}\n {`.`}
\n \n
{`Forewords`}
\n
{`If you are lucky enough to live in a place where an Amazon Go store is nearby, you definitely should visit it. It is packed full of esoteric technology that enthusiast like us get super intrigued. One item that caught our eyes was the labels on the salad and sandwiches in the ready-made food section.`}
{`There printed a matrix of cute little circles and squares! It must be a barcode of some sort. Our attempt to decipher the '`}\n {`Amazon Reversi Code`}\n {`' was futile, but this is the start of the story.`}
\n
{`Background`}
\n
{`A bit of context for the unfamiliar reader, the name Barcode is referring to the labels we still find in virtually every packaged products today. Barcode encode data by varying the widths and spacings of the vertical bars, which are to be scanned by a laser beam crossing horizontally. With the advent of image sensors, 2D barcodes were developed, where QR code and Aztec Code are some of the most well-known.`}
\n
{`In general, 2D barcode scanner works as follows:`}
\n \n\n
\n\n
{`Locate the finders. These finders act as a trigger to tell the scanner that 'here is a barcode!'. The finders have to be easily detectable yet not interfering with the data modules.`}
\n\n
\n\n\n
\n\n
{`Attempt to scan and read metadata surrounding the finder candidates. The metadata usually includes a format code and timing patterns.`}
\n\n
\n\n\n
\n\n
{`If the previous step succeeds, the scanner would sample the entire barcode to construct a raw bit string. A checksum operation is done to detect errors.`}
\n\n
\n\n\n
\n\n
{`If the previous step succeeds, the payload is extract from the raw bits, which usually involves some bit masking and shuffling.`}
\n\n
\n\n \n
{`This framework has aged well and they are quite robust and low-cost to implement in embedded systems. However, they are meant to be read by machines and not by our naked eyes. They are not designed for aesthetics and are impossible for humans to comprehend.`}
\n
{`Goals and Constraints`}
\n
{`Our goal is to design a barcode system that are both human-readable and machine-readable. We now turn our attention to human readability by studying human languages.`}
\n
{`Machines run on bits, so we can say the alphabet consists of 0 and 1. In human languages, we have a larger set of alphabets. In English, we have 26 distinct lowercase letters. In Korean, characters are constructed by composing 2 to 6 elements from a set of 40 distinct Jamo.`}
\n
{`There is a direct tradeoff between information density and visual ambiguity. If the symbol set is too large, humans would have difficulty in remembering all of them. In addition, there may be some visually similar symbols that are hard to disambiguate. If the symbol set is too small, the same message has to be encoded with more symbols, which again, humans often have a hard time in processing long strings.`}
\n
{`We determined that a symbol set in the range of 16 to 64 symbols is a good balance.`}
\n
{`Design of symbols`}
\n
{`What makes good symbols?`}
\n \n\n
\n\n
{`Prominence`}
\n\n\n
{`First, the symbols have to stand out from the natural world, to manifest that they are created deliberately to convey a message but not a result of some natural phenomenon.`}
\n\n
\n\n\n
\n\n
{`Repeatable`}
\n\n\n
{`Symbols are constructed with specific tools and processes that can be taught to other people. The meaning of a symbol should remain the same when reproduced, in which variation is tolerated.`}
\n\n
\n\n\n
\n\n
{`Distinctive`}
\n\n\n
{`Symbols within a set should not be similar with each other and have distinctive features allowing the reader to resolve ambiguity between symbols.`}
\n\n
\n\n\n
\n\n
{`Aesthetics`}
\n\n\n
{`Finally, good symbols should observe the aesthetics rules of the human eye, including being anti-symmetric, rotational symmetric, balanced and consistent. As a pointer, the `}\n {`Gestalt Principles`}\n {` are fantastic rules of thumb.`}
\n\n
\n\n \n
{`With the above rules in mind, we designed a minimalistic symbol set. Each symbol is composed of multiple triangles, the basic geometric primitive. Each symbol is symmetric or anti-symmetric in overall, but exhibits asymmetry internally. They are like Tangram, in which a human child can easily reproduce the symbols by assembly some triangular pieces together.`}
\n
{`The next section would quantitatively analyze and justify this design methodology.`}
\n
{`Trace of shapes`}
\n
{`The naive way to match a shape against a symbol set is to perform a linear search, XOR it with every symbol and then chooses the one with the lowest delta. It works in principle, but is inefficient. Ideally, an algorithm can condense a complex shape into a feature vector, which we can lookup in the feature space of the symbol set for the nearest neighbour and arrive to a match.`}
\n
{`Instead of using an array of real numbers, we devised that an array of bits are sufficient to capture the essence of symbols, and from now on we refer this bit string as 'trace'.`}
\n
{`The English alphabet`}
\n
{`Now, let us take a closer look at the lowercase English alphabet set to illustrate this idea.`}
\n \n
{`First off, we can classify the 26 alphabets as either tall or short, giving:`}
\n
{`Tall: b d f g h j k l p q t y\nShort: a c e i m n o r s u v w x z\n`}
\n
{`Next, we can divide the letter into two halfs horizontally and compare their weights:`}
\n
{`Left > right: b c f h k p r t y\nLeft < right: a d g q\nLeft ~ right: e i j l m n o s u v w x z\n`}
\n
{`Then, we can divide the letter into two halfs vertically and compare their weights:`}
\n
{`Up > down: f m n p q r y\nUp < down: b d h k u\nUp ~ down: a c e g i j l o s t v w x z\n`}
{`SR=: a\nTLD: b h k\nSL=: c\nTRD: d\nS==: e i o s v w x z\nTL=: t\nTR=: g\nT==: j l \nS=U: m n\nTLU: f p y\nTRU: q\nSLU: r\nS=D: u\n`}
\n
{`Which is a surprisingly good classifier using only three comparisons. We can do more trinary comparisons on smaller partitions to further differentiate the collisions, but our investigation on English alphabets ends here for the scope of this article.`}
\n
{`SymCode`}
\n
{`Our trace for SymCode symbols follows a similar scheme. We define the symbol traces starting from the three comparisons defined in the previous section.`}
\n
{`The pixel count in each of the four quadrants of a symbol (in order of top-left, top-right, bottom-left, bottom-right) are denoted by non-negative quantities `}\n {`a`}\n {`,`}\n {`b`}\n {`,`}\n {`c`}\n {`,`}\n {`d`}\n {` respectively, the three comparisons are defined as follows:`}
\n \n
\n\n
{`Vertical comparison = `}\n {`(a+b)`}\n {` vs `}\n {`(c+d)`}
\n\n\n
{`Horizontal comparison = `}\n {`(a+c)`}\n {` vs `}\n {`(b+d)`}
\n\n\n
{`Diagonal comparison = `}\n {`(a+d)`}\n {` vs `}\n {`(b+c)`}
\n\n
\n
{`Below are two examples:`}
\n \n
\n\n
{`Vertical comparison = Top ~ Bottom`}
\n\n\n
{`Horizontal comparison = Left > Right`}
\n\n\n
{`Diagonal comparison = Backslash ~ Slash`}
\n\n
\n \n
\n\n
{`Vertical comparison = Top ~ Bottom`}
\n\n\n
{`Horizontal comparison = Left ~ Right`}
\n\n\n
{`Diagonal comparison = Backslash > Slash`}
\n\n
\n
{`Each comparison has `}\n {`3 possible outcomes (`}\n {`<`}\n {`, `}\n {`>`}\n {`, ~)`}\n {`. For simplicity, we assign `}\n {`2 bits`}\n {` to encode each comparison. Therefore, this naive implementation uses `}\n {`3 * 2 = 6 bits`}\n {` to store each trace.`}
\n
{`The above worked well enough for sets of 16 symbols, but it was found inadequate for 32 symbols. Acute32 requires more comparisons for traces.`}
\n
{`Trace of SymCode`}
\n
{`The following figure is Acute32's alphabet set.`}
\n \n
{`Apart from the three basic comparisons explained in the previous section (`}\n {`V,H,D`}\n {`), we also compare every pair of the four quadrants (each quadrant is compared to every other quadrant exactly once), requiring `}\n {`an extra of `}\n {`4 choose 2`}\n {` = 6 comparisons`}\n {` (`}\n {`ab, cd, ac`}\n {`, ...). The last comparison `}\n {`efgh`}\n {` is explained in the details below.`}
\n \n \n
{`Side note: `}\n {`efgh`}
\n
{`It is important to note that not any arbitrary extra comparisons are effective. The rule of thumb is each extra comparison should introduce new information than the existing ones, making them `}\n {`(at least partially) independent`}\n {` of each other. In general, comparisons that use `}\n {`different numbers of blocks`}\n {` should be independent. For example, in the previous section all comparisons used 2 blocks vs 2 blocks, so the extra ones in this section, which use 1 block vs 1 block, are all (partially) independent of the previous ones. This is because as more blocks are being considered at once, the scope of analysis becomes irreversibly broader - just like how you cannot retrieve neither `}\n {`x`}\n {` nor `}\n {`y`}\n {` from the sum `}\n {`x+y`}\n {`.`}
\n
{`Adding the extra ones results in a total of 3 + 6 = 9 comparisons. Using 2 bits to encode each, we are using 9 * 2 = `}\n {`18 bits`}\n {` to store each trace in Acute32.`}
\n
{`Denote a comparison operation by \"U vs V\". The vertical, horizontal, and diagonal comparisons become \"Top vs Bottom\", \"Left vs Right\", and \"Backslash vs Slash\" respectively. The rest of the comparisons become \"a vs b\", \"c vs d\", and so on. We set the bits as follows:`}
\n
{`For each comparison U vs V`}
\n
\n\n
{`11 means \"U `}\n {`~`}\n {` V\"`}
\n\n\n
{`10 means \"U `}\n {`>`}\n {` V\"`}
\n\n\n
{`01 means \"U `}\n {`<`}\n {` V\"`}
\n\n\n
{`00 is invalid`}
\n\n
\n
{`The trace of all 1's, which happens when all four quadrants of the symbol share (approximately) the same number of dots, is shared by about one-third of the entire Acute32. This makes approximately one-third of the searches scan through 12 symbol images, which is 4 times that of most of the rest, which only scan through 3 symbol images.`}
\n
{`Our solution here is one more comparison.`}
\n \n
{`We further partition the grid of each symbol image so that there are 4x4 = 16 small blocks, and denote the top/bottom/left/right blocks along the edge by `}\n {`e,f,g,h`}\n {` respectively, as illustrated in the figure above. Next, we define an \"`}\n {`ef/gh`}\n {` comparison\" which compares `}\n {`e+f`}\n {` to `}\n {`g+h`}\n {`.`}
\n
{`Now we have a more balanced distribution.`}
\n \n
{`Advantages of traces`}
\n
{`The trace partitions the symbol set into smaller, mutually-exclusive subsets. This enhances robustness against noisy inputs.`}
\n
{`As searching through traces is much more efficient than comparing with each symbol individually, traces also help with performance.`}
\n
{`Most importantly, it provides us a guidance in designing the symbols to maximize differences within the symbol set.`}
{`Each stage below begins with a more general description of the processing stage (general across any SymCode), followed by explanation of the implementation of Acute32.`}
\n
{`Stage 1: Locating Finder Candidates`}
\n
{`We first binarize the input color image using an adaptive thresholding strategy, shapes are then extracted from a binary image by clustering.`}
\n
{`The goal of this stage is to find the positions of all finder candidates in the frame.`}
\n
{`A minimum of 4 feature points is needed because at least `}\n {`4 point correspondences`}\n {` are required to fit a `}\n {`perspective transform`}\n {`.`}
\n
{`Acute32`}
\n
{`Acute32 uses circles as finders because they are distinct from the heavily cornered symbol set. The 4 finders are arranged in a Y shaped manner to break the rotational symmetry with itself.`}
\n
{`The advantage of using a circle is that, in general, it always transforms into an ellipse under any perspective distortion, making it relatively easy to detect.`}
\n
{`The disadvantage is the `}\n {`lack of corners in circles`}\n {`. It is too easy to detect false positives because there are indeed many circles in real life.`}
\n
{`Stage 2: Fitting Perspective Transform`}
\n
{`We define the \"`}\n {`image space`}\n {`\" as the space of pixels on the `}\n {`input frame`}\n {`, and the \"`}\n {`object space`}\n {`\" as the space of pixels on the `}\n {`code image`}\n {` (whose boundary is predefined). An image space is simply the input frame image. An object space can either be generated using a code instance, or by rectifying the corresponding image space.`}
\n \n
{`In essence, this stage chooses the correct perspective transform to be used in the next stage.`}
\n
{`Each perspective transform `}\n {`converts the image space into `}\n {`an`}\n {` object space`}\n {` (but not necessarily the correct one) and `}\n {`is defined by (at least) 4 point pairs`}\n {` (source and destination points), where each pair consists of `}\n {`a point in the image space`}\n {` and `}\n {`the other one in the object space`}\n {`.`}
\n
{`Since we have obtained a list of finder candidates from the previous stage, we can extract `}\n {`n`}\n {` feature points in the image space`}\n {` from them. Matching the 4-permutations of them to the `}\n {`4 predefined feature points in the object space`}\n {` gives us at most `}\n {`n permute 4 = k`}\n {` perspective transforms.`}
\n
{`It is inefficient to accept each transformation and generate `}\n {`k`}\n {` object spaces to proceed to next stage. Therefore, we need to design a method to evaluate each transform and chooses which ones to proceed. The simplest way is to `}\n {`define some extra feature points in the object space as `}\n {`check points`}\n {`, which are `}\n {`re-projected to the image space`}\n {`, and `}\n {`check if the feature exists there`}\n {`, and reject the candidate otherwise.`}
\n
{`Acute32`}
\n
{`The re-projection method mentioned above could not work for us because circles do not provide extra feature points for us to verify.`}
\n
{`Instead, we take each of the `}\n {`k`}\n {` perspective transforms and calculates an error value.`}
\n
{`We define 4 `}\n {`object check points`}\n {` as the top of the 4 circle finders in the object space. Re-projecting these 4 points to the image space, we obtain the `}\n {`image check points`}\n {` (`}\n {`i1`}\n {` to `}\n {`i4`}\n {`). Furthermore, we denote the `}\n {`centres of the circle finders`}\n {` in the `}\n {`image space`}\n {`, in the same order as the previously defined points, by `}\n {`c1`}\n {` to `}\n {`c4`}\n {`.`}
\n
{`Our metric of evaluation relies on `}\n {`magnitude and direction`}\n {` of the difference vector. 4 vectors interest us here: the vectors from the centres of finders to the image check points, and we denote them by `}\n {`v1`}\n {` to `}\n {`v4`}\n {` respectively.`}
\n \n
{`We observed that given a correct transform, the 4 green vectors are consistently pointing in the same direction, with their magnitudes less uniform but still very similar. We found it effective to choose the transform with the `}\n {`minimum variance`}\n {` among `}\n {`v1`}\n {` to `}\n {`v4`}\n {`.`}
\n
{`Stage 3: Recognizing Symbols`}
\n
{`In the previous stage, we have obtained a perspective transform which converts between the image and object spaces. Next, we're going to rectify the input image into a code image, and recognize the symbols on it.`}
\n
{`Rectify the image space`}
\n
{`Once we have a transform that we believe is correct, the object space can be obtained by applying it on the image space. We sample the image with bilinear interpolation.`}
\n
{`Recognition`}
\n \n
{`Assuming the transform is correct, the coordinates of the symbols on the code image should be close to the `}\n {`anchors`}\n {` we defined in the object space.`}
\n \n
{`The bounding boxes in blue are all clusters found on the code image. The boxes in red are the grouped clusters used to recognize each symbol.`}
\n
{`Once we have the images, we can `}\n {`evaluate their traces`}\n {` and compare them with the ones in our symbol library, obtaining `}\n {`a small number of candidates`}\n {` for each symbol image. Each of these candidates is compared to the symbol image and the one with the `}\n {`lowest delta`}\n {` is the final recognized symbol.`}
\n
{`Each symbol is mapped to a unique bit string, and they are concatenated into a longer bit string.`}
\n
{`Stage 4: Decoding the SymCode`}
\n
{`This stage performs error detection (possibly correction) and extract the payload.`}
\n
{`Acute32`}
\n
{`As there are 32 symbols in the set of Acute32, each symbol can encode 5 bits. There are 5 data symbols and 4 finders in the 3x3 configuration. It can easily be extended to a 6x4 to carry more data. For now, each `}\n {`SymCode`}\n {` instance encodes `}\n {`25`}\n {` raw bits, where `}\n {`20`}\n {` bits are payload and `}\n {`5`}\n {` bits are CRC checksum.`}
\n
{`Conclusion`}
\n
{`We developed a theoretic as well as a programming framework for designing and implementing symbolic barcodes. The programming library is available as `}\n {`symcode`}\n {`. For those who simply want to integrate SymCode into your application, please refer to the wasm distribution `}\n {`acute32`}\n {` which is ready to use in any browser based Javascript projects.`}
\n
{`About Vision Cortex`}
\n
{`This story is a manifest of the philosophy behind Vision Cortex, and a glimpse of the ongoing research and development by the Vision Cortex Research Group.`}
\n
{`If you enjoyed the reading and appreciate our work, consider starring (on `}\n {`GitHub`}\n {`), discuss (on `}\n {`Reddit`}\n {` / `}\n {`GitHub`}\n {`), share (on `}\n {`Twitter`}\n {`) and collaborate.`}
\n\n ;\n}\n;\nMDXContent.isMDXComponent = true;\n "],"sourceRoot":""}
\ No newline at end of file
diff --git a/docs/component---src-vtracer-mdx-0593babbb68bc0de1d9d.js b/docs/component---src-vtracer-mdx-0593babbb68bc0de1d9d.js
new file mode 100644
index 0000000..22b5e6b
--- /dev/null
+++ b/docs/component---src-vtracer-mdx-0593babbb68bc0de1d9d.js
@@ -0,0 +1,2 @@
+(window.webpackJsonp=window.webpackJsonp||[]).push([[10],{ahi8:function(e,t,i){"use strict";i.r(t),i.d(t,"_frontmatter",(function(){return l})),i.d(t,"default",(function(){return p}));var a=i("Fcif"),n=i("+I+c"),r=(i("mXGw"),i("/FXl")),o=i("TjRS"),s=i("zmid"),c=i("cry5"),l=(i("aD51"),{});void 0!==l&&l&&l===Object(l)&&Object.isExtensible(l)&&!l.hasOwnProperty("__filemeta")&&Object.defineProperty(l,"__filemeta",{configurable:!0,value:{name:"_frontmatter",filename:"src/vtracer.mdx"}});var h=o.a;function p(e){var t=e.components,i=Object(n.a)(e,["components"]);return Object(r.b)(h,Object(a.a)({},i,{components:t,mdxType:"MDXLayout"}),Object(r.b)("h1",{id:"vtracer"},"VTracer"),Object(r.b)("p",null,"Researcher: ",Object(r.b)("a",{href:"//github.com/shpun817",parentName:"p"},"Sanford Pun")," | Supervisor: ",Object(r.b)("a",{href:"//github.com/tyt2y3",parentName:"p"},"Chris Tsang")," | Published: 2020-11-01"),Object(r.b)("hr",null),Object(r.b)("p",null,"VTracer is a utility to convert raster images into vector graphics. Try the ",Object(r.b)("a",{href:"//www.visioncortex.org/vtracer/",target:"_blank"},"Demo Web App"),"."),Object(r.b)("img",{style:{width:"100%"},src:"/public/images/vtracer-screenshot.png"}),Object(r.b)("blockquote",null,Object(r.b)("p",{parentName:"blockquote"},"Graphic by sunshine-91 via ",Object(r.b)("a",{href:"https://www.vecteezy.com/vector-art/172177-gum-tree-vector",parentName:"p"},"Vecteezy"))),Object(r.b)("p",null,"The input image is first clustered by ",Object(r.b)("a",{href:"/impression-docs",parentName:"p"},"Hierarchical Clustering"),", and each of the output clusters are traced into vector."),Object(r.b)("p",null,"The algorithm of vector tracing involves 3 main stages:"),Object(r.b)("ol",null,Object(r.b)("li",{parentName:"ol"},Object(r.b)("p",{parentName:"li"},"Convert pixels into path")),Object(r.b)("li",{parentName:"ol"},Object(r.b)("p",{parentName:"li"},"Simplify the path into polygon")),Object(r.b)("li",{parentName:"ol"},Object(r.b)("p",{parentName:"li"},"Smoothen the polygon and approximate it with a curve-fitter"))),Object(r.b)("h2",{id:"path-walking"},"Path Walking"),Object(r.b)("p",null,"VTracer first obtains the raw paths of pixel clusters. A walker is used to trace the outlines of every cluster after building an image tree. The walker would combine consecutive steps in the same direction."),Object(r.b)(s.a,{src:"vtracer/WalkerOptim.svg",text:"2 results of path-walking. Left: Naïve, Unoptimized walker. Right: Optimized walker.",mdxType:"Figure"}),Object(r.b)(s.a,{src:"vtracer/PipelineShowcase/Pixel.png",text:"Raw pixelated polygon",height:"200px",mdxType:"Figure"}),Object(r.b)("h2",{id:"path-simplification"},"Path Simplification"),Object(r.b)("p",null,"Path simplification consists of 2 steps:"),Object(r.b)("ol",null,Object(r.b)("li",{parentName:"ol"},Object(r.b)("p",{parentName:"li"},"Remove staircases")),Object(r.b)("li",{parentName:"ol"},Object(r.b)("p",{parentName:"li"},"Simplify by limiting subpath penalties"))),Object(r.b)("h3",{id:"staircase-removal"},"Staircase Removal"),Object(r.b)("p",null,"From the previous stage, we have obtained a path whose consecutive edges must have different directions, i.e. the shape is represented by the minimum number of edges with 100% fidelity. However, to represent slant lines and curves in raster graphics, “jaggies” (or pixel staircases) inevitably occur. In this step, we aim at removing these artifacts."),Object(r.b)("p",null,"To replace a staircase with a straight line (hence “removing” it), one may adopt an outset (“additive”) or inset (“subtractive”) approach. Both approaches are justifiable in difference contexts, so the key is to maintain consistency in the same shape. In order to determine which points to keep, we make use of the ",Object(r.b)("em",{parentName:"p"},"signed area"),"."),Object(r.b)(s.a,{src:"vtracer/Staircase.svg",mdxType:"Figure"}),Object(r.b)("p",null,"The signed area of a right triangle is a metric used to determine whether the vertices are arranged clockwise or anti-clockwise geometrically. With this information, we can determine which points to keep on the staircase."),Object(r.b)("p",null,"For each point on the staircase, we calculate the signed area of the triangle formed by the previous point, the current point, and the next point on the path. The decision of whether the current point should be kept is then made by comparing the sign of the signed area and the clockwise-ness of the original path."),Object(r.b)(s.a,{src:"vtracer/SignedArea.svg",text:"Two examples of signed area. If outset(inset) is chosen, B(B’) is kept and B’(B) is discarded.",mdxType:"Figure"}),Object(r.b)("h3",{id:"simplify-by-limiting-penalties"},"Simplify by Limiting penalties"),Object(r.b)("p",null,"The path can be further simplified by evaluating the ",Object(r.b)("em",{parentName:"p"},"penalty")," from replacing a subpath with one long edge from the first to the last point."),Object(r.b)(s.a,{src:"vtracer/RemoveWobble.svg",text:"Left: Path with wobbles. Right: Path approximated by red line, removing wobbles",mdxType:"Figure"}),Object(r.b)("p",null,"Given a subpath, we would like to determine if a line drawn from the first point to the last can approximate the whole subpath with high fidelity. The idea is to make sure that all points in the subpath are close enough to the approximating line. To avoid all the complicated coordinate geometry, we can simply evaluate the areas of triangles formed by the first point, the last point, and each in-between point."),Object(r.b)("p",null,"Let ΔABC be one such triangle, with A and C being the first and last points of the subpath respectively, and B being any in-between point. Let h and b be the height and the base (length of AC) respectively. VTracer models the penalty of ΔABC as ",Object(r.b)(c.a,{src:"/public/vtracer/PenaltyEquation.svg",invSrc:"/public/vtracer/PenaltyEquation_inv.svg",mdxType:"InlineEquation"}),", as the area of ΔABC and b can be obtained by simple geometry. It is crucial for the penalty to be directly proportional to h and b."),Object(r.b)("p",null,"Once the penalty is clearly defined, the procedure of simplification is straightforward. VTracer greedily extends a subpath until the maximum penalty along it exceeds a specific tolerance, then all edges in the subpath are replaced by one line from the first to the second last point (or equivalently, remove in-between points from the path). After the replacement, the same process is performed starting from the next point in the path. When the last subpath is extended to the last point in the whole path, the simplification stage concludes."),Object(r.b)(s.a,{src:"vtracer/Penalty.svg",mdxType:"Figure"}),Object(r.b)("h2",{id:"path-smoothing"},"Path Smoothing"),Object(r.b)(s.a,{src:"vtracer/PipelineShowcase/Original.png",text:"Sample Simplified 'S' Shape",height:"200px",mdxType:"Figure"}),Object(r.b)("p",null,"What we have now is a simplified polygon with high fidelity. However, if we feed the path to the curve fitter as is, the curves will approximate the shape poorly. This is because the curves are underdetermined given the small number of points. In order to generate points that lie on our desired shape, subdivision smoothing is performed."),Object(r.b)("h3",{id:"4-point-scheme"},"4-Point Scheme"),Object(r.b)("p",null,"VTracer adapts from the ",Object(r.b)("em",{parentName:"p"},"4-Point Scheme")," subdivision algorithm, which is an interpolating smoothing method. The problem of the 4-Point Scheme is that all vertices would be smoothed into round corners, which, from our point of view, is a loss of fidelity."),Object(r.b)(s.a,{src:"vtracer/4-PointScheme.svg",text:"4-Point Scheme performed on segment A₁A₂",mdxType:"Figure"}),Object(r.b)(s.a,{src:"vtracer/4-PointSchemeProblem.svg",text:"If A2 is a corner, the 4-Point Scheme smooths it out iteratively.",mdxType:"Figure"}),Object(r.b)("h3",{id:"finding-corners"},"Finding corners"),Object(r.b)("p",null,"To preserve corners, we first have to locate them. VTracer finds corners by checking the ",Object(r.b)("em",{parentName:"p"},"angle difference")," at each vertex. If the absolute angle difference exceeds a certain threshold, the vertex is considered to be a corner."),Object(r.b)(s.a,{src:"vtracer/Corner.svg",mdxType:"Figure"}),Object(r.b)("h6",{style:{textAlign:"center"}},Object(r.b)("p",null,"Angle difference from A to B is small => a is not a corner\n",Object(r.b)("br",null),"\nAngle difference from B to C is large => b is a corner")),Object(r.b)("h3",{id:"corner-preserving-4-point-scheme"},"Corner-Preserving 4-Point Scheme"),Object(r.b)("p",null,"In the original 4-Point Scheme, 2 adjacent points are always used to generate the new point for each segment. In our adapted version, we do not take the adjacent point for corners, but instead we take the corners themselves. For segments whose points are both corners, we simply ignore them."),Object(r.b)(s.a,{src:"vtracer/3-PointScheme.svg",mdxType:"Figure"}),Object(r.b)("p",null,"Since A₂ is a corner, the smoothing procedure does not take the adjacent point as B₂. As a result, the corner will be (approximately) preserved after smoothing, even after iterations."),Object(r.b)("p",null,"VTracer applies a threshold on the length of each segment during subdivision smoothing, so that the result will not be over-dense. This threshold should be decided carefully (mainly based on the resolution of image), otherwise the resulting path will be a poor approximation."),Object(r.b)("p",null,"Shown below are examples smoothed with no iteration:"),Object(r.b)(s.a,{src:"vtracer/PipelineShowcase/LengthThresholds.svg",text:"Length Thresholds 3.5/5.0/7.5",height:"200px",mdxType:"Figure"}),Object(r.b)("p",null,"Introducing iterations, you can see more points are generated by subdivision:"),Object(r.b)(s.a,{src:"vtracer/PipelineShowcase/MaxIterations.svg",text:"Iterations 0/1/2",height:"200px",mdxType:"Figure"}),Object(r.b)("p",null,"VTracer's implementation defaults to 10 iterations, and exit early when no further smoothing can be done on the path."),Object(r.b)("h2",{id:"curve-fitting"},"Curve Fitting"),Object(r.b)("p",null,"The path is now populated with nicely positioned and sufficiently dense points that faithfully represent our desired shape. Before feeding it to a (Bezier) curve-fitter, VTracer determines where to cut curves (splice points)."),Object(r.b)("h3",{id:"finding-splice-points"},"Finding Splice Points"),Object(r.b)("p",null,"To define a splice point, we make use of the ",Object(r.b)("em",{parentName:"p"},"signed angle differences"),". Interpreting each edge along the path as vectors, we can define the signed angle difference from edge eᵢ to eᵢ₊₁ as the 2D angle of rotation θ ∊ (-π, π] required to rotate eᵢ to the same direction as eᵢ₊₁ (assume positive θ in clockwise direction)."),Object(r.b)("p",null,"It is sufficient for a vertex to be a splice point if it passes one of two tests: ",Object(r.b)("em",{parentName:"p"},"point of inflection test"),"and ",Object(r.b)("em",{parentName:"p"},"angle displacement test"),"."),Object(r.b)("p",null,"Points of inflection can be found by tracking the signs of angle differences along the path. When the concavity of the path changes at a certain point, the sign of the signed angle difference also changes at that point."),Object(r.b)(s.a,{src:"vtracer/PointOfInflection.svg",mdxType:"Figure"}),Object(r.b)("h6",{style:{textAlign:"center"}},Object(r.b)("p",null,"As the sign of angle difference changes at point P, P is a point of inflection and hence a splice point.\n",Object(r.b)("br",null),"\nTherefore, we cut (A..B) into (A..P) and (P..B).")),Object(r.b)("p",null,"Angle displacement at a point is defined as the signed angle differences accumulated from the previous splice point (exclusive) to the current point (inclusive). If the absolute value of angle displacement at a certain point exceeds a specific threshold, that point is a splice point. The smaller the threshold, the more curves are cut, and the resulting spline is more accurate."),Object(r.b)(s.a,{src:"vtracer/AngleDisplacement.svg",text:"Angle Displacements shown along a path.",mdxType:"Figure"}),Object(r.b)("p",null,"Once all splice points are found, VTracer feeds all subpaths between every consecutive pair of splice points into the curve-fitter."),Object(r.b)("p",null,"If the smoothed path from the previous step is fed into the curve-fitter, we get a spline like the following:"),Object(r.b)(s.a,{src:"vtracer/PipelineShowcase/Curve-fitted.png",text:"Final output",height:"200px",mdxType:"Figure"}),Object(r.b)("h2",{id:"vs-potrace"},"vs Potrace"),Object(r.b)("h3",{id:"fidelity"},"Fidelity"),Object(r.b)("p",null,Object(r.b)("a",{href:"http://potrace.sourceforge.net/",parentName:"p"},"Potrace")," is a popular bitmap tracing tool that also transforms bitmaps into vector graphics. Being able to produce high-quality output for low-resolution images, Potrace traces images by finding global optimal under certain assumptions. VTracer favours fidelity over simplification. Potrace and VTracer produce different results especially on small objects."),Object(r.b)(s.a,{src:"vtracer/VSPotraceCorner.svg",mdxType:"Figure"}),Object(r.b)("h6",{style:{textAlign:"center"}},Object(r.b)("p",null,"Illustration of how assumptions affect tracing.\n",Object(r.b)("br",null),"\nLeft: Original input.\n",Object(r.b)("br",null),"\nMiddle: Possible shape interpretation with the assumption that ambiguous “corners” in the original input are sharp corners.\n",Object(r.b)("br",null),"Right: Possible shape interpretation with no assumptions: “corners” in the original image are represented by curves or round corners.")),Object(r.b)("h3",{id:"efficiency"},"Efficiency"),Object(r.b)("p",null,"Potrace finds the global optimal way of tracing a shape, meaning it approximates parts using information from the entire shape, which leads to performance issues in high-resolution images. On the other hand, VTracer runs a linear algorithm on each cluster and has lower CPU and memory usage. In fact, we regularly use VTracer to process high resolution scans at 10 megapixels."),Object(r.b)("h2",{id:"vs-adobe-illustrator"},"vs Adobe Illustrator"),Object(r.b)("p",null,"Comparing to Adobe Illustrator's Live Trace, VTracer's output is much more compact thanks to a stacking strategy. VTracer's preprocessing stage utilizes Vision Cortex's image tree algorithm and never produces shapes with holes."))}void 0!==p&&p&&p===Object(p)&&Object.isExtensible(p)&&!p.hasOwnProperty("__filemeta")&&Object.defineProperty(p,"__filemeta",{configurable:!0,value:{name:"MDXContent",filename:"src/vtracer.mdx"}}),p.isMDXComponent=!0}}]);
+//# sourceMappingURL=component---src-vtracer-mdx-0593babbb68bc0de1d9d.js.map
\ No newline at end of file
diff --git a/docs/component---src-vtracer-mdx-0593babbb68bc0de1d9d.js.map b/docs/component---src-vtracer-mdx-0593babbb68bc0de1d9d.js.map
new file mode 100644
index 0000000..1efe1cc
--- /dev/null
+++ b/docs/component---src-vtracer-mdx-0593babbb68bc0de1d9d.js.map
@@ -0,0 +1 @@
+{"version":3,"sources":["webpack:///../src/vtracer.mdx"],"names":["_frontmatter","MDXLayout","DefaultLayout","MDXContent","components","props","mdxType","href","target","style","width","src","text","height","invSrc","textAlign","isMDXComponent"],"mappings":"gRAQaA,G,UAAe,S,iMAE5B,IAKMC,EAAYC,IACH,SAASC,EAAT,GAGZ,IAFDC,EAEC,EAFDA,WACGC,EACF,8BACD,OAAO,YAACJ,EAAD,eAAeI,EAAf,CAAsBD,WAAYA,EAAYE,QAAQ,cAC3D,iBAAQ,CACN,GAAM,WADR,WAGA,oCACE,gBAAO,CACL,KAAQ,wBACR,WAAc,KAFhB,eADF,kBAME,gBAAO,CACL,KAAQ,sBACR,WAAc,KAFhB,eANF,4BAWA,uBACA,oGACE,iBAAGC,KAAK,kCAAkCC,OAAO,UAAjD,gBADF,KAGA,mBAAKC,MAAO,CACVC,MAAO,QACNC,IAAK,0CACR,8BAEE,gBAAO,CACL,WAAc,cADhB,8BAGE,gBAAO,CACL,KAAQ,6DACR,WAAc,KAFhB,cAMJ,8DACE,gBAAO,CACL,KAAQ,mBACR,WAAc,KAFhB,2BADF,6DAMA,gFACA,sBAEE,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,6BAOF,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,mCAOF,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,iEAOJ,iBAAQ,CACN,GAAM,gBADR,gBAGA,uOACA,YAAC,IAAD,CAAQA,IAAI,0BAA0BC,KAAK,uFAAuFN,QAAQ,WAC1I,YAAC,IAAD,CAAQK,IAAI,qCAAqCC,KAAK,wBAAwBC,OAAO,QAAQP,QAAQ,WACrG,iBAAQ,CACN,GAAM,uBADR,uBAGA,iEACA,sBAEE,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,sBAOF,iBAAQ,CACN,WAAc,MAGd,gBAAO,CACL,WAAc,MADhB,4CAOJ,iBAAQ,CACN,GAAM,qBADR,qBAGA,uXACA,mVACE,iBAAQ,CACN,WAAc,KADhB,eADF,KAKA,YAAC,IAAD,CAAQK,IAAI,wBAAwBL,QAAQ,WAC5C,sPACA,mVACA,YAAC,IAAD,CAAQK,IAAI,yBAAyBC,KAAK,iGAAiGN,QAAQ,WACnJ,iBAAQ,CACN,GAAM,kCADR,kCAGA,6EACE,iBAAQ,CACN,WAAc,KADhB,WADF,kFAKA,YAAC,IAAD,CAAQK,IAAI,2BAA2BC,KAAK,kFAAkFN,QAAQ,WACtI,qbACA,6QACE,YAAC,IAAD,CAAgBK,IAAK,sCAAuCG,OAAQ,0CAA2CR,QAAQ,mBADzH,yIAGA,yjBACA,YAAC,IAAD,CAAQK,IAAI,sBAAsBL,QAAQ,WAC1C,iBAAQ,CACN,GAAM,kBADR,kBAGA,YAAC,IAAD,CAAQK,IAAI,wCAAwCC,KAAK,8BAA8BC,OAAO,QAAQP,QAAQ,WAC9G,2WACA,iBAAQ,CACN,GAAM,kBADR,kBAGA,gDACE,iBAAQ,CACN,WAAc,KADhB,kBADF,0NAKA,YAAC,IAAD,CAAQK,IAAI,4BAA4BC,KAAK,2CAA2CN,QAAQ,WAChG,YAAC,IAAD,CAAQK,IAAI,mCAAmCC,KAAK,oEAAoEN,QAAQ,WAChI,iBAAQ,CACN,GAAM,mBADR,mBAGA,iHACE,iBAAQ,CACN,WAAc,KADhB,oBADF,2HAKA,YAAC,IAAD,CAAQK,IAAI,qBAAqBL,QAAQ,WACzC,kBAAIG,MAAO,CACTM,UAAW,WAEX,oFAEE,uBAFF,6DAMF,iBAAQ,CACN,GAAM,oCADR,oCAGA,4TACA,YAAC,IAAD,CAAQJ,IAAI,4BAA4BL,QAAQ,WAChD,gNACA,4SACA,6EACA,YAAC,IAAD,CAAQK,IAAI,gDAAgDC,KAAK,gCAAgCC,OAAO,QAAQP,QAAQ,WACxH,sGACA,YAAC,IAAD,CAAQK,IAAI,6CAA6CC,KAAK,mBAAmBC,OAAO,QAAQP,QAAQ,WACxG,8IACA,iBAAQ,CACN,GAAM,iBADR,iBAGA,0PACA,iBAAQ,CACN,GAAM,yBADR,yBAGA,qEACE,iBAAQ,CACN,WAAc,KADhB,4BADF,8PAKA,0GACE,iBAAQ,CACN,WAAc,KADhB,4BADF,OAKE,iBAAQ,CACN,WAAc,KADhB,2BALF,KASA,oPACA,YAAC,IAAD,CAAQK,IAAI,gCAAgCL,QAAQ,WACpD,kBAAIG,MAAO,CACTM,UAAW,WAEX,kIAEE,uBAFF,uDAMF,oZACA,YAAC,IAAD,CAAQJ,IAAI,gCAAgCC,KAAK,0CAA0CN,QAAQ,WACnG,4JACA,sIACA,YAAC,IAAD,CAAQK,IAAI,4CAA4CC,KAAK,eAAeC,OAAO,QAAQP,QAAQ,WACnG,iBAAQ,CACN,GAAM,cADR,cAGA,iBAAQ,CACN,GAAM,YADR,YAGA,qBAAG,gBAAO,CACN,KAAQ,kCACR,WAAc,KAFf,WAAH,iWAKA,YAAC,IAAD,CAAQK,IAAI,8BAA8BL,QAAQ,WAClD,kBAAIG,MAAO,CACTM,UAAW,WAEX,yEAEE,uBAFF,4BAME,uBANF,kIAUE,uBAVF,0IAaF,iBAAQ,CACN,GAAM,cADR,cAGA,iZACA,iBAAQ,CACN,GAAM,wBADR,wBAGA,kQ,+LAKJZ,EAAWa,gBAAiB","file":"component---src-vtracer-mdx-0593babbb68bc0de1d9d.js","sourcesContent":["import * as React from 'react'\n /* @jsx mdx */\nimport { mdx } from '@mdx-js/react';\n/* @jsx mdx */\n\nimport DefaultLayout from \"/home/chris/visioncortex.github.io/node_modules/gatsby-theme-docz/src/base/Layout.js\";\nimport { Figure } from './components/Figure';\nimport { InlineEquation } from './components/InlineEquation';\nexport const _frontmatter = {};\n\nconst makeShortcode = name => function MDXDefaultShortcode(props) {\n console.warn(\"Component \" + name + \" was not imported, exported, or provided by MDXProvider as global scope\");\n return ;\n};\n\nconst MDXLayout = DefaultLayout;\nexport default function MDXContent({\n components,\n ...props\n}) {\n return \n
{`The input image is first clustered by `}\n {`Hierarchical Clustering`}\n {`, and each of the output clusters are traced into vector.`}
\n
{`The algorithm of vector tracing involves 3 main stages:`}
\n \n\n
\n\n
{`Convert pixels into path`}
\n\n
\n\n\n
\n\n
{`Simplify the path into polygon`}
\n\n
\n\n\n
\n\n
{`Smoothen the polygon and approximate it with a curve-fitter`}
\n\n
\n\n \n
{`Path Walking`}
\n
{`VTracer first obtains the raw paths of pixel clusters. A walker is used to trace the outlines of every cluster after building an image tree. The walker would combine consecutive steps in the same direction.`}
\n \n \n
{`Path Simplification`}
\n
{`Path simplification consists of 2 steps:`}
\n \n\n
\n\n
{`Remove staircases`}
\n\n
\n\n\n
\n\n
{`Simplify by limiting subpath penalties`}
\n\n
\n\n \n
{`Staircase Removal`}
\n
{`From the previous stage, we have obtained a path whose consecutive edges must have different directions, i.e. the shape is represented by the minimum number of edges with 100% fidelity. However, to represent slant lines and curves in raster graphics, “jaggies” (or pixel staircases) inevitably occur. In this step, we aim at removing these artifacts.`}
\n
{`To replace a staircase with a straight line (hence “removing” it), one may adopt an outset (“additive”) or inset (“subtractive”) approach. Both approaches are justifiable in difference contexts, so the key is to maintain consistency in the same shape. In order to determine which points to keep, we make use of the `}\n {`signed area`}\n {`.`}
\n \n
{`The signed area of a right triangle is a metric used to determine whether the vertices are arranged clockwise or anti-clockwise geometrically. With this information, we can determine which points to keep on the staircase.`}
\n
{`For each point on the staircase, we calculate the signed area of the triangle formed by the previous point, the current point, and the next point on the path. The decision of whether the current point should be kept is then made by comparing the sign of the signed area and the clockwise-ness of the original path.`}
\n \n
{`Simplify by Limiting penalties`}
\n
{`The path can be further simplified by evaluating the `}\n {`penalty`}\n {` from replacing a subpath with one long edge from the first to the last point.`}
\n \n
{`Given a subpath, we would like to determine if a line drawn from the first point to the last can approximate the whole subpath with high fidelity. The idea is to make sure that all points in the subpath are close enough to the approximating line. To avoid all the complicated coordinate geometry, we can simply evaluate the areas of triangles formed by the first point, the last point, and each in-between point.`}
\n
{`Let ΔABC be one such triangle, with A and C being the first and last points of the subpath respectively, and B being any in-between point. Let h and b be the height and the base (length of AC) respectively. VTracer models the penalty of ΔABC as `}\n \n {`, as the area of ΔABC and b can be obtained by simple geometry. It is crucial for the penalty to be directly proportional to h and b.`}
\n
{`Once the penalty is clearly defined, the procedure of simplification is straightforward. VTracer greedily extends a subpath until the maximum penalty along it exceeds a specific tolerance, then all edges in the subpath are replaced by one line from the first to the second last point (or equivalently, remove in-between points from the path). After the replacement, the same process is performed starting from the next point in the path. When the last subpath is extended to the last point in the whole path, the simplification stage concludes.`}
\n \n
{`Path Smoothing`}
\n \n
{`What we have now is a simplified polygon with high fidelity. However, if we feed the path to the curve fitter as is, the curves will approximate the shape poorly. This is because the curves are underdetermined given the small number of points. In order to generate points that lie on our desired shape, subdivision smoothing is performed.`}
\n
{`4-Point Scheme`}
\n
{`VTracer adapts from the `}\n {`4-Point Scheme`}\n {` subdivision algorithm, which is an interpolating smoothing method. The problem of the 4-Point Scheme is that all vertices would be smoothed into round corners, which, from our point of view, is a loss of fidelity.`}
\n \n \n
{`Finding corners`}
\n
{`To preserve corners, we first have to locate them. VTracer finds corners by checking the `}\n {`angle difference`}\n {` at each vertex. If the absolute angle difference exceeds a certain threshold, the vertex is considered to be a corner.`}
\n \n
\n
{`Angle difference from A to B is small => a is not a corner\n`}\n \n {`\nAngle difference from B to C is large => b is a corner`}
\n \n
{`Corner-Preserving 4-Point Scheme`}
\n
{`In the original 4-Point Scheme, 2 adjacent points are always used to generate the new point for each segment. In our adapted version, we do not take the adjacent point for corners, but instead we take the corners themselves. For segments whose points are both corners, we simply ignore them.`}
\n \n
{`Since A₂ is a corner, the smoothing procedure does not take the adjacent point as B₂. As a result, the corner will be (approximately) preserved after smoothing, even after iterations.`}
\n
{`VTracer applies a threshold on the length of each segment during subdivision smoothing, so that the result will not be over-dense. This threshold should be decided carefully (mainly based on the resolution of image), otherwise the resulting path will be a poor approximation.`}
\n
{`Shown below are examples smoothed with no iteration:`}
\n \n
{`Introducing iterations, you can see more points are generated by subdivision:`}
\n \n
{`VTracer's implementation defaults to 10 iterations, and exit early when no further smoothing can be done on the path.`}
\n
{`Curve Fitting`}
\n
{`The path is now populated with nicely positioned and sufficiently dense points that faithfully represent our desired shape. Before feeding it to a (Bezier) curve-fitter, VTracer determines where to cut curves (splice points).`}
\n
{`Finding Splice Points`}
\n
{`To define a splice point, we make use of the `}\n {`signed angle differences`}\n {`. Interpreting each edge along the path as vectors, we can define the signed angle difference from edge eᵢ to eᵢ₊₁ as the 2D angle of rotation θ ∊ (-π, π] required to rotate eᵢ to the same direction as eᵢ₊₁ (assume positive θ in clockwise direction).`}
\n
{`It is sufficient for a vertex to be a splice point if it passes one of two tests: `}\n {`point of inflection test`}\n {`and `}\n {`angle displacement test`}\n {`.`}
\n
{`Points of inflection can be found by tracking the signs of angle differences along the path. When the concavity of the path changes at a certain point, the sign of the signed angle difference also changes at that point.`}
\n \n
\n
{`As the sign of angle difference changes at point P, P is a point of inflection and hence a splice point.\n`}\n \n {`\nTherefore, we cut (A..B) into (A..P) and (P..B).`}
\n \n
{`Angle displacement at a point is defined as the signed angle differences accumulated from the previous splice point (exclusive) to the current point (inclusive). If the absolute value of angle displacement at a certain point exceeds a specific threshold, that point is a splice point. The smaller the threshold, the more curves are cut, and the resulting spline is more accurate.`}
\n \n
{`Once all splice points are found, VTracer feeds all subpaths between every consecutive pair of splice points into the curve-fitter.`}
\n
{`If the smoothed path from the previous step is fed into the curve-fitter, we get a spline like the following:`}
\n \n
{`vs Potrace`}
\n
{`Fidelity`}
\n
{`Potrace`}\n {` is a popular bitmap tracing tool that also transforms bitmaps into vector graphics. Being able to produce high-quality output for low-resolution images, Potrace traces images by finding global optimal under certain assumptions. VTracer favours fidelity over simplification. Potrace and VTracer produce different results especially on small objects.`}
\n \n
\n
{`Illustration of how assumptions affect tracing.\n`}\n \n {`\nLeft: Original input.\n`}\n \n {`\nMiddle: Possible shape interpretation with the assumption that ambiguous “corners” in the original input are sharp corners.\n`}\n \n {`Right: Possible shape interpretation with no assumptions: “corners” in the original image are represented by curves or round corners.`}
\n \n
{`Efficiency`}
\n
{`Potrace finds the global optimal way of tracing a shape, meaning it approximates parts using information from the entire shape, which leads to performance issues in high-resolution images. On the other hand, VTracer runs a linear algorithm on each cluster and has lower CPU and memory usage. In fact, we regularly use VTracer to process high resolution scans at 10 megapixels.`}
\n
{`vs Adobe Illustrator`}
\n
{`Comparing to Adobe Illustrator's Live Trace, VTracer's output is much more compact thanks to a stacking strategy. VTracer's preprocessing stage utilizes Vision Cortex's image tree algorithm and never produces shapes with holes.`}
\n 
\n 
\n 
\n 
\n 
.jpg)
.jpg){kind=link}