This repo holds contains source code for CAVA, or Columnar data Augmentation through Visual Analytics, a visual analytics tool for column-wise data augmentation using knowledge graphs. This work was developed by visual analytics research labs at Tufts University, Georgia Tech, and University of Wisconsin in relation to DARPA's Data Driven Discovery of Models program.
A preprint of the paper can be found on arXiv.
https://arxiv.org/abs/2009.02865
The paper will be presented at the IEEE Visualization Conference virtually in October 2020. It will be published in IEEE Transactions on Visualization and Computer Graphics at some point after that. Thank you to the full list of authors: Dylan Cashman (Tufts), Shenyu Xu (Georgia Tech), Subhajit Das (Georgia Tech), Florian Heimerl (Wisconsin), Cong Liu (Tufts), Shah Rukh Humayoun (San Francisco State), Michael Gleicher (Wisconsin), Alex Endert (Georgia Tech), Remco Chang (Tufts). Thank you also to all of our collaborators in DARPA's Data Driven Discovery of Models program.
The goal of this application is to make it easy to add on additional columns to a tabular dataset by mining knowledge graphs. Currently, this application only supports connecting to wikidata.
CAVA lets users upload a dataset and then search through wikidata for related attributes of different columns. For example, if a dataset has a column consisting of U.S. cities, CAVA will return a list of attributes shared by U.S. cities in wikidata, and let the user join them to their CSV and download the augmented dataset. It also supports more complex augmented attributes, like joining a country's average neighbor GDP.
CAVA consists of a front end user interface that lives in the browser, and a backend server that preprocesses data and calls out to the knowledge graph. Both front and backend need to be served in order to use CAVA.
This application contains a vueJS, nodeJS, and python project. In includes some legacy code from a larger application that included more data exploration and connection to a machine learning backend, but it has been pared down to only focus on column data augmentation. It has a web server done in node.js. It also contains some data processing scripts, in lib/external, that are called through node's child_process library. This node project includes a frontend in vueJS that communicates with the server.
This is the Summer 2020 version, and is not currently being developed or supported. Feel free to fork this and usethis in your own research.
Our application is split into two servers: The node.js middleware, found in node_middleware, and the vue.js frontend, found in vue_frontend. You need to run installation commands in both directores.
Once installation is done, test it out. Two servers need to be run, actually - the node server supporting GRPC and web socket communication, and a server for the vue assets. They are coupled in a single command. From the application root,
bash ta3_search shared/static/local_testing_data/185_baseball
If you run the ta3_search bash script without a dataset, the user is able to upload a CSV dataset themselves.
bash ta3_search
All instructions should be run from within the node_middleware directory.
We use python3 on the backend to run python scripts in child processes. You should be able to run CAVA without having python installed, but you may see that some features don't work.
We also require node version >= 8.0. If you get an error during the server startup, double check that you have the right node version.
Assuming you have pip installed,
pip3 install -r python-requirements.txt
Install the node dependencies.
yarn install
You might find that the server reports issues with the 'static' folder. This folder should be symlinked with the static folder in shared/, but sometimes this doesn't come through right in the git repo. If so, delete the empty file node_middleware/static, and then create a new symlink from the node_middleware/shared directory
ln -nfs ../../shared/static staticd
The same thing might occur with the output/ directory as well.
ln -nfs ../../shared/output output
All instructions should be run from within vue_frontend
Install the node dependencies.
yarn install
The middleware communicates with the frontend over two ports - 3000 for the web front end, and 8080 for the web socket. Those will need to be opened if this is run in a docker container. A compatible Dockerfile is included in this repository.
If you'd like to contribute to this repository, please create an issue for discussion, and fork the repository and make a pull request.
This is released under the MIT License.