Home

SFCTA Prospector

We have built a data warehouse and visualization platform called Prospector here at SFCTA. Find all kinds of details and documentation about it here. Please note that this is still a work in progress. Do not hesitate to reach out sfcta-prospector@googlegroups.com if you have any questions.

Please feel free to fork it and submit pull requests if you get a chance to develop some useful visualization tools.

The SFCTA data warehouse "Prospector" is composed of several components. The back-end systems include Linux virtual machines running geospatial database software, and front-end web pages which access that data using Javascript, JSON, and HTML. The various pieces are described below.

Step-by-step for preparing your machine

• Back-End Setup
• Setting up your development environment
• Building Prospector locally
• Publishing your changes

Technologies You'll Want to Learn

The system is based on two languages: Javascript and SQL. Without basic knowledge of those, you won't be able to do much, so you should jump in and learn them! The tutorials below worked for me.

• Javascript: the lingua franca of the web. This tutorial taught me everything -- and you really just need to get through "Part 1".
• SQL - specifically, PostgreSQL. This tutorial is comprehensive. Truth is you don't need a ton of SQL, the basics should get you pretty far.

Diving Deeper: Understanding the code

To really be productive you'll need to understand some of the components that we used to build the site. These are listed in the order I think you'll encounter/need them. Check out the Glossary too, for a brief description of each of the components.

1. Use PostgREST to fetch data from the database. It returns JSON and has a pretty flexible query language that you build into the URL of whatever you request.
2. Use Vue to build web components that do things -- it connects the widgets, buttons, dropdowns on your front end to the javascript code on the backend. The "Getting Started" guide on the Vue website tells you everything you need to know.
3. Use either Leaflet or MapBox GL to show maps with clickable things on them. I used Leaflet for the CMP tool, and MapBox for the TNC tool, since I didn't know which one I would like better. They are very similar, but only MapBox GL can do the 3D effects.
4. I found a small, lightweight charting library called Morris but I don't think it's particularly special in any way -- and there are lots of other ones out there if you decide you need something else.

Overall Architecture

SFCTA wanted a system built as much as possible on open-source tools, and so I've cobbled together these components based on my previous experiences as well as our research here this spring.

The world of website development is very fractured. There are hundreds of ways to do one thing. I've tried to pick some decent best-in-class options but the answers are pretty subjective. When confronted with choices, I tried to balance the most popular choice, the cleanest implementation, and the easiest to deploy for non-experts like us. I hope this has left SFCTA with a platform that is maintainable in the long term.

Back-End Systems

All three of the following systems are "virtual" linux server machiness (VMs), running in the SFCTA server room. Chien has all the passwords and backups.

Virtual Machine Name	Purpose
prospector	The VM running the NGINX web server. This is the data portal website itself.
prospector-db	The VM running the PostgreSQL database, with PostGIS geospatial extensions.
prospector-api	The VM running (1) an NGINX webserver/reverse-proxy, (2) the PostgREST API server, which exposes database tables using standard HTTP and web protocols, and (3) the GeoServer map tile server, currently unused but it's there in case we need it.
prospector-extra	A "staging web server" which is identical to prospector above, which allows you to test modifications to the data portal web site internally, without breaking the production server.

Front-End Development

Each product is essentially a 'static' website, which means it is just raw HTML and javascript that can be deployed just about anywhere. The code calls the postgREST API when it needs to fetch data from the database.

• Web pages are written in modern "ES2015" javascript, which is transpiled using NPM and Webpack to work on older browsers.
• UI interactivity built using Vuejs.org
• Lots and lots of helper components are used to build the site, each being a solid implementation of a particular thing that we need. This makes for a long list of things to learn, but that's better than writing any of this stuff from scratch ourselves:
  • Semantic UI widget elements, for a nice modern look
  • Leaflet + MapBox for mapping
  • Morris.js for charts