-
Notifications
You must be signed in to change notification settings - Fork 5
web application concept and context
CCCS Web Map Publishing Application—A Discussion of Current Challenges and Possible Future Directions
ALERT: CCCS' work to develop an application to publish maps is occurring in two silos. We wish to change this so that CCCS' development efforts [led by @pwhipp] is integrated directly with those of Kartoza. This issue is addressed briefly here, and in more detail under separate cover.
Status Report: Kartoza
Kartoza Pty has provided CCCS with a stack of OpenSource software bundled as a Docker application. This software stack responds to the following CCCS concerns:
-
presenting data for consumption in a web browser;
-
allowing maps to be simultaneously broadcast over the network; this is accomplished by creating parallel software environments that communicate over assigned ports
-
allowing users to load in data from designated repositories; this is accomplished via custom 'import' scripts being authored and maintained by @NyakudyaA
Upon a period of review of the Docker application bundle, CCCS has asked to discontinue efforts for Docker integration [01 December 2014]. Our reasons are as follows:
-
CCCS has a limited grap of many aspects for the Docker-based set-up. Firstly, it is unclear to us exactly what our 'Docker application' "is". The set-up scripts suggest that we're running not one "CCCS" application, but instead a "CCCS-bundle" comprised of a suite of individual Kartoza docker applications, each wrapping a pre-configured application entity (e.g. postgreSQL, QGIS Server. Looking through the scripts, we can see how certain configuration options are being passed into Docker, but it is hard to evaluate and learn from how different application are configured to communicate with one another. For example, Paul has been struggling to identify how, within docker, the system is configured to pass data manipulated with QGIS to present data layers to the browser. CCCS would like more direct access and exposure to the application system architecture, and to communicate with applications directly rather than via a docker layer.
-
The aspect of scripting data loading into the Docker-embedded database is also somewhat troublesome from CCCS' perspective. Scripting the psql data import means maintaining bundles of import scripts, each of which must be assiduously tracked kept up-to-date in order for the script to work. The is a level of maintenance that we cannot afford in the long term (though using our current scripts for tutorials and examples is fine; these scripts can be periodically updated as long as appropriate cautions declared to end users.)
-
CCCS has also had some difficulty learning how to interact with data via Docker. We are unsure how to point the Docker application to the specific *.qgis file that we wish to server (it is hard coded into the loader?). Do we have to re-load the docker app each time we make a change to the qgis file in the source repo? How is docker referencing a source repo outside of its own container? Via ssh / html paths? What happens when these paths change?
-
CCCS also asked to stop Docker-related development due to concerns for the large size of each Docker app (or, app suite). Each of CCS Docker repositories is rather large in size. [%%% NOTE SIZES %%%] While this is managable from the standpoint of sharing a development environment for web mappers (allowing them to see how the web mapping application presents data), it makes less sense other CCCS purposes, such as serving multiple maps for consumption on the site. With only a few maps online, we'd quickly start incurring unnecessary costs for excessive utilization of storage space.
In addition to the integrated software package, Kartoza has also been leading efforts to help get QGIS server (or any wms-based project) to show in our Django-based website. Toward this end, Kartoza [@timlinux and @ismailsunni] wrote a simple django application: django-wms-client to facilitate publication of QGIS Server (or any WMN-based project) within a Django project framework (e.g. Mezzanine CMS). This wms-client is currently configured as a standalone project (mainly for testing). For django integration, Kartoza proposes publishing the app to pypi and then provide instructions for deploying and running it as a Django app.
[NOTE: Deployment within Django is achieved by adding django-wms-client to INSTALLED_APPS, doing a django sync / migration and then adding in your maps using the django admininistration panel. An overview of django-wms-client functionality is provided in the README.]
@ismailsunni has been charged with testing django-wms-client integration wtih mezzanine. Kartoza has pledged to come back with more details of best practice on how to directly embed maps into your CMS documents.
Status Report: CCCS
CCCS wishes to continue our work to develop a python-centric software stack and deployment framework. Bundling of software and services will occur using the Django web framework. Our mapping software application should be deployable as a Django module.
@pwhipp has been working to ascertain out the Kartoza software stack and to and re-create it in the context of the Django web framework [see notes above concerning our challenges of the Docker set-up]. From my understanding, Paul's efforts include getting QGIS server to work under nginx a task related to the server's internal configuration.
From's his work, Paul's interpretation of the currently-available wms clients for consuming QGIS Server [namely
QGIS-Web-Client and django-wms-client] is that each is intended to serve a single *.qgs file. [A brief aside: Paul submitted a pull request to django-wms-client that brings its requirements up to date so that it can be installed more easily. THis includes a bug fix to address issue # 27.]
CCCS is still exploring how to present different maps for consumption to individual web pages (or map viewing frames). We have been working on the assumption that it would be 'best' (or at least, 'possible') to serve multiple qgs projects without needing a separate server running for each one (i.e., per the methods described in Tim's 2012 blog entry, with the challenge for Django-integration being that the QGIS Server wont serve other *.qgs files even if we clone the wms-client for each.
Tim has clarified that QGIS server presents a series of QGIS projects using the standard web services : Web Mapping Service (WMS) and Web Feature Service (WFS). These services can be consumed by various clients including OpenLayers and Leaflet. The WFS specification allow one to ask for available layers to and selectively display all or a subset thereof. Tim also explained that each QGIS-Web-Client can serve only a single QGIS project (i.e. a *.qgs file) due to how the web client is tied into the specific QGIS project it renders.
Tim also clarified that his concept for web publication architecture did not include the QGIS web client. He noted that our initial discussions were centered around using a 'pure' OpenLayers3 or Leaflet implementation with some added 'extras' for managing layer ordering. This is the underlying logic in the development of the django-wms-client. Following this approach, CCCS would publish your projects via QGIS server and then register those projects in the django-wms-client app, which would then deliver them as a Leaflet (or Openlayers) project. Tim notes that the app already does a decent job of service discovery when you point it to a WMS URL.
To carry our web application development work forward, we need to establish and agree upon team workflow and and production pipelines. From there, we need to provide a functional demonstration / "proof-of-concept" and then to extend our supporting documentation.
The following are CCCS' considerations of the two options Tim presented for future web application development:
Option 1
To use QGIS Web Client as the platform for map delivery we would need to update the QGIS-Web-Client to automatically generate and deploy WMS client configuration files. @timlinux recommends against this approach. The QGIS Web Client, Tim explains, is a 'heavy' app that is essentially trying to replicate a read only experience of the original desktop QGIS project.
The advantage of QGIS-Web-Client is that it supports some nice-to-have extended features, including 'print-to-PDF' capacity on server side which will be difficult to otherwise replicate.
Option 2
The second option is to further extend the django-wms-client to achieve the following:
- allow the inclusion of multiple WMS services into a single map
- to provide a legend widget where layers can be drag-dropped to reorder them and a checkbox is provided to hide / show objects
This Tim's preferred way as it is generic (e.g. not tied in to QGIS project files).
DISCUSSION OF OPTIONS 1 & 2:
It's hard for CCCS to evaluate the technical difficulties of either of the two approaches outlined above. @pwhipp : Paul made a comparison of these two options over the phone a couple of weeks ago. I cannot recall his specific assessment of each options. I recall some aspect of our discussion that explained how each application was built on top of a different code base: one that was open-source code base and another that was proprietary. Without knowing more, it's hard to evaluate the merits of either approach.
The follow section will present a third potential option that CCCS recently identified. Beyond that, we offer further explication of our usage scenarios and usability preferences. CCCS is disposed to follow Tim's lead and guidance [@timlinux], but ask the team to further consider how CCCS would like to interact with the data and to offer your insights and recommendations concerning the way forward.
Option 3
CCCS recently encountered a software application suite called OpenGeo Suite, ref 1, ref 2. We would appreciate hearing any insights you have into the application. The platform seems to be well suited to meet CCCS' needs.
The application seems to be designed to accommodate a centralized collection of data and to serve out multiple maps that make narrowed, independent presentations of those data to for specific purposes.
The Boundless team actively argue against having over-large mapping application, encouraging instead targeted map application creation and deployment. This fits with CCCS' current workflow [discussion below]. OpenGeo Suite is intended to facilitates such a development and deployment processes—with GeoExplorer replacing QGIS Server and the related presentation technology (either qgis-web-client or django-wms-client). Map applications are initially built using Boundless templates, but these can be modified and extended.
QGIS continues in much the same role, but supplies data to GeoServer—fulfilling the need for a GIPOT (Graphical Information Point of Truth). It appears that QGIS can be used to publish layers and styling to GeoServer, and also that GeoServer can be used as a source of styling data and styling for both QGIS Server (or GeoExplorer, the equivalent web UI for their deployment framework). As such, it appears that GeoServer may be the functional equivalent (and alternative) to GeoGig [though Boundless also maintains GeoGig, so perhaps it's another part of the stack?]. It is unclear how GeoServer preserves and supplies presentation parameters and symbology from QGIS (icons, layer transparency, etc.).
With regard to the "need" (or potential need) for printing multiple *.qgis files:
CCCS often needs to produce reports for client-sponsored research projects. We have been using *.qgis to print maps for these reports. Map printing need to conform to a general template[^1], which includes identification of the map creators, specification of map topic, attribution of data sources, scale, regional view indicators, and general legend. Presentation of web maps will differ from the printed template standards, but it's important to maintain and present appropriate provenience records for data sources.
[NOTE^1: We are actively developing a CCCS template.]
While it came up in discussion of presenting multiple *.qgis files. A point that Tim raised concerning the presentation of maps with QGIS server that confused me. Tim noted that QGIS has a feature allowing one to embed layers and groups from other projects. This feature would allow us to create numerous small projects [each focused on a particular theme (e.g. infrastructure)], which would be nested into projects on an ad hoc basis. Tim write, "With this approach you can 'pick and mix' different thematic elements into a master project and then publish that master project. There is an added benefit in that you have a canonical instance of each layer that propagates its changes into all projects where it is embedded."
In many respects, CCCS approaches our overall process of map creation from the opposite direction. CCCS currently compiles all data into a single "master" qgis file that we use to establish a 'template' color scheme, to assign styles and symbols, to define and populate meta data, etc. This *.qgs file is linked to the same underlying shapefile data as all other *.qgs files produced for a given client to illustrate specific points or to present targted data. To produce customized, issue-oriented maps, we usually start by coping the 'master' file, and then stripping from it any excessive data layers.
CCCS' first priority usage scenario for our online mapping tool is for it to allow our users to click on and off different data layers (and potentially to re-order these layers) and then to "print" their selection of visible layers (as well as any custom style adjustments that we allow, like layer transparency) to a PDF file referencing CCCS' custom-defined templates. We would like for these maps to print at a suitably high level of resolution to allow viewers to zoom in a bit (e.g. to more easily read data labels) without rapid deterioration of image quality. We would also like for these maps to contain appropriate legends / keys. [NOTE: Example template files can be accessed here.] In sum, the purpose of this "master template" [which includes all project layers] is to allow any CCCS user with access to the maps to to create a customized PDF map that is printed into a CCCS template.
Once that sort of a "project master" map is in place, our team could also go about the business of creating and publishing specific maps to illustrate specific points, such as would be embedded within Django as a page-level illustration. We can discuss specific needs and requirements later, and we can add capabilities like map querying, dynamic overlays, layer switchers, etc on an ad hoc basis—ideally we would do this in a generic and re-usable way. CCCS envisions the end results may be something like these examples from the Word Resources Institute (WRI)[]:
OpenGeo Suite is especially intriguing because it is an OpenSource software stack that neatly packages together all the components that we need--though it also means abandoning a lot of our current efforts. Your considered evaluation of the software and recommendations for a way forward would be highly appreciated.
CCCS originally envisioned that the /cccs-web/soc-maps/ repository would be used to assign tasks and raise issues related to development our our map application infrastructure/ The 'soc-maps' project would server as a Django module that could be ported into the /cccs-web/core/ in much the same way as our other application modules (e.g. /cccs-web/doc-meta/).
Currently, however, Paul's development has been occurring under the 'staging' branch of /cccs-web/soc-maps/ and the application development occurring in /cccs-web/soc-maps/ is not linked to our Django deployment framework. CCCS would for future development to be integrated and consolidated within under /soc-maps/ (unless an appropriate alternative is to work through staging; @pwhipp - please advise).
A more in-depth discussion of team integration can be found here.
A discussion on data repository management and version control can be found here). CCCS is awaiting an update from Admire about the work carried out to-date on GeoGig integration of shapefiles and our PostGIS database. CCCS would also like some clarification about the advantages of GeoGig (e.h. the "feature-level diffs" referenced by Tim).