-
Notifications
You must be signed in to change notification settings - Fork 5
web application concept and context
ALERT: CCCS' work to develop an application to publish maps is occurring in two silos. We wish to change this so that CCCS' development efforts [led by @PaulWhipp] is integrated directly with those of Kartoza.
STATUS: Kartoza
Kartoza Pty has provided CCCS with a stack of OpenSource software bundled as a Docker application. This software stack responds to the following CCCS concerns:
-
presenting data for consumption in a web browser;
-
allowing maps to be simultaniously broadcast over the network; this is accomplished by creating parallel software environments that communicate over assigned ports
-
allowing users to load in data from designated repositories; this is accomplished via custom 'import' scipts being authored and maintained by @Admire [!!!]
CCCS has asked to discontinue work with Docker [01 December 2014].
CCCS has been relatively unsuccessful in understaing many aspects for the Docker-based set-up. Firstly, it is unclear to us exactly whatour 'Docker application' "is". The set-up scripts suggest that we're running not one "CCCS" appliocation, but instead a "CCCS-bundle" comprised of a suite of individual Kartoza docker applications, each wrapping a pre-configured application entity (e.g. postgreSQL, QGIS Server. Looking through the scritps, we can see how certain configuration options are being passed into Docker, but it is hard to evaluate and learn from how different application are configured to communicate with one another. For example, Paul has been struggling to indentify how, within docker, the system is configured to pass data manipulated with QGIS to present data layers to the browser. CCCS would like more direct access and exposure to the application system architecture, and to communicate with applications directly rather than via a docker layer.
The aspect of scripting data loading into the Docker-embedded database is also somewhat troublesome from CCCS' perspective. Scriptiong the psql data import means maintaining bundles of import scripts, each of which must be assiduously tracked kept up-to-date in order for the script to work. The is a level of maintainance that we cannot afford in the long term (though using our current scripts for tutorials and examples is fine; these scripts can be periodically updated as long as appropriate cautions declared to end users.)
CCCS has also had some difficulty learning how to interact with data via Docker. We are unsure how to point the Docker application to the specific *.qgis file thhat we wish to server (it is hard coded into the loader?). Do we have to re-load the docker app each time we make a change to the qgis file in the source repo? How is docker referencing a source repo outside of its own container? Via ssh / html paths? What happens when these paths change?
CCCS also asked to cease work with Docker due to concerns for the large size of each Docker app (or, app suite). Each of my Docker repostories is rather large in size. [!!! NOTE !!!] While this is managable from the standpoint of sharing a development environment for web mappers (allowing them to see how the web mapping application presents data), it makes less sense other CCCS purposes, such as serving multiple maps for consumption on the site. With only a few maps online, we'd quickly start incuring unnecssary costs for excessive utilziations of storage space.
In other areas of development...
getting the QGIS server to show in Django website; Kartoza have written a simple django application that will let you do this.
@ismailsunni has joined work on this project and will support you to get it installed and running.
What is your preferred approach for this? I think it will be easiest if we publish the app to pypi and then provide you with the instructions needed to deploy it in your running django project.
@ismailsunni will you please take responsibility for this?
Just to be clear we built it as a complete standalone project mainly for testing, our intention was that just the wms-client app part should actually be deployed in your infrastructure (and again no need to use docker if you do not wish to).
https://github.com/kartoza/django-wms-client
Now that we have the basic mapping services in place, I have co-opted one of our developers, Ismail, (in cc and who hails from Yogyakarta).
Last week we built a little django app that allows you to easily publish your QGIS Server (or any wms based) projects - its available at
https://github.com/kartoza/django-wms-client
Its usable both as a simple standalone project (mainly for testing purposes) or as a django app that you can embed into any other django project (e.g. your mezzanine CMS). I've added an overview of the functionality to the README there:
https://github.com/kartoza/django-wms-client/blob/develop/README.md
Ismail is going to do some work this week to test it with mezzanine integration - we will come back with more details of best practice on how to directly embed maps into your CMS documents.
STATUS: CCCS
CCCS will continiue to develop our software application stack with a python-centric deployment framework. Bundling of software and services will occur using the Django web framework. Our maping software application should be deployable as a Django module.
Paul Whipp has been working to ascertain out the Kartoza software stack and to and re-create it in the context of the Django web framework [see notes above concerning our challengs of the Docker set-up]. From my understanding, Paul's efforts include getting QGIS server to work under nginx a task related to the server's internal configuration.
[Tim's elucidation: QGIS server presents a series of QGIS projects using the standard web services : Web Mapping Service (WMS) and Web Feature Service (WFS). These services can be consumed by various clients including OpenLayers, Leaflet. Within the WFS spec you can ask for available layers, and selectively display all or a subset of layers.
From's his work, Paul's interpretation of the currently-available wms clients for consuming QGIS server [namely
QGIS-Web-Client and django-wms-client] is that each is intended to serve a single *.qgs file. [A brief aside: Paul submitted a pull request to django-wms-client that brings its requirements up to date so that it can be installed more easily. THis includes a bug fix to address issue # 27.]
I need help understanding how I can get it to serve multiple qgs projects without needing a separate server running for each one (I looked at Tim's 2012 blog entry but it does not seem to help me because the server wont serve the other files even if I were prepared to clone the client for each project).
With regards to the hosting of multiple QGIS project files, I have mentioned before that there is no limitation on how many project files you host. Looking over the last messages from CCCS, can I surmise that what you are looking to do is host more than one QGIS project file in the context of the QGIS web client? In that case yes, it is correct the web client is limited to one project per web client instance. This is because the web client is deeply tied into the QGIS project it renders. However in our initial planning the idea was to use a feature of QGIS that allows you to embed layers and groups from other projects to bypass this issue. So in effect we would create numerous small projects, each focused on a particular theme (e.g. infrastructure), and then nest these into projects on an ad hoc basis. With this approach you can 'pick and mix' different thematic elements into a master project and then publish that master project. There is an added benefit in that you have a canonical instance of each layer that propagates its changes into all projects where it is embedded.
I wanted to add that I had not actually thought to use the QGIS web client in your architecture - our initial discussions were centered around using a 'pure' OpenLayers3 or Leaflet implementation with some added niceties to manage layer ordering. This is the route we have been following with the wms-client django app that Ismail and myself have been building. The idea is that you would publish your projects via QGIS server and then register those projects in the wms client django app, which would then deliver them as a leaflet (or openlayers if you prefer) project. The app already does a decent job of service discovery when you point it to a WMS url, though there are plenty of things that can be done to refine it.
====
It remains unclear how we should be integrating our teams' work on map application development. As noted, CCCS had originally envisioned that the /cccs-web/soc-maps/ repository would be used for map application development, with work from the 'soc-maps' repository being ported into the /cccs-web/core/ in much the same way as our other application modules.
Paul's work is currently occurring under the 'staging' branch of /cccs-web/soc-maps/.
I would appreciate if Paul could contribute some clarification for why we chose a different path.
As we are beginning work that involves essentially two endpoints for mapping work, I decided to make a simple graphic showing our understanding of the division between 'public' (CCCS) and private (client) data.
Admire will be back in action later this week and at the beginning of next month and we will work to get the GeoGIG side of things implemented.
So all the pieces of the puzzle are coming together - you will soon have a platform that allows you to easily create, upload, publish and embed your maps into your website. After that is in place we can switch our focus to providing some rich interactions for on the fly queries.
CCCS Web Map Publishing Application—A Discussion of Current Challenges and Possible Future Directions
There are two ways we can proceed here:
Option 1) If you are happy using QGIS Web Client as the platform for your map delivery, we can update the wms-client app to automatically generate and deploy the WMS client configuration files. This is not my preferred approach - the QGIS Web Client is a 'heavy' app essentially trying to replicate a read only experience of the original desktop QGIS project. Though it does have one nice feature in that it supports pdf 'print' generation on the server side which will be difficult to otherwise replicate.
Option 2) We further extend the wms-client to a) allow the inclusion of multiple WMS services into a single map and b) provide a legend widget where layers can be drag-dropped to reorder them and checkbox hidden / show. This is my preferred way as it is generic (e.g. not tied in to QGIS project files).
======
NOTES ON QGIS SOFTWARE STACK
QGIS is a tool that creates interactive map projects. The maps are presented and analysed within the QGIS tool.
Among other functionality, correct.
...
...
OpenLayers (OL) is a javascript appliction that presents data from map services. It can present complex interactive maps in layers. CCCS web maps will be presented using OL3.
Correct. Leaflet performs a similar function too.
...
QGIS and OL are not related but they process similar data. Both deal with aggregations of maps (layers) and their associated styling.
In terms of our architecture, QGIS Server provides a backend service, OL/Leaflet provides a client side front end to that service. They both 'speak' WMS as their interoperability layer.
....
There are three layer aspects of interest:
Features: Vector layer features and attributes These need to be handled using a database so that the features draw on a common CCCS set and projects may extend and modify that set over time. For this we need a single online CCCS POT for features. This POT will be a PostGIS database service. Yes. Though in our proposed architecture you would run two databases: one for public data, one for 'private' data.
...
Styling: Symbology The coloring, associated graphics and annotation of a layer. If developed in QGIS, this information needs to be represented within the PostGIS database service so that it can be transformed for appropriate presentation using OL.
The styling can be store in the database by QGIS, but typically also resides in the .qgs project file. When using the WMS service, the styling defined on the server side is rendered into map images on the server and sent pre-styled to the client. When using the WFS service, the client (e.g. OL3) would be completely responsible for styling. Typically any more complex styling activities would be done server side.
...
Layer Attributes Layer visibility, extent, precedence (for ordering). If developed in QGIS, this information needs to be represented within the PostGIS database service so that it can be transformed for appropriate presentation using OL.
No - these are purely artifacts of the QGIS project file and or the OL/Leaflet client. For example a QGIS project may define layers A, B, C but the WMS client may request them in order A, C, B and they will be rendered as such.
Its best to think of PostGIS as a 'smart data store' and it is not it's responsibility to do any styling of data. QGIS does have the ability to store styling information in the database, but PostGIS isn't intrinsically aware of this (it considers it to be just a stored xml document).
...
To effectively present project data as interactive maps in browsers, CCCS needs to operate at least one map service which will supply the data to browsers running javascript that extends OL.
Correct. In our proposed architecture you would run at least two (one for public, one for private) map services. In the OL/Leaflet client you can combine any number of these services into a single map. I would expect that over time you would actually host a number of QGIS projects, separated thematically or regionally. IMHO it is better to have a number of smaller thematic projects than one massive great big one with everything in it.
...
To prepare the data for presentation, tools such as QGIS will remain very useful. However a pipeline needs to be established to migrate the QGIS features, styling and layer attributes into the form needed to serve via the map service.
QGIS Server does this for us. It also distributes attribute data via the 'getFeatreInfo' WMS API.
...
Legacy QGIS projects have created shapefiles. These will need to be individually handled and imported into PostGIS. New QGIS projects should be able to create PostGIS layers via a CCCS PostGIS service. This PostGIS service will be related to the Map Service making the features immediately available over the map service.
Correct. An update to e.g. an attribute for a kabapaten made on the database would immediately be visible on refresh of the map either in terms of styling (if styling is linked to that attribute) or in terms of the raw attribute data returned to the client.
...
I am not yet sure how/whether it can also make the styling and layer attributes available. This is a possibility which is likely to involve some PyQGIS plugin work.
No, QGIS Server returns rendered layers according to the style rules in the QGIS project.
...
If this sounds good so far, I need to create a working pipeline along with a functional demonstration and supporting documentation. Once that is done, I'll be able to import existing QGIS projects and generate the required OL presentations of the maps.
So our intention with the django-wms-client app is to provide a simple connection point between a map you have published via QGIS Server (again it may be one of many you publish) and your (or any) django application. Taking a look at the screenshots in the README.md I created over here might help to clarify this.
https://github.com/kartoza/django-wms-client/blob/develop/README.md
Ismail is busy squaring up this django app and will post it in PyPi. From there it will be a simpled matter of adding it to your INSTALLED_APPS, doing a django sync / migration and then adding in your maps using the django admin panel.
Once that is in place we would like to come back to you to define further requirements - we can add capabilities like map querying, dynamic overlays, layer switchers, whatever you need basically - ideally we would do this in a generic and re-usable way.
One other thing not touched on above is data versioning. This is where GeoGig comes in (which Admire is now working on as a follow on to his initial work doing data ingestion from shapefiles into PostGIS). So each time your datasets change we should be able to commit that change, much like a geospatial git repository - with feature level diffs.
...
Another challenge that CCCS would like to resolve in the near term is the formalization of an appropriate communication protocol and team data management workflow.
Currently, Kartoza is managing CCCS-related development via a private repository hosted on GitHub. CCCS and Kartoza agreed to this approach to allow Kartoza to use maintain regular use of their waffle.io 'scrum' workflow. CCCS and Kartoza also agreed that CCCS would raise issues related to map application development on /cccs-web/soc-maps/, and that particular stylization and customization requests for our client projects would be managed via their respective, private directories hosted on GitLabs.
For tasking you can file your tickets directly on the soc-maps project and view them here on waffle:
https://waffle.io/cccs-web/soc-maps
I suggest to add as many requirements as you can currently think of into the issue tracker to start with and then we will use normal SCRUM workflows to group the work into sprints and work through the issues assigned to that sprint.
As discussed, we'd like all coding and development work related to extending and improving the application to occur on the CCCS-web GitHub master (https://github.com/cccs-web/soc-maps), and we will cascade this work down to our client project sites, including CCCS-Abadi (http://gitlab.crossculturalconsult.com/abadi/esms-maps).
Since we will need to discuss Abadi-specific issues from time to time, please instruct me about where these discussions should occur. Tim noted that waffle.io doesn't connect to GitLabs, so perhaps the solution is to invite CCCS to join your private branch of the Abadi folk?
Otherwise, we will raise issues to the CCCS-web/soc-maps/ GitHub repository (https://github.com/cccs-web/soc-maps), as previously agreed.
...
Going forward, however, I was considering just developing the mapping application on GitHub. My idea was to keep all development of the application public. To work on the Abadi Project, we would just pull and merge changes from the CCCS website (and public map application) to the private Abadi site, where we would do the work of styling / visualizing their proprietary data. In this way, CCCS could host publicly some of the work (like visualizing population data and public socio-economic indicators) while keeping private data private (like the locations site facilities).
This potentially means a bit of extra headache to pull into (and out of) the Abadi repository, but it would allow us to maintain a robust record of the application development process tied to the CCCS GitHub account.
Unfortunately, this approach has not lent itself to effective communication. The issue we assign on /cccs-web/soc-maps sometimes go unanswered for a long period of time. CCCS is not always aware of the questions being raised on /kartoza/cccs/, and activity there appears to have stalled out as of early November. Our email correspondences appear to sometimes get 'lost in the mix', which appears to have resulted in work being carried out on issues that we asked be put on hold in favor of more pressing concerns.
Going forward, CCCS hopes to channel all our development discussions through /cccs-web/soc-maps for issues related to application development, and to our respective client project repositories (e.g. /abadi/esms-maps/) for client- and project- specific requests.
===
In a previous email, Gavin raised the following:
How would you like to handle data editing, adding new data (like demographics) etc.? I see us establishing the infrastructure and SOPs (standard operating procedures) and facilitating your being able to do it yourselves. Such a workflow might be:
You obtain new data -> use tools incl QGIS and PostGIS to work the data into your local PostGIS database (your 'develop' branch) -> import changes into geogig once you're happy (= git add) -> commit -> push -> merge with 'master' -> export from geogig to PostGIS on the server such that the server database version that is supporting the web maps is always canonical.
And I propose that we have a public geogig repo that maps to the PostGIS public schema and a separate, private one for the abadi schema.
This sound very good to us, thank you. Let's please name both repos according to project, hence something like 'CCCS/soc-maps' (public) and 'Abadi/esms-maps' (private).
=======