Trash. StaticWebBackend

A static web backend is a module that takes a Python script defining a (possibly interactive) visualization with Vispy, and generates a HTML/Javascript/WebGL document that implements the same visualization. All binary data is included in the document (for instance as base64-encoded strings).

NR: An interesting paper on how to load external data using PNG (image) format and optionally store it locally: http://cg.alexandra.dk/wp-content/uploads/2012/11/optimizing_data_transfer.pdf.

I think that there would be a huge demand for something like this. Python is one of the best languages for data analysis, whereas HTML/Javascript is currently the best platform for sharing interactive documents. Python is definitely not the language you'd choose for sharing stuff. Javascript is definitely not the language you'd choose for analyzing data. Having a system that combines the best of both worlds would be great.

Combined with the IPython notebook, such a system would be extremely powerful and would have a dramatic impact. See the incredible buzz that Jake Vanderplas did with his "quick mpld3 hack" which is becoming less and less a hack (many contributors and pull requests in a few days).

A big advantage of Vispy is that, being based on OpenGL, it can scale to very large datasets (up to a million points in the browser), whereas SVG or canvas-based systems are limited to hundreds to thousands of points.

So, how can we do that?

Python and Javascript being two different languages, we need to define a common dialect that can be understood by both. This dialect can be defined at any level.

The conversion is simpler with a high-level dialect, but there is more work to do on the Javascript side (basically reimplementing entire Vispy in Javascript!). This is not an acceptable solution.

Also, I would strongly suggest we do not restrict ourselves to scientific plotting (like bokeh). Although this is what we are mostly interested in, Vispy can do so much more. Our architecture can be used for so many kinds of applications beyond scientific plotting (3D, maps, animations, demos, games, computer art...). Again, I have d3js in mind: it is much more than just a plotting library. Therefore, our dialect should be more general.

I propose a dialect based on GLOO. This layer is quite interesting, because it is lightweight, simple and flexible.

• On the one hand, we have a bunch of GLOO objects, coming with different kinds of variables and data (essentially numerical data: tuples, NumPy arrays).
• On the other hand, we have a few functions: initGL() (initialization of the objects), paintGL() (what we draw on screen based on these objects) and various event functions (mouse, keyboard interactions). The event functions are triggered by user actions, and basically modify the variables associated to the GLOO objects.

Not all visualizations may be of this form. The paint and event functions can be arbitrarily complex as they're written in Python (they may use NumPy, SciPy or any Python library). We cannot possibly convert any Python function in Javascript (NumPy contains parts of C and FORTRAN code...). Therefore, we'll probably need to restrict the kinds of visualizations we support. This has an important implication on Vispy's architecture, as we'll see below.

Now, what we probably need is a domain-specific language that describes these two things: the GLOO objects, and the few paint and interactive functions. These functions need to be simple enough to support the conversion to Javascript.

The main conclusion is that we probably need to think about this architecture before writing the Vispy high-level layers on top of GLOO. I fear that it will be extremely difficult to write such a backend if we create Vispy with Python in mind only. We want a layer in pure Python that is above GLOO, and that lets us build the rest of Vispy (visuals, transforms, plotting, etc.) on top of it. This layer will probably be quite lightweight.

The role of this layer is to serve as a relay point between GLOO and all the high-level layers. It will register all GLOO objects, all interactions and updates made on these objects during the event functions, and it will generate the paintGL and event functions. Basically, we'll want to have a serializable object that contains the whole interactive visualization (what to display, how to display it, and how to make it interact with the user).

There may be corner cases where it won't be possible to use this layer to implement very complex interactions, in which case the Javascript code generation module won't be available. The user should be free to bypass it.

But the idea is that most things can be done with this system.

Intermediate representation

Here is a proposition about how things could work. We define a sort of language-independent intermediate representation that describes how data is displayed, and how the user interacts with it. This representation is then compiled in a backend language: mainly Javascript, possibly C++ too. The challenge is to have a sufficiently rich representation to allow most of Vispy use-cases (primarily interactive data visualization), but not too rich so that it can be converted to non-Python language without too much hassle.

There are two elements in this representation: objects and functions.

Objects

An object is defined by:

• A globally unique identifier (GUID).
• A type.
• Data associated to the type's fields.

The type is one of the following:

• VBO
• Shader
• Texture
• other GLOO types...
• other user-defined types

Each type comes with a set of fields (attributes) and methods.

There can be custom objects: for example, an object containing data in the original coordinate system (whereas the VBO might contain transformed data). We should restrict the data types to basic types + arrays of basic types.

Functions

There are several functions:

• init (GL function, mandatory)
• paint (GL function, mandatory)
• resize (GL function, mandatory)
• mouse_move
• mouse_click
• timer (for fixed-clock animations)
• other vispy.app events

The three mandatory GL functions describe how the scene is rendered. The other functions describe how the objects are updated following user or automatic events. This information describes with perfect accuracy the entire interactive visualization.

Now the tricky part. Each of these functions needs to be represented in a language-independent way. Yet, in our use-case, it will be provided as a Python function. We need to convert a Python function into an abstract representation that could be compiled in Javascript or C++. Not particularly easy.

With static compilation, we would take the Python code of Vispy and the user script, and we would convert that to an abstract representation that could be easily compiled in other languages. This seems quite unrealistic to achieve.

With Just-In-Time (JIT) compilation, we would execute the Python script (Vispy + user script), and we would capture all GLOO actions at runtime. Those would be compiled into a language-independent intermediate representation. I think this is an interesting direction.

Intermediate language

The language is defined by variables, memory and instructions. The memory contains the objects belonging to the current Context. The objects are collections of elementary variables (numbers, arrays, strings). The functions of our representation are made of elementary instructions.

Instruction set

What can each function do? As far as the three GL functions are concerned, it is most of the time sufficient to allow them to:

• consider an object from its GUID
• call a method (VBO.bind(), Shader.compile(), draw(), etc.)

User actions are harder, because virtually anything could be done (think about a complex video game...). We need to restrict the possibilities. A solution could be to pre-specify the list of functions that can be called here:

• consider an object from its GUID
• retrieve data from a field of this object
• call a method of one of these objects (update part of VBO, update value of uniform, etc.)
• retrieve an argument of the function (with mouse position, key pressed, etc.)
• call one function among those defined in some instruction set

The instruction set could contain just the pure functions we'd need. Plus, it would be easy to add new functions to the set as we add new features/layers to Vispy, and as users use the library. To convert a visualization to Javascript, one would notably need to convert all these functions to Javascript as well. But this would only be done once. Then, any visualization could be converted "for free".

Here are examples of such functions:

• create a new variable
• update an existing variable
• arithmetic operations (Python syntax: +, x, -, /, mod...)
• bitwise operations (Python syntax)
• comparisons (Python syntax: equality, less-than, greater-than...)
• a few mathematical functions (cos, sin, exp...)
• control flow (special syntax: testif(condition, then, else), for loops, etc.)

Basically, we need to create the simplest language possible that allows us to write Vispy.

JIT compilation from Python to the intermediate language

We need a frontend compiler from Python to this intermediate language, and a backend compiler from this intermediate language to Javascript (and possibly C++, much later...). Although we can defer the backend to later, we still need the frontend before we write Vispy. At the very least, we should have a minimal instruction set. We probably need to have a working prototype to ensure that the intermediate language is sufficient.

Let's detail how the frontend compiler works. This compiler will convert all functions (GL functions and user action callbacks) to the intermediate language. At the top, we have Python code on top of GLOO (Vispy, user code). At the bottom, we have code written in the intermediate language.

Since I expect our GL functions to be quite simple, we have nothing particular to do for them. We will write Vispy normally in Python. To convert those functions, we will execute the code, execute these functions by rerouting dynamically all GLOO calls to the compiler. The compiler will track all created GLOO objects, assign GUIDs on the fly, and will register the lists of GLOO functions and methods called. This will only work if there is no complicated control flow in these functions. In other words, the init, paint and resize functions should only do simple things, such as looping through all GL objects, calling their methods, etc. There should not be conditional statements, or loops with an arbitrary number of iterations (that would change after initialization).

User actions are harder to tackle. Here are a few real-world examples:

• Zooming in a plot: in mouse_move, take e.mouse_position, take the exponential of it, and then update the uniform called translate with this value.
• Tooltip text when clicking on a data point: in mouse_click, take e.mouse_position, take the array user_data.transformed, find the index i of the closest point from the mouse position, take user_data.tooltip_text[i], convert that in GL data using some Vispy functions, and update a specific VBO with this data.
• Video game: the sequence of elementary operations performed on the scene when a user action is done can be arbitrarily complicated.

The instruction set could contain low-level operations (such as arithmetic operations, vector operations) and also high-level operations (for example, obtaining GPU data from a text string). There is a trade-off: with only low-level operations, we would have many things to re-implement in Javascript (some of NumPy functionality). With high-level operations, we would have high-level functionality to re-implement (like converting text to GPU data).

How to track these operations? One possibility would be to create a Drop class in Python: we would call the event functions with an instance of these classes instead of an actual number (like an actual mouse position). This function would implement __add__, __leq__ and all the other functions that can override most of the operations of our instruction set. Then, each operation would record it in an internal list.

For control flow, we could not use standard if statements with an object depending (directly or indirectly) on an event object. Rather, we would need to call our own testif(condition, then, else) function on this object. This severely restricts what we can do with event objects. However, sticking to these strong requirements ensures that everything we build on top of Vispy will be automatically convertable to pure Javascript for free!

If we need some complicated algorithm, it may be simpler to simply encapsulate it in a specific function that we add to our instruction set. As long as this function is not converted in Javascript, we won't be able to use the static WebGL backend.

Backend compiler

The Javascript backend compiler will take the intermediate representation, and generate Javascript code. This code will call GLOO functions that will need to be reimplemented (in other words, we'll need to rewrite GLOO in Javascript). Hopefully, this should not be too complicated, as this layer is quite thin. In addition, this Javascript module can be useful on its own.

We will also probably need a very lightweight Javascript module that handles array-like variables, and replicate a tiny portion of NumPy (mainly vectorized computations). There already are Javascript types for that (Typed arrays).

Conclusion

In conclusion, it is critical to think about these issues right now before we write Vispy on top of GLOO. Otherwise, it will be extremely difficult to implement a static WebGL backend after the higher-layer have been developed.

What do we need to do then?

• Design the minimal instruction set we need for now (we can extend it later).
• Write a Drop class that records every operation that is done on the event objects, during user actions.
• Create a sort of hijacked GLOO that does not execute the GLOO operations, but records them carefully. It should also track all GLOO objects that are created.
• Write a battery of tests that define some visualization using functions from a higher-level layer, and then call the GL functions init, paint and resize and the user action callbacks with Drop instances. This will ensure that only operations from our instruction set are being used.
• Later, write a Javascript backend compiler for our intermediate language.

This represents a lot of work, and puts some constraints while we work on pure Python Vispy. However, I think that ending up in an automatic way of generating a static pure HTML/JS WebGL visualization from Python would be extremely valuable. Think about the IPython notebook: you would do some data analysis with Vispy in the notebook, then convert it in HTML with nbconvert or nbviewer. You will continue to see your interactive plots without the need for a Python server. The plot would still be interactive, with any kind of interactivity we could imagine (pan/zoom, tooltips, etc.).

Finally, I should add that this whole approach may be advantageously decoupled from OpenGL visualization, as it is probably much more general. The reason is that the only OpenGL-related stuff is in GLOO, which is already abstracted away in this approach (we just call GLOO through a small set of functions). The same approach could work for non-OpenGL visualization applications. Then, we could have contributors that know nothing about OpenGL.