project.org

Data scientists and programmers in q/kdb+ (kdb+: world’s fastest database; q: functional programming language of APL heritage) can leverage Babel, an existing Emacs framework for literate programming and reproducible research.

ob-q provides Org-Babel support for evaluating q source code within .org documents. q source code blocks in Org Mode can be used to define functions, filter and analyze data, create graphics and figures, and produce reproducible research papers using a style similar to literate programming.

Setting up Babel support for q language

To develop support for q, take a ob-template.el as a starting point and implement integration with q-mode in the prepared ob-q.el file. Inspiration from other languages and guide how to develop support for new languages can be found here: https://orgmode.org/worg/org-contrib/babel/languages.html#fn.1.

Executing source code

Org captures the results of the source code block evaluation and inserts them in the Org file, right after the source code block. https://www.gnu.org/software/emacs/manual/html_node/org/Evaluating-code-blocks.html#Evaluating-code-blocks

If :session argument is not provided, one possible execution strategy is to pipe the string into a fresh invocation of q program. See org-babel-eval and https://stackoverflow.com/questions/10405461/org-babel-new-language

If :results value is provided:

echo -e -n "1+1\na:1\nb:2\na+b" | q

If :results value is provided, append \n to the string so that intermediate results are printed:

echo -e -n "1+1\na:1\nb:2\na+b\n" | q

session-based evaluation

It is possible to run an interactive session as an “inferior process” within Emacs. This means that an environment is created containing data objects that persist between different source code blocks. Babel supports evaluation of code within such sessions with the :session header argument. If the header argument is given a value then that will be used as the name of the session. Thus, it is possible to run separate simultaneous sessions in the same language. Sessions can be used as one way to preserve state accessed by several source codeblocks. In session mode, each block is run in the same long-running q interactive interpreter session, as if you had typed that block into q. You can have multiple sessions, all independent.

The function org-babel-pop-to-session can be used to switch to the process buffer running the session.

If :session argument is provided, perform evaluation using a persistently running inferior process to which the code block is submitted

(*\) 1+til 4

You can name the session if you wish to separate state.

([a:1 2] b:3 4.;c:`d`e)

You can pass the name of existing Q-Shell buffer (without the surrounding asterisks - *) as a session name in order to use it for the session.

1+1

You can pass connection string of remote instance as a session name. If the Q-Shell buffer *qcon-host:port* already exists, it will be used for the session. If it doesn’t exist, it will be created with qcon:

1+1

Exporting source code

The exported document should by default contain \code\ from the code blocks. It’s possible to change this to export just the \results\ of the evaluation, or \both\ the code and results, or \none. https://www.gnu.org/software/emacs/manual/html_node/org/Exporting-code-blocks.html#Exporting-code-blocks

Extracting source code

The q source code from the blocks can be extracted (tangled) into one or multiple q files. https://www.gnu.org/software/emacs/manual/html_node/org/Extracting-source-code.html#Extracting-source-code

[#C] Nice to have:

Show REPL

With the point in a q source block, command to show q instance buffer of the session.

Async execution

Asynchronous execution is supported:

\sleep 3
1+1

Variables and tables input

Support providing variables and tables to code

a	1	2
b	2	3
c	3	4

(x; data)

Error buffer

If evaluated code produces an error, this will be displayed nicely in a buffer using traceback support.

Stdout/err from code evaluation is displayed in a popup buffer. This is great for debugging or getting verbose output that is best left out of documents (e.g progress updates). If you wish to capture output in your document use the :results output SRC block header.

org settings

(add-hook 'org-babel-after-execute-hook 'org-display-inline-images 'append) ;; display/update images in the buffer after I evaluate

If export with the LaTeX backend using the minted package for source block highlighting fails with the error

Error: no lexer for alias ‘q’ found

(add-to-list 'org-latex-minted-langs q)

graphics

any files generated to ob-q-resources-dir, unless :file /path/file.ext is specified