Experimental port without numpy

[shuckc]

I have been maintaining an alternative kdb-ipc layer for python for a while now, aiokdb at https://github.com/TeaEngineering/aiokdb
It doesn't yet have support for compression, and it has no dependency on pandas or numpy.

I recently switched to an M1 arm Mac, and building numpy for Sublime's py3.3 seemed relatively difficult, so I back-ported my kdb library to py3.3 and then ported sublime-q over to use it. The result is https://github.com/shuckc/sublime-q - you can git clone this (with submodules) into the Sublime plugin folder and it should work.

There are a couple of rough edges - code completion for instance where my API doesn't match Excelleron's that I will fix up, but it's better than nothing.

[komsit37]

I like the idea of ditching numpy. It's a pain to build for different architectures.

I have a concern on not supporting compression. Do you know much bigger the data size would be without compression?

[shuckc]

OK great, I will have a think about how to support py33 without a fork of the aiokdb library.

with regards compression - it's worse than that. If the server enables compression on a particular message, and you don't support it, that message will be undecipherable garbage (I raise an Exception in the decoding code). The decompression does not look difficult to implement - here's what you have already and this is where it needs to be implemented, but I cannot find one single test case for it in any of the public implementations. It also selectively enabled which makes it tricky to reproduce locally, e.g. following https://code.kx.com/q/basics/ipc/#compression compression is only enabled when:

• Uncompressed serialized data has a length greater than 2000 bytes
• Connection is not localhost
• Connection is not 127.0.0.1
• Connection is not using UDS (Unix Domain Socket)
• Connection does not resolve to being localhost (since 4.1t 2021.06.04)
• Size of compressed data is less than ½ the size of uncompressed data

[shuckc]

Implementing compression turned out to be fairly simple and I managed to capture some compressed payloads for test cases.

I found that Sublime builds 4050 and above ship with a dual python 3.3 and 3.8 interpreter, that plugins can opt-in to use. See https://www.sublimetext.com/docs/api_environments.html#selecting-the-python-version for the details. My test suite now builds cleanly with python 3.8 so there is now a fair chance that it can be used as-is from master. I will take a look at fixing up the remaining issues.