Questions about ArcadeDB Data Structures

[Eric24]

I'm a long-time user of OrientDB. I have some questions after reviewing some of the key differences, especially as it relates to data structures:

ArcadeDB saves every type and property name in the dictionary to compress record size by storing only the names ids (as varint)

• Does this also apply to properties inside of an object, or only to "top level" properties in a record?
• Our OrientDB implementation uses a "hybrid" schema, where most of the properties are not part of the schema, specifically because they vary greatly on a per-record basis. If this does apply beyond the "top level" properties in a record, does the possibility of hundreds of thousands of properties names (or more) cause you any concerns? (If it only applies to "top level" properties, I don't see it as an issue.

ArcadeDB allows custom page size per bucket/index

• Since a single record can't currently exceed a page size...
  • What is the maximum practical page size?
  • What are the pros/cons to smaller or larger page sizes?
  • Since it is mentioned in the documentation that this won't be a permanent limitation, where does it fall on the road map?

ArcadeDB Server supports HTTP/JSON, Postgres, MongoDB and Redis protocols, while OrientDB supports only HTTP/JSON and a proprietary binary protocol

• What is the recommended way to access ArcadeDB from node.js?

It's not mentioned in the documentation...

• OrientDB creates a permanent "tombstone" in the database when a record is deleted (with no practical way to ever recover the space needed by the tombstone. Does ArcadeDB do this?
• Where is the time series model on the road map?

[lvca]

I'm a long-time user of OrientDB. I have some questions after reviewing some of the key differences, especially as it relates to data structures:

Hi @Eric24,
Glad to see you here.

ArcadeDB saves every type and property name in the dictionary to compress record size by storing only the names ids (as varint)

• Does this also apply to properties inside of an object, or only to "top level" properties in a record?

All of them, at any level.

• Our OrientDB implementation uses a "hybrid" schema, where most of the properties are not part of the schema, specifically because they vary greatly on a per-record basis. If this does apply beyond the "top level" properties in a record, does the possibility of hundreds of thousands of properties names (or more) cause you any concerns? (If it only applies to "top level" properties, I don't see it as an issue.

The dictionary works only to compress the property names. It doesn't matter if you use the same property name on different documents and with different types. Storing the name id as a var int (1-N bytes) instead of the property name as a string saves a huge amount of space.

ArcadeDB allows custom page size per bucket/index

• Since a single record can't currently exceed a page size...

  • What is the maximum practical page size?
  • What are the pros/cons to smaller or larger page sizes?
  • Since it is mentioned in the documentation that this won't be a permanent limitation, where does it fall on the road map?

This is already in our roadmap, but we haven't started yet because of other priorities from users that are sponsoring the project (through GitHub Sponsors or contracts). In terms of new features we're completing a couple of things and then there will be the support for GraphQL before the multi-page record split feature.

Anyway, now you can just set the page size to a big number. By default is 64K. If you have records bigger than 60K I suggest you to increase this number when you create the bucket. We have tested in production for a couple of users pages up to 1MB without problems.

Why 64K is the default? It's the best number in terms of speed and space occupied for most of the use cases.

ArcadeDB Server supports HTTP/JSON, Postgres, MongoDB and Redis protocols, while OrientDB supports only HTTP/JSON and a proprietary binary protocol

• What is the recommended way to access ArcadeDB from node.js?

It's not mentioned in the documentation...

it depends on your requirements. For basic CRUD operations, you could even use a Redis driver. If you need something more complicated I suggest HTTP/JSON, the API is very simple. You do 99% with SQL commands.

• OrientDB creates a permanent "tombstone" in the database when a record is deleted (with no practical way to ever recover the space needed by the tombstone. Does ArcadeDB do this?

ArcadeDB is also using a similar technique, but it's much more efficient with the space. The same database imported from OrientDB (use the OrientDB importer) usually is 1/3 or less of the original size with OrientDB. If your use case has tons of deletes I'd suggest to:

1. recycle the records: not delete them but mark them as "available" with a property in the document/vertex. Of course, you should unlink it first from any reference
2. if you have a use case like time-series where you store points/data and you want to drop it after a while, the best way is to create one bucker per month/week/day/hour based on the amount of data you store. In this way, it's easy to get rid of an entire period by just dropping the bucket without any "holes".

• Where is the time series model on the road map?

We have users in Fintech that are already in production with a time-series approach (soon a case study on our website) by using a graph to store time and embedding small units in embedded documents. We're analyzing a way to make this generic in order to be used by everybody without creating such structures from scratch.

ETA? >6 months

[lvca]

We're close to starting the official development of the Time Series module after a lot of thoughts and requirements form users. You can followup here: #1180.