diff --git a/docs/protocol/specification.md b/docs/protocol/specification.md index 56282cb9f..7cde769b1 100644 --- a/docs/protocol/specification.md +++ b/docs/protocol/specification.md @@ -103,7 +103,7 @@ When the client sends a [simple query](#simple-query), the server will respond w ### Pipeline -A pipeline is a type of query that is used to send multiple queries to the server at once, and the server will return the responses in the same order. There is nothing special about the structure of pipelines. Consider [reviewing this section on pipelines](/querying/#pipelines). +A pipeline is a type of query that is used to send multiple queries to the server at once, and the server will return the responses in the same order. There is nothing special about the structure of pipelines. Consider [reviewing this section on pipelines](/querying#pipelines). **⚠️ Illegal packet error escape**: If the client incorrectly encodes a pipeline (even though some of the first queries may be encoded correctly), the server will execute the correctly encoded queries and then instead of sending any further responses it will respond with a `0xFF` byte. This is equivalent to [*Query Error 25*](errors#query-errors). diff --git a/docusaurus.config.js b/docusaurus.config.js index 2009fe1cd..c5d9136bf 100644 --- a/docusaurus.config.js +++ b/docusaurus.config.js @@ -125,7 +125,7 @@ module.exports = { lastVersion: 'current', versions: { current: { - label: '0.8.2', + label: '0.8.1', path: '/', } }, diff --git a/versioned_docs/version-0.8.1/13.libraries.md b/versioned_docs/version-0.8.1/13.libraries.md new file mode 100644 index 000000000..e27772df6 --- /dev/null +++ b/versioned_docs/version-0.8.1/13.libraries.md @@ -0,0 +1,35 @@ +--- +id: libraries +title: Client drivers +--- + + +## Official drivers + +The drivers that are officially maintained can be expected to be feature complete and stable. We currently support the following drivers: + +| Language/Framework | Links | Status | Package manager command | +| ------------------------------ | --------------------------------------------------------------------------------------------------- | -------- | ----------------------- | +| Rust | [GitHub](https://github.com/skytable/client-rust), [crates.io](https://crates.io/crates/skytable) | Stable ✅ | `cargo add skytable` | +| NodeJS (JavaScript/TypeScript) | [GitHub](https://github.com/skytable/client-nodejs), [npm](https://npmjs.com/package/skytable-node) | Beta 🏗 | `npm i skytable-node` | + +:::info More lanugages +The team is always looking to support more languages and we wish we could ship more drivers. But due to limited +resources we haven't been able to. If you're willing to write a driver (it's super easy to), jump into our chat! +::: + +## Community powered + +List of community powered drivers (sorted alphabetically): + +| Language/Framework | Client source code | Upstreamed | License | Remarks | +| -------------------------------- | ------------------------------------------------------------- | ---------- | ---------- | --------------------------------------------------------------------------------- | +| **C#/.NET** | | ❗ | | | +| | [C# driver](https://github.com/martinmolin/skytable-dotnet) | ❗ | Apache-2.0 | Available on [NuGet](https://www.nuget.org/packages/Skytable.Client) | +| **Go** | | ❗ | | | +| | [Go driver](https://github.com/No3371/go-skytable) | ❗ | Apache-2.0 | Written from scratch | +| | [Go driver](https://github.com/satvik007/skytable-go) | ❗ | Apache-2.0 | Easy for migration from go-redis | +| **JavaScript/TypeScript (Node)** | | ❗ | | | +| | [NodeJS Driver](https://github.com/zhangyuannie/skytable.js) | ❗ | Apache-2.0 | Available on [npm](https://www.npmjs.com/package/skytable.js) | +| **PHP** | | ❗ | | | +| | [PHP client driver](https://github.com/hkulekci/skytable-php) | ❗ | MIT | Can be found on [Packagist](https://packagist.org/packages/hkulekci/skytable-php) | diff --git a/versioned_docs/version-0.8.1/14.benchmarking.md b/versioned_docs/version-0.8.1/14.benchmarking.md new file mode 100644 index 000000000..c6d0fe1a3 --- /dev/null +++ b/versioned_docs/version-0.8.1/14.benchmarking.md @@ -0,0 +1,47 @@ +--- +id: benchmarking +title: Benchmarking +--- + +All of Skytable's components are developed from scratch in-house, which makes it hard (if not impossible) to utilize other benchmarking suites. This is why all of Skytable's release bundles come with the `sky-bench` benchmarking tool. It's important to understand how the tool works to understand what you're benchmarking and what the results mean. + +The overall goal with the benchmark tool is to simulate how the database would perform in the real-world and reduce synthetic factors. + +:::info +The ability to simulate different workloads is currently being worked on. This means that across releases, the default workload that the benchmark tool runs may vary. See the release notes to see if the benchmark workload has changed. +::: + +## Benchmark workload + +The workload that the engine currently uses (as of v0.8.1) is the following: +- A model `bench.bench` is created with a primary key (of type `string` and with a column of type `uint8`) +- Multiple clients are created (simulating "application servers") +- Queries are run against **unique rows**. This means that unlike `redis-benchmark` **every query group touches a different row, as it would generally do in the real-world** +- The following queries (collectively a "query group") are run for each unique row: + - The row is first created with an `INSERT` + - All columns of the row are returned with a `SELECT` + - The integer column is incremented with an `UPDATE` + - The row is finally removed with a `DELETE` +- **By default, 1,000,000 rows are created and manipulated** +- **The time taken for each row to be sent, read back and decoded into a readable form is recorded** (time taken to parse into actual language structures such as maps and lists) towards the total time taken, once again unlike many other benchmark tools +- In total 4,000,000 queries are run (by default) + +:::caution +The benchmark tool will create a space `bench` and a model `bench` and will completely remove the space and associated data once the benchmark is complete. **Do not use this space!** +::: + +## Running the benchmark + +Now that you know how the benchmark engine works, go ahead and benchmark for yourself. + +1. Run: + ```sh + sky-bench --password + ``` + **Note**: You can ignore the `--password` argument if you have already set it using the `SKYDB_PASSWORD` environment variable +2. The benchmark engine will run the full workload (as described earlier) +3. Witness 4,000,000 queries being executed in real-time. Good luck and enjoy the results! + +:::tip +You can tune the number of threads, connections, rows created and such to your liking to simulate the environment that you think matches your production setting. +::: diff --git a/versioned_docs/version-0.8.1/15.deployment.md b/versioned_docs/version-0.8.1/15.deployment.md new file mode 100644 index 000000000..b85e92e0f --- /dev/null +++ b/versioned_docs/version-0.8.1/15.deployment.md @@ -0,0 +1,16 @@ +--- +id: deployment +title: Deployment +--- + +Here are some recommendations for deployment: +1. **Make sure you have enough memory and storage!** The server will start returning errors when your server runs out of resources, as you'd expect. +2. **When deploying on docker**: + - Try to map the volume to a local path. We've had unwarranted data losses when we accidentally ended up running a `docker system prune` + - Make sure you have your networking configured right. For example, if you don't map your ports correctly, you'll not be able to access the database outside docker (without running `docker inspect` to find the IP of the container) + - Keep a separate folder for all your docker containers: + - Create a directory in your home directory like: `$HOME/docker-data` + - Then for every Docker image (whether it is Skytable or any other container) create a subfolder in that directory and map + the subfolder as a volume for persistence +3. **Check your firewall**: You want to be very sure about this! If you're starting your database on a different server (as you should; ideally you keep your database server separate from the host running your application server) then make sure that the ports are open and allowed +4. **Set up virtual memory and monitoring**: To avoid exhausting resources, set up monitoring on your node and enable virtual memory to temporarily avoid OOMs diff --git a/versioned_docs/version-0.8.1/16.limits.md b/versioned_docs/version-0.8.1/16.limits.md new file mode 100644 index 000000000..45319e642 --- /dev/null +++ b/versioned_docs/version-0.8.1/16.limits.md @@ -0,0 +1,37 @@ +--- +id: limits +title: Limits +--- + +We've made every effort to provide a robust querying interface, but there are some **temporary limitations** that we think you +should know about. We aim to remove the limitations over the next few releases which we expect should happen fairly quickly. + +Skytable's limitations primarily come from a bunch of concerns: +- **Performance and scalability**: Most of our design decisions are influenced by concerns about performance. For example, it's very hard to efficiently scale multi-column indexes. +- **Reliability**: how reliable is the execution of the task? If it's like walking on eggshells, then we're not going to implement it (for example, unreliable distributed locking) +- **Security**: If it can't be run securely, then it's off our list + +## Temporary limitations + +- **DML in collections are still limited**: You'll be able to easily `INSERT` data into any collections but the manipulations on them are currently limited.: + - `SELECT` will return the entire collection and cannot yet return a single element + - `UPDATE` can append elements to *non-nested* collections but can't do the same for nested collections + - `DELETE` can't remove individual elements +- **Models cannot be `volatile` yet**. If you've used Skytable before, you'd know that you could previously create `volatile` + models which were used as *caching tables* as in they didn't persist data across restarts. The `volatile` feature has been + temporarily removed because we're working on integrating it with the new storage engine. + +:::tip Nested collections will get an upgrade +We understand that collections are fundamental to the complexity of today's data and hence we're working on this! The only reason our team spends so much time *perfecting* features is because it has match our design philosophy. + +> You can have 100 fancy but slow features, or 10 powerful and performant features. + +If it scales, we ship it. **We're on it!** +::: + +## Soft limitations + +Following Skytable's design philosophy that closely encompasses NoSQL systems, the following soft limitations are set: +- **Only one index**: Right now, the only index that you can use is the primary index (primary_key -> row). This is due to concerns about performance and scale +- **No mass updates**: We consider mass updates, such as setting `counter += 1` to every row in a model with multi-million rows +to be very slow and bad for performance. Hence, we do not allow mass updates at this time. diff --git a/versioned_docs/version-0.8.1/2.installation.md b/versioned_docs/version-0.8.1/2.installation.md new file mode 100644 index 000000000..787a3f6a0 --- /dev/null +++ b/versioned_docs/version-0.8.1/2.installation.md @@ -0,0 +1,101 @@ +--- +id: installation +title: Installation +--- + +Getting started with Skytable involves choosing a mode of installation, downloading any required files and then starting up the database. You can choose to either use: + +- [**Native binaries (recommended)**](#native-binaries): This is what is generally recommended for the best performance. You will need to download a bundle and then start the server binary; no expert knowledge required +- [**Using a Debian package (recommended)**](#debian-package): If you're deploying on Ubuntu or any other Debian based Linux distribution, then consider using this method. Configuration files, users and passwords are autogenerated. +- [**A Docker image**](#docker-image): We generally recommend using a Docker image for experimenting with Skytable on your local system during development and you want to keep your local system *clean*. If you want to use a Docker image for deployment, you're always free to do so! + > **Note:** You might experience slightly degraded performance from the storage engine due to Docker engine's abstractions. + +:::tip +All client tools (such as `skysh` and `sky-bench`) *can* use the `SKYDB_PASSWORD` variable for authentication. If you're using Skytable in a testing environment and frequently need to use `skysh`, you may consider setting this variable to your password to avoid having to pass the `--password` argument every time. + +However, we strongly recommend **not** using it outside testing environments. +::: + +## Native binaries + +To use native binaries you need to download a bundle which is simply a ZIP file with all the necessary binaries that you'll ever need to develop on and deploy Skytable. + +1. **First download the latest bundle** for your platform. You can find [download links on the releases page](https://github.com/skytable/skytable/releases). +2. **Unzip the ZIP file**. You'll find the following binaries in the extracted archive: + - `skyd`: This is the database server binary which when started runs as a daemon, serving requests + - `skysh`: This is the Skytable shell and it provides a very helpful interactive REPL database client + - `sky-bench`: This is the benchmarking tool that you can use to load test Skytable +3. **Start up the server**. You need to choose a `root` password for the `root` account which will have complete control over the database. + + ```bash + ./skyd --auth-root-password= + ``` + + **Replace with your own secure password!** + + Explanation: + - `--auth-root-password`: sets the root password + +The server starts up at `localhost:2003` and is ready to run queries. + +:::info +Your operating system might sometimes not let you run binaries directly. On Unix based systems, you'll need to run: `chmod +x skyd skysh sky-bench`. + +And on Windows systems you might need to right-click on the binaries and click on "unblock" +::: + +## Debian package + +Find the correct `*.deb` file [from the releases page](https://github.com/skytable/skytable/releases). Now simply run: + +```sh +sudo dpkg -i .deb +``` + +The package will: + +- **Generate a root password:** Watch the terminal output! +- **Create a `systemd` unit**: So you can start and stop the process using `systemd` like `systemd start skyd` +- **Generate a configuration**: Your configuration is stored in `/var/lib/skytable/config.yaml`. Go ahead and modify it if you need to! + +## Docker image + +:::info You must have docker set up! + +- Use [this great guide from Docker](https://docs.docker.com/engine/install/) to install and get started +- To be able to run `docker run` and related commands, you may need administrative privileges +::: + +### Simple setup + +1. **Download the bundle**: To be able to run queries you need to download the bundle as described above +2. **Start the container**: + + ```shell + docker run -d --name skydb -p 2003:2003 skytable/skytable:latest + ``` + +:::tip +The password for the Skytable instance on the Docker container is auto-generated. Run `docker logs -f skydb` and you'll see a log +message with the generated password. +::: + +### With persistence + +1. **Download the bundle**: To be able to run queries you need to download the bundle as described above +2. **Create the data directory**: To ensure that our database is persistent and all our data doesn't vanish as soon as the container is terminated, we'll map the data directory to an actual directory on our local system. + > **Note:** Create a folder called `skytable` in a convenient location. We recommend having a directory in `$HOME/docker-containers` where you can store the Skytable container's data and any other containers that you might use. It's a great way to keep things organized. +3. **Create your configuration**: [Download this template file](https://raw.githubusercontent.com/skytable/skytable/next/examples/config-files/template.yaml) and place it into the directory you created. Update the password with your `root` password of choice. +4. **Start the container**: + + ```shell + docker run -d --name skydb \ + -v $HOME/docker-containers/skytable:/var/lib/skytable \ + -p 2003:2003 \ + skytable/skytable:latest + ``` + + Explanation: + - This starts a container with name `skydb` + - It maps the folder (as discussed earlier) `$HOME/docker-containers/skytable` from your local file system to `/var/skytable` (in the container's file system) + - Maps port `2003` on the host to the containers port `2003` so that you can use the command-line client `skysh` without having to inspect the container's IP address diff --git a/versioned_docs/version-0.8.1/3.using-repl.md b/versioned_docs/version-0.8.1/3.using-repl.md new file mode 100644 index 000000000..e9d62d456 --- /dev/null +++ b/versioned_docs/version-0.8.1/3.using-repl.md @@ -0,0 +1,96 @@ +--- +id: using-the-repl +title: Using the REPL +--- + +Once you've set up Skytable [following our guide](installation), you can now get started using the REPL. Note that you must have downloaded the bundle. If not, go back to the installation guide and do it now. + +## Starting the repl + +Run this to start the command-line REPL: + +```sh +./skysh +``` + +The REPL will then ask you for a password which you need to type in. This is the root password that you would have set during your installation, so if you don't remember it go ahead and check what you set during the installation step. + +:::tip +Keep the root password safe! It's the admin access to your database and without it you won't be able to create or drop or alter +or ... do anything administrative. But if you do happen to lose it, there's an easy way to reset the password (this however requires you to stop the database, which is also the case with many other databases, and that's primarily for security). + +You can read more in the [configuration](system/configuration) page. +::: + +## Using the REPL + +You will now see a welcome message and the REPL will prompt you to run something. Now is a good time to run `sysctl report status` which should just print out `(Okay)` in a cyan shade: + +```sh +> sysctl report status +(Okay) +``` + +You can also run queries like: `inspect global` to see available global system information. + +:::info Quick notes + +- The REPL currently stores history in the `.sky_history` file. If you want to remove history, you can simply delete the file +- The REPL will automatically parameterize queries. Don't worry about what this means; you'll learn about it ahead. +- The REPL applies custom formatting to `DDL` queries. For example, even though `inspect global` returns a JSON as a string, +the REPL formats it and outputs it without quotes, to improve readability +- To connect using different settings (the REPL will attempt to authenticate as `root` by default), see the options using `skysh --help` +::: + +## First steps + +Skytable's data model is discussed in depth on [this page](architecture#data-model), but let us understand some basics. If you've used a SQL +database, you would be used to the idea of a `database` — just like this, Skytable has `space`s. A `space` is a collection +of `models` (which are like SQL's `table`s with slightly different functionality) and other containers. + +### Create a space + +Let us create a `space`: + +```sql +CREATE SPACE myspace +``` + +### Create a model + +Let us create a `model`. We want to store something that resembles the structure: + +```json +{"username": "string username", "password": "string password", "notes": []} +``` + +To do this, we create the following Skytable model: + +```sql +CREATE MODEL myspace.mymodel(username: string, password: string, notes: list { type: string }) +``` + +### Add, update and remove some data + +- **Insert some data**: + ```sql + INSERT INTO myspace.mymodel('sayan', 'password123', []) + ``` +- **Update some data**: + ```sql + UPDATE myspace.mymodel SET notes += "mynewnote" WHERE username = 'sayan' + ``` +- **Select some data**: + ```sql + SELECT notes, password FROM myspace.mymodel WHERE username = 'sayan' + ``` + +## Use in your own apps + +While we would recommend you to learn BlueQL and more about Skytable's architecture by using the REPL, you can always start using Skytable +in your own apps. [Find a client driver for your language or framework here](libraries). + +Once you've found the driver for your framework, you can come back here and follow on with the guide. Be sure to check the driver's +documentation to see how the client driver should be used. + +Good luck! diff --git a/versioned_docs/version-0.8.1/4.architecture.md b/versioned_docs/version-0.8.1/4.architecture.md new file mode 100644 index 000000000..b61069a86 --- /dev/null +++ b/versioned_docs/version-0.8.1/4.architecture.md @@ -0,0 +1,120 @@ +--- +id: architecture +title: Architecture +--- + +Skytable is a modern NoSQL database that prioritises performance, scalability and reliability while providing a rich and powerful querying interface. We are generally targetting an audience that wants to build high performance, large-scale, low latency applications, such as social networking services, auth services, adtech and such. Skytable is designed to work with +both **structured and semi-structured data**. + +Our goal is to provide you with a powerful and solid foundation for your application with no gimmicks — just a solid core. That's why, every component in Skytable has been engineered from the ground up, from scratch. + +And all of that, without you having to be an expert, and with the least maintenance that you can expect. + +## Fundamental differences from relational systems + +BlueQL kind of looks and feels like using SQL with a relational database but that doesn't make Skytable's internals the same, with the most important distinction being the fact that Skytable has a NoSQL engine! But Skytable's evaluation and execution of queries is fundamentally different from SQL counterparts and even NoSQL engines. Here are some key differences: + +- All DML queries are point queries and **not** range queries: + - This means that they will either return atleast one row or error + - If you intend to do a multi-row query, then it won't work unless you add `ALL`. `ALL` by itself indicates that you're applying (or selecting) a large range and can be inefficient +- Multi-row DML queries are slow and inefficient and are discouraged +- You can **only** query on the primary index, once again because of speed (and the problem with scaling multiple indexes) with a fixed set of operators. +- **Remember, in NoSQL systems we denormalize.** Hence, no `JOIN`s or foreign keys as with many other NoSQL systems +- A different transactional model: + - All DDL and DCL queries are ACID transactions + - However, DML transactions are not ACID and instead are efficiently batched and are automatically made part of a batch + transaction. The engine decides *when* it will execute them, for example based on the pressure on cache. That's because our + focus is to maximize throughput + - All these differences mean that **DDL and DCL transactions are ACID transactions** while **DML queries are ACI and *eventually* D** (we call it a *delayed durability transaction*). This delay however can be adjusted to make sure that the DML + queries *emulate* ACID transactions but that defeats the point of the eventually durable system which aims to heavily increase throughput. + - The idea of eventually durable transactions relies on the idea that hardware failure even though prominent is still rare, + thanks to the extreme hard work that cloud vendors put into reliability engineering. If you plan to run on unreliable hardware, then the delay setting (reliability service) is what you need to change. + - For extremely unreliable hardware on the other hand, we're working on a new storage driver `rtsyncblock` that is expected to be released in Q1'24 +- The transactional engine powering DDL and DCL queries might often choose to demote a transaction to a virtual transaction which makes sure that the transaction is obviously durable but not necessarily actually executed but is eventually executed. If the system crashes, the engine will still be able to actually execute the transaction (even if it crashed halfway) + +:::tip +We believe you can now hopefully see how Skytable's workings are fundamentally different from traditional engines. And, we know +that you might have a lot of questions! If you do, please reach out. We're here to help. +::: + +## Data model + +Just like SQL has `DATABASE`s, Skytable has `SPACE`s which are collections of what we call data containers like tables. +In Skytable lingo, we don't call them tables but call them `MODEL`s which enable you to *define your data model*. + +:::info +While a `MODEL` is the only data container for now, many more are set to come. Now is a good time to [join our discord server](https://discord.gg/QptWFdx) where you can directly chat with the developers working on Skytable and all our awesome community members. +::: + +### A `space` is like a `database` + +A `space` in Skytable is a collection of `model`s and other objects, and settings. It is different from a traditional SQL +Database (that is created with SQL's `CREATE DATABASE`) because it is not meant for tables only, but many other things. + +Spaces have "space local" settings that can be set for the space (and will be respected by all its `models` and other items). You'll learn more about this in the section on DDL. + +### A `model` is like a `table` + +A `model` in Skytable is like a `table` in SQL but is vastly different because of certain concepts such as: +- DML queries are point and not range queries by default +- Restrictions on indexes +- Settings (which can't be created in traditional `table`s) +- Semi-structured data: with collection types in columns such as lists and dictionaries that violates some of the ideas of + traditional schema enforcement + +## Query language + +Skytable has it's own query language BlueQLTM which takes a lot of inspiration from SQL but makes several different (and sometimes vastly different) design choices, focused on clarity, speed, simplicity and most importantly, security. + +For example, Skytable's BlueQLTM *only* allows the parameterization of queries. All the queries you ran previously with strings and numbers directly were only possible because the REPL client smartly does the paramterization behind the scenes. This is done for security. You'll learn more about BlueQL next. + +## Transactions + +Skytable's DDL and DCL queries are all ACID transactions. However, DML queries use an AOF-style storage driver that periodically +records, analyses and then intelligently syncs the changes to disk. We're working on making ACID transactions widely available +across DML queries as well. + +## Storage + +Skytable's storage engine is collectively called the Skytable Disk Storage Subsystem or SDSS for short. The storage engine uses +several different storage drivers, using ones appropriate for the task. We do not use RocksDB or any other engine but we +implement everything in house, engineering them piece by piece. + +:::info Features on track +At this point, Skytable is primarily in-memory which means that while it uses disk storage for durability, most data is stored in-memory. **This is going to change in the near future as the team is working on building a custom log-based engine.** As you might understand, this is not an everyday task and as we incorporate new ideas it will take some time. But if you're seeing this in 2023, you can expect us to ship something by Q2 2024. +::: + +DDL and DCL transactions use a log-based append-only driver while DML queries use a custom log-based append-only driver that is able to intelligently handle concurrent changes. The team will implement new and updated storage drivers from time to time but you do not have to worry about anything, due to our promise for backwards compatibility (see below). + +## Scalability + +Skytable is heavily multithreaded enabling incredible vertical scalability and you can witness it for yourself by running benchmarks (or looking at ones that we publish). Clustering and replication are on track to be released soon. + +:::info Implementations on track +Clustering and replication are right on track and we're expecting to deliver them by May, 2024. We'd also like to note that +clustering is too important to ignore so you can be assured that we're hard at work on it. +::: + +Skytable will use atleast as many threads as the number of logical CPUs present on the host. At this moment it is not possible to limit multithreading because multithreading is a part of Skytable's design principles and every attempt is made to exploit available CPU cores to the fullest. + +## Networking + +Skytable its own in-house Skyhash protocol that is built on top of TCP enabling any programming language that has a TCP client to use it without issues. There are three phases in the connection: +- Handshake: All auth data, compatibility information and other data is exchanged at this step +- Connection mode selection: based on the handshake parameters a connection mode is chosen and the server responds with the chosen exchange mode +- Data exchange: This is where the real querying happens +- Termination: there is no special step; just a `TCP FIN` + +## Backwards compatibility + +We make the promise to you that no matter what changes in Skytable, you will always be able to: +- Upgrade from one version to another without data loss or too many hoops +- Export data from Skytable at any time + +More technically: +- **For minor/patch releases**: The minor/patch is just in the name but it indicates that no data migration effort is needed. **No minor releases ever need data migration, and any migration is done automatically** +- **For major releases**: Major releases generally introduce breaking changes (just like the upgrade from `0.7.x` to `0.8.0` is a largely breaking change). **Major releases will either automatically upgrade the data files or require you to use a migration tool that is shipped with the bundle**. +- Definitions (closely following semantic versioning): + - **A major release** is something like `1.0.0` to `2.0.0` or `0.8.0` to `0.9.0` (in development versions, 0.8.0 to 0.9.0 is a major version bump) + - **A minor release** is something like `1.0.0` to `1.1.0` or `0.8.0` to `0.8.1` + - **A patch release** is something like `1.0.0` to `1.0.1` or `0.8.0` to `0.8.1` (note that in development versions there is no distinction between a minor and patch release) diff --git a/versioned_docs/version-0.8.1/blueql/1.overview.md b/versioned_docs/version-0.8.1/blueql/1.overview.md new file mode 100644 index 000000000..e94745de5 --- /dev/null +++ b/versioned_docs/version-0.8.1/blueql/1.overview.md @@ -0,0 +1,185 @@ +--- +title: Overview +id: overview +--- + +In this document we explore some of the meta parts of BlueQL. If you want to look at how you can use BlueQL, consider looking at the sections that follow. + +Design principles: +- **Simplicity and clarity**: The language shouldn't be overwhelming to understand +- **Security with mandatory parameterization**: We want to reduce the surface of injection attacks. For this reason, [parameterization is mandatory](#parameters). + +Just like SQL, BlueQL has three categories of commands/queries inside it: + +- **DDL**: Data definition language is used to define, modify and/or remove DDL objects such as `space`s and `model`s +- **DCL**: Data control language is used to control the access to data, and perform other administrative tasks +- **DML**: Data manipulation language is used to manipulate data + +Jump to [differences from SQL](#differences-from-sql). + +:::info +This text is *not* a detailed, formal guide. It's meant for developers and users who want to work with +Skytable. If you need a more formal specification, like a grammar definition, please ask us, and we'll create +it. We haven't published it yet because no one has requested it. +::: + +## Identifiers +Can begin with any ASCII alphabet or an underscore (`_`) and then have any number of alphanumeric characters and/or underscores. + +## Keywords + +Keywords are identifiers with special meanings and hence can't be used as identifiers in other places. Here's a full-list of +keywords: + +```ts +[ + "sysctl", "create", "alter", "drop", "use", "inspect", "describe", "insert", "select", "update", + "delete", "exists", "table", "model", "space", "index", "type", "function", "rename", "add", + "remove", "transform", "set", "return", "order", "sort", "group", "limit", "asc", "desc", "all", + "by", "with", "on", "from", "into", "as", "to", "in", "of", "and", "or", "not", "if", "else", + "where", "when", "allow", "auto", "default", "null", "transaction", "batch", "lock", "read", + "write", "begin", "end", "key", "value", "primary", "truncate" +] +``` + +## Data types + +### Boolean +A boolean value, either `true` or `false` + +### Unsigned integers + +- `uint8`: unsigned 8-bit integer +- `uint16`: unsigned 16-bit integer +- `uint32`: unsigned 32-bit integer +- `uint64`: unsigned 64-bit integer + +### Signed integers + +- `sint8`: signed 8-bit integer +- `sint16`: signed 16-bit integer +- `sint32`: signed 32-bit integer +- `sint64`: signed 64-bit integer + +### Floating point values + +- `float32`: a single-precision float +- `float64`: a double-precision float + +### Simple collections + +- `binary`: a binary blob represented as a sequence of `uint8` values +- `string`: an UTF-8 string + +### Complex collections + +- `list`: a list of any of the data types, including nested lists + - A list is represented as: `[]` with values inbetween. For example, a `list { type:string }` would be represented as: + ```sql + ["sayan", "loves", "dogs"] + ``` + - **Lists cannot contain null values** + - **List can be nested**: You can have heavily nested lists like: `[[[]], [["another one"]]]` + - **List can only have one base type**: This means that if you have a list like `[[[string]]]` each element must either be the same nested list, or an empty list + +:::info Note +New data types are frequently added, so treat this list as non-exhaustive. +::: + +## Literals + +- Null literal: `null` +- Numeric literals: + - Unsigned: `1234` + - Signed: `[-]1234` +- Floating point literals: `[-]1234.5678` +- String literals: `"hello"` +- Binary literals: \`binary\` +- List literals: `[..]` +- Dictionaries: `{: }` + +:::warning Literals are not available everywhere +It is very important for you to know that literals are not allowed everywhere. The only literals allowed everywhere are: +- Lists +- Dictionaries + +Read below to understand why. +::: + +## Parameters + +All literals apart from dictionaries and lists must be used as parameters. **BlueQL only allows literals as parameters**. For example, using the Rust client, if you were to run this: + +```sql +insert into myspace.mymodel('sayan', 'pass123', ['myfirstnote']) +``` + +You are required to parameterize the query like this: + +```rust +use skytable::query; + +let query = query!("insert into myspace.mymode(?, ?, [?])", "sayan", "pass123", "myfirstnote") +``` + +If you try to run it without any parameters (don't forget that `skysh` automatically parameterizes for convenience) the query +won't even compile. + +:::tip +Just so you know, parameterizing involves passing a separate parameter list, with each parameter encoded. You wouldn't need to worry about this because the client driver does all of that for you! + +**The question is why? The answer is security.** SQL-injection vulernabilties have already costed companies a lot, so we don't +want to inherit that from SQL. + +**Also, parameterization is exclusively possible for literals**. This means that whenever you're accepting data from an untrusted +source, it becomes a parameter. If you try to not use parameters, the query will not even compile. + +On a final note, BlueQL doesn't support comments of any form also for security reasons. +::: + +## Expressions + +- `+=`: add RHS to LHS + - Can be used outside arithmetic contexts + - Add a char to a field `mystring`: `mystring += ",world"` + - Add a list to a nested list field: `mylist += ["item in nested list"]` +- `-=`: subtract RHS from LHS +- `/=`: divide LHS by RHS +- `*=`: multiply LHS by RHS + +## DDL + +Queries include: +- Spaces: + - `CREATE SPACE myspace [WITH { property: value, ... }]` + - `ALTER SPACE myspace [WITH { property: updated_value, ... }]` + - `DROP SPACE [allow not empty] myspace` +- Models: + - `CREATE MODEL myspace.mymodel([primary] [null] field: field_type, ...) [WITH { property: value, ... }]` + - `ALTER MODEL myspace.mymodel (ADD ... | UPDATE ... | REMOVE ...)` + - `DROP MODEL [allow not empty] myspace.mymodel` +- `INSPECT GLOBAL`: inspects global state, shows a list of spaces and users +- `INSPECT SPACE `: inspects a single space, shows a list of models and other things +- `INSPECT MODEL `: inspects a single model, shows information about stored data and other things + +## DML + +- `INSERT INTO myspace.mymodel(...)` +- `SELECT col1, ... FROM myspace.mymodel WHERE ...` +- `UPDATE myspace.mymodel SET counter += 1 WHERE ...` +- `DELETE FROM myspace.mymode WHERE ...` + +## DCL + +Queries include: +- `SYSCTL REPORT STATUS`: returns the status of the system. (Not a control query per se) +- `SYSCTL CREATE USER "username" WITH { password: ... }`: create a new user +- `SYSCTL DROP USER "username"`: removes the user in question + +## Differences from SQL + +- No literals (see above) +- Mandatory parameterization (see above) +- No semicolons! +- Only one statement per query. For multiple statements batches must be used +- DML queries are point queries (hence must contain a `WHERE` clause) diff --git a/versioned_docs/version-0.8.1/blueql/2.ddl.md b/versioned_docs/version-0.8.1/blueql/2.ddl.md new file mode 100644 index 000000000..a279b2883 --- /dev/null +++ b/versioned_docs/version-0.8.1/blueql/2.ddl.md @@ -0,0 +1,255 @@ +--- +id: ddl +title: DDL +--- + +DDL short for Data Definition Language is used to create, alter and drop DDL objects such as `space`s and `model`s. + +## Generic + +### USE + +- **Syntax**: + - `USE `: sets the space that will be used for all other queries + - `USE null`: unsets the space + - `USE $current`: returns the name of the space currently set **if it still exists** or otherwise returns an error +- **Access control**: any +- **Returns**: empty or error + +:::tip DDL actions don't respect `USE` +This is so that you're specific about what DDL object you want to manipulate. + +Exception: `INSPECT` queries will respect the currently set `SPACE` if required. +::: + +### INSPECT: + +- **Syntax**: + - `INSPECT GLOBAL`: returns a JSON with a list of all spaces, users and other information. For example: + ```json + {"spaces":["space1"],"users":["root"],"settings":{}} + ``` + - `INSPECT SPACE `: returns a JSON with a list of all models in the space and other information, For example: + ```json + {"models":["model1"],"properties":{}} + ``` + - `INSPECT MODEL .`: returns a JSON with information about the model such as the declaration, row count and such: + ```json + {"decl":"{*username: string, !password: string, ?notes: [string]}","size":12345678,"properties":{}} + ``` +- **Access control**: any +- **Returns**: string or error + +## Space + +### CREATE SPACE + +**Syntax:** + +```sql +-- without any properties +CREATE SPACE [IF NOT EXISTS] +-- with properties +CREATE SPACE [IF NOT EXISTS] WITH { property_name: property_value } +``` + +- **Access control**: `root` only +- **Properties**: None +- **Returns**: + - empty or error + - when used with `if not exists`: bool indicating whether the space was actually created or not (false indicates that it was already present and hence not created) + +#### Examples + +```sql +CREATE SPACE apps +CREATE SPACE IF NOT EXISTS apps +``` + +### ALTER SPACE + + +**Syntax:** +```sql +ALTER SPACE WITH { property_name: property_value } +``` + +- **Access control**: `root` only +- **Properties**: None +- **Returns**: empty or error + +#### Examples + +```sql +ALTER SPACE apps WITH { new_cache_value: 1234 } +``` + +### DROP SPACE + +**Syntax:** +```sql +DROP SPACE [IF EXISTS] [ALLOW NOT EMPTY] +``` + +- **Access control**: `root` only +- **Operational notes**: + - **A non-empty space cannot be dropped** + To avoid catastrophic `DROP`s, a `SPACE` can only be dropped directly if it is non-empty. To drop a non-empty space, you must + run: + ```sql + DROP SPACE ALLOW NOT EMPTY + ``` +- **Returns**: + - empty or error + - when used with `if exists`: bool indicating whether the space was actually present or not + +#### Examples + +```sql +DROP SPACE myspace +DROP SPACE ALLOW NOT EMPTY myspace +``` + +## Models + +### Type definitions + +- **When used in create**: + - **A simple type**: `field_name: string` + - **A list**: `field_name: list {type: string}` + - **A nested list**: `field_name: list{type: string}` +- **When used in alter**: + - **A simple type**: `field_name { type: string }` + - **A list**: `field_name { type: list {type: string} }` + - **A nested list**: `field_name {type: list{type:list{type:string}}}` + +### CREATE MODEL + +**Syntax**: +```sql +CREATE MODEL [IF NOT EXISTS] .([primary] [null] field_name: field_type) [ WITH {property_name: property_value} ] +``` +- **Access control**: `root` only +- **Properties**: None +- **Operational notes**: + - The **first field specified automatically becomes the primary key** + - All **fields are not nullable by default** + - To make a field nullable add the `null` keyword before the field name + - If you do not want the first field to become the primary key, then you must add the `primary` keyword before the field that you want to use as the primary key + - The field order is preserved. When doing an `INSERT` you will have to supply values in the same order +- **Returns**: + - empty or error + - when used with `if not exists`: bool indicating whether the model was actually created or not (false indicates that it was already present and hence not created) + +#### Examples + +```sql +CREATE MODEL myspace.mymodel(username: string, password: string, null email_id: string, notes: list { type: string }) +CREATE MODEL IF NOT EXISTS myspace.mymodel(username: string, password: string, null email_id: string, notes: list { type: string }) +``` + +### ALTER MODEL ADD + +**Syntax:** +```sql +-- add a single field +ALTER MODEL . ADD one_field { type: string } +-- add multiple fields +ALTER MODEL . ADD ( + first_field { type: string }, + second_field { type: binary }, +) +``` + +- **Access control**: `root` only +- **Returns**: empty or error + +#### Examples +```sql +ALTER MODEL myspace.mymodel ADD one_field { type: list { type: string } } +ALTER MODEL myspace.mymodel ADD ( + email_id: { + type: string, + nullable: true, + }, + phone_number: { + type: uint64, + nullable: true, + }, +) +``` + +### ALTER MODEL UPDATE + +**Syntax** +```sql +-- update one field +ALTER MODEL . UPDATE one_field { type: string } +-- update multiple fields +ALTER MODEL . UPDATE ( + first_field { type: string }, + second_field { type: binary }, +) +``` + +- **Access control**: `root` only +- **Operational notes** + - The primary key field cannot be modified +- **Returns**: empty or error + +#### Examples + +```sql +ALTER MODEL myspace.mymodel UPDATE my_int_field { type: uint64 } +ALTER MODEL myspace.mymodel UPDATE ( + my_int_field { type: uint64 }, + my_float_field { type: float64 } +) +``` + +### ALTER MODEL REMOVE + +**Syntax**: +```sql +-- remove one field +ALTER MODEL . REMOVE one_field +-- remove multiple fields +ALTER MODEL . REMOVE (first_field, second_field, ...) +``` + +- **Access control**: `root` only +- **Operational notes** + - The primary key field cannot be removed (as is expected) +- **Returns**: empty or error + +#### Examples + +```sql +ALTER MODEL myspace.mymodel REMOVE useless_field +ALTER MODEL myspace.mymodel REMOVE (useless_field_1, useless_field2) +``` + +### DROP MODEL + +**Syntax**: +```sql +DROP MODEL [IF EXISTS] [ALLOW NOT EMPTY] . +``` + +- **Access control**: `root` only +- **Operational notes** + - **Non-empty models cannot be dropped** to avoid catastrophic drops + - **To drop a non-empty model**: you must run: + ```sql + DROP MODEL ALLOW NOT EMPTY . + ``` +- **Returns**: + - empty or error + - when used with `if exists`: bool indicating whether the model was actually present or not + +#### Examples + +```sql +DROP MODEL myspace.mymodel +DROP ALLOW NOT EMPTY myspace.mymodel +``` diff --git a/versioned_docs/version-0.8.1/blueql/3.dml.md b/versioned_docs/version-0.8.1/blueql/3.dml.md new file mode 100644 index 000000000..f1ceb6376 --- /dev/null +++ b/versioned_docs/version-0.8.1/blueql/3.dml.md @@ -0,0 +1,124 @@ +--- +id: dml +title: DML +--- + +DML or Data Manipulation Language is what you'll be using for the most of your queries once you finish defining your data model. DML enables you to insert rows, manipulate the data in them, access their data or delete them. + +:::tip Indexes and performance +NB: Due to performance, at this point only the primary index (aka the primary key column) can be used to run DML queries. This +restriction will be eased in upcoming relases. +::: + +:::info +As you've already been told, don't forget to parameterize. The examples directly use literals to make it easier for you to understand! +::: + +## INSERT + +### Syntax + +```sql +-- insert in declaration order +INSERT INTO (?, ?, ?, ...) +-- insert using field names +INSERT INTO { + field1_name: ?, + field2_name: ?, + ... +} +``` +### Requirements and responses + +- **Access control**: any +- **Returns**: empty or error + +### Examples + +```sql +INSERT INTO mymodel('sayan', true) +INSERT INTO mymodel { + username: 'sayan', + verified: true, +} +``` + +## SELECT + +### Syntax + +```sql +-- select wildcard +SELECT * FROM WHERE = +-- select specific columns +SELECT profile_pic, phone_num FROM WHERE = +-- select several records (wildcard) +SELECT ALL * FROM LIMIT +-- select several records (named columns) +SELECT ALL username FROM LIMIT 100 +``` + +### Requirements and responses + +- **Access control**: any +- **Operational notes**: + - **Unlike SQL**, a standard `SELECT` in BlueQL is a **point query** and hence can only access a single row + - This means that if the row does not match an error is raised + - **To select multiple rows**: you need to use `SELECT ALL`. This ***must*** be accompanied by a `LIMIT` clause. +- **Access control**: any +- **Returns**: row or error + +### Examples + +```sql +SELECT * FROM apps.socialapp WHERE username = 'sayan' +SELECT password, profile_url FROM apps.logins WHERE username = 'sayan' +``` + +## UPDATE + +### Syntax + +```sql +UPDATE SET , ... WHERE = +``` + +### Requirements and responses + +- **Access control**: any +- **Operational notes**: + - **Unlike SQL**, a standard `UPDATE` in BlueQL is a **point query** and hence can only mutate a single row + - Wildcard queries aren't yet permitted. **NB: This will be fixed very soon!** +- **Access control**: any +- **Returns**: row or error + +### Examples + +```sql +UPDATE apps.socialapp SET followers += 1 WHERE username = 'sayan' +UPDATE apps.socialapp SET followers += 200, following += 5 WHERE username = 'sayan' +``` + +## DELETE + +### Syntax + +```sql +DELETE FROM WHERE = +``` + +### Requirements and responses + +- **Access control**: any +- **Operational notes**: + - **Unlike SQL**, a standard `DELETE` in BlueQL is a **point query** and hence can only remove a single row + - This means that if the row does not match an error is raised + - Wildcard queries aren't yet permitted. **NB: This will be fixed very soon!** +- **Access control**: any +- **Returns**: row or error + +### Examples + +```sql +DELETE FROM apps.socialapp WHERE username = 'aboslutely mid user' +``` diff --git a/versioned_docs/version-0.8.1/blueql/4.dcl.md b/versioned_docs/version-0.8.1/blueql/4.dcl.md new file mode 100644 index 000000000..7abe8c834 --- /dev/null +++ b/versioned_docs/version-0.8.1/blueql/4.dcl.md @@ -0,0 +1,51 @@ +--- +id: dcl +title: DCL +--- + +Data Control Language or DCL can be used to perform administrative tasks on the database. Currently, all DCL commands are +available under the `SYSCTL` query. + +## SYSCTL REPORT STATUS + +**Syntax**: +```sql +SYSCTL REPORT STATUS +``` +- **Access control**: any +- **Operational notes**: + - This returns the current overall health of the system +- **Returns**: empty or string or error + +## SYSCTL CREATE USER + +**Syntax**: +```sql +SYSCTL CREATE USER WITH { password: 'password' } +``` +- **Access control**: `root` only +- **Returns**: empty or error + +## SYSCTL ALTER USER + +**Syntax**: +```sql +SYSCTL ALTER USER WITH { password: 'new password' } +``` + +- **Access control**: `root` only +- **Returns**: empty or error + +:::warning +Trying to change the `root` account password will throw an error. You can only change the root by +[following the procedure here](../system/user-management#resetting-the-root-password) +::: + +## SYSCTL DROP USER + +**Syntax**: +```sql +SYSCTL DROP USER +``` +- **Access control**: `root` only +- **Returns**: empty or error diff --git a/versioned_docs/version-0.8.1/blueql/index.md b/versioned_docs/version-0.8.1/blueql/index.md new file mode 100644 index 000000000..1a387c6f5 --- /dev/null +++ b/versioned_docs/version-0.8.1/blueql/index.md @@ -0,0 +1,20 @@ +--- +title: BlueQL +--- + +import DocCardList from '@theme/DocCardList'; + +## Introduction + +BlueQLTM is Skytable's own query language that is based on SQL, with a focus on modern syntax and security. The design attempts to deter injection attacks at development time through the use of mandatory parameterization of variable data, strict LHS-RHS expression enforcement and several other measures. Besides this, BlueQL supports modern syntax such as lists and dictionaries. + +If you've used SQL before you should find BlueQL very easy to learn. There are some important differences to be aware of however, and you may wish to [review them here](/architecture#fundamental-differences-from-relational-systems) and some [more information here](overview#differences-from-sql). + +## Guides + +Here's an overview of the different BlueQL guides in this section: + +- **[Overview](overview)**: Understand BlueQL's design and language items such as parameters, literals, expressions and so forth +- **[DDL](ddl)**: Understand data definition with BlueQL in Skytable +- **[DML](dml)**: Understand data manipulation with BlueQL in Skytable +- **[DCL](dcl)**: Understand data control and administration with BlueQL in Skytable diff --git a/versioned_docs/version-0.8.1/index.md b/versioned_docs/version-0.8.1/index.md new file mode 100644 index 000000000..c02587b2f --- /dev/null +++ b/versioned_docs/version-0.8.1/index.md @@ -0,0 +1,52 @@ +--- +id: index +title: Introduction +sidebar_label: Home +slug: / +--- + +Welcome to Skytable's documentation! Skytable is a free and open-source modern NoSQL database that builds on the foundations of performance, scalability, powerful querying and a robust type system. Skytable can be deployed on multiple platforms as just a single binary file with no special system dependencies. You can either follow the guide below to learn Skytable in depth or jump to any sections that you need for reference. + +## Guide + +To develop using Skytable and maintain your deployment you will want to learn about Skytable's architecture, data organization (such as what `model`s and `space`s are), model definition, querying using BlueQLTM and system administration. We recommend you to follow the guides in the order given below (but feel free to skip any sections): + - [**Installation**](installation) and [**using the CLI**](using-the-repl): Get everything installed on your local system + - [**System overview**](architecture): A brief introduction to Skytable, including an overview of the data model, query systems and storage + - [**BlueQL**](blueql): + - [**Overview**](blueql/overview): Serves as a basic introduction to the query language and an (currently incomplete) informal specification with information on keywords, syntax and stuctures. + - [**DDL**](blueql/ddl): Data definition with BlueQL + - [**DML**](blueql/dml): Data manipulation with BlueQL + - [**DCL**](blueql/dcl): Data control with BlueQL + - [**Querying**](querying): Introduces different query modes and when to choose a specific query mode + - [**System administration**](system): + - [**Configuration**](system/configuration): Information to help you configure Skytable with custom settings such as custom ports, hosts, TLS, and etc. + - [**User management**](system/user-management): Information on access control, user and other administration features + - [**Global management**](system/global-management): Global settings management + - [**Data recovery**](system/recovery): Database recovery + - **Resources**: + - [**Useful links**](resources/useful-links): Links to helpful resources + - [**Migration**](resources/migration): For old our returning Skytable users who are coming from older versions + - [**Benchmarking**](benchmarking): A guide for load testing Skytable + - [**Deployment**](deployment): An useful guide for deploying + - [**Limits**](limits): An useful guide on system limits + +## Client drivers + +Looking to integrate Skytable in your application? Great! [Find a driver for your language/framework here](libraries). + +## Errors + +Hit an error? Check out the [error code index](protocol/errors). If you need further help please check the resources below. + +## Getting help + +We have a collection of resources [on this page](resources/useful-links). If you need in help in real-time, we recommend that you join our [Discord Community](https://discord.gg/QptWFdx) where you can get help directly from the developers and our community members. +Most queries are usually answered there in a few hours! + +## Contributing + +If you find any typos, mistakes or any other scope of improvement - please don't hesitate to bring it up [here](https://github.com/skytable/docs/issues). Thank you ❤️! + +## License + +The documentation is licensed under the [CC-BY-SA-4.0 License](https://github.com/skytable/docs/tree/master/LICENSE) diff --git a/versioned_docs/version-0.8.1/protocol/errors.md b/versioned_docs/version-0.8.1/protocol/errors.md new file mode 100644 index 000000000..39507af03 --- /dev/null +++ b/versioned_docs/version-0.8.1/protocol/errors.md @@ -0,0 +1,58 @@ +--- +id: errors +title: Error Index +--- + +This document provides an exhaustive listing of all the error codes that can be returned by the server. Do note that the [handshake errors](#handshake-errors) are only returned in the *connection stage* and are not the same as the [query errors](#query-errors) which occcur at the *exchange* stage. + +## Handshake errors + +These errors occur **before the connection is sucessfully established**. + +| Code | Meaning | Likely explanation | +| ---- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------- | +| 0 | Illegal packet | Client library bug because it sent an incorrect handshake packet to the server | +| 1 | Incompatible handshake version | Server does not support handshake version used by client | +| 2 | Incompatible protocol version | Server does not support protocol version used by client | +| 3 | Incompatible exchange mode | Server does not support the chosen data exchange mode | +| 4 | Incompatible query mode | Server does not support the chosen query mode | +| 5 | Invalid authentication data | Either an incorrect username or password (or both) or the client library generated an illegal authentication header | + +## Query errors + +These errors can occur **when running a query**. + +| Category | Code | Meaning | Likely explanation | +| ------------ | ---- | ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | +| **System** | | | | +| | 0 | Server error | An error ocurred on the server side. Most likely a problem with I/O on the host | +| | 1 | OOM | The server is out of memory | +| | 2 | Unknown error | An unknown error occurred and the server did not provide any information on why | +| | 3 | Authentication error | Returned when a DCL query tries to do something invalid | +| | 4 | Transaction error | A transactional operation failed. Reasons could include an I/O error, too many retries, long running threshold timeout | +| | 5 | Permission denied | The query tried to do something that isn't allowed for the user | +| | 6 | Illegal client packet | The client sent a malformed query | +| **BlueQL** | | | | +| | 25 | Invalid input | Returned when a literal (or parameter) is incorrectly encoded. **This indicates a bug with the client library** | +| | 26 | Unexpected byte | Illegal character in the query | +| | 27 | Unexpected end of statement | Expected more tokens for the statement | +| | 28 | Invalid syntax | Invalid syntax used in the query | +| | 29 | Invalid collection syntax | A collection was not correctly formatted | +| | 30 | Invalid type definition | Invalid DDL type definition | +| | 31 | Expected entity name | An entity name was expected (such as a `model` name) but was not found | +| | 32 | Expected statement | A statement was expected but was not found | +| | 33 | Unknown statement | Attempted to execute unrecognized statement | +| **Database** | | | | +| | 100 | DDL object not found | The referenced DDL object (space or model or any other DDL object) was not found | +| | 101 | Unknown field | An unknown model field was referenced | +| | 102 | Invalid DDL properties | Invalid DDL properties were used | +| | 103 | DDL object already exists | Trying to create a duplicate DDL object returns this error | +| | 104 | DDL object not empty | A manipulation that requires the DDL object to be empty, was run on a nonempty DDL object | +| | 105 | DDL Illegal field declaration | The field declaration was not correct. Either because the type was invalid or because invalid properties were set. | +| | 106 | DDL illegal model definition | The model definition is incorrect | +| | 107 | DDL illegal `alter model` statement | The `alter model` statement (or any other related statement) is invalid (for example, trying to remove a primary key column) | +| | 108 | DML duplicate constraint violation | A row with the primary key already exists | +| | 109 | DML data validation error | Input data validation failed | +| | 110 | DML unindexed column | The column which was referenced as an index is not indexed | +| | 111 | DML row not found | The referenced row was not found | +| | 112 | DML needs lock | The operation needs a global lock | \ No newline at end of file diff --git a/versioned_docs/version-0.8.1/protocol/index.md b/versioned_docs/version-0.8.1/protocol/index.md new file mode 100644 index 000000000..b1067802a --- /dev/null +++ b/versioned_docs/version-0.8.1/protocol/index.md @@ -0,0 +1,41 @@ +--- +title: 'Skyhash 2: Overview' +--- + +## Introduction + +Skyhash is the protocol used for communication between Skytable clients (client drivers or libraries) and the Skytable server, written on top of TCP. If you're building an application that needs to *talk to* Skytable, you'll usually choose a [client library](/libraries) that works with your programming language and then start building your application. + +However, if a client driver is not available for your language then you might choose to implement a client driver yourself and while it might sound like a complicated task, we've worked hard to make it as easy for for you to do so. If you've worked with string processing, what you're going to encounter is very similar. + +:::tip Looking for error codes? +Did the server return an error code and you're trying to figure out what happened? [**Jump to this page on error codes**](errors). +::: + +:::info New data types are constantly added! +We keep adding more and more data types with every release. So, if you're working to build a client, then consider keeping the type matching non-exhaustive, and if you encounter an unknown signature (that identifies the data type) then throw some sort of an exception indicating that your client does not support this data type yet. +::: + +Here are some good to know things, before a deep dive into the various pieces of the protocol: +- The protocol is based on top of TCP, so if your programming language standard library has a TCP client (as most of them do) you can build a client +- You might find the [official implementation of the Rust client](https://github.com/skytable/client-rust) to be a good reference +- If you need help, ask! (on Discord, ping us on Twitter, just reach out!) +- If the documentation doesn't seem helpful or you find it confusing, open an issue on the docs repository. We'll fix it. + +## Protocol revisions and compatibility + +- **What constitutes an incompatible protocol version?** A protocol version is considered incompatible if and only if: + - the queries sent by an older client can't be processed by the server or + - the **same** responses sent by *the server in question* can't be decoded by the older client +- If newer protocol versions introduce newer data types that is *not* considered an incompatible version since as long as the encoding for the types introduced in the earlier protocol(s) remains unchanged +- Hence, the compatibility code for a protocol only changes when it is considered incompatible + +### Version matrix + +Please note this list is only maintained post 0.8.0. + +| Server version (release tag) | Skyhash Version | Compatibility Code | Notes | +| ---------------------------- | --------------- | ------------------ | ----------------------------------- | +| 0.8.0 | 2.0-beta | `0` | Initial release of Skyhash/2 (beta) | +| 0.8.1 | 2.0-beta | `0` | | +| 0.8.2 | 2.0-beta | `0` | | diff --git a/versioned_docs/version-0.8.1/protocol/specification.md b/versioned_docs/version-0.8.1/protocol/specification.md new file mode 100644 index 000000000..a3cf6acde --- /dev/null +++ b/versioned_docs/version-0.8.1/protocol/specification.md @@ -0,0 +1,129 @@ +--- +id: specification +title: 'Specification' +--- + +# Skyhash 2: Specification + +## Network layer + +As noted earlier, Skyhash is a client/server protocol built on top of TCP, that enables communication between a Skytable database server and a Skytable client. The Skyhash protocol uses a very simple data exchange model with the following three connection stages: + +- Connection stage: + - **Client handshake**: The client sends a handshake packet + - The handshake contains all necessary information to successfully establish a connection + - The structure of the client handshake depends on the authentication plugin in use (since authentication data has to be exchanged before the connection can be established) + - For the `pwd` plugin the client handshake looks like this: + ``` + H0000\n\n + ``` + - For the protocol compatibility code, [see the version matrix](/protocol/#version-matrix) + - Please note that `0` means integer value `0` (`0x00`) and NOT the ASCII digit 0 + - Both the `` and `` are the respective lengths encoded as ASCII strings + - **Server handshake**: + - **Accepted:** If the server accepts the handshake information then it will respond with: `H000` (all the numeric values are integers and NOT ASCII digits) + - **Rejected**: If the server rejects the handshake information then it will respond with `H01<8-bit error code>` (all the numeric values are integers and NOT ASCII digits). You can find out what happened using [the error code index](errors) +- Data exchange stage: This is where the client and server send and receive data. A client and server will spend the majority of their time in this stage. +- Termination stage: Once the connection is no longer needed, the client (or in exceptional cases, the server) will close the connection using a simple TCP FIN. + +:::tip Is my client compatible? +If the server has accepted your connection then you can be sure that the protocol version and other extensions that your client intends to use is supported by the server. If this isn't the case, then the server will respond with a handshake error code indicating why the connection was rejected. [See all the handshake error codes here](errors#handshake-errors). +::: + +### Exchange modes + +The Skyhash/2.0 specification defines the following exchange mode: +- **Query-time**: This works like a request/response action where the client sends a request and the server responds with a response to the query (or an error) + +Expect other exchange modes (bi-directional) to be added in future protocol revisions. + +## Query-time exchange + +In query-time exchange, the protocol works like a "request-response" action where queries are sent by the client and the server sends a repsonse. Queries can be simple queries or pipelines (specification to be finalized[^1]). + +### Client data types + +The client side needs to send encoded data types to the server (in the form of parameters) so that the server can process them. The following types are defined: +- Null: Encoded as `0` +- Bool: Encoded as `1<0 or 1>\n` +- Unsigned integer: A 64-bit unsigned integer. Encoded as `2\n` +- Signed integer: A 64-bit unsigned integer. Encoded as `3\n` +- Float: A 64-bit (double precision) floating point value. Encoded as `4\n` +- Binary: A binary blob. Encoded as `5\n` +- String: An UTF-8 string. Encoded as `6\n` + +> **Note**: A `` is the value of the length in question converted to an ASCII string. + +### Server data types + +The server will respond with different data types, depending on the context. The following types are defined: +- Null: Encoded as `0` +- Bool: encoded as `1<0 or 1>\n` +- Unsigned integers: + - First byte: 2 -> 8-bit, 3 -> 16-bit, 4 -> 32-bit, 5 -> 64-bit + - Payload: `\n` +- Signed integers: + - First byte: 6 -> 8-bit, 7 -> 16-bit, 8 -> 32-bit, 9 -> 64-bit + - Payload: `\n` +- Simple collections: + - First byte: 10 -> binary, 11 -> string + - Payload: `\n` +- Complex collections: + - First byte: 11 -> list + - Payload: `\n` + +> **Note**: A `` is the value of the length in question converted to an ASCII string. + +### Simple query/response + +#### Simple query + +A simple query sends a single BlueQL query to the server to which the server responds with a [simple response](#simple-response). It has three sections: +- The metaframe: + - Contains metadata about the query + - Encoded as: `S\n` (total size of the other two sections) +- The dataframe header: Encoded as `\n` +- The dataframe: + - First part contains the query body, encoded as: `` (the query body is simply appended here) + - Second part contains the payload body with all the parameters, encoded end-on as: `...` (repeat for all parameters). See [the data types for parameters above](#client-data-types). + +#### Simple response + + +When the client sends a [simple query](#simple-query), the server will respond with a simple response using any of the [response data types](#server-data-types), or it can respond with any of the following response structures: + +- **Error**: Encoded as `0x10<16-bit error code>` +- **Row**: The server has returned a row. Encoded as `0x11\n` +- **Empty**: This indicates that the query ran successfully but nothing appropriate can be returned (like HTTP's 200 status). Encoded as `0x12` +- **Multirow**: The server has returned multiple rows. Encoded as `0x13\n` + +> **Note**: A `` or `` is the value of the length in question converted to an ASCII string. + +[^1]: See the [discussion here](https://github.com/skytable/skytable/issues/332) + +### Pipeline + +A pipeline is a type of query that is used to send multiple queries to the server at once, and the server will return the responses in the same order. There is nothing special about the structure of pipelines. + +**⚠️ Illegal packet error escape**: If the client incorrectly encodes a pipeline (even though some of the first queries may be encoded correctly), the server will execute the correctly encoded queries and then instead of sending any further responses it will respond with a `0xFF` byte. This is equivalent to [*Query Error 25*](errors#query-errors). + +#### Pipeline query + +The client is expected to encode the query in the following way: + +``` +P\n +\n\n +\n\n +... +``` + +#### Pipeline response + +The server will return multiple responses: + +``` + + +... +``` diff --git a/versioned_docs/version-0.8.1/querying.md b/versioned_docs/version-0.8.1/querying.md new file mode 100644 index 000000000..31a0147d8 --- /dev/null +++ b/versioned_docs/version-0.8.1/querying.md @@ -0,0 +1,24 @@ +--- +id: querying +title: Querying +--- + +## Simple queries + +A simple query is a type of query in which the client sends one query to the server and the server returns an appropriate response (or an error). There are no explicit guarantees provided about the execution of a simple query, other than what was described earlier in the [section on BlueQL](blueql). DDL and DCL queries are guaranteed to be ACID. DML queries are guaranteed to be atomic but have delayed durability. + +**When should you use simple queries?** The majority of use-cases will involve running simple queries. + +> **Note**: The execution guarantees for queries only apply when the are executed as simple queries. + +:::info +Fully ACID transactions are being worked on and will be available for use soon. +::: + +## Pipelines + +A pipeline is a method of sending multiple queries to the server at once, instead of being sent individually. The server keeps the queries in a queue and then executes them one-by-one in order, writing each response to the connection. **Pipelines do not provide any concurrency and execution guarantees** and should not be used in place of ACID transactions. It is merely a convenience provided to reduce round-trip-times (RTTs) associated with multiple requests to the server. + +**When should you use pipelines?** Pipelines are to be used when you have to run queries that can run *independently* of one another. For example, if you're "banning some users" you could just put all your `UPDATE` queries in a pipeline and send them to the server. The server will serially execute all the `UPDATE`s independently of one another. If someone `DELETE`s their "account" (i.e the row is gone) that's still fine because you were banning them anyway. + +> **Note**: To use pipelines, both your client and server should support pipelines. Pipelines were re-added in 0.8.2 so any client that supports pipelines should be paired with server versions >= 0.8.2 diff --git a/versioned_docs/version-0.8.1/resources/1.useful-links.md b/versioned_docs/version-0.8.1/resources/1.useful-links.md new file mode 100644 index 000000000..0470695a7 --- /dev/null +++ b/versioned_docs/version-0.8.1/resources/1.useful-links.md @@ -0,0 +1,24 @@ +--- +id: useful-links +title: Useful links +--- + +## Community and chat + +- [**Discord Community**](https://discord.gg/QptWFdx): The community platform that we use the most is Discord. Both our developers and users hang out almost all day around here, so you can expect to get your query resolved fairly quickly. +- [**GitHub Discussions**](https://github.com/skytable/skytable/discussions): If you prefer a more forum like discussion, then please make use of GitHub Discussions. We're fairly responsive here as well. + +## News + +- [**Skytable Blog**](https://skytable.io/blog): This is where you'll hear from us about major updates such as new releases, changes in maintenance and such +- [**X/Twitter**](https://twitter.com/skytabledb): On X (formerly Twitter), we post live updates as we work on new features. If there's a quick update or milestone, you'll be the first to know here +- [**E-mail**](https://cdn.forms-content.sg-form.com/e4a9719e-d94e-11eb-89d2-9232e6ded8b5): While we haven't ever sent out any e-mails, if you're keen on signing up for updates in the (near) future, it's a good idea to subscribe to this list. Just like we said, we haven't yet sent out anything! We're building software, not sending you fancy email! + +## Examples + +- [**Jotsy (rust)**](https://github.com/ohsayan/jotsy): Jotsy is a very small "blog" implementation that is written in Rust with Skytable and the Axum framework. If you're building something in Rust, do check it out +- [**Client repo (rust)**](https://github.com/skytable/client-rust): While this is in Rust, you might find the examples useful regardless of which language you're developing your application in + +## Client drivers + +Please head over to the [libraries](/libraries) page. diff --git a/versioned_docs/version-0.8.1/resources/2.migration.md b/versioned_docs/version-0.8.1/resources/2.migration.md new file mode 100644 index 000000000..7986704a3 --- /dev/null +++ b/versioned_docs/version-0.8.1/resources/2.migration.md @@ -0,0 +1,26 @@ +--- +id: migration +title: Migrating from an older release +--- + +Firstly, thank you patron! You're a part of an exclusive *club* which has seen Skytable take shape from *just another key-value* +store into a powerful database and we couldn't be more honored to be a choice in your stack. This document is meant for people +coming from version prior to Skytable Octave (0.8). + +Let's get you up to speed with the new semantics: + +- **Keyspaces are now spaces** +- **Tables are now models** and can store multiple fields +- **Nested lists are now supported** +- **All the actions are gone!** +- **There is no `default:default` keyspace and model**: You will need to create your own `space` and `model` + +## From actions to BlueQL + +- `SET x y` becomes: `INSERT INTO mymodel(x, ?)` +- `GET x` becomes `SELECT * FROM mymodel WHERE = x` +- `UPDATE x y` becomes `UPDATE mymodel SET field_name = ? WHERE = x` +- `DEL X` becomes `DELETE FROM mymodel WHERE = x` + +Essentially, you no longer have the key/value semantics but a more SQL-like interface for querying with BlueQL. You can continue +to use lists (create, update) but individual element access is currently limited. See [this page for more information](/limits). diff --git a/versioned_docs/version-0.8.1/resources/platform_support.md b/versioned_docs/version-0.8.1/resources/platform_support.md new file mode 100644 index 000000000..d36061feb --- /dev/null +++ b/versioned_docs/version-0.8.1/resources/platform_support.md @@ -0,0 +1,39 @@ +--- +id: platform-support +title: Platform Support +--- + +When you're building on or using Skytable on an officially supported platform, you can be certain that we've tested Skytable to work on these platforms, depending on their support level. The support level of a platform largely depends on the testing capabilities of our CI runners and tooling infrastructure and whether they support the platform or not besides our own codebase requirements. As such, we've divided platform support into multiple tiers. + +:::note +Support tiering is subject to change. It is likely that in the near future Skytable will exclusively build for 64-bit platforms and will deprecate or downgrade the support level for 32-bit targets, due to increasing complexity in maintaining 32-bit targets in the same codebase and due to the prevalence of 64-bit systems. +::: + +## Tier 1 + +Tier 1 platforms can be thought of as "guaranteed to work." On such platforms, we regularly run CI tests, regular on-demand stress tests and build and release binaries for. If you find any bugs or run into issues with Skytable on any of these platforms then your request is prioritized relative to platforms that are not Tier-1. + +| Platform | Architecture | Toolchain | +| ------------------ | ------------ | -------------- | +| Linux 64-bit (GNU) | x86_64 | GNU | +| Linux 64-bit (GNU) | arm 64 | GNU | +| macOS 64-bit | x86_64 | Apple (Darwin) | +| Windows 64-bit | x86_64 | MSVC | + +### Tier 1X + +These are just like Tier-1 platforms but due to toolchain limitations we can't yet provide Tier 1 support. We intend to fully upgrade these targets to Tier 1. + +| Platform | Architecture | Toolchain | +| ------------ | ------------ | -------------- | +| macOS 64-bit | arm64 | Apple (Darwin) | + +## Tier 2 + +For these platforms, we run a part of the full test suite but not all tests due to platform limitations and/or toolchain limitations. Binaries are built and released for these platforms, but are very often cross-compiled due to CI infrastructure limitations. + +| Platform | Architecture | Toolchain | +| ------------------- | ------------ | --------- | +| Linux 64-bit (MUSL) | x86_64 | MUSL | +| Linux 32-bit | i686 | GNU | +| Windows 32-bit | i686 | MSVC | diff --git a/versioned_docs/version-0.8.1/system/1.configuration.md b/versioned_docs/version-0.8.1/system/1.configuration.md new file mode 100644 index 000000000..5fe3d2f35 --- /dev/null +++ b/versioned_docs/version-0.8.1/system/1.configuration.md @@ -0,0 +1,96 @@ +--- +id: configuration +title: Configuration +--- + +Skytable can be configured to run per your needs. You can use either of command-line arguments, environment variables or a +configuration file. We don't have any specific recommendation for a mode of configuration but we generally consider using a +configuration file to be the best practice since it is something that won't get lost (which in other cases can get lost due +to lost shell history and such). + +## Configuration file + +The configuration file is a simple YAML file. Use this as your configuration file template: + +```yaml +system: + mode: prod + rs_window: 300 + +auth: + plugin: pwd + # replace with your root password of choice + root_pass: password + +endpoints: + secure: + host: 127.0.0.1 + port: 2004 + # replace `cert` with the path to your self-signed certificate + cert: cert.pem + # replace `private_key` with the path to your private key + private_key: private.key + # replace `passphrase.txt` with the path to your private key passphrase + pkey_passphrase: passphrase.txt + insecure: + host: 127.0.0.1 + port: 2003 +``` + +:::info Starting the server with a configuration file +To start the server with a configuration file, simply run `skyd --config .yaml` +::: + +Here's an explanation of all the keys: +- `system`: + - `mode`: set to either `dev` / `prod` mode. `prod` mode will generally make some things stricters (such as background services) + - `rs_window`: **This is a very important setting!** It is set to `300` by default and is called the "reliability service window" which ensures that if any changes are observed in `300` (or whatever value you set) seconds, then they reach the disk as soon as that time elapses. For example, in the default configuration the system checks for changes every 5 minutes and if there are any dataset changes, they are immediately synced. [Read more here](recovery#understanding-data-loss) +- `auth`: + - `plugin`: this is the authentication plugin. we currently only have `pwd` that is a simple password based authentication system where the password is stored as an [`rcrypt` hash](https://github.com/ohsayan/rcrypt) on disk. More `plugin` options are set to be implemented for more advanced authentication, especially in enterprise settings + - `root_pass`: this is the root account password. **It must have atleast 16 characters** +- `endpoints`: + - `secure`: + - `host`: the host for your secure (TLS) endpoint + - `port`: the port for your secure (TLS) endpoint + - `cert`: path to the PEM certificate to use for TLS + - `private_key`: path to the PEM private key to use for TLS + - `pkey_passphrase`: all private keys must be secured with a passphrase; this key should be set to **the path of a plaintext file containing your passphrase** and **not the passphrase itself** + - `insecure`: + - `host`: the host for your TCP endpoint + - `port`: the port for your TCP endpoint + +## Command-line arguments + +You can use the following command line arguments: + +| Category | Argument | Example | Meaning | +| ------------- | ---------------------- | ------------------------------- | ------------------------------------------------------------------------------ | +| | `--config` | `--config=config.yaml` | Path to the configuration file. | +| **System** | | | | +| | `--service-window` | `--service-window=300` | Set the reliability service window (see above) | +| | `--mode` | `--mode=prod` | Set `dev` or `prod` mode | +| **Auth** | | | | +| | `--auth-plugin` | `--auth-plugin=pwd` | Sets the authentication plugin (see above) | +| | `--auth-root-password` | `--auth-root-password=mypass` | Sets the root account password (see above) | +| **Endpoints** | | | | +| | `--endpoint` | `--endpoint=tcp@localhost:2003` | Repeat this for all your endpoints such as `tls@host:port` and `tcp@host:port` | +| | `--tlscert` | `--tlscert=cert.pem` | Path to TLS cert | +| | `--tlskey` | `--tlskey=private.key` | Path to TLS private key | +| | `--tls-passphrase` | `--tls-passphrase=pass.txt` | Path to plaintext file containing TLS private key password | + +## Environment variables + +You can use the following environment variables: + +| Category | Variable | Example | Meaning | +| ------------- | -------------------------- | ------------------------------------------------- | --------------------------------------------------------------------------- | +| **System** | | | | +| | `SKYDB_SERVICE_WINDOW` | `SKYDB_SERVICE_WINDOW=300` | Sets the reliability service window (see above) | +| **Auth** | | | | +| | `SKYDB_AUTH_PLUGIN` | `SKYDB_AUTH_PLUGIN=pwd` | Sets the authentication plugin | +| | `SKYDB_AUTH_ROOT_PASSWORD` | `SKYDB_AUTH_ROOT_PASSWORD=mypass` | Sets the root account password | +| **Endpoints** | | | | +| | `SKYDB_ENDPOINTS` | `SKYDB_ENDPOINTS=tcp@myhost:2003,tls@myhost:2004` | Sets the endpoints to use | +| | `SKYDB_TLS_CERT` | `SKYDB_TLS_CERT=cert.pem` | Sets the path to the TLS cert | +| | `SKYDB_TLS_KEY` | `SKYDB_TLS_KEY=private.key` | Sets the path to the TLS private key | +| | `SKYDB_TLS_KEY_PASS` | `SKYDB_TLS_KEY_PASS=pass.txt` | Sets the path to the plaintext file containing the TLS private key password | diff --git a/versioned_docs/version-0.8.1/system/2.user-management.md b/versioned_docs/version-0.8.1/system/2.user-management.md new file mode 100644 index 000000000..f2617a4ac --- /dev/null +++ b/versioned_docs/version-0.8.1/system/2.user-management.md @@ -0,0 +1,60 @@ +--- +id: user-management +title: User management +--- + +Skytable's access control is very simple: +- there is only one `root` account +- there are zero or more standard users + +## Types of users + +### Root users + +As noted earlier there can be only one `root` account and this is primarily for security concerns. We however do plan to support +creating multiple users (the implementation isn't hard but security is key). + +#### Exclusive rights + +Root users have exclusive rights to use plus everything that standard users have access to: +- `CREATE` +- `ALTER` +- `DROP` +- `SYSCTL CREATE` +- `SYSCTL DROP` + +#### Resetting the root password + +We strongly recommend that you keep the root password in someplace safe, but if you happen to lose it — just like many other +databases, you will first need to stop the server to reset it. Once you have stopped the server, you will need to modify the root +password that you set in your [configuration source](configuration) (such as CLI/ENV/configuration file). You will then +need to restart the server. The server will issue a warning in the logs that the root password has changed but since that is +what you intended, you can safely ignore it. + +### Standard users + +You can have any number of standard users. Standard users can essentially manipulate data but can't modify the objects that store them. + +#### Creating user accounts + +To create an user account, run the following BlueQL query: + +```sql +SYSCTL CREATE USER WITH { password: 'password' } +``` + +Read more about [creating users here](/blueql/dcl#sysctl-create-user). + +#### Updating and removing user accounts + +To remove or update an account, you can follow the [instructions on BlueQL's DCL guide here](/blueql/dcl). + +#### Rights + +Standard users can access the following query types: +- `SYSCTL REPORT STATUS` +- `INSERT` +- `SELECT` +- `UPDATE` +- `DELETE` +- `INSPECT` diff --git a/versioned_docs/version-0.8.1/system/3.global-management.md b/versioned_docs/version-0.8.1/system/3.global-management.md new file mode 100644 index 000000000..8de014f25 --- /dev/null +++ b/versioned_docs/version-0.8.1/system/3.global-management.md @@ -0,0 +1,33 @@ +--- +id: global-management +title: Global management +--- + +## Checking system health + +The following query returns an `Empty` response or an error code depending on the status of the database: + +```sql +SYSCTL REPORT STATUS +``` + +If you receive an error code, we recommend you to connect to the host and check logs. If the server has crashed, you may need to [recover the database](recovery). + +## Inspecting all spaces + +The single DDL query that lets you do a "sneak peek" into the status of the entire system is the `INSPECT GLOBAL` query. It +returns a JSON string like this: +```json +{ + "spaces:"["space1", "space2"], + "users":["root", "staging_server"], + "settings:{}, +} +``` + +- `spaces`: lists all the spaces in the system +- `users`: lists all users + :::info Access control note + This is only returned if you are the `root` user. Standard users cannot see the other users in the system + ::: +- `settings`: returns system settings (currently an empty dictionary is returned) diff --git a/versioned_docs/version-0.8.1/system/4.recovery.md b/versioned_docs/version-0.8.1/system/4.recovery.md new file mode 100644 index 000000000..c71c6708b --- /dev/null +++ b/versioned_docs/version-0.8.1/system/4.recovery.md @@ -0,0 +1,31 @@ +--- +id: recovery +title: Recovery +--- + +In the unforeseen event that a power failure or other catastrophic system failure causes the database to crash, the Skytable server will fail to start normally. Usually it will exit with a nonzero code and an error message such as "journal-corrupted." In such cases, you will need to recover the journal(s) and/or any other corrupted file(s). + +## Understanding data loss + +All DDL and DCL queries are immediately written to disk when they're run and hence usually no data loss will occur due to a runtime crash (unless a crash occurs in the middle of a disk write). On the other hand, DML queries are written in optimized delayed-durability batches, i.e when the engine determines that either there are too many pending changes or if too much memory is being used (alongside other factors). This however means that in the case of a runtime crash with pending changes, some of these changes may be lost. + +This is why it is so important to tune the [`rs_window`] value or the "Reliability Service" window which ensures that irrespective of the number of changes, all changes will be flushed in that given duration. We're further working on supporting optimized immediate writes for DML queries (which however as expected would come with a significant performance penalty). + +## Recovering database files + +To repair the database, simply run this on the command line **in the working directory of the database**: + +```sh +skyd repair +``` +The recovery system will first create a full backup of the current data files in a subdirectory in the `backups/` directory. It will then go over each database file, try to detect any errors and make any approriate repairs. + +## Important notes + +- The recovery system is *very conservative* and will attempt to restore the database to the most recent working state. Any remaining data is deemed unreliable and not loaded +- Please ensure that you have sufficient disk space before attempting a repair +- The earlier in the file the corruption happens, the greater the amount of data lost + +## Post recovery + +After running a repair operation, if a signficant amount of data loss has occurred (as reported by `skyd`) then we strongly recommend you to manually look through your datasets. The recovery process guarantees that the *restored data* is intact. If this failure resulted from power loss, in the future you may consider installing power backup systems if self-hosting or choosing a reliable cloud provider. diff --git a/versioned_docs/version-0.8.1/system/index.md b/versioned_docs/version-0.8.1/system/index.md new file mode 100644 index 000000000..cb82b341e --- /dev/null +++ b/versioned_docs/version-0.8.1/system/index.md @@ -0,0 +1,14 @@ +--- +title: System administration +--- + +In the following sections, we explore general system administration options with Skytable such as configuration, user management, system health and others. Please note that to use some of these guides you may need direct access to the host on which the database server is running; operations such as recovery and changing the root password require host access. + +## Guides + +Here's an overview of the different administration guides: + +- [**Configuration**](configuration): Understand how Skytable can be configured using command-line arguments, environment variables or a configuration file and what all configuration options are available +- [**User management**](user-management): Learn about account types, permissions and how you can manage multiple users +- [**Global management**](global-management): Learn how to check system health and manage the global state of your database instances +- [**Data recovery**](recovery): Understand what to do after a system crash and how to recover data if needed diff --git a/versioned_docs/version-0.8.1/versions.md b/versioned_docs/version-0.8.1/versions.md new file mode 100644 index 000000000..65e1002b8 --- /dev/null +++ b/versioned_docs/version-0.8.1/versions.md @@ -0,0 +1,34 @@ +--- +id: versions +title: All versions +--- + +# All versions + +A maintained version is one that actively receives security, performance and feature updates. Please see the +[versioning policy](https://github.com/skytable/rfcs/blob/next/rfcs/rfc-1.md) to know which versions are maintained. +As we follow a "ship early, ship often" release cycle, combined with Semver — a _specific version_ isn't +maintained. Instead, the last major version as defined by the versioning policy is maintained. + +## Active versions + +Major version `0.8` is being actively maintained. + +:::tip +Looking for information on compatibility and data migration? Please check the [deployment notes section](deployment). +::: + +## Unmaintained versions + +The following major versions have reached end-of-life (EOL) and will not receive any security, performance or feature +updates: + +- Version 0.7: Including 0.7.1, 0.7.2, 0.7.3, 0.7.4, 0.7.5, 0.7.6, 0.7.7 +- Version 0.6: Including 0.6.0, 0.6.1, 0.6.2, 0.6.3 and 0.6.4 +- Version 0.5: Including 0.5.0, 0.5.1 and 0.5.2 +- Version 0.4: Including 0.4.0, 0.4.1, 0.4.2, 0.4.3 and 0.4.5 +- Version 0.3: Including 0.3.0, 0.3.1 and 0.3.2 +- Version 0.2 +- Version 0.1 + +You can find archived docs by version that have been kept for reference [in this repository](https://github.com/skytable/docs/tree/next/versioned_docs). diff --git a/versioned_sidebars/version-0.8.1-sidebars.json b/versioned_sidebars/version-0.8.1-sidebars.json new file mode 100644 index 000000000..6c801ea0c --- /dev/null +++ b/versioned_sidebars/version-0.8.1-sidebars.json @@ -0,0 +1,61 @@ +{ + "docs": [ + "index", + "installation", + "using-the-repl", + "architecture", + { + "type": "category", + "label": "BlueQL", + "items": [ + "blueql/overview", + "blueql/ddl", + "blueql/dml", + "blueql/dcl" + ], + "collapsed": false, + "link": { + "type": "doc", + "id": "blueql/index" + } + }, + "querying", + { + "type": "category", + "label": "System Administration", + "items": [ + "system/configuration", + "system/user-management", + "system/global-management", + "system/recovery" + ], + "link": { + "type": "doc", + "id": "system/index" + } + }, + "libraries", + { + "type": "category", + "label": "Resources", + "items": [ + "resources/useful-links", + "resources/migration", + "resources/platform-support" + ], + "collapsed": true + }, + { + "type": "category", + "label": "Protocol", + "items": [ + "protocol/index", + "protocol/specification", + "protocol/errors" + ] + }, + "benchmarking", + "deployment", + "limits" + ] +} diff --git a/versions.json b/versions.json index 87ff77e91..fcba54ecc 100644 --- a/versions.json +++ b/versions.json @@ -1,4 +1,5 @@ [ + "0.8.1", "0.8.0", "0.7.5", "0.7.4"