|
1 | | -# duckdb-sql-tools |
2 | | -DuckDB SQL Tools |
| 1 | +# DuckDB SQL Tools |
| 2 | + |
| 3 | +[](https://marketplace.visualstudio.com/items?itemName=RandomFractalsInc.duckdb-sql-tools) |
| 4 | +[](https://marketplace.visualstudio.com/items?itemName=RandomFractalsInc.duckdb-sql-tools) |
| 5 | +[](https://marketplace.visualstudio.com/items?itemName=RandomFractalsInc.duckdb-sql-tools) |
| 6 | +<a href='https://ko-fi.com/F1F812DLR' target='_blank' title='support: https://ko-fi.com/dataPixy'> |
| 7 | + <img height='24' style='border:0px;height:20px;' src='https://az743702.vo.msecnd.net/cdn/kofi3.png?v=2' alt='https://ko-fi.com/dataPixy' /></a> |
| 8 | + |
| 9 | +[DuckDB SQL Tools](https://marketplace.visualstudio.com/items?itemName=RandomFractalsInc.duckdb-sql-tools) extension enables using [DuckDB](https://duckdb.org/) in [VSCode](https://code.visualstudio.com/) IDE with the popular [SQLTools](https://vscode-sqltools.mteixeira.dev/en/home/) VSCode extension and SQL query editing and processing tools. |
| 10 | + |
| 11 | + |
| 12 | + |
| 13 | +# Features |
| 14 | + |
| 15 | +[DuckDB](https://duckdb.org/docs/) is an in-process SQL [OLAP](https://en.wikipedia.org/wiki/Online_analytical_processing) database management system that uses vecotrized data engine for optimized analytics and parallel query processing with extensive SQL support and direct `Parquet` and `CSV` querying capabilites. |
| 16 | + |
| 17 | +[DuckDB SQL Tools](https://marketplace.visualstudio.com/items?itemName=RandomFractalsInc.duckdb-sql-tools) extension adds the following capabilities to VSCode IDE to work with DuckDB instances: |
| 18 | + |
| 19 | +- Connect to a local DuckDB instance |
| 20 | +- Create in-memory DuckDB instance |
| 21 | +- View DuckDB tables, columns, and views |
| 22 | +- Run SQL queries on open DuckDB instances |
| 23 | +- Attach SQLite database files to in-memory DuckDB instances |
| 24 | +- Query remote `CSV` and `Parquet` data files using [DuckDB HTTPFS](https://duckdb.org/docs/extensions/httpfs.html) extension and create in-memory DuckDB tables from remote data results |
| 25 | +- Create named SQL query Bookmarks |
| 26 | +- View SQL Query History |
| 27 | + |
| 28 | +See [SQLTools documentation](https://vscode-sqltools.mteixeira.dev/en/home/) for a comprehensive list of SQLTools extension features contributed to VSCode IDE. |
| 29 | + |
| 30 | +# Installation |
| 31 | + |
| 32 | +Install [DuckDB SQL Tools](https://marketplace.visualstudio.com/items?itemName=RandomFractalsInc.duckdb-sql-tools) extension from [VSCode marketplace](https://marketplace.visualstudio.com/items?itemName=RandomFractalsInc.duckdb-sql-tools) in your browser by clicking on **Install** button in the extension info page header. |
| 33 | + |
| 34 | +Alternatively, you can install this extension directly in VSCode IDE from Extensions tab (`ctrl+shift+x`) by searching for `DuckDB`. |
| 35 | + |
| 36 | + |
| 37 | + |
| 38 | +Users of [VSCodium](https://vscodium.com/) and other VSCode-based IDEs can install DuckDB SQL Tools extension using `.vsix` extension package attached to the Assets section in published [releases](https://github.com/RandomFractals/duckdb-sql-tools/releases) of this extension on Github. Follow [install from .vsix](https://code.visualstudio.com/docs/editor/extension-marketplace#_install-from-a-vsix) instructions in your VSCode extensions compatible IDE or online container service to install it. |
| 39 | + |
| 40 | +# DuckDB Extensions |
| 41 | + |
| 42 | +DuckDB egnine also provides a number of [Extensions](https://duckdb.org/docs/extensions/overview) you can install and load to work with remote `CSV` and `Parquet` data files over [HTTPFS](https://duckdb.org/docs/extensions/httpfs), enable [Full Text Search](https://duckdb.org/docs/extensions/full_text_search), attach SQLite database with [SQLite Scanner](https://duckdb.org/docs/extensions/sqlite_scanner), or attach Postgres database instance with [Postgres Scanner](https://duckdb.org/docs/extensions/postgres_scanner). |
| 43 | + |
| 44 | +You can check the list of core and installed DuckDB extensions by running the following SQL query on an open database instance: |
| 45 | + |
| 46 | +```sql |
| 47 | +select * from duckdb_extensions(); |
| 48 | +``` |
| 49 | + |
| 50 | + |
| 51 | + |
| 52 | +## DuckDB HTTPFS |
| 53 | + |
| 54 | +DuckDB SQL Tools VSCode extension installs and loads [HTTPFS](https://duckdb.org/docs/extensions/httpfs) DuckDB extension by default for all the open DuckDB instances. You can add other [DuckDB Extensions](https://duckdb.org/docs/extensions/overview) to an open database instance by using [`INSTALL` and `LOAD`](https://duckdb.org/docs/extensions/overview#remote-installation) extension SQL statements. |
| 55 | + |
| 56 | +Bellow is an example of loading reported Chicago crimes data into in-memory DuckDB instance from a `.parquet` data file hosted in our [Chicago Crimes](https://github.com/RandomFractals/chicago-crimes) data and analytical tools demo Github repository. This example uses implicitly loaded DuckDB HTTPFS extension to query reported Chicago crimes parquet data file with over 210K recorded crime reports, creates a `CrimeReports` table in DuckDB `:memory:` instance from loaded parquet data results, and queries imported data: |
| 57 | + |
| 58 | + |
| 59 | + |
| 60 | +## DuckDB SQLite Scanner |
| 61 | + |
| 62 | +Use the following SQL statements to add [SQLite Scanner](https://duckdb.org/docs/extensions/sqlite_scanner) extension to in-memory DuckDB instance: |
| 63 | + |
| 64 | +```sql |
| 65 | +INSTALL sqlite; |
| 66 | +LOAD sqlite; |
| 67 | +``` |
| 68 | + |
| 69 | +With the loaded SQLite Scanner extension you can attach SQLite database to DuckDB database instance. Attached SQLite database tables will show up as views in DuckDB instance. |
| 70 | + |
| 71 | +Use the following SQL `CALL` function to attach SQLite database instance: |
| 72 | + |
| 73 | +```sql |
| 74 | +CALL sqlite_attach('E:\\projects\\data\\tools\\duckdb-tools\\data\\chinook\\sqlite\\chinook.sqlite'); |
| 75 | +``` |
| 76 | + |
| 77 | + |
| 78 | + |
| 79 | +## DuckDB File References |
| 80 | + |
| 81 | +**Note**: DuckDB SQL Tools extension uses [DuckDB NodeJS Client API](https://duckdb.org/docs/api/nodejs/overview). In order to work with local data files, you need to specify full path to your local database or data files in SQL statements that reference local file paths. |
| 82 | + |
| 83 | +Future versions of this extension might simplify local file path references by deducing absolute file path form the local DuckDB file connection string or open VSCode project workspace URI and replace relatIve database or data file references with the corresponding absolute path in an open VSCode project Workspace. |
| 84 | + |
| 85 | +Also, note in the `sqlite_attach` SQL call statement above we are escaping `\` file path delimiters on Windows OS by using `\\` characters sequence. |
| 86 | + |
| 87 | +# Demo Data |
| 88 | + |
| 89 | +DuckDB SQL Tools extension github repository contains sample [/data](https://github.com/RandomFractals/sql-duckdb-tools/tree/main/data/chinook/duckdb) folder with `chinook.duckdb` and `chinook.sqlite` database files, csv, json and parquet data files, and some test SQL queries you can try running on this well-known sample database wih our VSCode extension. |
| 90 | + |
| 91 | + |
| 92 | + |
| 93 | +# Limitations |
| 94 | + |
| 95 | +Due to the limited time and minimal development effort invested into building this free trial DuckDB SQL Tools extension, our first Preview alpha version of this extension comes with the following known limitations and supported usage scenarios. |
| 96 | + |
| 97 | +## DuckDB Storage |
| 98 | + |
| 99 | +Preview release of this extension supports only local database instances created using [DuckDB v0.6.1](https://github.com/duckdb/duckdb/releases) engine. Database instances and files created with prior versions of DuckDB are not supported as they use different compression and storage formats and the structure of `.duckdb` file has been changing as DuckDB engine is evolving. |
| 100 | + |
| 101 | +Use [DuckDB CLI](https://duckdb.org/docs/api/cli.html) to export data from the older database file versions and create new `.duckdb` file using the latest DuckDB storage implemenation. Read [Announcing DuckDB 0.6.0](https://duckdb.org/2022/11/14/announcing-duckdb-060.html) blog post for more information about DuckDB storage improvements. |
| 102 | + |
| 103 | +## Read-Only DuckDB |
| 104 | + |
| 105 | +Additionaly, our first release opens `.duckdb` database files in `read-only` mode for the time being to ensure other data processing tools and [DuckDB CLI](https://duckdb.org/docs/api/cli) can open and update the same `.duckdb` file as DuckDB supports only **one** active `write` connection in its latest incarnation. |
| 106 | + |
| 107 | +Future paid Premium and/or Pro versions of this extension might provide write capabilities and better integration with DuckDB CLI in VSCode Terminal to our [Github Pro Sponsors](https://github.com/sponsors/RandomFractals/sponsorships?sponsor=RandomFractals&tier_id=18884), depending on the popularity of this extension in VSCode marketplace, user feedback and requests you can post in our [DuckDB SQL Tools Github Discussions](https://github.com/RandomFractals/duckdb-sql-tools/discussions), and users willing to pay for the development of **Premium DuckDB Tools** and additional **Pro** features. |
| 108 | + |
| 109 | +## In-Memory DuckDB |
| 110 | + |
| 111 | +Meanwhile, you can experiment with writable `:memory:` DuckDB instances in this DuckDB SQL Tools Preview extension version. In-Memory DuckDB instances function similar to [In-Memory SQLite Databases](https://www.sqlite.org/inmemorydb.html). You can use [DuckDB Import Data](https://duckdb.org/docs/data/overview) and [Atach](https://duckdb.org/docs/sql/statements/attach) features available via DuckDB SQL statments, and use [Export Database](https://duckdb.org/docs/sql/statements/export) SQL statements to export created in-memory DuckDB instances. |
| 112 | + |
| 113 | +DuckDB SQL Tools extension lets you create in-memory database instances by specifying `:memory:` in the Database File field of the new DuckDB connection in SQLTools Connection Assistant as demonstrated in DuckDB HTTPFS and SQLite Scanner extension usage examples above. |
| 114 | + |
| 115 | +**Note**: **only** in-memory DuckDB database instances are open in `read/write` mode in this DuckDB SQL Tools extension Preview release to avoid DuckDB instance file access locking. |
| 116 | + |
| 117 | +## VSCode Memory Limit |
| 118 | + |
| 119 | +You can adjust the amount of RAM allocated to VSCode IDE to enable opening large files and load more data into memory. Go to `File -> Preferences -> Settings` and type `files.maxMemoryForLargeFilesMB` in the Setting search field to change it. For example, users with 64GB of RAM can change it to `49152` MB to allow VSCode use 48GB of available memory. |
| 120 | + |
| 121 | + |
| 122 | + |
| 123 | +# Configuration |
| 124 | + |
| 125 | +[SQLTools](https://marketplace.visualstudio.com/items?itemName=mtxr.sqltools) extension provides many configuration [Settings](https://vscode-sqltools.mteixeira.dev/en/settings/) users can toggle to change database connection and tree view display options, sql formatting, and results display. |
| 126 | + |
| 127 | +The following [SQLTools Settings](https://vscode-sqltools.mteixeira.dev/en/settings/) were used while creating and testing this DuckDB SQL Tools extension and are recommended for working with DuckDB instances efficiently. We suggest you set these preferences in User Settings in VSCode by navigating to `File -> Preferences -> Settings -> User -> Extensions -> SQLTools`, or adding them to your global VSCode `settings.json` config file using the JSON code snippet below: |
| 128 | + |
| 129 | +```json |
| 130 | +{ |
| 131 | + ... |
| 132 | + "sqltools.useNodeRuntime": true, |
| 133 | + "sqltools.disableNodeDetectNotifications": true, |
| 134 | + "sqltools.autoOpenSessionFiles": false, |
| 135 | + "sqltools.results.limit": 10000, |
| 136 | + "sqltools.results.location": "current", |
| 137 | +} |
| 138 | +``` |
| 139 | + |
| 140 | +| Setting | Description | |
| 141 | +| --- | --- | |
| 142 | +| ```"sqltools.useNodeRuntime": true``` | Enable Node runtime in order to use [DuckDB NodeJS API](https://github.com/duckdb/duckdb/tree/master/tools/nodejs) this database SQL tools extension depends on. | |
| 143 | +| ```"sqltools.disableNodeDetectNotifications": true``` | Disable Node runtime detection notifications after initial SQLTools extension installation to prevent Node runtime information message display on every new VSCode session start. | |
| 144 | +| ```"sqltools.autoOpenSessionFiles": false``` | Prevent auto open of new session SQL editor instance after connecting to the databaase instance. | |
| 145 | +| ```"sqltools.results.limit": 10000``` | Maximum number of records to return in results. SQLTools defaults to displaying only 50 records in query results view. Changing this limit setting to 1000 or 10000 will show more data rows to inspect in result views. | |
| 146 | +| ```"sqltools.results.location": "current"``` | Defines the edigor group to use for result table views. SQLTools displays all results in the next editor group to show results on the side next to the active SQL query editor. Changing this setting to `current` will display results in the same editor group and display more result colulmns. | |
| 147 | + |
| 148 | +# Feedback |
| 149 | + |
| 150 | +Use [DuckDB SQL Tools Github Discussions](https://github.com/RandomFractals/duckdb-sql-tools/discussions) portal to submit your feedback, share examples of how you are using this VSCode extension, or request new trivial and premium features. Our aim is to make DuckDB more accessible and easier to use in VSCode IDE and Terminal. |
| 151 | + |
| 152 | +# Support |
| 153 | + |
| 154 | +Become a [Fan and Sponsor](https://github.com/sponsors/RandomFractals/sponsorships?sponsor=RandomFractals&tier_id=18883) our dev efforts on this and other [Random Fractals, Inc. code and data viz extensions](https://marketplace.visualstudio.com/publishers/RandomFractalsInc) if you find them useful, educational, or enhancing your daily dataViz dev code workflows and exploratory data analysis: |
| 155 | + |
| 156 | +☕️ https://ko-fi.com/dataPixy 💖 https://github.com/sponsors/RandomFractals |
0 commit comments