Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: estimate the cost of running a tag from web view #87

Merged
merged 50 commits into from
Jan 13, 2025
Merged
Show file tree
Hide file tree
Changes from 45 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
819f5fa
feat: estimate the cost of running a tag
ashish10alex Jan 9, 2025
63f7d64
fix: cost block showing in other blocks
ashish10alex Jan 9, 2025
6e71a03
fix: long descriptions not visible in schema webview
ashish10alex Jan 9, 2025
49db79e
Merge branch 'main' into cost_estimator
ashish10alex Jan 9, 2025
b3efe9a
feat: add costs of operations and assertions
ashish10alex Jan 9, 2025
24f08b0
chore: add gap btw table and button
ashish10alex Jan 9, 2025
fe99cc0
chore: give error colunm dark red color
ashish10alex Jan 9, 2025
01c04f1
feat: make ui more user intuitive
ashish10alex Jan 9, 2025
c55b5aa
feat: add GB processed column
ashish10alex Jan 9, 2025
836d1aa
chore: lower num rows in pagination
ashish10alex Jan 9, 2025
0ee8699
feat: order the table by error so user sees the incase there are any
ashish10alex Jan 9, 2025
48b0fc8
feat: make button more clickable
ashish10alex Jan 9, 2025
5bd389f
Merge branch 'main' into cost_estimator
ashish10alex Jan 9, 2025
0abfd52
chore: bump up font size
ashish10alex Jan 10, 2025
7054a18
Merge branch 'main' into cost_estimator
ashish10alex Jan 10, 2025
67e32fc
feat: show loading icon when cost estimator is loading
ashish10alex Jan 10, 2025
e2bcfad
feat: persist tag value when cost estimation has loaded
ashish10alex Jan 10, 2025
9ad0fbf
chore: change wording
ashish10alex Jan 11, 2025
cfeddee
feat: sort tags alphabetically
ashish10alex Jan 11, 2025
ca7ee70
feat: hide other metadata from navs when cost estimator nav is selected
ashish10alex Jan 11, 2025
05cad23
docs: update readme
ashish10alex Jan 11, 2025
9812203
docs: update readme
ashish10alex Jan 11, 2025
8b4cb84
feat: add check to see if BigQuery is accessible by user
ashish10alex Jan 11, 2025
44ab094
fix: error when trying to access tag if not there in model
ashish10alex Jan 11, 2025
459e4c2
feat: propagate error to web view
ashish10alex Jan 11, 2025
d5dab32
Merge branch 'main' into cost_estimator
ashish10alex Jan 11, 2025
2fd2392
chore: remove todo
ashish10alex Jan 11, 2025
df0220f
Merge branch 'main' into cost_estimator
ashish10alex Jan 11, 2025
dd29644
Merge branch 'main' into cost_estimator
ashish10alex Jan 12, 2025
c1ff590
feat: increase number of rowss in pagination
ashish10alex Jan 12, 2025
82c6b34
Merge branch 'main' into cost_estimator
ashish10alex Jan 12, 2025
1129620
Merge branch 'main' into cost_estimator
ashish10alex Jan 12, 2025
f41b999
Merge branch 'main' into cost_estimator
ashish10alex Jan 12, 2025
c7748e0
chore: fix markdown linting complains
ashish10alex Jan 12, 2025
47ef55f
ci: markdown lint json config file
ashish10alex Jan 12, 2025
4ef75df
chore: add `prerelease` script
ashish10alex Jan 12, 2025
c8a350c
chore: update markdown lint config
ashish10alex Jan 12, 2025
50048b5
docs: update CONTRIBUTING.md
ashish10alex Jan 12, 2025
91245f8
fix: cost estimator throws error when pre ops ends with `;`
ashish10alex Jan 12, 2025
aeb3188
feat: add support for multiple currencies
ashish10alex Jan 12, 2025
b1f6598
chore: cleanup
ashish10alex Jan 12, 2025
77505db
chore: symbol comes before amount
ashish10alex Jan 12, 2025
0329d5e
chore: cleanup
ashish10alex Jan 12, 2025
f2f5f69
refactor: more self explanatory variable names
ashish10alex Jan 12, 2025
564fd44
fix: handling of `;` in incremental pre_operation query
ashish10alex Jan 13, 2025
3362ab6
feat: add unit conversion depending on cost size
ashish10alex Jan 13, 2025
ac9c9c2
feat: add unit conversion to preview results panel
ashish10alex Jan 13, 2025
05973da
feat: add test to ensure correctness of `formatBytes` function
ashish10alex Jan 13, 2025
dc3550e
docs: add illustration of cost estimator
ashish10alex Jan 13, 2025
6cfc757
docs: update compiled query web view image
ashish10alex Jan 13, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .markdownlint.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@

{
"MD033": false, // Inline HTML
"MD013": false, // Line length
"MD051": false, // Link fragments should be valid"
"MD042": false // No empty links
}
1 change: 1 addition & 0 deletions .vscodeignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,4 @@ vsc-extension-quickstart.md
**/*.ts
**/.vscode-test.*
.versionrc
.markdownlint.json
38 changes: 20 additions & 18 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,33 +1,35 @@

### Requirements
# Requirements

* `node v18` or higher
* `npm` (gets installed alongside node)

## How to contribute

[1. Build and make changes to the extension]()

### How to contribute
* Fork this repository, clone it to your desktop and open it in VSCode
* Run `npm install` in your terminal to install all the dependencies
* Click the `Run and debug` icon on the left hand pane of your editor and click on `Run Extension` button

#### Build the project
<img src="media/images/dataform_tools_run_and_debug.png" alt="compilation" width="300">

1. To build and make changes to the extension
- Fork this repository, clone it to your desktop and open it in VSCode
- Run `npm install` in your terminal to install all the dependencies
- Click the `Run and debug` icon on the left hand pane of your editor and click on `Run Extension` button.
* This should open a new VSCode window where you can open a Dataform project. Make sure that you folder opened in the workspace is at the root of the Dataform project. For example if your Dataform project is located at `~/repos/my_dataform_project` open the workspace at `~/repos/my_dataform_project`, **NOT** `~/repos`. Such that either `workflow_settings.yaml` or `dataform.json` depending on the Dataform version you are using is located at the root of the VSCode workspace.

<img src="media/images/dataform_tools_run_and_debug.png" alt="compilation" width="300">
[2. Make your changes]()

2. Doing step (1) will open a new VSCode window where you can open a Dataform project. Make sure that you folder opened in the workspace is at the root of the Dataform project. For example if your Dataform project is located at `~/repos/my_dataform_project` open the workspace at `~/repos/my_dataform_project`, **NOT** `~/repos`. Such that either `workflow_settings.yaml` or `dataform.json` depending on the Dataform version you are using is located at the root of the VSCode workspace.
Make the desired changes to the `vscode-dataform-tools` repo and re-run/refresh the compilation to see the desired outcome in the new VSCode window

#### Make your changes
[Test your changes]()

3. Make the desired changes to the `vscode-dataform-tools` repo and re-run/refresh the compilation to see the desired outcome in the new VSCode window that opens when doing step (2)
* **Test your changes** on your Dataform repository. If you are running linux based operating system run `npm run test` on your terminal to verify if the exsisiting tests are pasing. There are some caveats with running tests, so do not panic if the test fail to run. The test would not be able to run if your project path is very long this is a [known issue reported here](https://github.com/microsoft/vscode-test/issues/232). Also, we are having to remove `.vscode-test/user-data` before running `vscode-test` in the `npm run test` script in `package.json`. These tests currently are only tested to be running on Mac OS. We will need to change the script for `npm run test` in `package.json` for it to work in multiple operating systems.

#### Test your changes
* Run `npm install markdownlint-cli2 --global` to install markdown linter and run `markdownlint-cli2 README.md` to verify the Markdown is correctly formatted if you have made any changes there.

4. **Test your changes** on your Dataform repository. If you are running linux based operating system run `npm run test` on your terminal to verify if the exsisiting tests are pasing. There are some caveats with running tests, so do not panic if the test fail to run. The test would not be able to run if your project path is very long this is a [known issue reported here](https://github.com/microsoft/vscode-test/issues/232). Also, we are having to remove `.vscode-test/user-data` before running `vscode-test` in the `npm run test` script in `package.json`. These tests currently are only tested to be running on Mac OS. We will need to change the script for `npm run test` in `package.json` for it to work in multiple operating systems.
### Open an issue / pull request

#### Open an issue / pull request
[If you'd like the feature or bug fix to be merged]()

5. If you'd like the feature or bug fix to be merged
- Check the exsisting issues to make sure that if it has not been already raised
- [Create an issue here](https://github.com/ashish10alex/vscode-dataform-tools/issues)
- [Create a pull request here](https://github.com/ashish10alex/vscode-dataform-tools/pulls)
* Check the exsisting issues to make sure that if it has not been already raised
* [Create an issue here](https://github.com/ashish10alex/vscode-dataform-tools/issues)
* [Create a pull request here](https://github.com/ashish10alex/vscode-dataform-tools/pulls)
31 changes: 17 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,18 +12,17 @@

⚠️ ***This is not an officially supported Google product.***


[VS Code extension](https://marketplace.visualstudio.com/items?itemName=ashishalex.dataform-lsp-vscode) for [Dataform](https://github.com/dataform-co/dataform) with following features


| Feature | Description |
|---------|-------------|
| [Compiled Query & Dry run stats](#compilation) | Compiled query in a vertical split |
| [Compiled Query & Dry run stats](#compilation) | Compiled query with dry run stats in a vertical split |
| [Dependancy graph](#depgraph) | Interative dependancy graph with external sources higlighted in distinct colors |
| [Inline diagnostics on `.sqlx` file](#diagnostics) | Native lsp like experience with diagnostics being directly put on both the sqlx file & compiled query |
| [Inline diagnostics on `.sqlx` file](#diagnostics) 🚨 | Native LSP like experience with diagnostics being directly put on sqlx file |
| [Preview query results](#preview_query_results) | Preview query results in a table by running the file |
| [Cost estimator](#cost_estimator) 💸 | Estimate the cost of running a Tag|
| [Go to definition](#definition) | Go to definition for source in `$ref{("my_source")}` and javascript blocks in `.sqlx` files |
| [Auto-completion](#autocomplete) | - Column names of current model <br><br> - Dependencies and declarations in `${ref("..")}` trigger when `$` character is typed <br><br> - Dependencies when `"` or `'` is typed inside the config block which has `dependencies` keyword is in the line prefix <br><br> - `tags` when `"` or `'` is typed inside the config block which has `tags` keyword is in the line prefix |
| [Auto-completion](#autocomplete) | - Column names of current model <br> - Dependencies and declarations in `${ref("..")}` trigger when `$` character is typed <br> - Dependencies when `"` or `'` is typed inside the config block which has `dependencies` keyword is in the line prefix <br> - `tags` when `"` or `'` is typed inside the config block which has `tags` keyword is in the line prefix |
| [Code actions](#codeactions) | Apply dry run suggestions at the speed of thought |
| [Run file(s)/tag(s)](#filetagruns) | Run file(s)/tag(s), optionally with dependencies/dependents/full refresh using vscode command pallet / menu icons |
| [Format using Sqlfluff](#formatting) 🪄 | Fromat `.sqlx` files using [sqlfluff](https://github.com/sqlfluff/sqlfluff)|
Expand All @@ -45,22 +44,22 @@
```bash
gcloud init
```

```bash
gcloud auth application-default login
```

```bash
gcloud config set project <project_id> #replace with your gcp project id
```


3. To enable formatting using [sqlfluff](https://github.com/sqlfluff/sqlfluff) install [sqlfluff](https://github.com/sqlfluff/sqlfluff)

```bash
# install python and run
pip install sqlfluff
```


4. To enable prettier diagnostics install [Error Lens](https://marketplace.visualstudio.com/items?itemName=usernamehw.errorlens) extension [ **optional** ]

> [!NOTE]
Expand All @@ -72,19 +71,27 @@ Trouble installing ? Please see [FAQ section](FAQ.md), if you are still stuck, p
## Features

### <a id="compilation">Compiled query & Dry run stats</a>

![compilation](media/images/compiled_query_preview.png)

### <a id="depgraph">Dependency graph</a>

![depgraph](/media/images/dependancy_tree.png)

### <a id="diagnostics">Inline diagnostics errors on `.sqlx` files ❗</a>

![diagnostics](media/images/diagnostics.png)

### <a id="preview_query_results">Preview query results</a>

![preview_query_results](/media/images/preview_query_results.png)

### <a id="cost_estimator">Estimate cost of running a Tag</a>

![cost_estimator]()

### <a id="definition">Go to definition</a>

Go to definition for source in `$ref{("my_source")}`. Takes you to `my_source.sqlx` or `sources.js` at the line where `my_source` is defined. There is also support for go to definiton
from a javascript variable/module from a `.sqlx` file to `js` block or `.js` file where the virable or module declaration exsists

Expand All @@ -108,8 +115,6 @@ Auto completion support for `tags` when `"` or `'` is typed inside the config bl

![formatting](media/images/formatting.gif)



### <a id="filetagruns">Run file/tag with dependencies/dependents</a>

Open vscode command pallet by pressing <kbd>CTLR</kbd> + <kbd>SHIFT</kbd> + <kbd>p</kbd> or <kbd>CMD</kbd> + <kbd>SHIFT</kbd> + <kbd>p</kbd> on mac and run one of the required commands
Expand All @@ -127,11 +132,9 @@ Open vscode command pallet by pressing <kbd>CTLR</kbd> + <kbd>SHIFT</kbd> + <kbd

## Known Issues

- [ ] Features such as go to definition / dependancy graph might not work with consistantly with `${ref("dataset", "table")}` or when it is multiline or a different format works best with `${ref('table_name')}` format
* [ ] Features such as go to definition / dependancy graph might not work with consistantly with `${ref("dataset", "table")}` or when it is multiline or a different format works best with `${ref('table_name')}` format

## TODO

- [ ] Improved UI to support light mode users
- [ ] Handle case where user is not connected to internet or on vpn where network request for dry run cannot be made


* [ ] Add option to include dependents / dependencies when running cost estimator for tag
* [ ] Handle case where user is not connected to internet or on vpn where network request for dry run cannot be made
24 changes: 24 additions & 0 deletions media/css/query.css
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,7 @@ button:disabled:hover {

.topnav a {
color: #b8b5b5;
box-shadow: 0 1px 1px gray;
text-decoration: none;
font-size: 16px;
padding: 4px 5px;
Expand Down Expand Up @@ -427,4 +428,27 @@ form {
align-items: center;
cursor: pointer;
margin: 10px;
}


.cost-model {
background-color: #007acc;
color: #ffffff;
padding: 5px 10px;
font-size: 16px;
border-radius: 5px;
border: none;
cursor: pointer;
}

.cost-model:hover {
background-color: #3291d0;
}

#costEstimator {
margin-bottom: 20px;
}

.tag-dropdown {
font-size: 14px;
}
4 changes: 4 additions & 0 deletions media/css/tabulator_custom.css
Original file line number Diff line number Diff line change
Expand Up @@ -26,4 +26,8 @@
/* Optional: Add alternating row colors for better readability */
.tabulator .tabulator-row:nth-child(even) {
background-color: #24272a; /* Slightly different background for even rows */
}

.error-column {
color: #D32F2F;
}
Loading