|
| 1 | +# Using Native Parameters |
| 2 | + |
| 3 | +This connector supports native parameterized query execution. When you execute a query that includes variable markers, then you can pass a collection of parameters which are sent separately to Databricks Runtime for safe execution. This prevents SQL injection and can improve query performance. |
| 4 | + |
| 5 | +This behaviour is distinct from legacy "inline" parameterized execution in versions below 3.0.0. The legacy behavior is preserved behind a flag called `use_inline_params`, which will be removed in a future release. See [Using Inline Parameters](#using-inline-parameters) for more information. |
| 6 | + |
| 7 | +See **[below](#migrating-to-native-parameters)** for details about updating your client code to use native parameters. |
| 8 | + |
| 9 | +See `examples/parameters.py` in this repository for a runnable demo. |
| 10 | + |
| 11 | +## Requirements |
| 12 | + |
| 13 | +- `databricks-sql-connector>=3.0.0` |
| 14 | +- A SQL warehouse or all-purpose cluster running Databricks Runtime >=14.2 |
| 15 | + |
| 16 | +## Limitations |
| 17 | + |
| 18 | +- A query executed with native parameters can contain at most 255 parameter markers |
| 19 | +- The maximum size of all parameterized values cannot exceed 1MB |
| 20 | + |
| 21 | +## SQL Syntax |
| 22 | + |
| 23 | +Variables in your SQL query can use one of three PEP-249 [paramstyles](https://peps.python.org/pep-0249/#paramstyle). A parameterized query can use exactly one paramstyle. |
| 24 | + |
| 25 | +|paramstyle|example|comment| |
| 26 | +|-|-|-| |
| 27 | +|`named`|`:param`|Parameters must be named| |
| 28 | +|`qmark`|`?`|Parameter names are ignored| |
| 29 | +|`pyformat`|`%(param)s`|Legacy syntax. Will be deprecated. Parameters must be named.| |
| 30 | + |
| 31 | +#### Example |
| 32 | + |
| 33 | +```sql |
| 34 | +-- named paramstyle |
| 35 | +SELECT * FROM table WHERE field = :value |
| 36 | + |
| 37 | +-- qmark paramstyle |
| 38 | +SELECT * FROM table WHERE field = ? |
| 39 | + |
| 40 | +-- pyformat paramstyle (legacy) |
| 41 | +SELECT * FROM table WHERE field = %(value)s |
| 42 | +``` |
| 43 | + |
| 44 | +## Python Syntax |
| 45 | + |
| 46 | +This connector follows the [PEP-249 interface](https://peps.python.org/pep-0249/#id20). The expected structure of the parameter collection follows the paramstyle of the variables in your query. |
| 47 | + |
| 48 | +### `named` paramstyle Usage Example |
| 49 | + |
| 50 | +When your SQL query uses `named` paramstyle variable markers, you need specify a name for each value that corresponds to a variable marker in your query. |
| 51 | + |
| 52 | +Generally, you do this by passing `parameters` as a dictionary whose keys match the variables in your query. The length of the dictionary must exactly match the count of variable markers or an exception will be raised. |
| 53 | + |
| 54 | +```python |
| 55 | +from databricks import sql |
| 56 | + |
| 57 | +with sql.connect(...) as conn: |
| 58 | + with conn.cursor() as cursor(): |
| 59 | + query = "SELECT field FROM table WHERE field = :value1 AND another_field = :value2" |
| 60 | + parameters = {"value1": "foo", "value2": 20} |
| 61 | + result = cursor.execute(query, parameters=parameters).fetchone() |
| 62 | +``` |
| 63 | + |
| 64 | +This paramstyle is a drop-in replacement for the `pyformat` paramstyle which was used in connector versions below 3.0.0. It should be used going forward. |
| 65 | + |
| 66 | +### `qmark` paramstyle Usage Example |
| 67 | + |
| 68 | +When your SQL query uses `qmark` paramstyle variable markers, you only need to specify a value for each variable marker in your query. |
| 69 | + |
| 70 | +You do this by passing `parameters` as a list. The order of values in the list corresponds to the order of `qmark` variables in your query. The length of the list must exactly match the count of variable markers in your query or an exception will be raised. |
| 71 | + |
| 72 | +```python |
| 73 | +from databricks import sql |
| 74 | + |
| 75 | +with sql.connect(...) as conn: |
| 76 | + with conn.cursor() as cursor(): |
| 77 | + query = "SELECT field FROM table WHERE field = ? AND another_field = ?" |
| 78 | + parameters = ["foo", 20] |
| 79 | + result = cursor.execute(query, parameters=parameters).fetchone() |
| 80 | +``` |
| 81 | + |
| 82 | +The result of the above two examples is identical. |
| 83 | + |
| 84 | +### Legacy `pyformat` paramstyle Usage Example |
| 85 | + |
| 86 | +Databricks Runtime expects variable markers to use either `named` or `qmark` paramstyles. Historically, this connector used `pyformat` which Databricks Runtime does not support. So to assist assist customers transitioning their codebases from `pyformat` → `named`, we can dynamically rewrite the variable markers before sending the query to Databricks. This happens only when `use_inline_params=False`. |
| 87 | + |
| 88 | + This dynamic rewrite will be deprecated in a future release. New queries should be written using the `named` paramstyle instead. And users should update their client code to replace `pyformat` markers with `named` markers. |
| 89 | + |
| 90 | +For example: |
| 91 | + |
| 92 | +```sql |
| 93 | +-- a query written for databricks-sql-connector==2.9.3 and below |
| 94 | + |
| 95 | +SELECT field1, field2, %(param1)s FROM table WHERE field4 = %(param2)s |
| 96 | + |
| 97 | +-- rewritten for databricks-sql-connector==3.0.0 and above |
| 98 | + |
| 99 | +SELECT field1, field2, :param1 FROM table WHERE field4 = :param2 |
| 100 | +``` |
| 101 | + |
| 102 | + |
| 103 | +**Note:** While named `pyformat` markers are transparently replaced when `use_inline_params=False`, un-named inline `%s`-style markers are ignored. If your client code makes extensive use of `%s` markers, these queries will need to be updated to use `?` markers before you can execute them when `use_inline_params=False`. See [When to use inline parameters](#when-to-use-inline-parameters) for more information. |
| 104 | + |
| 105 | +### Type inference |
| 106 | + |
| 107 | +Under the covers, parameter values are annotated with a valid Databricks SQL type. As shown in the examples above, this connector accepts primitive Python types like `int`, `str`, and `Decimal`. When this happens, the connector infers the corresponding Databricks SQL type (e.g. `INT`, `STRING`, `DECIMAL`) automatically. This means that the parameters passed to `cursor.execute()` are always wrapped in a `TDbsqlParameter` subtype prior to execution. |
| 108 | + |
| 109 | +Automatic inferrence is sufficient for most usages. But you can bypass the inference by explicitly setting the Databricks SQL type in your client code. All supported Databricks SQL types have `TDbsqlParameter` implementations which you can import from `databricks.sql.parameters`. |
| 110 | + |
| 111 | +`TDbsqlParameter` objects must always be passed within a list. Either paramstyle (`:named` or `?`) may be used. However, if your query uses the `named` paramstyle, all `TDbsqlParameter` objects must be provided a `name` when they are constructed. |
| 112 | + |
| 113 | +```python |
| 114 | +from databricks import sql |
| 115 | +from databricks.sql.parameters import StringParameter, IntegerParameter |
| 116 | + |
| 117 | +# with `named` markers |
| 118 | +with sql.connect(...) as conn: |
| 119 | + with conn.cursor() as cursor(): |
| 120 | + query = "SELECT field FROM table WHERE field = :value1 AND another_field = :value2" |
| 121 | + parameters = [ |
| 122 | + StringParameter(name="value1", value="foo"), |
| 123 | + IntegerParameter(name="value2", value=20) |
| 124 | + ] |
| 125 | + result = cursor.execute(query, parameters=parameters).fetchone() |
| 126 | + |
| 127 | +# with `?` markers |
| 128 | +with sql.connect(...) as conn: |
| 129 | + with conn.cursor() as cursor(): |
| 130 | + query = "SELECT field FROM table WHERE field = ? AND another_field = ?" |
| 131 | + parameters = [ |
| 132 | + StringParameter(value="foo"), |
| 133 | + IntegerParameter(value=20) |
| 134 | + ] |
| 135 | + result = cursor.execute(query, parameters=parameters).fetchone() |
| 136 | +``` |
| 137 | + |
| 138 | +In general, we recommend using `?` markers when passing `TDbsqlParameter`'s directly. |
| 139 | + |
| 140 | +**Note**: When using `?` markers, you can bypass inference for _some_ parameters by passing a list containing both primitive Python types and `TDbsqlParameter` objects. `TDbsqlParameter` objects can never be passed in a dictionary. |
| 141 | + |
| 142 | +# Using Inline Parameters |
| 143 | + |
| 144 | +Since its initial release, this connector's `cursor.execute()` method has supported passing a sequence or mapping of parameter values. Prior to Databricks Runtime introducing native parameter support, however, "parameterized" queries could not be executed in a guaranteed safe manner. Instead, the connector made a best effort to escape parameter values and and render those strings inline with the query. |
| 145 | + |
| 146 | +This approach has several drawbacks: |
| 147 | + |
| 148 | +- It's not guaranteed to be safe from SQL injection |
| 149 | +- The server could not boost performance by caching prepared statements |
| 150 | +- The parameter marker syntax conflicted with SQL syntax in some cases |
| 151 | + |
| 152 | +Nevertheless, this behaviour is preserved in version 3.0.0 and above for legacy purposes. It will be removed in a subsequent major release. To enable this legacy code path, you must now construct your connection with `use_inline_params=True`. |
| 153 | + |
| 154 | +## Requirements |
| 155 | + |
| 156 | +Rendering parameters inline is supported on all versions of DBR since these queries are indistinguishable from ad-hoc query text. |
| 157 | + |
| 158 | + |
| 159 | +## SQL Syntax |
| 160 | + |
| 161 | +Variables in your SQL query can look like `%(param)s` or like `%s`. |
| 162 | + |
| 163 | +#### Example |
| 164 | + |
| 165 | +```sql |
| 166 | +-- pyformat paramstyle is used for named parameters |
| 167 | +SELECT * FROM table WHERE field = %(value)s |
| 168 | + |
| 169 | +-- %s is used for positional parameters |
| 170 | +SELECT * FROM table WHERE field = %s |
| 171 | +``` |
| 172 | + |
| 173 | +## Python Syntax |
| 174 | + |
| 175 | +This connector follows the [PEP-249 interface](https://peps.python.org/pep-0249/#id20). The expected structure of the parameter collection follows the paramstyle of the variables in your query. |
| 176 | + |
| 177 | +### `pyformat` paramstyle Usage Example |
| 178 | + |
| 179 | +Parameters must be passed as a dictionary. |
| 180 | + |
| 181 | +```python |
| 182 | +from databricks import sql |
| 183 | + |
| 184 | +with sql.connect(..., use_inline_params=True) as conn: |
| 185 | + with conn.cursor() as cursor(): |
| 186 | + query = "SELECT field FROM table WHERE field = %(value1)s AND another_field = %(value2)s" |
| 187 | + parameters = {"value1": "foo", "value2": 20} |
| 188 | + result = cursor.execute(query, parameters=parameters).fetchone() |
| 189 | +``` |
| 190 | + |
| 191 | +The above query would be rendered into the following SQL: |
| 192 | + |
| 193 | +```sql |
| 194 | +SELECT field FROM table WHERE field = 'foo' AND another_field = 20 |
| 195 | +``` |
| 196 | + |
| 197 | +### `%s` paramstyle Usage Example |
| 198 | + |
| 199 | +Parameters must be passed as a list. |
| 200 | + |
| 201 | +```python |
| 202 | +from databricks import sql |
| 203 | + |
| 204 | +with sql.connect(..., use_inline_params=True) as conn: |
| 205 | + with conn.cursor() as cursor(): |
| 206 | + query = "SELECT field FROM table WHERE field = %s AND another_field = %s" |
| 207 | + parameters = ["foo", 20] |
| 208 | + result = cursor.execute(query, parameters=parameters).fetchone() |
| 209 | +``` |
| 210 | + |
| 211 | +The result of the above two examples is identical. |
| 212 | + |
| 213 | +**Note**: `%s` is not compliant with PEP-249 and only works due to the specific implementation of our inline renderer. |
| 214 | + |
| 215 | +**Note:** This `%s` syntax overlaps with valid SQL syntax around the usage of `LIKE` DML. For example if your query includes a clause like `WHERE field LIKE '%sequence'`, the parameter inlining function will raise an exception because this string appears to include an inline marker but none is provided. This means that connector versions below 3.0.0 it has been impossible to execute a query that included both parameters and LIKE wildcards. When `use_inline_params=False`, we will pass `%s` occurrences along to the database, allowing it to be used as expected in `LIKE` statements. |
| 216 | + |
| 217 | +### Passing sequences as parameter values |
| 218 | + |
| 219 | +Parameter values can also be passed as a sequence. This is typically used when writing `WHERE ... IN` clauses: |
| 220 | + |
| 221 | +```python |
| 222 | +from databricks import sql |
| 223 | + |
| 224 | +with sql.connect(..., use_inline_params=True) as conn: |
| 225 | + with conn.cursor() as cursor(): |
| 226 | + query = "SELECT field FROM table WHERE field IN %(value_list)s" |
| 227 | + parameters = {"value_list": [1,2,3,4,5]} |
| 228 | + result = cursor.execute(query, parameters=parameters).fetchone() |
| 229 | +``` |
| 230 | + |
| 231 | +Output: |
| 232 | + |
| 233 | +```sql |
| 234 | +SELECT field FROM table WHERE field IN (1,2,3,4,5) |
| 235 | +``` |
| 236 | + |
| 237 | +**Note**: this behavior is not specified by PEP-249 and only works due to the specific implementation of our inline renderer. |
| 238 | + |
| 239 | +### Migrating to native parameters |
| 240 | + |
| 241 | +Native parameters are meant to be a drop-in replacement for inline parameters. In most use-cases, upgrading to `databricks-sql-connector>=3.0.0` will grant an immediate improvement to safety. Plus, native parameters allow you to use SQL LIKE wildcards (`%`) in your queries which is impossible with inline parameters. Future improvements to parameterization (such as support for binding complex types like `STRUCT`, `MAP`, and `ARRAY`) will only be available when `use_inline_params=False`. |
| 242 | + |
| 243 | +To completely migrate, you need to [revise your SQL queries](#legacy-pyformat-paramstyle-usage-example) to use the new paramstyles. |
| 244 | + |
| 245 | + |
| 246 | +### When to use inline parameters |
| 247 | + |
| 248 | +You should only set `use_inline_params=True` in the following cases: |
| 249 | + |
| 250 | +1. Your client code passes more than 255 parameters in a single query execution |
| 251 | +2. Your client code passes parameter values greater than 1MB in a single query execution |
| 252 | +3. Your client code makes extensive use of [`%s` positional parameter markers](#s-paramstyle-usage-example) |
| 253 | +4. Your client code uses [sequences as parameter values](#passing-sequences-as-parameter-values) |
| 254 | + |
| 255 | +We expect limitations (1) and (2) to be addressed in a future Databricks Runtime release. |
0 commit comments