Prepare for a release with telemetry on by default (#717)

* Prepare for a release with telemetry on by default

Signed-off-by: samikshya-chand_data <samikshya.chand@databricks.com>

* Make edits

Signed-off-by: samikshya-chand_data <samikshya.chand@databricks.com>

* Update version

Signed-off-by: samikshya-chand_data <samikshya.chand@databricks.com>

* Fix CHANGELOG formatting to match previous style

Signed-off-by: samikshya-chand_data <samikshya.chand@databricks.com>

* Fix telemetry e2e tests for default-enabled behavior

- Update test expectations to reflect telemetry being enabled by default
- Add feature flags cache cleanup in teardown to prevent state leakage between tests
- This ensures each test runs with fresh feature flag state

* Add wait after connection close for async telemetry submission

* Remove debug logging from telemetry tests

* Mark telemetry e2e tests as serial - must not run in parallel

Root cause: Telemetry tests share host-level client across pytest-xdist workers,
causing test isolation issues with patches. Tests pass serially but fail with -n auto.

Solution: Add @pytest.mark.serial marker. CI needs to run these separately without -n auto.

* Split test execution to run serial tests separately

Telemetry e2e tests must run serially due to shared host-level
telemetry client across pytest-xdist workers. Running with -n auto
causes test isolation issues where futures aren't properly captured.

Changes:
- Run parallel tests with -m 'not serial' -n auto
- Run serial tests with -m 'serial' without parallelization
- Use --cov-append for serial tests to combine coverage
- Mark telemetry e2e tests with @pytest.mark.serial
- Update test expectations for default telemetry behavior
- Add feature flags cache cleanup in test teardown

* Mark telemetry e2e tests as serial - must not run in parallel

The concurrent telemetry e2e test globally patches telemetry methods
to capture events. When run in parallel with other tests via pytest-xdist,
it captures telemetry events from other concurrent tests, causing
assertion failures (expected 60 events, got 88).

All telemetry e2e tests must run serially to avoid cross-test
interference with the shared host-level telemetry client.

---------

Signed-off-by: samikshya-chand_data <samikshya.chand@databricks.com>

Author: samikshya-db

.github/workflows/code-coverage.yml

@@ -61,17 +61,32 @@ jobs:
       - name: Install library
         run: poetry install --no-interaction --all-extras
       #----------------------------------------------
-      #    run all tests with coverage
+      #    run parallel tests with coverage
       #----------------------------------------------
-      - name: Run all tests with coverage 
+      - name: Run parallel tests with coverage
         continue-on-error: false
         run: |
           poetry run pytest tests/unit tests/e2e \
+            -m "not serial" \
             -n auto \
             --cov=src \
             --cov-report=xml \
             --cov-report=term \
             -v
+
+      #----------------------------------------------
+      #    run serial tests with coverage
+      #----------------------------------------------
+      - name: Run serial tests with coverage
+        continue-on-error: false
+        run: |
+          poetry run pytest tests/e2e \
+            -m "serial" \
+            --cov=src \
+            --cov-append \
+            --cov-report=xml \
+            --cov-report=term \
+            -v
           
       #----------------------------------------------
       #         check for coverage override

CHANGELOG.md

@@ -1,5 +1,12 @@
 # Release History
 
+# 4.2.2 (2025-12-01)
+- Change default use_hybrid_disposition to False (databricks/databricks-sql-python#714 by @samikshya-db)
+- Circuit breaker changes using pybreaker (databricks/databricks-sql-python#705 by @nikhilsuri-db)
+- perf: Optimize telemetry latency logging to reduce overhead (databricks/databricks-sql-python#715 by @samikshya-db)
+- basic e2e test for force telemetry verification (databricks/databricks-sql-python#708 by @nikhilsuri-db)
+- Telemetry is ON by default to track connection stats. (Note : This strictly excludes PII, query text, and results) (databricks/databricks-sql-python#717 by @samikshya-db)
+
 # 4.2.1 (2025-11-20)
 - Ignore transactions by default (databricks/databricks-sql-python#711 by @jayantsing-db)
 

README.md

@@ -13,7 +13,7 @@ You are welcome to file an issue here for general use cases. You can also contac
 
 ## Requirements
 
-Python 3.8 or above is required.
+Python 3.9 or above is required.
 
 ## Documentation
 

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "databricks-sql-connector"
-version = "4.2.1"
+version = "4.2.2"
 description = "Databricks SQL Connector for Python"
 authors = ["Databricks <databricks-sql-connector-maintainers@databricks.com>"]
 license = "Apache-2.0"
@@ -62,7 +62,10 @@ exclude = ['ttypes\.py$', 'TCLIService\.py$']
 exclude = '/(\.eggs|\.git|\.hg|\.mypy_cache|\.nox|\.tox|\.venv|\.svn|_build|buck-out|build|dist|thrift_api)/'
 
 [tool.pytest.ini_options]
-markers = {"reviewed" = "Test case has been reviewed by Databricks"}
+markers = [
+    "reviewed: Test case has been reviewed by Databricks",
+    "serial: Tests that must run serially (not parallelized)"
+]
 minversion = "6.0"
 log_cli = "false"
 log_cli_level = "INFO"

src/databricks/sql/__init__.py

@@ -71,7 +71,7 @@ def __repr__(self):
 DATE = DBAPITypeObject("date")
 ROWID = DBAPITypeObject()
 
-__version__ = "4.2.1"
+__version__ = "4.2.2"
 USER_AGENT_NAME = "PyDatabricksSqlConnector"
 
 # These two functions are pyhive legacy

src/databricks/sql/auth/common.py

@@ -51,7 +51,7 @@ def __init__(
         pool_connections: Optional[int] = None,
         pool_maxsize: Optional[int] = None,
         user_agent: Optional[str] = None,
-        telemetry_circuit_breaker_enabled: Optional[bool] = None,
+        telemetry_circuit_breaker_enabled: Optional[bool] = True,
     ):
         self.hostname = hostname
         self.access_token = access_token

src/databricks/sql/client.py

@@ -328,7 +328,7 @@ def read(self) -> Optional[OAuthToken]:
         self.ignore_transactions = ignore_transactions
 
         self.force_enable_telemetry = kwargs.get("force_enable_telemetry", False)
-        self.enable_telemetry = kwargs.get("enable_telemetry", False)
+        self.enable_telemetry = kwargs.get("enable_telemetry", True)
         self.telemetry_enabled = TelemetryHelper.is_telemetry_enabled(self)
 
         TelemetryClientFactory.initialize_telemetry_client(

tests/e2e/test_concurrent_telemetry.py

@@ -26,6 +26,7 @@ def run_in_threads(target, num_threads, pass_index=False):
         t.join()
 
 
+@pytest.mark.serial
 class TestE2ETelemetry(PySQLPytestTestCase):
     @pytest.fixture(autouse=True)
     def telemetry_setup_teardown(self):

tests/e2e/test_telemetry_e2e.py

@@ -43,8 +43,9 @@ def connection(self, extra_params=()):
             conn.close()
 
 
+@pytest.mark.serial
 class TestTelemetryE2E(TelemetryTestBase):
-    """E2E tests for telemetry scenarios"""
+    """E2E tests for telemetry scenarios - must run serially due to shared host-level telemetry client"""
 
     @pytest.fixture(autouse=True)
     def telemetry_setup_teardown(self):
@@ -58,6 +59,14 @@ def telemetry_setup_teardown(self):
             TelemetryClientFactory._stop_flush_thread()
             TelemetryClientFactory._initialized = False
 
+            # Clear feature flags cache to prevent state leakage between tests
+            from databricks.sql.common.feature_flag import FeatureFlagsContextFactory
+            with FeatureFlagsContextFactory._lock:
+                FeatureFlagsContextFactory._context_map.clear()
+                if FeatureFlagsContextFactory._executor:
+                    FeatureFlagsContextFactory._executor.shutdown(wait=False)
+                    FeatureFlagsContextFactory._executor = None
+
     @pytest.fixture
     def telemetry_interceptors(self):
         """Setup reusable telemetry interceptors as a fixture"""
@@ -142,7 +151,7 @@ def verify_events(self, captured_events, captured_futures, expected_count):
         else:
             assert len(captured_events) == expected_count, \
                 f"Expected {expected_count} events, got {len(captured_events)}"
-            
+
             time.sleep(2)
             done, _ = wait(captured_futures, timeout=10)
             assert len(done) == expected_count, \
@@ -163,7 +172,7 @@ def verify_events(self, captured_events, captured_futures, expected_count):
         (True, False, 2, "enable_on_force_off"),
         (False, True, 2, "enable_off_force_on"),
         (False, False, 0, "both_off"),
-        (None, None, 0, "default_behavior"),
+        (None, None, 2, "default_behavior"),
     ])
     def test_telemetry_flags(self, telemetry_interceptors, enable_telemetry, 
                             force_enable, expected_count, test_id):
@@ -185,6 +194,8 @@ def test_telemetry_flags(self, telemetry_interceptors, enable_telemetry,
                     cursor.execute("SELECT 1")
                     cursor.fetchone()
 
+            # Give time for async telemetry submission after connection closes
+            time.sleep(0.5)
             self.verify_events(captured_events, captured_futures, expected_count)
 
             # Assert statement execution on latency event (if events exist)