Skip to content

Conversation

@fjakobs
Copy link
Contributor

@fjakobs fjakobs commented Nov 20, 2025

Implements intelligent warehouse auto-discovery removing need for configuration.

Changes

  • Add GetWarehouseID() with smart fallback chain
  • Auto-discover warehouses via API
  • Prefer RUNNING, fall back to STOPPED (auto-start)
  • Cache warehouse ID in session
  • Remove warehouse ID from config
  • Update providers to use GetWarehouseID

Dependencies

Testing

  • Databricks provider tests pass

@eng-dev-ecosystem-bot
Copy link
Collaborator

eng-dev-ecosystem-bot commented Nov 20, 2025

Commit: b31acf9

Run: 19573554747

Env 🟨​KNOWN 🔄​flaky 🙈​SKIP ✅​pass 🙈​skip Time
🟨​ aws linux 7 2 363 614 22:42
🔄​ gcp linux 3 4 357 616 23:00
11 failing tests:
Test Name aws linux gcp linux
TestAccept 🟨​K 🔄​f
TestAccept/bundle/resources/permissions 🙈​S 🙈​S
TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions 🟨​K 🙈​S
TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K
TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K
TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions 🟨​K 🙈​S
TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K
TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K
TestAccept/bundle/run/app-with-job 🙈​S 🙈​S
TestAccept/bundle/templates/default-python/combinations/classic ✅​p 🔄​f
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=yes/NBOOK=yes/PY=yes ✅​p 🔄​f
Top 9 slowest tests (at least 2 minutes):
duration env testname
7:29 gcp linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:14 gcp linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
2:48 gcp linux TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=yes/PY=no
2:27 aws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
2:27 gcp linux TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=no/PY=no
2:26 aws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
2:21 gcp linux TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=no/PY=yes
2:19 gcp linux TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=no/PY=no
2:05 gcp linux TestAccept/bundle/resources/models/basic/DATABRICKS_BUNDLE_ENGINE=direct

Implements intelligent warehouse auto-discovery:
- Add GetWarehouseID() with smart fallback chain:
  1. Check DATABRICKS_WAREHOUSE_ID environment variable
  2. Query available warehouses via API
  3. Prefer RUNNING warehouses
  4. Fall back to STOPPED warehouses (auto-start)
  5. Use first available warehouse
- Cache warehouse ID in session after first resolution
- Remove warehouse ID from CLI flags and config
- Update NewDatabricksRestClient and ResourcesFromEnv to use GetWarehouseID

Eliminates need to specify warehouse ID upfront.
@fjakobs fjakobs force-pushed the mcp/lazy-warehouse-resolution branch from a20f35d to ced4c41 Compare November 20, 2025 20:37
@fjakobs fjakobs force-pushed the mcp/lazy-warehouse-resolution branch from f5c131f to ced4c41 Compare November 20, 2025 20:38
}
}

// Return first available warehouse regardless of state

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we cover STOPPED + STOPPING together with the same priority and raise err here already? From what I read, the left states are DELETING / DELETED, not a good choice.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've improve the lookup code. I also consider starting and stopping now and ignore the deleting and deleted states.

@fjakobs fjakobs force-pushed the mcp/lazy-warehouse-resolution branch from 70756b8 to b31acf9 Compare November 21, 2025 14:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants