Databricks Connector
4 minute read
Overview
The Databricks connector enables AI Team members to interact with your Databricks analytics platform, providing capabilities for SQL query execution, cluster management, job monitoring, and notebook access. By connecting Databricks to AI Team, AI teammates can execute analytical queries, track job execution status, analyze failures, and manage cluster resources.
The connector provides access to Delta Lake tables, SQL warehouses, cluster metrics, job execution details, and workspace notebooks, allowing AI teammates to help investigate data pipeline issues, optimize query performance, and monitor resource utilization.
Add the Databricks Connector
To add the Databricks connector, you obtain a personal API token from Databricks and configure authentication in Edge Delta.
Prerequisites
Before configuring the connector, ensure you have:
- Databricks workspace with API access
- Personal Access Token from Databricks
- Appropriate workspace permissions for the operations you need
- Network connectivity from Edge Delta to the workspace
Configuration Steps
- Navigate to AI Team > Connectors in the Edge Delta application
- Find the Databricks connector
- Click the connector card to open the configuration panel
- Enter your Databricks Token
- Click Save
The connector is now available for use by AI Team members who have been assigned this connector.
General Options
Databricks Token
Personal Access Token for authenticating with Databricks. To create a token, log into your Databricks workspace, navigate to User Settings > Developer > Access tokens, and generate a new token. The token should have appropriate permissions for workspace access, cluster management, SQL execution, and job monitoring based on your needs. See Databricks personal access tokens for detailed instructions.
Tools
list_clusters
Lists all Databricks clusters in the workspace, showing cluster status, configuration, and resource utilization.
create_cluster
Creates a new Databricks cluster with specified configuration parameters.
terminate_cluster
Terminates a running Databricks cluster to free up resources.
get_cluster
Retrieves detailed information about a specific Databricks cluster including status, configuration, and metrics.
start_cluster
Starts a previously terminated Databricks cluster.
list_jobs
Lists all Databricks jobs in the workspace with their current status and recent run history.
run_job
Triggers execution of a Databricks job.
list_notebooks
Lists notebooks in a specified workspace directory path.
list_files
Lists files and directories in a DBFS (Databricks File System) path.
execute_sql
Executes SQL statements against Databricks SQL warehouses or clusters.
How to Use the Databricks Connector
The Databricks connector integrates with AI Team, enabling AI teammates to interact with your data platform based on natural language queries. Once configured, AI teammates can execute SQL queries, monitor job status, manage clusters, and investigate data pipeline issues.
Use Case: Data Pipeline Investigation
When data pipelines fail or produce unexpected results, AI teammates can retrieve job run details, analyze error logs, and identify the root cause. For example, when asked “Why did the nightly ETL job fail?”, the AI can check job status, retrieve failure logs, and provide context about what went wrong.
Use Case: Query Execution
AI teammates can execute SQL queries against Delta Lake tables to retrieve data for analysis. When investigating incidents or answering questions about data patterns, the AI can query tables directly and provide insights based on the results.
Use Case: Cluster Management
AI teammates can monitor cluster status, check resource utilization, and manage cluster lifecycle. When investigating performance issues or optimizing costs, the AI can check which clusters are running, review their configurations, and provide recommendations.
Troubleshooting
Connection errors: Verify your Databricks token is valid and hasn’t been revoked. Check that the workspace URL is correct and accessible from Edge Delta.
Permission errors: Ensure your token has the required permissions for the operations AI teammates need to perform (workspace access, cluster management, SQL execution, job monitoring).
Query failures: Verify SQL warehouse or cluster availability. Check that tables exist and the token has access to query them.
Next Steps
- Learn about creating custom teammates that can use this connector
- Explore other connectors to integrate additional tools
For additional help, visit AI Team Support.