Quick answer: Databricks secret scopes provide a controlled way to store and reference secrets from notebooks and jobs. Azure Databricks supports Databricks-backed scopes and Azure Key Vault-backed scopes. Choose the backend based on where you want the secret to be stored and how you manage access.
Prerequisites:
1. An Azure Databricks workspace.
2. A secret scope or permission to create one.
3. For an Azure Key Vault-backed scope, an Azure Key Vault and the required permissions to create the scope.
4. If you use the Key Vault-backed option, the Key Vault permission model must be Vault access policy; Azure RBAC is not supported for this secret-scope integration.
What are secret scopes in Databricks?
Secrets such as database passwords, API keys, and tokens should not be hard-coded in notebooks or application code. A Databricks secret scope is a named collection of secrets that can be referenced securely from Databricks workloads.
Azure Databricks supports two secret-scope backends:
- Databricks-backed: secrets are stored in an encrypted database owned and managed by Azure Databricks.
- Azure Key Vault-backed: the scope provides a read-only interface to secrets stored in Azure Key Vault. The secrets themselves are managed in Azure Key Vault.
Databricks-backed vs Azure Key Vault-backed scopes
| Feature | Databricks-backed | Azure Key Vault-backed |
|---|---|---|
| Secret storage | Encrypted database managed by Azure Databricks | Azure Key Vault |
| Secret management | Managed through Databricks | Managed in Azure Key Vault |
| Scope permissions | Databricks secret ACLs | Databricks secret ACLs plus Key Vault access configuration |
| Key Vault required | No | Yes |
| Key Vault permission model | Not applicable | Vault access policy |
There is no general rule that a Key Vault-backed scope is automatically “more secure.” The important design choice is where secrets are managed and which identities and users receive access. Secret ACLs are applied at the scope level.
Create an Azure Key Vault-backed scope
The following steps use the Azure Databricks workspace UI.
1. Open https://<databricks-instance>#secrets/createScope.
2. In Azure Key Vault, open Settings > Access configuration and set the permission model to Vault access policy. Microsoft documents this requirement for Azure Key Vault-backed secret scopes.
3. In Databricks, provide the following details:
Scope Name: Enter a unique scope name.
Manage Principal: Select who receives the MANAGE permission on the scope. The available choices depend on the workspace and plan.
DNS Name and Resource ID: Copy these values from the Azure Key Vault properties in the Azure portal.
4. Click Create. You can verify the scope with databricks secrets list-scopes.
Access a secret from Databricks
Use the Databricks Secrets utility from a notebook or job:
password = dbutils.secrets.get(scope = "<scope-name>", key = "<key-name>")
Available on Microsoft Store
SSRS Reports Migration Wizard
A simple Windows tool for migrating SSRS reports, data sources, and related configurations between report servers.
Manage secret-scope permissions
Secret ACLs are assigned at the scope level. The main permission levels are READ, WRITE, and MANAGE. A user with access to an Azure Key Vault-backed scope can access the secrets available through that scope; Microsoft specifically notes that access to a Key Vault-backed scope gives access to all secrets in the associated Key Vault. Use separate Key Vault instances when you need stronger isolation between sets of secrets.
For the current Databricks CLI, you can inspect and manage scope ACLs with commands such as:
databricks secrets list-acls <scope-name>
databricks secrets put-acl <scope-name> <principal> <permission>
Delete a secret scope
Deleting a secret scope removes the scope and the secrets and ACLs associated with it. The current Databricks CLI command is:
databricks secrets delete-scope <scope-name>
FAQ
Pro tips:
1. Keep secret scopes aligned to applications or roles rather than individual users where practical.
2. Grant the minimum required READ, WRITE, or MANAGE permission.
3. Remember that secret ACLs are scope-level, not individual-secret-level.
4. For Azure Key Vault-backed scopes, use the Vault access policy permission model required by this integration.
5. If you are automating Databricks infrastructure, see Automate Databricks Infrastructure as Code with Terraform.
See more
Visual Studio Marketplace
SSIS Catalog Migration Wizard
Extend Visual Studio with an easy way to migrate SSIS Catalog projects.
Pavan Bangad
9+ years of experience in building data warehouse and big data application.
Helping customers in their digital transformation journey in cloud.
Passionate about data engineering.
