Databricks is a unified big data processing and analytics cloud platform that transforms and processes huge volumes of data. Apache Spark is the building block of Databricks, an in-memory analytics engine for big data and machine learning. Databricks can connect to various sources for data ingestion. This article describes how to manage secret scopes in Databricks using GUI.
Pre-requisites:
To mount a location, you would need:
1. Databricks service in Azure, GCP, or AWS cloud.
2. A Databricks cluster.
3. Azure subscription with Azure Key Vault service created.
What are Secret scopes in Databricks?
When working with various applications, the Databricks platform comes in handy. To establish connections, credentials or secrets are necessary, which can be securely stored in Databricks or Azure Key Vault. Secret scopes are responsible for managing these secrets in either Azure Key Vault or Databricks.y Vault or Databricks.y Vault or Databricks.
Databricks supports two secret scopes :
1. Azure Key Vault backed scopes: to manage secrets stored in the Azure Key Vault.
2. Databricks-backed scopes: to manage secrets stored in Databricks.
This article will focus on how to manage Azure Key Vault-backed secret scopes.
Create an Azure Key Vault-backed scope
Follow the below steps to create an Azure Key Vault-backed secret scope.
1. Open https://<databricks-instance>#secrets/createScope URL
2. Provide the below details:
Scope Name: <Name of the scope>
Manage Principal: Using this option, you can specify what all users can manage the secret scope. You can either select “All Users” or “Create’.
DNS Name and Resource ID: Both these properties can be found in Azure Key Vault service properties.
3. Click on Create. This will create secret scope.
Access a secret from the Azure Key Vault in Databricks
We can access secrets in Databricks using the below command.
password = dbutils.secrets.get(scope = <name_of_scope>, key = "<name_of_secret>)
Delete a secret scope from Databricks
Unfortunately, it is not possible to delete a secret scope using GUI. The alternative option is to use either Databricks CLI or Databricks Rest API for deletion. You will see more details about this in my next blog.
Pro tips:
1. Databricks provide a free community version where you can learn and explore Databricks. You can signup here.
2. By managing secret scopes in Databricks, you can keep your sensitive data secure while allowing authorized users and applications to access it when needed.
3. If you’re aiming to obtain the Databricks certified Data Engineer Associate certification, take a look at these helpful tips.
4. Learn how to mount and unmount data lake gen2 storage in Databricks.
Pavan Bangad
9+ years of experience in building data warehouse and big data application.
Helping customers in their digital transformation journey in cloud.
Passionate about data engineering.