Pre-requisites:
1. Azure DevOps project and a key vault.
2. Permission to access Microsoft Entra ID applications.

What is Azure DevOps project service principal?

To know the service principle (entra application ID) associated with Azure DevOps project

1. navigate to ‘Project Settings’ – > ‘Service connections’.

2. Open any service connection and click on the ‘Manage Service Principal’ link as shown in the image below.

3. This will open the Microsoft Entra ID application associated with the DevOps project in Azure portal.

4. Note the Application (client) ID. This is the application ID to which we would need to grant access to Azure key vault.

Grant access to Azure DevOps application ID on Azure key vault

1. Navigate to the key vault in Azure portal that you want to access in Azure DevOps project pipelines.

2. Click on ‘Access policies’ on the left side navigation list and click ‘Create’ to create a new access policy.

3. Select the required permissions for the pipeline and Click ‘Next’.

4. Search for the application ID (from the previous section) and select the application.

5. Authorise the app to perform the specified permissions on the User’s or Group’s behalf.

6. ‘Review’ the steps and click ”Create’ to grant access to the DevOps project on Azure key vault.

Access the key vault secrets in Azure DevOps YAML pipeline

Once the access is set up, you can add a Azure Key Vault Task in your YAML pipeline to access key vault secrets.

          - task: AzureKeyVault@1
            inputs:
              azureSubscription: 'service-connection-name'
              KeyVaultName: 'key-vault-name'
              SecretsFilter: 'secret1,secret2'
              RunAsPreJob: true

Pro tips:
1. Learn how to access Key Vault secrets in Azure Data Factory.

See more

The post Access Key Vault from Azure DevOps Pipeline appeared first on AzureOps.

Pre-requisites:
1. Azure Az PowerShell module installed and you have the appropriate permissions to access and modify the Azure Key Vault.
2. Azure key vault with appropriate permissions.

Update key vault secrets using PowerShell

Run below PowerShell command in any editor like PowerShell ISE.

$azureTenantId = "tenantid"
$subscriptionId = "subscriptionId"
$keyVaultName = "vaultname"
"'Log in to Azure..."
Connect-AzAccount -Tenant $azureTenantId  -Subscription $subscriptionId
Set-AzContext -TenantId $azureTenantId
$secretAccessKeyID = Set-AzKeyVaultSecret -VaultName $keyVaultName -Name <key name> -SecretValue <secret value> 

Here,

Connect-AzAccount -Tenant $azureTenantId -Subscription $subscriptionId
This cmdlet connects to your Azure account using the specified Azure Tenant ID and Subscription ID. It initiates an authentication process to sign in to Azure.

Set-AzContext -TenantId $azureTenantId
This cmdlet sets the context for the current session to the specified Azure Tenant ID. It ensures that it can perform subsequent operations in the context of the specified Azure tenant.

$secretAccessKeyID = Set-AzKeyVaultSecret -VaultName $keyVaultName -Name <key name> -SecretValue <secret value>
This line updates a secret in the Azure Key Vault specified by $keyVaultName. It uses the Set-AzKeyVaultSecret cmdlet to perform this task. Replace <key name> with the name of the secret you want to update, and <secret value> with the new value for the secret.

Update secret in azure key vault using managed identity

$azureTenantId =  "tenantid"
$keyVaultName = "vaultname"
"'Log in to Azure…"
Connect-AzAccount -Identity
Set-AzContext -TenantId $azureTenantId
$secretAccessKeyID = Set-AzKeyVaultSecret -VaultName $keyVaultName -Name <key name> -SecretValue <secret value> 

Here,

Connect-AzAccount -Identity
To connect to Azure, this command uses the Managed Service Identity (MSI) of the current environment where the script is running. With MSI, you can access Azure resources without the need for explicit credentials in the code. However, it is important to note that this requires the environment to have a system-assigned or user-assigned managed identity with the necessary permissions to access the Azure Key Vault.

Set-AzContext -TenantId $azureTenantId
This cmdlet sets the context for the current session to the specified Azure Tenant ID. It ensures that subsequent operations are performed in the context of the specified Azure tenant. The value for $azureTenantId was retrieved from the Azure Automation variable.

$secretAccessKeyID = Set-AzKeyVaultSecret -VaultName $keyVaultName -Name <key name> -SecretValue <secret value>
This line updates a secret in the Azure Key Vault specified by $keyVaultName. It uses the Set-AzKeyVaultSecret cmdlet to perform this task. Replace <key name> with the name of the secret you want to update, and <secret value> with the new value for the secret.

Pro tips:
1. Be cautious while updating secrets; they are sensitive information and should be handled securely.
2. While we can only change secret attributes such as expiration date, activation date. However, we can add a new version of the existing secret
3. Follow this article if you want to learn how to access key vault secrets in Azure Data Factory.

See more

The post Update Secret in Azure Key Vault using PowerShell appeared first on AzureOps.

Pre-requisites
1. Azure subscription with access to deploy Azure resources.
2. Az PowerShell module installed on the local machine.
3. Azure Key Vault soft-delete should be enabled on the vaults to be recovered.

What is a key vault soft delete?

Key vault’s soft delete feature enables the recovery of deleted vaults and deleted key vault objects (like keys, secrets, and certificates), Microsoft Azure retains soft-deleted key vault for a specified period (90 days by default). The service further provides a mechanism to recover the deleted object. Read more about it here.

How to restore a deleted key vault?

Follow the below three steps to recover the deleted Key Vault in Azure using the Az PowerShell module.

1. Connect to the Azure account

Execute the below command in PowerShell to connect to a specific Azure Tenant.

Connect-AzAccount

2. Check what key vaults are eligible for recovery

Let us now check what Azure Key Vaults are eligible for recovery from the deleted state. The below command will list down all the soft deleted key vaults in the last 90 days. Specify VaultName parameter to check a specific key vault for recovery or specify Location parameter to check all eligible vaults from a particular location within the logged-in Tenant.

Get-AzKeyVault -InRemovedState

3. Undo key vault deletion

Run the below command to recover a specific soft deleted key vault from a particular location and resource group.

Undo-AzKeyVaultRemoval -VaultName VaultName -ResourceGroupName ResourceGroupName -Location Location

We have just seen how to recover a deleted key vault in Azure.

How to recover deleted key vault objects?

Follow below quick steps to recover soft deleted key vault keys, secrets, or certificates.

1. Log in to the Azure portal.
2. Navigate to the key vault containing soft-deleted secrets, keys, or certificates.
3. Select the blade corresponding to the secret type you want to manage (keys, secrets, or certificates).
4. At the top of the screen, click on “Manage deleted (keys, secrets, or certificates).
5. A context pane will appear on the right side of your screen.
6. Select the secret, key, or certificate you want to recover and select the recovery option.

How to enable secrets in key vault

In case the secret in the key vault is disabled and you want to enable it. Right click on the secret and click Enable.

Pro tips:
1. If you also have deleted the resource group along with the Key Vault, you would need to create the resource group with the same name before recovering the Key Vault.
2. Soft-Delete feature in Azure Key-Vault would be compulsory by February 2025.

We have seen steps we can follow for azure key vault recovery.

See more

The post Recover Deleted Key Vault in Azure appeared first on AzureOps.

Pre-requisites:
To mount a location, you would need:
1. Databricks service in Azure, GCP, or AWS cloud.
2. A Databricks cluster.
3. Azure subscription with Azure Key Vault service created.

What are Secret scopes in Databricks?

When working with various applications, the Databricks platform comes in handy. To establish connections, credentials or secrets are necessary, which can be securely stored in Databricks or Azure Key Vault. Secret scopes are responsible for managing these secrets in either Azure Key Vault or Databricks.y Vault or Databricks.y Vault or Databricks.

Databricks supports two secret scopes :
1. Azure Key Vault backed scopes: to manage secrets stored in the Azure Key Vault.
2. Databricks-backed scopes: to manage secrets stored in Databricks.

Secret Scopes vs Key Vault-Backed Scopes

This article will focus on how to manage Azure Key Vault-backed secret scopes.

Create an Azure Key Vault-backed scope

Follow the below steps to create an Azure Key Vault-backed secret scope.

1. Open https://<databricks-instance>#secrets/createScope URL

2. Provide the below details:

Scope Name: <Name of the scope>

Manage Principal: Using this option, you can specify what all users can manage the secret scope. You can either select “All Users” or “Create’.

DNS Name and Resource ID: Both these properties can be found in Azure Key Vault service properties.

3. Click on Create. This will create secret scope.

Access a secret from the Azure Key Vault in Databricks

We can access secrets in Databricks using the below command.

password =  dbutils.secrets.get(scope = <name_of_scope>, key = "<name_of_secret>)

Delete a secret scope from Databricks

Unfortunately, it is not possible to delete a secret scope using GUI. The alternative option is to use either Databricks CLI or Databricks Rest API for deletion.

FAQ

Pro tips:
1. Databricks provides a free community version where you can learn and explore Databricks. You can signup here.
2. By managing secret scopes in Databricks, you can keep your sensitive data secure while allowing authorized users and applications to access it when needed.
3. If you’re aiming to obtain the Databricks certified Data Engineer Associate certification, take a look at these helpful tips.
4. Learn how to mount and unmount data lake gen2 storage in Databricks.
5. Learn how to automate Databricks IAAC using Terraform.

The post Databricks Secret Scopes – How to Create, Manage, and Use Securely appeared first on AzureOps.

What is Azure Key Vault?

Azure Key Vault is a cloud service used to securely save and access secrets. The secret could be anything we want to secure, like API keys, credentials, etc. It provides data encryption when it’s moving from a key vault to a client application, making it more secure. Read more about it here.

Let’s see it in action

Pre-requisites:
To setup Azure key vault for storing Azure Data Factory credentials, we need:
1. Azure subscription and a resource group with Azure Data Factory and Azure key-vault.
2. Permission on key vault for setting up Access policies.

Follow the below steps to use Azure key vault secrets in Azure Data Factory.

1. Create linked service for the key vault in Azure Data Factory (ADF).

Key vault base URL is https://<keyvaultName>.vault.azure.net

2. Grant access to Azure Data Factory service principal on Azure key vault. Follow the below steps to do this.

a. Navigate to key vault resource in Azure portal and click on Access policies

b. Click on Add Access Policy followed by Select principal and search for Azure Data Factory resource by its name. Here, the service principal is nothing but an app (managed identity) created for the data factory in Azure active directory,

In case the managed identity for Azure Data Factory does not exist in the Azure active directory, you can create it by running below Azure CLI.

#Generate managed identity for Data Factory.
Set-AzDataFactoryV2 -ResourceGroupName $resourceGroupName -Name $factoryName -Location $location

Select at least get and set permission for Key permissions and Secret permissions, as shown in the image below.

3. After permission configuration, the connection with the key vault should be successful.

Pro tips:
1. Data Factory cannot store credentials in a Git repository; hence, it is advisable to use Azure key vault to store credentials. This will avoid the immediate publishing of linked services during the development.
2. Follow this article, if you want to learn how to update key vault secrets using PowerShell.
3. Learn how to access Key Vault secrets in Azure DevOps deployment pipelines.

See more

The post Use Key Vault Secrets in Azure Data Factory appeared first on AzureOps.