Pre-requisites:
To implement Azure Data Factory automation using Azure automation account you would need,
1. Azure subscription and a resource group with Azure Data Factory and Azure Automation Account.
2. Admin permission to grant access to Azure automation account on Azure Data Factory.

What is Managed Identity?

Managed identity is a service principal associated with resources in Azure. When we create a Data Factory in Azure, it automatically creates an app in Microsoft Entra ID (Azure active directory).To connect Azure Data Factory from Azure automation account, we need to grant Azure automation account’s system-assigned managed identity access to the Azure Data Factory. This makes access management more secure and easy.

Benefits of using managed identity authentication

1. You don’t need to manage credentials.
2. You can use managed identities to authenticate to any resource that supports Entra ID authentication, including your applications.
3. Managed identities can be used at no extra cost.

Types of managed identities

There are two types of managed identities:
1. System assigned managed identity – This is the identity that is associated with Azure resources like Azure automation account.
2. User-assigned managed identity – This identity is created and managed by the user in Azure portal. Learn more about it here.

Let us now see how the system-assigned managed identity of Azure automation account can be used to connect and manage Azure Data Factory integration runtime.

Enable the System assigned managed identity for Azure automation account

1. Navigate to Azure automation account resource in Azure portal.

2. Click ‘Identity’ under ‘Account Settings’ as shown in the image below.

3. Under the System assigned tab, enable Status and click Save.

Grant access to Azure automation account on Azure Data Factory

1. Navigate to Data Factory resource in Azure portal and Click on Access control (IAM). Click Add -> Add role assignment as shown in the image below.

2. Grant sufficient role to Azure automation account resource on Data Factory resource under the Role tab. Job function role – ‘Data Factory Contributor’ should be sufficient to perform management activities on Data Factory.

3. Under the Members tab, select ‘Managed Identity’. Search for the managed identity of Azure automation account to whom you want to grant access.

Start And Stop Azure Data Factory IR using PowerShell runbook

1. Import Azure Az PowerShell modules in automation account. Under the automation account resource, navigate to ‘Shared resources’ -> ‘Modules’.

2. Create ‘datafactory-name’ and ‘resourcegroup-name’ variables in the automation account and assign appropriate values to them.

3. Navigate to ‘Process Automation’ left side section under Azure automation account and click ‘Runbooks’ . Create a new PowerShell runbook and provide necessary details.

4. You can use the code below to connect and stop/start Integration runtime in Azure Data Factory.

Stop ADF integration runtime

# Retrieve Automation Variables
$DataFactoryName = Get-AzAutomationVariable -Name 'datafactory-name'
$ResourceGroup = Get-AzAutomationVariable -Name 'resourcegroup-name'

# Authenticate to Azure
Connect-AzAccount

# Get Integration Runtime in Azure Data Factory
$IntegrationRuntime = Get-AzDataFactoryV2IntegrationRuntime `
                            -DataFactoryName $DataFactoryName `
                            -ResourceGroupName $ResourceGroup

# Check the current status of the Integration Runtime
$IRStatus = $IntegrationRuntime.Status
Write-Output "Current Integration Runtime Status: $IRStatus"

# Stop the Integration Runtime if it is running
if ($IRStatus -eq 'Online') {
    Stop-AzDataFactoryV2IntegrationRuntime `
        -DataFactoryName $DataFactoryName `
        -ResourceGroupName $ResourceGroup `
        -Name $IntegrationRuntime.Name

    Write-Output "Integration Runtime '$($IntegrationRuntime.Name)' has been stopped."
} else {
    Write-Output "Integration Runtime '$($IntegrationRuntime.Name)' is already stopped or in a state that cannot be stopped."
}

Start ADF integration runtime

# Retrieve Automation Variables
$DataFactoryName = Get-AzAutomationVariable -Name 'datafactory-name'
$ResourceGroup = Get-AzAutomationVariable -Name 'resourcegroup-name'

# Authenticate to Azure
Connect-AzAccount

# Get Integration Runtime in Azure Data Factory
$IntegrationRuntime = Get-AzDataFactoryV2IntegrationRuntime `
                            -DataFactoryName $DataFactoryName `
                            -ResourceGroupName $ResourceGroup

# Check the current status of the Integration Runtime
$IRStatus = $IntegrationRuntime.Status
Write-Output "Current Integration Runtime Status: $IRStatus"

# Start the Integration Runtime if it is not running
if ($IRStatus -ne 'Online') {
    Start-AzDataFactoryV2IntegrationRuntime `
        -DataFactoryName $DataFactoryName `
        -ResourceGroupName $ResourceGroup `
        -Name $IntegrationRuntime.Name

    Write-Output "Integration Runtime '$($IntegrationRuntime.Name)' is starting."
} else {
    Write-Output "Integration Runtime '$($IntegrationRuntime.Name)' is already running."
}

Notice in the above code blocks that we have not used any credentials to authenticate to Azure. This is because authentication is taken care of using managed identity.

Pro tips:
1. Learn how to connect to Azure SQL from Azure Data Factory using managed identity.
2. Azure Automation Run as accounts, including Classic Run as accounts have retired on 30 September 2023. It is now mandatory to use the managed identity.

See more

The post Programmatically Start and Stop Azure Data Factory Integration Runtime appeared first on AzureOps.

Pre-requisites:
1. Access to Data Factory resources in the Azure portal.
2. Az PowerShell module installed on the local machine.

Understanding Azure Data Factory Triggers

Azure Data Factory utilizes triggers to initiate data pipeline executions based on specific conditions automatically. These conditions can be set on a schedule, event, or through manual intervention. Triggers play a crucial role in ensuring that data pipelines run at the correct time, keeping data processing up-to-date and accurate. Turning triggers on-demand on and off gives data engineers the flexibility to control the data flow efficiently.

Script to disable and enable triggers in Azure Data Factory

Use the below PowerShell script to disable all the triggers in Azure Data Factory.

$adf_name = 'adfname'
$resourcegroup_name = 'resourcegroupname'

$triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName $adf_name -ResourceGroupName $resourcegroup_name; 
$triggersADF | ForEach-Object { Stop-AzDataFactoryV2Trigger -ResourceGroupName $resourcegroup_name -DataFactoryName $adf_name -Name $_.name -Force }

Use the below PowerShell script to enable all the triggers in Azure Data Factory.

$adf_name = 'adfname'
$resourcegroup_name = 'resourcegroupname'

$triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName $adf_name -ResourceGroupName $resourcegroup_name; 
$triggersADF | ForEach-Object { Start-AzDataFactoryV2Trigger -ResourceGroupName $resourcegroup_name -DataFactoryName $adf_name -Name $_.name -Force }

Manage ADF triggers in YAML based Azure DevOps pipeline

Use the script below to enable and disable triggers in adf using a YAML-based]deployment pipeline in Azure DevOps.

          deploy:  #Configure steps under deploy hook.
            steps:
              #task to disable triggers in Azure Data Factory.
              - task: AzurePowerShell@5     
                displayName: Stop Triggers
                inputs:
                  azureSubscription: 'subscriptionname'
                  ScriptType: 'InlineScript'
                  Inline: |
                    $triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName "$(dataFactoryName)" -ResourceGroupName "$(resourceGroup)"; 
                    $triggersADF | ForEach-Object { Stop-AzDataFactoryV2Trigger -ResourceGroupName "$(resourceGroup)" -DataFactoryName "$(dataFactoryName)" -Name $_.name -Force }
                  azurePowerShellVersion: 'LatestVersion'             
             
              #task to enable triggers in Azure Data Factory.
              - task: AzurePowerShell@5    
                displayName: Start Triggers
                inputs:
                  azureSubscription: 'subscriptionname'
                  ScriptType: 'InlineScript'
                  Inline: |
                    $triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName "$(dataFactoryName)" -ResourceGroupName "$(resourceGroup)";
                    $triggersADF | ForEach-Object { Start-AzDataFactoryV2Trigger -ResourceGroupName "$(resourceGroup)" -DataFactoryName "$(dataFactoryName)" -Name $_.name -Force }
                  azurePowerShellVersion: 'LatestVersion'

Pro tips:
1. It is important to disable triggers before deploying data factory code using Azure DevOps pipelines. Refer to this article if you want to automate Azure data factory deployments using the DevOps pipeline.

See more

The post Disable and Enable triggers in Azure Data Factory using PowerShell appeared first on AzureOps.

Pre-requisites:
1. Azure Az PowerShell module installed and you have the appropriate permissions to access and modify the Azure Key Vault.
2. Azure key vault with appropriate permissions.

Update key vault secrets using PowerShell

Run below PowerShell command in any editor like PowerShell ISE.

$azureTenantId = "tenantid"
$subscriptionId = "subscriptionId"
$keyVaultName = "vaultname"
"'Log in to Azure..."
Connect-AzAccount -Tenant $azureTenantId  -Subscription $subscriptionId
Set-AzContext -TenantId $azureTenantId
$secretAccessKeyID = Set-AzKeyVaultSecret -VaultName $keyVaultName -Name <key name> -SecretValue <secret value> 

Here,

Connect-AzAccount -Tenant $azureTenantId -Subscription $subscriptionId
This cmdlet connects to your Azure account using the specified Azure Tenant ID and Subscription ID. It initiates an authentication process to sign in to Azure.

Set-AzContext -TenantId $azureTenantId
This cmdlet sets the context for the current session to the specified Azure Tenant ID. It ensures that it can perform subsequent operations in the context of the specified Azure tenant.

$secretAccessKeyID = Set-AzKeyVaultSecret -VaultName $keyVaultName -Name <key name> -SecretValue <secret value>
This line updates a secret in the Azure Key Vault specified by $keyVaultName. It uses the Set-AzKeyVaultSecret cmdlet to perform this task. Replace <key name> with the name of the secret you want to update, and <secret value> with the new value for the secret.

Update secret in azure key vault using managed identity

$azureTenantId =  "tenantid"
$keyVaultName = "vaultname"
"'Log in to Azure…"
Connect-AzAccount -Identity
Set-AzContext -TenantId $azureTenantId
$secretAccessKeyID = Set-AzKeyVaultSecret -VaultName $keyVaultName -Name <key name> -SecretValue <secret value> 

Here,

Connect-AzAccount -Identity
To connect to Azure, this command uses the Managed Service Identity (MSI) of the current environment where the script is running. With MSI, you can access Azure resources without the need for explicit credentials in the code. However, it is important to note that this requires the environment to have a system-assigned or user-assigned managed identity with the necessary permissions to access the Azure Key Vault.

Set-AzContext -TenantId $azureTenantId
This cmdlet sets the context for the current session to the specified Azure Tenant ID. It ensures that subsequent operations are performed in the context of the specified Azure tenant. The value for $azureTenantId was retrieved from the Azure Automation variable.

$secretAccessKeyID = Set-AzKeyVaultSecret -VaultName $keyVaultName -Name <key name> -SecretValue <secret value>
This line updates a secret in the Azure Key Vault specified by $keyVaultName. It uses the Set-AzKeyVaultSecret cmdlet to perform this task. Replace <key name> with the name of the secret you want to update, and <secret value> with the new value for the secret.

Pro tips:
1. Be cautious while updating secrets; they are sensitive information and should be handled securely.
2. While we can only change secret attributes such as expiration date, activation date. However, we can add a new version of the existing secret
3. Follow this article if you want to learn how to access key vault secrets in Azure Data Factory.

See more

The post Update Secret in Azure Key Vault using PowerShell appeared first on AzureOps.