Azure Data Factory is a Microsoft cloud ETL service for serverless data integrations and orchestrations. Azure automation account is a service by Microsoft to automate various management activities in Azure cloud. You can leverage an automation account if you want to start and stop Azure Data Factory SSIS runtime programmatically. This article describes how to programmatically start and stop Azure Data Factory integration runtime using managed identity and Azure automation account.
Pre-requisites:
To implement Azure Data Factory automation using Azure automation account you would need,
1. Azure subscription and a resource group with Azure Data Factory and Azure Automation Account.
2. Admin permission to grant access to Azure automation account on Azure Data Factory.
What is Managed Identity?
Managed identity is a service principal associated with resources in Azure. When we create a Data Factory in Azure, it automatically creates an app in Microsoft Entra ID (Azure active directory).To connect Azure Data Factory from Azure automation account, we need to grant Azure automation account’s system-assigned managed identity access to the Azure Data Factory. This makes access management more secure and easy.
Benefits of using managed identity authentication
1. You don’t need to manage credentials.
2. You can use managed identities to authenticate to any resource that supports Entra ID authentication, including your applications.
3. Managed identities can be used at no extra cost.
Types of managed identities
There are two types of managed identities:
1. System assigned managed identity – This is the identity that is associated with Azure resources like Azure automation account.
2. User-assigned managed identity – This identity is created and managed by the user in Azure portal. Learn more about it here.
Let us now see how the system-assigned managed identity of Azure automation account can be used to connect and manage Azure Data Factory integration runtime.
Enable the System assigned managed identity for Azure automation account
1. Navigate to Azure automation account resource in Azure portal.
2. Click ‘Identity’ under ‘Account Settings’ as shown in the image below.
3. Under the System assigned tab, enable Status and click Save.
Grant access to Azure automation account on Azure Data Factory
1. Navigate to Data Factory resource in Azure portal and Click on Access control (IAM). Click Add -> Add role assignment as shown in the image below.
2. Grant sufficient role to Azure automation account resource on Data Factory resource under the Role tab. Job function role – ‘Data Factory Contributor’ should be sufficient to perform management activities on Data Factory.
3. Under the Members tab, select ‘Managed Identity’. Search for the managed identity of Azure automation account to whom you want to grant access.
Start And Stop Azure Data Factory IR using PowerShell runbook
1. Import Azure Az PowerShell modules in automation account. Under the automation account resource, navigate to ‘Shared resources’ -> ‘Modules’.
2. Create ‘datafactory-name’ and ‘resourcegroup-name’ variables in the automation account and assign appropriate values to them.
3. Navigate to ‘Process Automation’ left side section under Azure automation account and click ‘Runbooks’ . Create a new PowerShell runbook and provide necessary details.
4. You can use the code below to connect and stop/start Integration runtime in Azure Data Factory.
Stop ADF integration runtime
# Retrieve Automation Variables
$DataFactoryName = Get-AzAutomationVariable -Name 'datafactory-name'
$ResourceGroup = Get-AzAutomationVariable -Name 'resourcegroup-name'
# Authenticate to Azure
Connect-AzAccount
# Get Integration Runtime in Azure Data Factory
$IntegrationRuntime = Get-AzDataFactoryV2IntegrationRuntime `
-DataFactoryName $DataFactoryName `
-ResourceGroupName $ResourceGroup
# Check the current status of the Integration Runtime
$IRStatus = $IntegrationRuntime.Status
Write-Output "Current Integration Runtime Status: $IRStatus"
# Stop the Integration Runtime if it is running
if ($IRStatus -eq 'Online') {
Stop-AzDataFactoryV2IntegrationRuntime `
-DataFactoryName $DataFactoryName `
-ResourceGroupName $ResourceGroup `
-Name $IntegrationRuntime.Name
Write-Output "Integration Runtime '$($IntegrationRuntime.Name)' has been stopped."
} else {
Write-Output "Integration Runtime '$($IntegrationRuntime.Name)' is already stopped or in a state that cannot be stopped."
}
Start ADF integration runtime
# Retrieve Automation Variables
$DataFactoryName = Get-AzAutomationVariable -Name 'datafactory-name'
$ResourceGroup = Get-AzAutomationVariable -Name 'resourcegroup-name'
# Authenticate to Azure
Connect-AzAccount
# Get Integration Runtime in Azure Data Factory
$IntegrationRuntime = Get-AzDataFactoryV2IntegrationRuntime `
-DataFactoryName $DataFactoryName `
-ResourceGroupName $ResourceGroup
# Check the current status of the Integration Runtime
$IRStatus = $IntegrationRuntime.Status
Write-Output "Current Integration Runtime Status: $IRStatus"
# Start the Integration Runtime if it is not running
if ($IRStatus -ne 'Online') {
Start-AzDataFactoryV2IntegrationRuntime `
-DataFactoryName $DataFactoryName `
-ResourceGroupName $ResourceGroup `
-Name $IntegrationRuntime.Name
Write-Output "Integration Runtime '$($IntegrationRuntime.Name)' is starting."
} else {
Write-Output "Integration Runtime '$($IntegrationRuntime.Name)' is already running."
}
Notice in the above code blocks that we have not used any credentials to authenticate to Azure. This is because authentication is taken care of using managed identity.
Pro tips:
1. Learn how to connect to Azure SQL from Azure Data Factory using managed identity.
2. Azure Automation Run as accounts, including Classic Run as accounts have retired on 30 September 2023. It is now mandatory to use the managed identity.
See more
Kunal Rathi
With over 13 years of experience in data engineering and analytics, I've assisted countless clients in gaining valuable insights from their data. As a dedicated supporter of Data, Cloud and DevOps, I'm excited to connect with individuals who share my passion for this field. If my work resonates with you, we can talk and collaborate.