Azure Data Factory is a Microsoft cloud ETL service for serverless data integrations and orchestrations. Azure automation account is a service by Microsoft to automate various management activities in Azure cloud. If you want to start and stop Azure Data Factory SSIS runtime programmatically, you can leverage an automation account. This article describes how to perform Azure Data Factory automation with managed identity and Azure automation account.
Pre-requisites:
To implement Azure Data Factory automation using Azure automation account you would need,
1. Azure subscription and a resource group with Azure Data Factory and Azure Automation Account.
2. Admin permission to grant access to Azure automation account on Azure Data Factory.
What is Managed Identity?
Managed identity is a service principal associated with resources in Azure. When we create a Data Factory in Azure, it automatically creates an app in Microsoft Entra ID (Azure active directory).To connect Azure Data Factory from Azure automation account, we need to grant Azure automation account’s system-assigned managed identity access to the Azure Data Factory. This makes access management more secure and easy.
Benefits of using managed identity authentication
1. You don’t need to manage credentials.
2. You can use managed identities to authenticate to any resource that supports Entra ID authentication, including your applications.
3. Managed identities can be used at no extra cost.
Types of managed identities
There are two types of managed identities:
1. System assigned managed identity – This is the identity that is associated with Azure resources like Azure automation account.
2. User-assigned managed identity – This identity is created and managed by the user in Azure portal. Learn more about it here.
Let us now see how the system-assigned managed identity of Azure automation account can be used to connect and manage Azure Data Factory integration runtime.
Enable the System assigned managed identity for Azure automation account
1. Navigate to Azure automation account resource in Azure portal.
2. Click ‘Identity’ under ‘Account Settings’ as shown in the image below.
3. Under the System assigned tab, enable Status and click Save.
Grant access to Azure automation account on Azure Data Factory
1. Navigate to Data Factory resource in Azure portal and Click on Access control (IAM). Click Add -> Add role assignment as shown in the image below.
2. Grant sufficient role to Azure automation account resource on Data Factory resource under the Role tab. Job function role – ‘Data Factory Contributor’ should be sufficient to perform management activities on Data Factory.
3. Under the Members tab, select ‘Managed Identity’. Search for the managed identity of Azure automation account to whom you want to grant access.
Manage Azure Data Factory using PowerShell runbook
Create a new PowerShell runbook in Azure automation account and import necessary Azure modules. Depending on what PowerShell modules (AzureRM or Az) you choose, you can use the below code to connect and manage Integration runtime in Azure Data Factory.
For AzureRM module
$DataFactoryName = Get-AutomationVariable -Name 'datafactory-name'
$ResourceGroup = Get-AutomationVariable -Name 'resourcegroup-name'
Connect-AzureRMAccount -Identity
'Log in to Azure...'
# Get Integration Runtime in Azure Data Factory
$IntegrationRuntime = Get-AzureRmDataFactoryV2IntegrationRuntime `
-DataFactoryName $DataFactoryName `
-ResourceGroupName $ResourceGroup
For Az module
$DataFactoryName = Get-AzAutomationVariable -Name 'datafactory-name'
$ResourceGroup = Get-AzAutomationVariable -Name 'resourcegroup-name'
Connect-AzAccount
# Get Integration Runtime in Azure Data Factory
$IntegrationRuntime = Get-AzDataFactoryV2IntegrationRuntime `
-DataFactoryName $DataFactoryName `
-ResourceGroupName $ResourceGroup
Notice in the above code blocks that we have not used any credentials to authenticate to Azure. This is because authentication is taken care of using managed identity.
Pro tips:
1. Learn how to connect to Azure SQL from Azure Data Factory using managed identity.
2. Azure Automation Run as accounts, including Classic Run as accounts have retired on 30 September 2023. It is now mandatory to use the managed identity.
See more
Kunal Rathi
With over a decade of experience in data engineering and analytics, I've assisted countless clients in gaining valuable insights from their data. As a dedicated supporter of Data, Cloud and DevOps, I'm excited to connect with individuals who share my passion for this field. If my work resonates with you, we can talk and collaborate.
I am always interested in new challenges so if you need consulting help, reach me at kunalrathi55@gmail.com.