Pre-requisites:
To implement Azure Data Factory automation using Azure automation account you would need,
1. Azure subscription and a resource group with Azure Data Factory and Azure Automation Account.
2. Admin permission to grant access to Azure automation account on Azure Data Factory.

What is Managed Identity?

Managed identity is a service principal associated with resources in Azure. When we create a Data Factory in Azure, it automatically creates an app in Microsoft Entra ID (Azure active directory).To connect Azure Data Factory from Azure automation account, we need to grant Azure automation account’s system-assigned managed identity access to the Azure Data Factory. This makes access management more secure and easy.

Benefits of using managed identity authentication

1. You don’t need to manage credentials.
2. You can use managed identities to authenticate to any resource that supports Entra ID authentication, including your applications.
3. Managed identities can be used at no extra cost.

Types of managed identities

There are two types of managed identities:
1. System assigned managed identity – This is the identity that is associated with Azure resources like Azure automation account.
2. User-assigned managed identity – This identity is created and managed by the user in Azure portal. Learn more about it here.

Let us now see how the system-assigned managed identity of Azure automation account can be used to connect and manage Azure Data Factory integration runtime.

Enable the System assigned managed identity for Azure automation account

1. Navigate to Azure automation account resource in Azure portal.

2. Click ‘Identity’ under ‘Account Settings’ as shown in the image below.

3. Under the System assigned tab, enable Status and click Save.

Grant access to Azure automation account on Azure Data Factory

1. Navigate to Data Factory resource in Azure portal and Click on Access control (IAM). Click Add -> Add role assignment as shown in the image below.

2. Grant sufficient role to Azure automation account resource on Data Factory resource under the Role tab. Job function role – ‘Data Factory Contributor’ should be sufficient to perform management activities on Data Factory.

3. Under the Members tab, select ‘Managed Identity’. Search for the managed identity of Azure automation account to whom you want to grant access.

Start And Stop Azure Data Factory IR using PowerShell runbook

1. Import Azure Az PowerShell modules in automation account. Under the automation account resource, navigate to ‘Shared resources’ -> ‘Modules’.

2. Create ‘datafactory-name’ and ‘resourcegroup-name’ variables in the automation account and assign appropriate values to them.

3. Navigate to ‘Process Automation’ left side section under Azure automation account and click ‘Runbooks’ . Create a new PowerShell runbook and provide necessary details.

4. You can use the code below to connect and stop/start Integration runtime in Azure Data Factory.

Stop ADF integration runtime

# Retrieve Automation Variables
$DataFactoryName = Get-AzAutomationVariable -Name 'datafactory-name'
$ResourceGroup = Get-AzAutomationVariable -Name 'resourcegroup-name'

# Authenticate to Azure
Connect-AzAccount

# Get Integration Runtime in Azure Data Factory
$IntegrationRuntime = Get-AzDataFactoryV2IntegrationRuntime `
                            -DataFactoryName $DataFactoryName `
                            -ResourceGroupName $ResourceGroup

# Check the current status of the Integration Runtime
$IRStatus = $IntegrationRuntime.Status
Write-Output "Current Integration Runtime Status: $IRStatus"

# Stop the Integration Runtime if it is running
if ($IRStatus -eq 'Online') {
    Stop-AzDataFactoryV2IntegrationRuntime `
        -DataFactoryName $DataFactoryName `
        -ResourceGroupName $ResourceGroup `
        -Name $IntegrationRuntime.Name

    Write-Output "Integration Runtime '$($IntegrationRuntime.Name)' has been stopped."
} else {
    Write-Output "Integration Runtime '$($IntegrationRuntime.Name)' is already stopped or in a state that cannot be stopped."
}

Start ADF integration runtime

# Retrieve Automation Variables
$DataFactoryName = Get-AzAutomationVariable -Name 'datafactory-name'
$ResourceGroup = Get-AzAutomationVariable -Name 'resourcegroup-name'

# Authenticate to Azure
Connect-AzAccount

# Get Integration Runtime in Azure Data Factory
$IntegrationRuntime = Get-AzDataFactoryV2IntegrationRuntime `
                            -DataFactoryName $DataFactoryName `
                            -ResourceGroupName $ResourceGroup

# Check the current status of the Integration Runtime
$IRStatus = $IntegrationRuntime.Status
Write-Output "Current Integration Runtime Status: $IRStatus"

# Start the Integration Runtime if it is not running
if ($IRStatus -ne 'Online') {
    Start-AzDataFactoryV2IntegrationRuntime `
        -DataFactoryName $DataFactoryName `
        -ResourceGroupName $ResourceGroup `
        -Name $IntegrationRuntime.Name

    Write-Output "Integration Runtime '$($IntegrationRuntime.Name)' is starting."
} else {
    Write-Output "Integration Runtime '$($IntegrationRuntime.Name)' is already running."
}

Notice in the above code blocks that we have not used any credentials to authenticate to Azure. This is because authentication is taken care of using managed identity.

Pro tips:
1. Learn how to connect to Azure SQL from Azure Data Factory using managed identity.
2. Azure Automation Run as accounts, including Classic Run as accounts have retired on 30 September 2023. It is now mandatory to use the managed identity.

See more

The post Programmatically Start and Stop Azure Data Factory Integration Runtime appeared first on AzureOps.

Pre-requisites:
1. Access to Data Factory resources in the Azure portal.
2. Az PowerShell module installed on the local machine.

Understanding Azure Data Factory Triggers

Azure Data Factory utilizes triggers to initiate data pipeline executions based on specific conditions automatically. These conditions can be set on a schedule, event, or through manual intervention. Triggers play a crucial role in ensuring that data pipelines run at the correct time, keeping data processing up-to-date and accurate. Turning triggers on-demand on and off gives data engineers the flexibility to control the data flow efficiently.

Script to disable and enable triggers in Azure Data Factory

Use the below PowerShell script to disable all the triggers in Azure Data Factory.

$adf_name = 'adfname'
$resourcegroup_name = 'resourcegroupname'

$triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName $adf_name -ResourceGroupName $resourcegroup_name; 
$triggersADF | ForEach-Object { Stop-AzDataFactoryV2Trigger -ResourceGroupName $resourcegroup_name -DataFactoryName $adf_name -Name $_.name -Force }

Use the below PowerShell script to enable all the triggers in Azure Data Factory.

$adf_name = 'adfname'
$resourcegroup_name = 'resourcegroupname'

$triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName $adf_name -ResourceGroupName $resourcegroup_name; 
$triggersADF | ForEach-Object { Start-AzDataFactoryV2Trigger -ResourceGroupName $resourcegroup_name -DataFactoryName $adf_name -Name $_.name -Force }

Manage ADF triggers in YAML based Azure DevOps pipeline

Use the script below to enable and disable triggers in adf using a YAML-based]deployment pipeline in Azure DevOps.

          deploy:  #Configure steps under deploy hook.
            steps:
              #task to disable triggers in Azure Data Factory.
              - task: AzurePowerShell@5     
                displayName: Stop Triggers
                inputs:
                  azureSubscription: 'subscriptionname'
                  ScriptType: 'InlineScript'
                  Inline: |
                    $triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName "$(dataFactoryName)" -ResourceGroupName "$(resourceGroup)"; 
                    $triggersADF | ForEach-Object { Stop-AzDataFactoryV2Trigger -ResourceGroupName "$(resourceGroup)" -DataFactoryName "$(dataFactoryName)" -Name $_.name -Force }
                  azurePowerShellVersion: 'LatestVersion'             
             
              #task to enable triggers in Azure Data Factory.
              - task: AzurePowerShell@5    
                displayName: Start Triggers
                inputs:
                  azureSubscription: 'subscriptionname'
                  ScriptType: 'InlineScript'
                  Inline: |
                    $triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName "$(dataFactoryName)" -ResourceGroupName "$(resourceGroup)";
                    $triggersADF | ForEach-Object { Start-AzDataFactoryV2Trigger -ResourceGroupName "$(resourceGroup)" -DataFactoryName "$(dataFactoryName)" -Name $_.name -Force }
                  azurePowerShellVersion: 'LatestVersion'

Pro tips:
1. It is important to disable triggers before deploying data factory code using Azure DevOps pipelines. Refer to this article if you want to automate Azure data factory deployments using the DevOps pipeline.

See more

The post Disable and Enable triggers in Azure Data Factory using PowerShell appeared first on AzureOps.

Background

When working with ETL or orchestration tools, it is crucial to have the ability to enable or disable tasks or activities within a pipeline for various reasons. Firstly, it allows one to disable specific activities while debugging the code temporarily. Secondly, it enables running only a portion of the code to accomplish specific tasks.

Microsoft recently announced that they are adding a feature to enable and disable activity in Azure Data Factory. This feature is in the preview currently.

How to disable activity in ADF?

1. Open Azure data factory studio and navigate to the pipeline.

2. Select the activity which you want to disable, and under the general property window, set the Activity state radio button to inactive, as shown in the image below.

You can also deactivate the activity with a right click on the activity and click on Deactivate (preview) as shown in the below image. Make sure to save the changes to take them into effect.

If you wish to run this pipeline using a trigger, you would need to publish changes.

Pro tips:
1. While the activity state will help enable or disable activity in ADF, it seems to lack a feature to programmatically or conditionally enable or disable it.
2. Know how to connect to Azure SQL Database from Azure Data Factory using the managed identity.

See more

The post Activate and Deactivate Activity in Azure Data Factory appeared first on AzureOps.

Cause

Azure Data Factory is missing a managed identity. This error might appear when trying to grant access to Azure Data Factory on the Azure SQL database.

Solution

System Assigned Managed identity for Azure Data Factory will be automatically created when we create it through the Azure portal or PowerShell. If you do not find the managed identity of Azure Data Factory, you can create it using the below PowerShell command. You would need the Az PowerShell module installed on the local machine.

#Generate managed identity for Data Factory.
Set-AzDataFactoryV2 -ResourceGroupName $resourceGroupName -Name $factoryName -Location $location

You can also achieve this in the Azure portal. You can go to the Azure Data Factory in the Azure portal and enable the managed identity for the data factory, as shown in the image below. This will create a managed identity for the data factory in Azure Active Directory.

Pro tips:
1. Learn how to access Azure SQL from Azure Data Factory using the managed identity.

The post Principle could not be found or this principal type is not supported appeared first on AzureOps.

Pre-requisites:
To implement linked service to Azure SQL database from Azure data factory, you would need:
1. Azure subscription and a resource group with Azure Data Factory and Azure SQL database.
2. Admin permission to grant access to Azure SQL database.
3. Allow Azure services and resources to access this server checkbox is checked on Azure SQL database. This will enable data factory to access Azure SQL database.

What is Managed Identity?

Managed identity is a service principal associated with resources in Azure. When we create a data factory in Azure, it automatically creates an app in Azure Active Directory. Suppose we want to connect Azure SQL using Azure Data Factory. In that case, we need to grant Azure Data Factory app access to the Azure SQL database like any other Azure Active Directory user. This makes access management more secure and easy.

Benefits of using managed identity authentication

1. You don’t need to manage credentials. Credentials aren’t even accessible to you.
2. You can use managed identities to authenticate to any resource that supports Azure AD authentication, including your own applications.
3. Managed identities can be used at no extra cost.

Types of managed identities

There are two types of managed identities:
1. System assigned managed identity – This is the identity that is associated with Azure resources like Azure Data Factory.
2. User assigned managed identity – This identity is created and managed by user in Azure portal. Learn more about it here.

This article will cover how to grant access to Azure data factory’s system assigned managed identity on Azure SQL database.

Enable Managed Identity on Azure Data Factory

System Assigned Managed, identity for Azure Data Factory will be automatically created when we create it through the Azure portal or PowerShell. If you do not find the managed identity of Azure Data Factory, you can create it using the below PowerShell command. You would need the Az PowerShell module installed on the local machine.

#Generate managed identity for Data Factory.
Set-AzDataFactoryV2 -ResourceGroupName $resourceGroupName -Name $factoryName -Location $location

You can also achieve this in the Azure portal. You can go to the Azure Data Factory in the Azure portal and enable the managed identity for the data factory, as shown in the image below. This will create a managed identity for the data factory in Azure Active Directory.

Note: The managed identity of Azure Data Factory must be in the same Azure Entra ID tenant as the Azure SQL database server. If the managed identity is not in the same tenant, you can use a service principal with an access token instead.

Permission to grant Azure Data Factory managed identity access to Azure SQL database

Azure Active Directory users with admin permission on Azure SQL database can only grant access to the managed identity of Azure Data Factory. In case you do not have an Active Directory Admin account set for Azure SQL database, you can do this with the below steps.

1. Navigate to Azure SQL database resource in the Azure portal and Click on Azure Active Directory under the Settings section.
2. Click on the Set admin button.
3. Search for the Azure active directory user or security group to which you want to grant admin access.
4. Click Select, and this will select the admin account.
5. Click Save to enable the admin.

Grant Azure Data Factory managed identity access to Azure SQL database

Now that we have an Azure Active Directory admin account set for Azure SQL database, we can grant access to Azure Data Factory Managed Identity to Azure SQL database. You can use Azure SQL database client tools like SQL Server Management Studio 18 or Directly from Azure SQL database portal. If you want to grant permission using client tools like SSMS 18, log in to Azure SQL database using Active Active Directory – Universal with MFA and use an Azure SQL admin account.

Create a user in Azure SQL database for Azure Data Factory and grant roles to this newly created user based on your needs.

--Add Azure Data Factory managed identity as an account to Azure SQL database.
CREATE USER [name_of_the_adf] FROM EXTERNAL PROVIDER;
--Grant appropriate role to Azure Data Factory based on your needs. 
ALTER ROLE [db_owner] ADD MEMBER [name_of_the_adf];

Create a Linked service to Azure SQL Database in Azure Data Factory

We are all set to connect Azure SQL from Data Factory using managed identity. Follow the below steps to create a linked service in Azure Data Factory for Azure SQL using managed identity authentication.

1. Navigate to Azure Data Factory and Click on New Linked Service.
2. Search for Azure SQL Database.
3. Click Continue.

1. On the new linked service page, specify Azure subscription, Azure SQL Server name, and Database name.
2. In the Authentication type, select System Assigned Managed Identity.
3. Test the connection and click on Create.

Retrieve Azure SQL connection string from key vault.

Note: The key vault based connection string option is only available in legacy version of Azure SQL database Linked Service. Recommended version only has Manual connection string option instead.

If you want to use Azure key vault instead of choosing the Azure SQL database details from Azure subscription, you would need to:

1. Create a Azure key vault with a secret containing connection string of the Azure SQL database. Format of the connection string should be like below.

Server=tcp:<yourservername>.database.windows.net,1433;Initial Catalog=<yourdatabasename>;

2. Add access policy in Azure key vault for Azure data factory. Minimum access could be ‘get’, ‘list’ on secrets.

3. Now, on Data Factory linked service to Azure SQL database, you can choose Azure Key Vault instead of Connection string and choose the secret from the key vault as shown in the below image.

You can now use this linked service in ADF activities to perform various operations on the Azure SQL database.

Pro tips:
1. If you use version control (Azure DevOps or git integration) for Azure Data Factory, you need to use a Key vault-based connection string instead of manually specifying Azure SQL database details in the linked service.
2. While creating a user for Azure Data Factory in Azure SQL database, if you get an error ‘Principal ” could not be found or this principal type is not supported.’. This means Azure Data Factory does not have managed identity set. Please refer to this article to fix it.
3. Learn how to manage Azure Data Factory using Azure automation account using managed identity authentication.

See more

The post Connect Azure SQL from Data Factory using Managed Identity appeared first on AzureOps.

What is Integration Runtime?

Integration Runtime (IR) in Azure Data Factory provides a computational infrastructure for pipelines. It is also instrumental in securing data movement in the cloud. Read more about it here.

What are the different types of integration runtime in ADF?

Azure Data Factory consists of three types of Integration runtimes:

1. Azure integration runtime: This is the default one that Azure fully manages. Suitable to connect to Azure resources. Since Azure manages it, the data movement is in the public network.
2. Self-hosted integration runtime: This IR can connect to on-prem sources and secure data movement between the systems. This IR supports the data movement in the private network.
3. Azure-SSIS integration runtime: This is used for running SSIS packages in Azure Data Factory

Let’s see it in Action.

This section describes setting up self-hosted shared integration runtime in Azure Data Factory.

Pre-requisites:
1. Azure subscription and a resource group with Azure Data Factory.
2. Self Hosted Integration runtime setup in Azure Data Factory.

1. Select the integration runtime that needs to be shared.

Select the integration runtime to share with other Azure Data Factories and Click edit.

Copy the ResourceID and save it. We will use it in our next steps.

Click on Grant Permission to another Data Factory or user-assigned managed identity button, as shown in the above image.

2. Grant permission to target Azure data factory:

Grant access to target Azure Data Factory which will use shared integration runtime.

3. Now, go to the Target Data Factory and add an Integration Runtime

Click New and select Azure, self-hosted

Under External Resources, select Linked Self-Hosted and Click Continue.

It will ask few details about the shared IR.

Name: put the name of the new IR.

Resource ID Copy and paste the resource ID we previously saved and click Create.

And we are done!

Now we are using the same integration runtime in two different Azure Data Factory instances.

Pro tips:
1. Please make sure you have Microsoft.Authorization/roleAssignments/write permission on subscription.
2. Self-hosted IR requires a virtual machine and Integration runtime software. This means we need to pay the cost of the hosted virtual machine. By sharing the IR between the multiple Data Factory instances, we can use the same virtual machine amongst multiple Data Factories.
3. If you’re aiming to obtain the Databricks certified Data Engineer Associate certification, take a look at these helpful tips.

See more

The post Shared Integration Runtime in Azure Data Factory appeared first on AzureOps.

What is Azure Key Vault?

Azure Key Vault is a cloud service used to securely save and access secrets. The secret could be anything we want to secure, like API keys, credentials, etc. It provides data encryption when it’s moving from a key vault to a client application, making it more secure. Read more about it here.

Let’s see it in action

Pre-requisites:
To setup Azure key vault for storing Azure Data Factory credentials, we need:
1. Azure subscription and a resource group with Azure Data Factory and Azure key-vault.
2. Permission on key vault for setting up Access policies.

Follow the below steps to use Azure key vault secrets in Azure Data Factory.

1. Create linked service for the key vault in Azure Data Factory (ADF).

Key vault base URL is https://<keyvaultName>.vault.azure.net

2. Grant access to Azure Data Factory service principal on Azure key vault. Follow the below steps to do this.

a. Navigate to key vault resource in Azure portal and click on Access policies

b. Click on Add Access Policy followed by Select principal and search for Azure Data Factory resource by its name. Here, the service principal is nothing but an app (managed identity) created for the data factory in Azure active directory,

In case the managed identity for Azure Data Factory does not exist in the Azure active directory, you can create it by running below Azure CLI.

#Generate managed identity for Data Factory.
Set-AzDataFactoryV2 -ResourceGroupName $resourceGroupName -Name $factoryName -Location $location

Select at least get and set permission for Key permissions and Secret permissions, as shown in the image below.

3. After permission configuration, the connection with the key vault should be successful.

Pro tips:
1. Data Factory cannot store credentials in a Git repository; hence, it is advisable to use Azure key vault to store credentials. This will avoid the immediate publishing of linked services during the development.
2. Follow this article, if you want to learn how to update key vault secrets using PowerShell.
3. Learn how to access Key Vault secrets in Azure DevOps deployment pipelines.

See more

The post Use Key Vault Secrets in Azure Data Factory appeared first on AzureOps.

What is Microsoft Graph API?

Microsoft Graph is a RESTful web API that enables you to access Microsoft Cloud service resources like Users, Groups, Mail, Calendars, Contacts, Files, etc. More information is available on the Microsoft docs.

Authentication

There are two ways to authenticate Graph API from Azure Data Factory.

Using Service principal

The service principle in simple terms is a service account. This enables programmatic access to resources in Azure. To create a service principal, we need to register an App in Azure Active Directory. This application has clientID, client Secret and app ID. These details can also be used to grant access to Microsoft Graph API.

Using Managed Identity

Managed identity is a service principal associated with resources in Azure. When we create a data factory in Azure, it automatically creates an app in Azure Active Directory. This means if we want to access Graph API using the data factory, we just need to grant Azure Data Factory app access to Graph API. This makes access management more secure and easy, isn’t it?

Let’s see it in action

This section describes how to call Graph API from Azure Data Factory using the managed identity. For demo purposes, we will get Azure active directory users’ data using Graph API and copy it to blob storage.

1. Providing Graph API access to Azure data factory.

a. Navigate to Azure active directory on Azure portal and search the data factory application (managed identity) under Enterprise applications.

b. Grant access to Data Factory app on Graph API as shown in the below image.

Note: Granting access to Graph API requires Azure active directory admin consent.

Azure data factory requires “user.read.all” permission to read users’ data from Graph API.

2. Setting up linked service for API calls.

a. Connect to the Azure portal and open the data factory. Create a new pipeline.

b. Create a new Linked service for REST API. Provide Authentication Type and AAD resource values as mentioned below.

Authentication Type : System Assigned Managed Identity

AAD resource : https://graph.microsoft.com/

3. Configure Copy Activity to get data from API and push it to blob Storage.

URL : https://graph.microsoft.com/v1.0/users

Pagination Rules : AbsolutionUrl = ['@odata.nextLink']

Execution of this ADF pipeline should generate a file in the target blob storage.

Pro tips:
1. Microsoft Graph API provides various API filtering parameters. More details can be found at this link.

See more

The post Call Graph API from Azure Data Factory appeared first on AzureOps.

Pre-requisites
1. Azure subscription with access to deploy Azure resources.
2. Az PowerShell module installed on the local machine.

Export Azure Data Factory

Let’s first export an existing data factory ARM template. Follow the below steps to do it.

1. Connect to the Azure portal and open-source Azure Data Factory Studio.
2. Navigate to Manage options.
3. Click on the Export button under ‘Export ARM Template’ as shown in the above image.
4. This will download Azure Data Factory code in a file named arm_template.zip. Unzip the file. Files of our interest are
   • ARMTemplateForFactory.json – This file contains the entire Data Factory code.
   • ARMTemplateParametersForFactory.json – This file contains parameters for the Data Factory.

Transform ARM template

Let’s change the factory parameter values in the ARMTemplateParametersForFactory.json file as per the target environment. The mandatory change you need to make in the parameter file is factoryName. Perhaps, it is also a good idea to set other factory parameters in this file.

Note: Include Global parameters in the ARM template is deprecated and only available through git integration. You may need to manually recreate global parameters in the target data factory once the import is done.

Import Azure Data Factory

1. Open PowerShell ISE and copy the below script in the new window. Ensure you have installed the Az module as mentioned in the prerequisites.

$templateFile = "Full path of ARMTemplateForFactory.json"
$parameterFile="Full path of ARMTemplateParametersForFactory.json"
$deploymentName = "Name of the deployment"
$tenantId = "Tenant id"
$subscriptionName = "Subscription name"
$resourceGroupName = "Name of the resource group"
$factoryName = "target factory name"
$location = "Location of resource group"

#Connect to Azure account
Connect-AzAccount -Tenant $tenantId -Subscription $subscriptionName

#Deploy Data Factory
New-AzResourceGroupDeployment `
  -Name $deploymentName `
  -ResourceGroupName $resourceGroupName `
  -TemplateFile $templateFile `
  -TemplateParameterFile $parameterFile `
  -Tag @{"key1"="value1"; "key2"="value2";}

#Generate managed identity for Data Factory.
Set-AzDataFactoryV2 -ResourceGroupName $resourceGroupName -Name $factoryName -Location $location

2. Provide values to the variables below
$templateFile and $parameterFile variables refer to the files we have downloaded and transformed in the above steps.
$tenantId – Tenant id of the target subscription. You can get tenantId details from here.
$subscriptionName-Subscription name of the target resource group.
$resourceGroupName-Target resource group.
$factoryName – Name of the target Data Factory. Here, we need to ensure that the name is the same one we used in the ARMTemplateParametersForFactory.json file.
$location-Location of the target resource group.

That’s it! You should see the Data Factory in the target resource group if the above script executes without error.

We have seen how adf import export can be done using PowerShell ISE and Azure Data Factory Studio in Azure portal.

Pro tips:
1. Azure Data Factory export from the Azure portal may not include global parameters in the arm template unless you have git integration enabled.
2. If you want to edit data factory triggers before the deployment, you can edit them in the arm_template.json file under the resources node.
3. If your organization has defined a policy to add tags with Azure resources, you need to include them as a parameter of the New-AzResourceGroupDeployment command.
4. You can refer to this post if you want to enable automatic publishing and deployment of the Data Factory code with Azure DevOps.
5. Learn how to export and import PowerApps package in just a few clicks.

See more

The post Export and Import Azure Data Factory appeared first on AzureOps.

Background

In Azure Data Factory, continuous integration and continuous delivery (CI/CD) mean moving Data Factory artifacts from one environment, such as dev, test, production, etc. We have been using the Data Factory CI/CD process with the adf_publish branch. The main issue with this approach is that the developer must manually click the publish button on the Data Factory UI master branch to deploy the code to the dev Data Factory. This generates the ARM Template artifacts in the adf_publish branch for higher environment deployments.

With the Automated publish improvement, we can create an ARM Template artifact using the Azure DevOps build pipeline with a trigger of master branch update (pull request). Deploying these artifacts to the dev and other environments can then be managed with the release pipeline (CD).

Microsoft will support both Azure Data Factory CI CD flows for Data Factory deployments, So below are some of the key differences between the old and the new approach.

Build pipeline (CI)

Pre-requisites:
1. Azure DevOps Account.
2. Data Factory in a dev environment with Azure Repos Git integration.

Before we start with the build pipeline, we need to create a file named package.json in the master branch of the Data Factory repository and copy the code below.

{
    "scripts":{
        "build":"node node_modules/@microsoft/azure-data-factory-utilities/lib/index"
    },
    "dependencies":{
        "@microsoft/azure-data-factory-utilities":"^0.1.3"
    }
}

The file should look like the below image.

Follow the below steps to create CI (build) pipeline for automated Azure Data Factory publish.

1. Create a new build pipeline in the Azure DevOps project.
2. Select Azure Repos Git as your code repository.
3. From the Azure Repos, select the repo that contains the Data Factory code. This is the repository where you have Data Factory DevOps integration.
4. Select Start Pipeline as your build pipeline type.
5. Replace the default YAML code with the below code. You would also make some changes to the code: replace <subscription-id> with your subscriptionid, <resourcegroup-name> with the resource group name, and <dev data factory name> with Dev Data Factory name.

trigger:
- master #change master to something else if master is not your main branch
pool:
  vmImage: 'ubuntu-latest'
steps:
- task: NodeTool@0
  inputs:
    versionSpec: '18.x'
  displayName: 'Install Node.js'

- task: Npm@1
  inputs:
    command: 'install'
    verbose: true
  displayName: 'Install npm package'
- task: Npm@1
  inputs:
    command: 'custom'
    customCommand: 'run build validate $(Build.Repository.LocalPath) /subscriptions/<subscription-id>/resourceGroups/<resourcegroup-name>/providers/Microsoft.DataFactory/factories/<dev data factory name>'
  displayName: 'Validate'

# Validate and then generate the ARM template into the destination folder. Same as clicking "Publish" from UX
# The ARM template generated is not published to the ‘Live’ version of the factory. Deployment should be done using a release pipeline. 
- task: Npm@1
  inputs:
    command: 'custom'
    customCommand: 'run build export $(Build.Repository.LocalPath) /subscriptions/<subscription-id>/resourceGroups/<resourcegroup-name>/providers/Microsoft.DataFactory/factories/<dev data factory name> "ArmTemplate"'
  displayName: 'Validate and Generate ARM template'

# Publish the Artifact to be used as a source for a release pipeline
- task: PublishPipelineArtifact@1
  inputs:
    targetPath: '$(Build.Repository.LocalPath)/ArmTemplate'
    artifact: 'ArmTemplates'
    publishLocation: 'pipeline'

6. Click Save.

The outcome of the above build process will generate the ARM Template as a linked artifact. Note here that the source of this artifact is the Data Factory master branch.

Now that we have the build pipeline setup, let us create a release pipeline to deploy code to the dev Data Factory. In this approach, the release pipeline for Data Factory utilizes the build pipeline linked artifact instead of the adf_publish  branch.

Release pipeline (CD)

Pre-requisites:
1. Service connection to dev Azure resource group.

Follow the below steps to create a release pipeline for Data Factory build deployment.

1. Select Pipelines, and then select Releases from the left side of the Azure DevOps.
2. Click New release pipeline under New dropdown. Select the Empty job template.
3. Provide Stage name. This generally denotes a dev environment.
4. Click Add an artifact and then Build. And select the appropriate Project & Source from the drop-down. Source alias can be the default (typically _name_of_the_build).
5. Click on the Stage (dev) box hyperlink. Add a new task. Search for ARM Template Deployment.

6. Selec Service connection to dev Azure resource group in Azure resource manager connection. Provide relevant Subscription, Resource group & Location. Select Template location as Linked artifact. Browse ARMTemplateForFactory.json file in Template, and ARMTemplateParametersForFactory.json in Template parameters. In the case of Overwrite template parameters, we need to manually provide all parameter names and values from ARMTemplateForFactory.json file. This is because build artifacts are generated at runtime. Give a suitable Deployment name, for example, ‘adf_deployment’. Click Save.
7. You can also enable a continuous deployment trigger from the icon on the Artifacts box.

Handle ADF triggers

Refer to this article to disable and enable triggers in the data factory deployment pipeline.

Pro tips:
1. In the real world, you must set Data Factory and global parameters in a release pipeline. Setting up global parameters differs slightly in the ADFUtilities NPM package-based deployment. Read more about it here.
2. Similar to dev Data Factory deployment, we can add more stages for higher environments like test, production, etc. deployment.
3. If you get an error “Resource <adf_name> was  disallowed by  policy.  Error Type:  PolicyViolation, Policy Definition Name : Restrict resources without specific tag, Policy Assignment Name “. Your organization has defined a policy to disallow resource creation in Azure without specific tags. In such a case, Azure DevOps CI-CD pipeline cannot be used for ADF deployment. As per Microsoft docs, we need to use the Export and Import Data factory approach.
4. If you wish to export and import data factories between resource groups, refer to this article.
5. Refer to this post to know how to implement CICD pipeline for Azure SQL Database deployments.

See more

The post Azure Data Factory Deployment using Azure DevOps appeared first on AzureOps.