Azure Data Factory Deployment using Azure DevOps

Recently, Microsoft announced that they now support automated Azure Data Factory publish using the ADFUtilities NPM (@microsoft/azure-data-factory-utilities) library. This library provides basic functionality to validate and generate an ARM template given a set of Data Factory resources. This article describes how to implement Azure Data Factory Deployment using Azure DevOps and ADFUtilities NPM library.

Background

In Azure Data Factory, continuous integration and continuous delivery (CI/CD) mean moving Data Factory artifacts from one environment, such as dev, test, production, etc. We have been using the Data Factory CI/CD process with the adf_publish branch. The main issue with this approach is that the developer must manually click the publish button on the Data Factory UI master branch to deploy the code to the dev Data Factory. This generates the ARM Template artifacts in the adf_publish branch for higher environment deployments.

With the Automated publish improvement, we can create an ARM Template artifact using the Azure DevOps build pipeline with a trigger of master branch update (pull request). Deploying these artifacts to the dev and other environments can then be managed with the release pipeline (CD).

Microsoft will support both Azure Data Factory CI CD flows for Data Factory deployments, So below are some of the key differences between the old and the new approach.

Build pipeline (CI)

Pre-requisites:
1. Azure DevOps Account.
2. Data Factory in a dev environment with Azure Repos Git integration.

Before we start with the build pipeline, we need to create a file named package.json in the master branch of the Data Factory repository and copy the code below.

{
    "scripts":{
        "build":"node node_modules/@microsoft/azure-data-factory-utilities/lib/index"
    },
    "dependencies":{
        "@microsoft/azure-data-factory-utilities":"^0.1.3"
    }
}

The file should look like the below image.

Follow the below steps to create CI (build) pipeline for automated Azure Data Factory publish.

1. Create a new build pipeline in the Azure DevOps project.
2. Select Azure Repos Git as your code repository.
3. From the Azure Repos, select the repo that contains the Data Factory code. This is the repository where you have Data Factory DevOps integration.
4. Select Start Pipeline as your build pipeline type.
5. Replace the default YAML code with the below code. You would also make some changes to the code: replace <subscription-id> with your subscriptionid, <resourcegroup-name> with the resource group name, and <dev data factory name> with Dev Data Factory name.

trigger:
- master #change master to something else if master is not your main branch
pool:
  vmImage: 'ubuntu-latest'
steps:
- task: NodeTool@0
  inputs:
    versionSpec: '18.x'
  displayName: 'Install Node.js'

- task: Npm@1
  inputs:
    command: 'install'
    verbose: true
  displayName: 'Install npm package'
- task: Npm@1
  inputs:
    command: 'custom'
    customCommand: 'run build validate $(Build.Repository.LocalPath) /subscriptions/<subscription-id>/resourceGroups/<resourcegroup-name>/providers/Microsoft.DataFactory/factories/<dev data factory name>'
  displayName: 'Validate'

# Validate and then generate the ARM template into the destination folder. Same as clicking "Publish" from UX
# The ARM template generated is not published to the ‘Live’ version of the factory. Deployment should be done using a release pipeline. 
- task: Npm@1
  inputs:
    command: 'custom'
    customCommand: 'run build export $(Build.Repository.LocalPath) /subscriptions/<subscription-id>/resourceGroups/<resourcegroup-name>/providers/Microsoft.DataFactory/factories/<dev data factory name> "ArmTemplate"'
  displayName: 'Validate and Generate ARM template'

# Publish the Artifact to be used as a source for a release pipeline
- task: PublishPipelineArtifact@1
  inputs:
    targetPath: '$(Build.Repository.LocalPath)/ArmTemplate'
    artifact: 'ArmTemplates'
    publishLocation: 'pipeline'

6. Click Save.

The outcome of the above build process will generate the ARM Template as a linked artifact. Note here that the source of this artifact is the Data Factory master branch.

Now that we have the build pipeline setup, let us create a release pipeline to deploy code to the dev Data Factory. In this approach, the release pipeline for Data Factory utilizes the build pipeline linked artifact instead of the adf_publish  branch.

Release pipeline (CD)

Pre-requisites:
1. Service connection to dev Azure resource group.

Follow the below steps to create a release pipeline for Data Factory build deployment.

1. Select Pipelines, and then select Releases from the left side of the Azure DevOps.
2. Click New release pipeline under New dropdown. Select the Empty job template.
3. Provide Stage name. This generally denotes a dev environment.
4. Click Add an artifact and then Build. And select the appropriate Project & Source from the drop-down. Source alias can be the default (typically _name_of_the_build).
5. Click on the Stage (dev) box hyperlink. Add a new task. Search for ARM Template Deployment.

6. Selec Service connection to dev Azure resource group in Azure resource manager connection. Provide relevant Subscription, Resource group & Location. Select Template location as Linked artifact. Browse ARMTemplateForFactory.json file in Template, and ARMTemplateParametersForFactory.json in Template parameters. In the case of Overwrite template parameters, we need to manually provide all parameter names and values from ARMTemplateForFactory.json file. This is because build artifacts are generated at runtime. Give a suitable Deployment name, for example, ‘adf_deployment’. Click Save.
7. You can also enable a continuous deployment trigger from the icon on the Artifacts box.

Handle ADF triggers

Refer to this article to disable and enable triggers in the data factory deployment pipeline.

Pro tips:
1. In the real world, you must set Data Factory and global parameters in a release pipeline. Setting up global parameters differs slightly in the ADFUtilities NPM package-based deployment. Read more about it here.
2. Similar to dev Data Factory deployment, we can add more stages for higher environments like test, production, etc. deployment.
3. If you get an error “Resource <adf_name> was  disallowed by  policy.  Error Type:  PolicyViolation, Policy Definition Name : Restrict resources without specific tag, Policy Assignment Name “. Your organization has defined a policy to disallow resource creation in Azure without specific tags. In such a case, Azure DevOps CI-CD pipeline cannot be used for ADF deployment. As per Microsoft docs, we need to use the Export and Import Data factory approach.
4. If you wish to export and import data factories between resource groups, refer to this article.
5. Refer to this post to know how to implement CICD pipeline for Azure SQL Database deployments.

See more

Kunal Rathi

With over 13 years of experience in data engineering and analytics, I've assisted countless clients in gaining valuable insights from their data. As a dedicated supporter of Data, Cloud and DevOps, I'm excited to connect with individuals who share my passion for this field. If my work resonates with you, we can talk and collaborate.