Shared Integration Runtime in Azure Data Factory

Azure Data Factory is an ETL and orchestrator tool for building cloud-native data engineering pipelines. It has a lot of source connectors available, and this list is overgrowing. Azure Data Factory uses Integration Runtimes as a computing infrastructure to execute data movement and transformation activities. This article describes setting up self-hosted shared integration runtime in Azure Data Factory.

What is Integration Runtime?

Integration Runtime (IR) in Azure Data Factory provides a computational infrastructure for pipelines. It is also instrumental in securing data movement in the cloud. Read more about it here.

What are the different types of integration runtime in ADF?

Azure Data Factory consists of three types of Integration runtimes:

1. Azure integration runtime: This is the default one that Azure fully manages. Suitable to connect to Azure resources. Since Azure manages it, the data movement is in the public network.
2. Self-hosted integration runtime: This IR can connect to on-prem sources and secure data movement between the systems. This IR supports the data movement in the private network.
3. Azure-SSIS integration runtime: This is used for running SSIS packages in Azure Data Factory

Let’s see it in Action.

This section describes setting up self-hosted shared integration runtime in Azure Data Factory.

Pre-requisites:
1. Azure subscription and a resource group with Azure Data Factory.
2. Self Hosted Integration runtime setup in Azure Data Factory.

1. Select the integration runtime that needs to be shared.

Select the integration runtime to share with other Azure Data Factories and Click edit.

Copy the ResourceID and save it. We will use it in our next steps.

Click on Grant Permission to another Data Factory or user-assigned managed identity button, as shown in the above image.

2. Grant permission to target Azure data factory:

Grant access to target Azure Data Factory which will use shared integration runtime.

3. Now, go to the Target Data Factory and add an Integration Runtime

Click New and select Azure, self-hosted

Under External Resources, select Linked Self-Hosted and Click Continue.

It will ask few details about the shared IR.

Name: put the name of the new IR.

Resource ID Copy and paste the resource ID we previously saved and click Create.

And we are done!

Now we are using the same integration runtime in two different Azure Data Factory instances.

Pro tips:
1. Please make sure you have Microsoft.Authorization/roleAssignments/write permission on subscription.
2. Self-hosted IR requires a virtual machine and Integration runtime software. This means we need to pay the cost of the hosted virtual machine. By sharing the IR between the multiple Data Factory instances, we can use the same virtual machine amongst multiple Data Factories.
3. If you’re aiming to obtain the Databricks certified Data Engineer Associate certification, take a look at these helpful tips.

See more