Azure Data Factory lets you share a self-hosted integration runtime (IR) across multiple data factories. This allows several factories to use the same self-hosted IR infrastructure instead of installing a separate IR on another machine for every factory. In this guide, we’ll walk through how to create a shared self-hosted IR and link it to another Azure Data Factory.
Important: Microsoft calls the original IR the shared self-hosted integration runtime and the IR created in another factory a linked self-hosted integration runtime. Sharing is currently supported between Azure Data Factory instances in the same Microsoft Entra tenant; it isn’t supported between Synapse workspaces or across different tenants.
What is an Integration Runtime in Azure Data Factory?
An Integration Runtime (IR) is the compute infrastructure Azure Data Factory uses to perform data integration activities. The type of IR you choose depends on where your data and compute resources are located.
Types of Integration Runtime
- Azure integration runtime: Fully managed by Azure and used for cloud-based data integration scenarios.
- Self-hosted integration runtime: Runs on customer-managed Windows machines and is used when Data Factory needs to access data in a private network or other scenarios requiring customer-managed compute.
- Azure-SSIS integration runtime: Used to run SSIS packages in Azure Data Factory.
This article focuses specifically on sharing a self-hosted integration runtime between Azure Data Factory instances.
Why share a self-hosted Integration Runtime?
A self-hosted IR runs on one or more customer-managed machines. Sharing it can reduce infrastructure, maintenance, and upgrade effort when multiple Azure Data Factories in the same environment need access to the same private-network resources.
The original factory owns the physical self-hosted IR infrastructure. Other factories reference that infrastructure through a linked self-hosted IR. Microsoft notes that increasing the number of factories using a shared IR can increase queue times; if workload grows, you can scale up the machine or add additional IR nodes.
Prerequisites:
1. An Azure subscription with the required permissions.
2. A source Azure Data Factory containing an existing self-hosted integration runtime.
3. A target Azure Data Factory that will use the shared IR.
4. The target Data Factory must have a managed identity.
5. Permission to create or manage the required role assignment on the shared IR resource. Microsoft documents Microsoft.Authorization/roleAssignments/write as a required permission when adding access.
How to share a self-hosted Integration Runtime
1. Select the self-hosted IR to share
In the source Azure Data Factory, open Manage > Integration runtimes and select the self-hosted integration runtime that you want to share.
Open the sharing options and copy the IR’s Resource ID. You will use this resource ID when creating the linked IR in the target factory.
Select Grant Permission to another Data Factory or user-assigned managed identity and choose the target Data Factory.
2. Grant the target Data Factory access
Grant access to the target Data Factory so its managed identity can use the shared self-hosted IR.
The target factory’s managed identity is used to access the shared IR. Microsoft documents role-assignment permissions for adding and removing access to the shared IR.
3. Create a linked self-hosted IR in the target factory
Open the target Azure Data Factory and go to Manage > Integration runtimes.
Select New, choose Azure, self-hosted, and continue.
Under External Resources, select Linked Self-Hosted.
Enter a name for the linked IR and paste the Resource ID copied from the source factory.
Select Create. The target Data Factory now has a linked self-hosted IR that uses the infrastructure of the shared IR.
How the shared and linked IRs work
The architecture is straightforward:
- Source Data Factory: contains the physical self-hosted IR and its registered nodes.
- Target Data Factory: contains a linked self-hosted IR that references the shared IR.
- Linked services: in the target factory can reference the linked IR for activities that require that integration runtime.
- Infrastructure: the linked IR does not require another self-hosted IR installation; it uses the shared IR infrastructure.
Important limitations and considerations
- Same tenant: self-hosted IR sharing is supported between Azure Data Factories in the same Microsoft Entra tenant. Cross-tenant sharing isn’t supported.
- ADF only: a self-hosted IR can’t currently be shared between Synapse workspaces or between Azure Data Factory and a Synapse workspace.
- Managed identity: the Data Factory receiving the linked IR needs a managed identity.
- Capacity: sharing the IR across more factories can increase workload and queue times. A self-hosted IR can have up to four nodes for high availability and scalability.
- CI/CD: be careful when deploying ARM templates to environments that use linked self-hosted IRs. Microsoft documents a scenario where generated linked templates can overwrite a linked IR with a standalone IR definition. Remove or otherwise handle the IR resource definitions during deployment as recommended by Microsoft.
Shared Integration Runtime vs separate self-hosted IRs
| Consideration | Shared self-hosted IR | Separate self-hosted IR |
|---|---|---|
| Infrastructure | Shared across Data Factories | Dedicated infrastructure per factory |
| Maintenance | Centralized | Managed separately |
| Cost | Can reduce duplicated infrastructure | Higher infrastructure duplication |
| Isolation | Workloads share IR infrastructure | Greater infrastructure isolation |
| Scaling | Scale the shared IR as workload grows | Scale each IR independently |
The appropriate design depends on workload isolation, capacity, network architecture, and operational requirements. Sharing is particularly useful when multiple Data Factories need to use the same self-hosted infrastructure within an environment.
Summary
Azure Data Factory’s shared self-hosted integration runtime feature lets multiple Data Factories reuse the same self-hosted IR infrastructure. The basic workflow is:
- Select the existing self-hosted IR in the source Data Factory.
- Copy its Resource ID and grant the target Data Factory access.
- Create a Linked Self-Hosted IR in the target Data Factory.
- Paste the shared IR Resource ID and create the linked IR.
- Use the linked IR from the target factory’s linked services and activities that require it.
For the latest requirements and supported scenarios, see Microsoft’s documentation for shared self-hosted integration runtimes and Integration Runtime concepts.
Pavan Bangad
9+ years of experience in building data warehouse and big data application.
Helping customers in their digital transformation journey in cloud.
Passionate about data engineering.
