Terraform Environment Setup

Databricks is a cloud-based tool built on Apache Spark used in machine learning, big data analytics, and data engineering to enable AI model deployment, fast data processing, and collaboration. Its scalable Delta Lake infrastructure supports its efficient, secure data management.

To manage Databricks infrastructure with Terraform, you have to install and configure some of the systems; the first step is installing Terraform and the Databricks provider. Download Terraform from the official website and ensure it is available in the system’s path, and then initialize a Terraform working directory by creating a new directory and running :

terraform init

Authentication in Databricks is configured using the credentials from cloud providers. A service principal is used to set up authentication for Azure:

provider "databricks" {
 host  = "https://<your-databricks-instance>"
 azure_client_id     = var.azure_client_id
 azure_client_secret = var.azure_client_secret
 azure_tenant_id     = var.azure_tenant_id
}

An Access key or service accounts will be required for AWS and GCP

Databricks Infrastructure with Terraform Setup

Terraform allows configuration using HCL (Hashicorp Configuration Language). and configurations are made in the primary configuration file, main.tf, declares a Databricks workspace:

resource "databricks_workspace" "example" {
 name          = "example-workspace"
 resource_group = "example-rg"
 location      = "East US"
}

Configuring clusters within a workspace requires some important setup, like versioning, autoscaling, and specific nodes:

resource "databricks_cluster" "example" {
 cluster_name            = "example-cluster"
 spark_version           = "12.0.x-scala2.12"
 node_type_id            = "Standard_D3_v2"
 autotermination_minutes = 20
 autoscale {
   min_workers = 2
   max_workers = 8
 }
}

Access Control and Permissions

Terraform will be able to control user roles and access control lists using the Databricks permission control user and group access

resource "databricks_group" "data_scientists" {
 display_name = "Data Scientists"
}

resource "databricks_user" "analyst" {
 user_name  = "analyst@example.com"
 groups     = [databricks_group.data_scientists.id]
}

Integrating Terraform with a cloud IAM system like Microsoft Entra Id ensures authentication and access policies are in line with enterprise governance best practices.

Deploying Databricks Jobs with Terraform

Databricks jobs are used to automate batch and streaming workloads; the databricks_job resources define the job execution parameters:

resource "databricks_job" "example_job" {
 name = "Example Job"
 new_cluster {
   spark_version = "12.0.x-scala2.12"
   node_type_id  = "Standard_D3_v2"
   num_workers   = 4
 }
 notebook_task {
   notebook_path = "/Shared/example_notebook"
 }
}

Scheduling jobs and dependencies are managed within Terraform, allowing seamless automation of recurring tasks.

Versioning with Terraform

Terraform uses a state file to track all deployed resources to ensure collaborations and prevent conflicts, so remote state storage must be configured.

terraform {
 backend "azurerm" {
   resource_group_name  = "terraform-state-rg"
   storage_account_name = "tfstateaccount"
   container_name       = "tfstate"
   key                  = "databricks.tfstate"
 }
}

This ensures changes made to infrastructure are managed safely using terraform plan and terraform apply, preventing unintended changes.

Automating Deployment with CI/CD

To ensure infrastructure with consistency, Terraform needs to integrate with CI/CD; a Github action pipeline can be used for automated deployments:

name: Terraform Deployment
on: [push]
jobs:
 deploy:
   runs-on: ubuntu-latest
   steps:
     - uses: actions/checkout@v3
     - name: Setup Terraform
       uses: hashicorp/setup-terraform@v1
     - name: Terraform Init
       run: terraform init
     - name: Terraform Apply
       run: terraform apply -auto-approve

This pipeline can also be configured in GitLab CI/CD or Azure DevOps for streamlined deployments.

Monitoring and Scaling Databricks infrastructure

Databricks environments require monitoring for performance and cost efficiency; Terraform can be used to configure monitoring solutions:

resource "databricks_instance_profile" "monitoring" {
 instance_profile_arn = "arn:aws:iam::123456789012:instance-profile/DatabricksMonitor"
}
Autoscaling is enabled on the clusters to optimize resource utilization without over-provisioning:
autoscale {
 min_workers = 4
 max_workers = 16
}

Errors and debugging Terraform Deployments

Some common Terraform errors include authentication failures and state mismatches. Provider authentication issues can be resolved by verifying permissions and credentials. State mismatches often require manual state adjustments:

terraform state rm databricks_cluster.example
terraform apply

Quota restrictions and API rate limits must be handled by adjusting Terraform configurations to include retries and back-off mechanisms.

Databricks Deployments Optimization.

More resources like Unity Catalog for data governance can be managed with Terraform:

resource "databricks_metastore" "unity" {
 name = "example-metastore"
}

MLflow tracking servers can also be set up to manage ML models and notebooks through Git integration, ensuring streamlined workflows.

This tutorial has shown how to automate Databricks infrastructure with Terraform, including infra provisioning, access control, job automation, CI/CD integration, and monitoring, but more optimizations like cost management, security best practices, and additional Databricks features can be included.

Pro tips:
1. Follow this guide to know how to manage secret scopes in databricks.

See more

The post Automate Databricks Infrastructure as Code with Terraform appeared first on AzureOps.

Prerequisites:
1. Visual Studio with data tools installed on your system.
2. An Azure SQL Database set up and accessible.
3. A DACPAC file ready for deployment.

Create a DACPAC File

In this post, we cover the process of creating a DACPAC file, a critical artifact that encapsulates the schema and objects of a SQL Server database. Learn how to build and validate your DACPAC files to ensure seamless deployments.

Open a DACPAC File in Visual Studio

Discover how to open and inspect a DACPAC file in Visual Studio. This guide provides details of how to open DACPAC file in Visual Studio.

Deploy DACPAC to Azure SQL Database from Visual Studio

Follow the below steps to deploy the DACPAC project from visual Studio. You can follow the same below steps to deploy database project to any SQL Server database or Azure Managed Instance. Make sure to select correct target platform and provide applicable connection details.

1. Open Visual Studio

1. Launch Visual Studio and open your solution or project.
2. Ensure you have the SQL Server Data Tools (SSDT) installed.

2. Change target platform of database project

1. Right-click on database project and click properties.
2. Change the Target platform to Microsoft Azure SQL Database as shown in the below image.

3. Publish the Database Project

Right-click on the project and select Publish from the context menu.

In Solution Explorer, locate your database project.

4. Configure the Deployment Target

1. In the Publish Database dialog, click Edit to specify the target database.

2. In the Edit Target Database Connection window:

• Select Azure SQL Database as the target.
• Enter the server name, authentication details, and select the target database.
• Test the connection to ensure it is valid.
• Click OK to confirm.

5. Review and Publish

1. Back in the Publish Database dialog, review the settings to ensure the correct DACPAC file and target database are selected.

2. Optionally, click Advanced to configure deployment options such as excluding certain schema changes as shown in the below image.

3. Click Publish to start the deployment process.

6. Monitor and Verify Deployment

1. Monitor the progress of dacpac publish in Visual Studio as shown in the below image.
2. Once the deployment is complete, open SQL Server Management Studio (SSMS) or use Visual Studio’s built-in tools to verify the schema changes in your Azure SQL Database.
3. Run test queries to confirm the database is functioning as expected.

Pro tips:
1. Refer to this post if you wish to Automate Azure SQL Database deployment using Azure DevOps CICD pipeline.
2. If you skip target platform change step during the deployment, you may get an error “A project which specifies SQL Server 2022 or Azure SQL Database Managed Instance as the target platform cannot be published to Microsoft Azure SQL Database v12.”. Refer to this post to learn more about it.
3. Test your DACPAC deployment on a local environment before pushing it to Azure.
4. Always take a backup of your target Azure SQL Database before deployment.
5. Use tools like SQL Compare to identify schema differences between the DACPAC and the target database.

FAQ

See more

The post Deploy DACPAC to Azure SQL Database appeared first on AzureOps.

Error

##[error]*** An error occurred during deployment plan generation. Deployment cannot continue.
##[error]A project which specifies SQL Server 2022 or Azure SQL Database Managed Instance as the target platform cannot be published to Microsoft Azure SQL Database v12.
##[error]The Azure SQL DACPAC task failed. SqlPackage.exe exited with code 1.Check out how to troubleshoot failures at https://aka.ms/sqlazuredeployreadme#troubleshooting-

Solution

1. Open the database dacpac project in Visual Studio.

2. Right Click on the database project and Click on ‘Properties’.

3. Open ‘Target platform’ drop down and select ‘Microsoft Azure SQL Database’.

4. Save the project and build it.

5. If the build is successful, raise a PR to merge this change with main branch and initiate the Azure DevOps CICD pipeline.

See more

Pro tips:
1. Follow the steps mentioned in the article deploy azure sql database using devops to automate Azure SQL database deployments.

The post A project which specifies SQL Server 2022 or Azure SQL Database Managed Instance as the target platform cannot be published to Microsoft Azure SQL Database v12. appeared first on AzureOps.

Prerequisites:
1.Service connections in Azure DevOps project for each resource groups containing Azure SQL databases.

Create a database project and link it to Azure Repos

When setting up a CI/CD pipeline for Azure SQL Database, it’s essential to understand how to create a DACPAC file and utilize Visual Studio to a database project from it. This project then needs to be integrated with Azure Repos for setting up an automated build and deployment pipeline. In case you already have this setup, you can skip this section.

Create a DACPAC file

A DACPAC (Data-tier Application Component Package) file is a portable representation of a SQL Server database schema, making it crucial for efficiently managing database changes. Follow the steps mentioned in this post to learn how to create it. If you already have a database project, you can skip this step.

Create a database project from the DACPAC file

Once the DACPAC file is ready, integrating it into a Visual Studio project becomes imperative for further development and deployment. Detailed instructions on opening a DACPAC file in Visual Studio can be found in the guide

Link database project with Azure Repos

Create a repo in Azure DevOps under the DevOps organization project and link the database project with it. Follow the steps mentioned in this post to learn about it.

Setup Azure SQL Server CICD pipeline

Once we have Azure repo ready for Azure SQL database projects, let us now implement SQL DACPAC deployment Azure DevOps pipeline.

Enable access to Azure Key Vault from Azure DevOps pipelines

To connect to Azure SQL Database from Azure DevOps deployment pipeline, we need to fetch database credentials from Azure Key Vault. Follow the steps mentioned in this article to enable access to Azure Key Vault secrets from Azure DevOps pipeline.

Create a new pipeline in Azure DevOps project

It’s time to set up a deployment pipeline for the Azure SQL database.

1. Login to Azure DevOps portal and navigate to the DevOps organization. Select the DevOps project containing the database Azure repo we created in the above step.

2. Select Pipelines and Click ‘New pipeline’ button to create a new pipeline.

3. Select ‘Azure Repos Git’ as a source of your code.

4. Select the Azure repo containing the database project we created in previous steps.

5. On the Configure screen, select the ‘Starter pipeline’ option as shown in the image below.

6. Add below YAML code to the editor on Review screen and make necessary changes as per your environment. You can use YAML pipeline tasks to generate the code snippets based on your parameter values.

trigger:
# Name of the main (master) branch of the Azure DevOps repos. 
- main

variables:
  vmImageName: 'windows-latest'
  solution: '**/*.sln'
  buildPlatform: 'Any CPU'
  buildConfiguration: 'Release'

stages:
- stage: build
  displayName: Build Stage
  jobs:
  - job: Build
    displayName: Build
    pool:
      vmImage: $(vmImageName)
    steps:
    - task: NuGetToolInstaller@1
    - task: NuGetCommand@2
      inputs:
        restoreSolution: '$(solution)'
    - task: VSBuild@1
      inputs:
        solution: '$(solution)'
        platform: '$(buildPlatform)'
        configuration: '$(buildConfiguration)'
    - task: CopyFiles@2
      inputs:
        SourceFolder: '$(agent.builddirectory)'
        Contents: '**/bin/$(buildConfiguration)/**'
        TargetFolder: '$(build.artifactstagingdirectory)'
        CleanTargetFolder: true
    - task: PublishBuildArtifacts@1
      inputs:
        PathtoPublish: '$(build.artifactstagingdirectory)'
        ArtifactName: 'drop'
        publishLocation: 'Container'

- stage: dev
  displayName: Dev Deploy Stage
  dependsOn: build
  condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))
  jobs: 
  - deployment: Deploy
    displayName: Deploy Dev
    pool:
      vmImage: $(vmImageName)
    environment: dev
    strategy:
      runOnce:
        deploy:
          steps:
          - task: AzureKeyVault@1
            inputs:
             # Enter name of the service connection created to deploy resources to dev Azure SQL database. 
              AzureSubscription: 'dev-service-connection-name'
             # Enter name of the key vault that stores Azure SQL database credentials. 
              KeyVaultName: 'dev-keyva'
             # Enter comma separated names of the key vault secrets that store the Azure SQL database credentials.
              SecretsFilter: 'azuresqldb-dbconnstring'
              RunAsPreJob: true

          - task: SqlAzureDacpacDeployment@1
            inputs:
             # Enter name of the service connection created to deploy resources to dev Azure SQL database. 
              AzureSubscription: 'dev-service-connection-name'
              AuthenticationType: 'connectionString'
             # Provide the database connection string variable value to ConnectionString parameter. (key vault secret name)
              ConnectionString: '$(azuresqldb-dbconnstring)'
              DeployType: 'DacpacTask'
              DeploymentAction: 'Publish'
              # Provide path of the dacpac file. Replace 'AzureOps.Sql' with name of your AzureRepo and AzureOps.Sql.dacpac with name of your database dacpac file.
              DacpacFile: '$(Pipeline.Workspace)/drop/s/AzureOps.Sql/bin/Release/AzureOps.Sql.dacpac'
              AdditionalArguments: '/p:DropObjectsNotInSource=true /p:BlockOnPossibleDataLoss=true /p:IgnorePermissions=true /p:ExcludeObjectTypes="Users;Permissions"'
              IpDetectionMethod: 'AutoDetect'

# similarly, customise below block of code for test environment deployment.
- stage: test
  displayName: Test Deploy Stage
  dependsOn: dev
  condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))
  jobs: 
  - deployment: Deploy
    displayName: Deploy Test
    pool:
      vmImage: $(vmImageName)
    environment: test
    strategy:
      runOnce:
        deploy:
          steps:
          - task: AzureKeyVault@1
            inputs:
              AzureSubscription: 'test-service-connection-name'
              KeyVaultName: 'test-keyva'
              SecretsFilter: 'azuresqldb-dbconnstring'
              RunAsPreJob: true

          - task: SqlAzureDacpacDeployment@1
            inputs:
              azureSubscription: 'test-service-connection-name'
              AuthenticationType: 'connectionString'
              ConnectionString: '$(azuresqldb-dbconnstring)'
              DeployType: 'DacpacTask'
              DeploymentAction: 'Publish'
              DacpacFile: '$(Pipeline.Workspace)/drop/s/AzureOps.Sql/bin/Release/AzureOps.Sql.dacpac'
              AdditionalArguments: '/p:DropObjectsNotInSource=true /p:BlockOnPossibleDataLoss=true /p:IgnorePermissions=true /p:ExcludeObjectTypes="Users;Permissions"'     
              IpDetectionMethod: 'AutoDetect'

Note:
You can add AdditionalArguments to SqlAzureDacpacDeployment task like DropObjectsNotInSource=true which will drop objects from the database which do not exists in DACPAC file. Read more about all the properties supported in SqlAzureDacpacDeployment. Exercise this with caution.

Review and test the pipeline

Click on ‘Run pipeline’ button to manually trigger the pipeline and test if the pipeline works without any issues.

How to Manually Deploy DACPAC project to Azure SQL Database

Refer to this post if you want to deploy DACPAC project to Azure SQL database using Visual Studio.

Pro tips:
1. When publishing a DACPAC, the pipeline might fail with potential data loss errors. Customize the publish parameters according to your specific requirements in such cases.
2. Before implementing a block for production deployment, it’s advisable to thoroughly test the pipeline in lower environments. This ensures that any issues or bugs are identified and resolved before reaching critical stages of deployment.
3. To prevent unintended code deployments to higher environments, incorporate mandatory approvals at each stage of deployment. This ensures that changes are reviewed and approved before progressing to the next environment, maintaining the integrity of the deployment process.
4. If you come access error ‘A project which specifies SQL Server 2022 or Azure SQL Database Managed Instance as the target platform cannot be published to Microsoft Azure SQL Database v12., follow this article to resolve the issue.

See more

The post Implement Azure SQL Database Deployment CI CD pipeline appeared first on AzureOps.

Pre-requisites:
1. Azure DevOps project and a key vault.
2. Permission to access Microsoft Entra ID applications.

What is Azure DevOps project service principal?

To know the service principle (entra application ID) associated with Azure DevOps project

1. navigate to ‘Project Settings’ – > ‘Service connections’.

2. Open any service connection and click on the ‘Manage Service Principal’ link as shown in the image below.

3. This will open the Microsoft Entra ID application associated with the DevOps project in Azure portal.

4. Note the Application (client) ID. This is the application ID to which we would need to grant access to Azure key vault.

Grant access to Azure DevOps application ID on Azure key vault

1. Navigate to the key vault in Azure portal that you want to access in Azure DevOps project pipelines.

2. Click on ‘Access policies’ on the left side navigation list and click ‘Create’ to create a new access policy.

3. Select the required permissions for the pipeline and Click ‘Next’.

4. Search for the application ID (from the previous section) and select the application.

5. Authorise the app to perform the specified permissions on the User’s or Group’s behalf.

6. ‘Review’ the steps and click ”Create’ to grant access to the DevOps project on Azure key vault.

Access the key vault secrets in Azure DevOps YAML pipeline

Once the access is set up, you can add a Azure Key Vault Task in your YAML pipeline to access key vault secrets.

          - task: AzureKeyVault@1
            inputs:
              azureSubscription: 'service-connection-name'
              KeyVaultName: 'key-vault-name'
              SecretsFilter: 'secret1,secret2'
              RunAsPreJob: true

Pro tips:
1. Learn how to access Key Vault secrets in Azure Data Factory.

See more

The post Access Key Vault from Azure DevOps Pipeline appeared first on AzureOps.

Export PowerApps package

Follow the below steps to export an existing PowerApps package.

1. Launch the PowerApps portal and navigate to the apps.

2. Navigate the PowerApps you want to export and click the Elips icon {…}. Click on the Export package, as shown in the image below.

3. Under ‘Review Package Content,’ provide the package name and environment where the app will be stored. A Power Platform environment is a space to store, manage, and share your organization’s business data, apps, chatbots, and flows. It also serves as a container for apps with different roles, security requirements, or audiences.

Import PowerApps package

Follow the below steps to import the PowerApps package we exported in the previous steps.

1. Navigate to the PowerApps portal where you want to import the PowerApps. Click on ‘Import canvas app’ as shown in the image below.

2. On the import package screen, browse the package we exported in the previous steps and click on the Upload button.

3. Once the package is uploaded, review the package content. Click the ACTION button and update the settings.

4. You will see them under the Related resources section if you are connected to resources like the Azure SQL database. Click the ACTION button to update connection details.

5. Once all settings are okay, click the Import button to configure the PowerApps in the target environment.

6. Review the app and share it with end users.

We have seen how to export and import PowerApps package for migrations.

Pro tips:
1. Learn how to implement row-level security in Power BI.
2. Learn how to delete multiple records from a PowerApps data source.

See more

The post Export PowerApps App to another environment appeared first on AzureOps.

How to create a DACPAC file.

To create a DACPAC file from an existing SQL Server or Azure SQL database, you must ensure certain things. Refer to this post to learn how to create a DACPAC file.

Follow the below steps to open DACPAC file in Visual Studio.

Pre-requisites:
1. Visual Studio with SSDT extension installed.

1. Create a new blank project in Visual Studio

1. Open Visual Studio, click on the File menu item, and Select ‘New’ – > ‘Project’ as shown in the below image.

2. Select the project type as ‘SQL Server Database Project’. Provide the name to the project and Click OK.

This should create a blank database project in Visual Studio.

2. Import DACPAC file into the database project

1. Right-click on the project we just created in the previous step. Click ‘Import’ – > ‘Data tier Application (*.dacpac)’ option.

2. Browse the .dacpac file you want to unpack in the Visual Studio project. Click Start to begin the import process.

3. if there are no errors, you will see the progress shown in the image below.

4. You should see all database objects from the DACPAC under the project.

Pro tips:
1. It is advisable to always review the content of a DACPAC file before publishing it to a SQL Server or Azure SQL database.
2. If Visual Studio is not installed on your system and you wish to access the contents of a DACPAC file, you can do so by renaming the file with a .zip extension. This will enable you to extract the contents of the file and directly view the database objects in File Explorer.

See more

The post Open DACPAC File in Visual Studio appeared first on AzureOps.

Pre-requisites:
1. Access to Data Factory resources in the Azure portal.
2. Az PowerShell module installed on the local machine.

Understanding Azure Data Factory Triggers

Azure Data Factory utilizes triggers to initiate data pipeline executions based on specific conditions automatically. These conditions can be set on a schedule, event, or through manual intervention. Triggers play a crucial role in ensuring that data pipelines run at the correct time, keeping data processing up-to-date and accurate. Turning triggers on-demand on and off gives data engineers the flexibility to control the data flow efficiently.

Script to disable and enable triggers in Azure Data Factory

Use the below PowerShell script to disable all the triggers in Azure Data Factory.

$adf_name = 'adfname'
$resourcegroup_name = 'resourcegroupname'

$triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName $adf_name -ResourceGroupName $resourcegroup_name; 
$triggersADF | ForEach-Object { Stop-AzDataFactoryV2Trigger -ResourceGroupName $resourcegroup_name -DataFactoryName $adf_name -Name $_.name -Force }

Use the below PowerShell script to enable all the triggers in Azure Data Factory.

$adf_name = 'adfname'
$resourcegroup_name = 'resourcegroupname'

$triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName $adf_name -ResourceGroupName $resourcegroup_name; 
$triggersADF | ForEach-Object { Start-AzDataFactoryV2Trigger -ResourceGroupName $resourcegroup_name -DataFactoryName $adf_name -Name $_.name -Force }

Manage ADF triggers in YAML based Azure DevOps pipeline

Use the script below to enable and disable triggers in adf using a YAML-based]deployment pipeline in Azure DevOps.

          deploy:  #Configure steps under deploy hook.
            steps:
              #task to disable triggers in Azure Data Factory.
              - task: AzurePowerShell@5     
                displayName: Stop Triggers
                inputs:
                  azureSubscription: 'subscriptionname'
                  ScriptType: 'InlineScript'
                  Inline: |
                    $triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName "$(dataFactoryName)" -ResourceGroupName "$(resourceGroup)"; 
                    $triggersADF | ForEach-Object { Stop-AzDataFactoryV2Trigger -ResourceGroupName "$(resourceGroup)" -DataFactoryName "$(dataFactoryName)" -Name $_.name -Force }
                  azurePowerShellVersion: 'LatestVersion'             
             
              #task to enable triggers in Azure Data Factory.
              - task: AzurePowerShell@5    
                displayName: Start Triggers
                inputs:
                  azureSubscription: 'subscriptionname'
                  ScriptType: 'InlineScript'
                  Inline: |
                    $triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName "$(dataFactoryName)" -ResourceGroupName "$(resourceGroup)";
                    $triggersADF | ForEach-Object { Start-AzDataFactoryV2Trigger -ResourceGroupName "$(resourceGroup)" -DataFactoryName "$(dataFactoryName)" -Name $_.name -Force }
                  azurePowerShellVersion: 'LatestVersion'

Pro tips:
1. It is important to disable triggers before deploying data factory code using Azure DevOps pipelines. Refer to this article if you want to automate Azure data factory deployments using the DevOps pipeline.

See more

The post Disable and Enable triggers in Azure Data Factory using PowerShell appeared first on AzureOps.

Pre-requisites:
1. Confluent account with a confluent cluster and ksqlDB cluster.
2. Confluent platform installed locally using docker.
3. Confluent CLI installed locally.
4. A Basic understanding of the Confluent platform and KSQL.

Setup GitHub repository

First thing first!

1. Fork the git repository jzaralim/ksqldb-migrations-action. This repository has most of the things we need to implement CICD for ksqlDB.

2. Clone the above repository locally using any client tools like VS Code.

Create a Confluent ksqlDB cluster API key.

Login to confluent.io and navigate to the confluent cluster and ksqlDB cluster. Click on CLI instructions as shown in the below image. These instructions are specific to your selected Kafka and ksqlDB clusters, so you can copy and paste them into the Confluent CLI.

Follow the below steps to get the API key, API secret, and endpoint URL for the ksqlDB cluster.

1. Log in to the Confluent CLI,

confluent login

2. Set the context to your environment and cluster.

confluent environment use env-v7ymd0
confluent kafka cluster use lkc-0xyop2

3. Create an API key and secret for your ksqlDB cluster.

confluent api-key create --resource lksqld-pgx9ro

4. Get the ksqlDB cluster endpoint URL.

Save this information somewhere, as we will use them in the below steps.

Initialize ksql-migrations metadata

The confluent ksqldb-migrations tool uses MIGRATION_EVENTS & MIGRATION_SCHEMA_VERSIONS topics to maintain the deployment log for the deployments made through ksql-migrations. To create these topics in the confluent cluster, follow the below steps.

1. Connect to the local ksqlDB-server docker container using bash. If you are using windows bash, you can execute the below command.

docker-compose exec ksqldb-server bash

The ksql-migrations tool is available with all ksqlDB versions starting from ksqlDB 0.17 or Confluent Platform 6.2.

2. Execute the below command on the local ksqlDB-server container.

ksql-migrations new-project orders https://pksqld-zjyqd.westeurope.azure.confluent.cloud:443

Here, orders is the project’s name, and https://pksqlc-zjyqd.westeurope.azure.confluent.cloud:443 is the ksqldb cluster endpoint.

The above command will create order\migrations\ksql-migrations.properties file under the home directory in the ksqldb-server container as shown in the below image.

3. Edit the ksql-migrations.properties file with the following details. Values assigned to the variables are based on the name of the variables in the GitHub workflow file.

ksql.server.url=${CONFLUENT_KSQLDB_ENDPOINT}
ksql.migrations.topic.replicas=3
ssl.alpn=true
ksql.auth.basic.username=${CONFLUENT_KSQLDB_API_KEY} 
ksql.auth.basic.password=${CONFLUENT_KSQLDB_API_SECRET}
ksql.migrations.dir.override=${CONFLUENT_MIGRATION_DIR}

4. Now, run the below command to initiate migration metadata in the confluent cloud based on the configurations set in the above steps.

ksql-migrations initialize-metadata -c orders/ksql-migrations.properties

You should be able to see the below topic in the Confluent cloud.

Note: GitHub collaboration branch name and the workflow trigger branch name should be the same.

Set up GitHub secrets for GitHub actions.

Create GitHub secrets for the ksqlDB cluster endpoint, API key, and API secret, respectively named:

• CONFLUENT_KSQLDB_ENDPOINT
• CONFLUENT_KSQLDB_API_KEY
• CONFLUENT_KSQLDB_API_SECRET

You have retrieved these secrets in the above steps.

We are all set to automate Confluent ksqlDB deployments

It’s time to verify our setup. Start committing changes to the migrations folder in the GitHub repository. For this demo, we will commit the three migrations files inside the ksqldb-migrations-action Github repository.

1. After the initial commit, the migrate-cloud-ksqldb workflow will be triggered. It will deploy the initial three ksql migration versions to the Confluent cloud cluster.
2. After migration ORDER_EVENTS stream would look like in the below image.

Let’s say, we need to add a column DESCRIPTION to the ORDER_EVENTS stream.

1. Add a new version file V000004__add_column_to_order_events.sql.
2. Commit the new version file, triggering a Github action, as shown in the image below.

Pro tips:
1. The steps mentioned in this article must be done only once by a DevOps person, and individual developers need only to commit changes to the migrations folder.
2. Format of the migrations version file is Vxxxxxx__name_of_the_file.sql. That means ‘V’ followed by an incremental six digits number with leading 0s. E.g., V000012__new_streams.SQL.
3. It is advisable to add the IF NOT EXISTS clause in CREATE STREAM/TABLE ksql statements. And IF EXISTS in DROP STREAM/TABLE ksql statements. This will avoid failures of version files in case of object availability conflict in the ksqlDB cluster.
4. Learn how to filter kstream by rowtime using ksql .

The post Automate Confluent ksqlDB deployments with ksql-migrations appeared first on AzureOps.

Pre-requisites
1. Azure subscription with access to deploy Azure resources.
2. Az PowerShell module installed on the local machine.

Export Azure Data Factory

Let’s first export an existing data factory ARM template. Follow the below steps to do it.

1. Connect to the Azure portal and open-source Azure Data Factory Studio.
2. Navigate to Manage options.
3. Click on the Export button under ‘Export ARM Template’ as shown in the above image.
4. This will download Azure Data Factory code in a file named arm_template.zip. Unzip the file. Files of our interest are
   • ARMTemplateForFactory.json – This file contains the entire Data Factory code.
   • ARMTemplateParametersForFactory.json – This file contains parameters for the Data Factory.

Transform ARM template

Let’s change the factory parameter values in the ARMTemplateParametersForFactory.json file as per the target environment. The mandatory change you need to make in the parameter file is factoryName. Perhaps, it is also a good idea to set other factory parameters in this file.

Note: Include Global parameters in the ARM template is deprecated and only available through git integration. You may need to manually recreate global parameters in the target data factory once the import is done.

Import Azure Data Factory

1. Open PowerShell ISE and copy the below script in the new window. Ensure you have installed the Az module as mentioned in the prerequisites.

$templateFile = "Full path of ARMTemplateForFactory.json"
$parameterFile="Full path of ARMTemplateParametersForFactory.json"
$deploymentName = "Name of the deployment"
$tenantId = "Tenant id"
$subscriptionName = "Subscription name"
$resourceGroupName = "Name of the resource group"
$factoryName = "target factory name"
$location = "Location of resource group"

#Connect to Azure account
Connect-AzAccount -Tenant $tenantId -Subscription $subscriptionName

#Deploy Data Factory
New-AzResourceGroupDeployment `
  -Name $deploymentName `
  -ResourceGroupName $resourceGroupName `
  -TemplateFile $templateFile `
  -TemplateParameterFile $parameterFile `
  -Tag @{"key1"="value1"; "key2"="value2";}

#Generate managed identity for Data Factory.
Set-AzDataFactoryV2 -ResourceGroupName $resourceGroupName -Name $factoryName -Location $location

2. Provide values to the variables below
$templateFile and $parameterFile variables refer to the files we have downloaded and transformed in the above steps.
$tenantId – Tenant id of the target subscription. You can get tenantId details from here.
$subscriptionName-Subscription name of the target resource group.
$resourceGroupName-Target resource group.
$factoryName – Name of the target Data Factory. Here, we need to ensure that the name is the same one we used in the ARMTemplateParametersForFactory.json file.
$location-Location of the target resource group.

That’s it! You should see the Data Factory in the target resource group if the above script executes without error.

We have seen how adf import export can be done using PowerShell ISE and Azure Data Factory Studio in Azure portal.

Pro tips:
1. Azure Data Factory export from the Azure portal may not include global parameters in the arm template unless you have git integration enabled.
2. If you want to edit data factory triggers before the deployment, you can edit them in the arm_template.json file under the resources node.
3. If your organization has defined a policy to add tags with Azure resources, you need to include them as a parameter of the New-AzResourceGroupDeployment command.
4. You can refer to this post if you want to enable automatic publishing and deployment of the Data Factory code with Azure DevOps.
5. Learn how to export and import PowerApps package in just a few clicks.

See more

The post Export and Import Azure Data Factory appeared first on AzureOps.