In this article, we will see how to compare Azure SQL database schema using sqlpackage.exe and generate a delta script that shows the differences in a controlled and repeatable way. The approach and flow are similar to the DACPAC-based comparison strategy discussed earlier on AzureOps.

Why use sqlpackage for database comparison?

sqlpackage.exe is a command-line utility provided by Microsoft that works with DACPAC and BACPAC files. Using sqlpackage, you can:

• Compare two databases using DACPACs
• Generate a deployment (delta) script
• Control destructive changes explicitly
• Automate comparisons in CI/CD pipelines

For Azure SQL Database scenarios, this method is especially useful when direct schema comparison tools are not available or when automation is required.

Creating a DACPAC

To compare database schemas using sqlpackage, you first need a DACPAC file, which represents the database schema.

You can create a DACPAC using sqlpackage.exe, SSMS, or Visual Studio. For a quick, step-by-step guide covering all methods, refer to this article.

Installing sqlpackage Before You Begin

Before you can compare two Azure SQL database schemas using sqlpackage, you need to install the sqlpackage command-line utility on your machine. sqlpackage.exe is the same tool that Powers the DACPAC deployments discussed in this site’s previous article on deploying DACPACs to Azure SQL Database.

Here are the recommended ways to install sqlpackage, depending on your environment:

Option 1 — Install as a .NET Global Tool (Cross-Platform)

The easiest and most modern way to install sqlpackage on Windows, macOS, or Linux is using the .NET SDK tool installer. This gives you the sqlpackage command directly on your PATH. Run below command in command prompt.

dotnet tool install --global Microsoft.SqlPackage

If you already have it installed and want to update to the latest version, run:

dotnet tool update --global Microsoft.SqlPackage

This method works well for local development and automation scripts, including PowerShell and CI/CD pipelines.

Option 2 — Download Standalone Zip (Windows / macOS / Linux)

If you don’t use the .NET SDK or prefer a self-contained download, Microsoft also publishes standalone builds of sqlpackage:

1. Visit the official SqlPackage download page on Microsoft Docs.
2. Download the appropriate zip for your OS.
3. Extract and place the folder somewhere on your system (e.g., C:\tools\sqlpackage).
4. Add the folder containing sqlpackage.exe to your system PATH so you can call it from any terminal.

This method is useful if you want to avoid installing .NET tools or need to bundle sqlpackage with deployment scripts.

High-level approach

The comparison process consists of three simple steps:

1. Export both Azure SQL databases to DACPAC files
2. Use sqlpackage.exe with /Action:Script
3. Review the generated delta script before applying it

In this article, we focus on step 2, assuming you already have DACPAC files for both databases.

Sample scenario

Let us assume the following setup:

• Source (baseline) database: sales-test-db
• Target (to be aligned) database: sales-prod-db

Our goal is to compare Azure SQL Database schema and generate a SQL script that shows what changes are required to make Production match Test.

Sample sqlpackage command

Below is a sample sqlpackage.exe command to compare two DACPAC files and generate a delta script.

sqlpackage.exe ^
  /Action:Script ^
  /SourceFile:"C:\Dacpacs\sales-test-db.dacpac" ^
  /TargetFile:"C:\Dacpacs\sales-prod-db.dacpac" ^
  /TargetDatabaseName:"sales-prod-db" ^
  /OutputPath:"C:\Dacpacs\sales_test_vs_prod_delta.sql" ^
  /p:DropObjectsNotInSource=True

In case you want to use PowerShell ISE, you can use below script.

# Path to sqlpackage.exe
$sqlPackagePath = "C:\Program Files\Microsoft SQL Server\160\DAC\bin\sqlpackage.exe"

# DACPAC paths
$SourceDacpacPath = "C:\Dacpacs\sales-test-db.dacpac"
$TargetDacpacPath = "C:\Dacpacs\sales-prod-db.dacpac"

# Target database name
$TargetDatabaseName = "sales-prod-db"

# Output script path
$OutputScriptPath = "C:\Dacpacs\sales_test_vs_prod_delta.sql"

# Publish options
$DropObjectsNotInSource = "True"

# Execute sqlpackage
& $sqlPackagePath `
    /Action:Script `
    /SourceFile:$SourceDacpacPath `
    /TargetFile:$TargetDacpacPath `
    /TargetDatabaseName:$TargetDatabaseName `
    /OutputPath:$OutputScriptPath `
    /p:DropObjectsNotInSource=$DropObjectsNotInSource

This command does not apply any changes. It only generates a SQL script that you can review and execute manually if needed.

Understanding each parameter

Let us break down all the parameters used in the command.

/Action:Script

Specifies that sqlpackage should generate a deployment script instead of publishing changes directly to a database. This is the safest option when comparing environments like Test and Production.

/SourceFile

Path to the source DACPAC file. This represents the desired state of the database.

/SourceFile:"C:\Dacpacs\sales-test-db.dacpac"

In our example, the Test database schema is treated as the source of truth.

/TargetFile

Path to the target DACPAC file. This represents the database you want to compare against.

/TargetFile:"C:\Dacpacs\sales-prod-db.dacpac"

The generated script will contain changes required to make the target match the source.

/TargetDatabaseName

Logical name of the target database.

/TargetDatabaseName:"sales-prod-db"

This parameter is mandatory for the Script action, even when you are comparing DACPAC-to-DACPAC.

/OutputPath

Specifies where the generated delta script should be written.

/OutputPath:"C:\Dacpacs\sales_test_vs_prod_delta.sql"

Always review this script carefully before executing it in Production.

/p:DropObjectsNotInSource

Controls whether objects that exist in the target but not in the source should be dropped.

/p:DropObjectsNotInSource=True

• True – Generates DROP statements for extra objects in the target
• False – Preserves objects that exist only in the target

⚠️ Use this option with caution, especially for Production databases.

Common additional publish parameters

Depending on your requirements, you may also consider these commonly used parameters:

• /p:BlockOnPossibleDataLoss=False
• /p:IgnorePermissions=True
• /p:IgnoreRoleMembership=True
• /p:ExcludeObjectTypes=Users;Logins
• /p:ExcludeSchemaValidation=True

These options help fine-tune the comparison and avoid unwanted differences.

Best practices

• Always generate a script first (/Action:Script)
• Review the delta script line by line
• Never auto-apply changes to Production without validation
• Store DACPACs and scripts in source control
• Use the same approach in CI/CD pipelines for consistency

Conclusion

Comparing two Azure SQL databases using sqlpackage.exe is a reliable and automation-friendly approach. By working with DACPAC files and generating delta scripts, you gain full visibility and control over schema changes before they reach Production.

This method fits well into modern DevOps workflows and provides a safe alternative to manual or GUI-based comparisons.

If you regularly manage schema changes across multiple environments, sqlpackage should be a core part of your database deployment tool.

Pro tips
1. If you want to generate BACPAC file of an Azure SQL database, you can refer to this post

See more

The post Compare Azure SQL Database Schema Using sqlpackage appeared first on AzureOps.

Use Case – Imagine this scenario:

• You have a main transactional database named app-db hosted on Azure SQL Server app-db.database.windows.net.
• You also have a reference or lookup database named source-db, hosted on a separate Azure SQL Server: source-server.database.windows.net.

You want your app-db to query the table dbo.external_table from the source-db — without importing or copying the data. The goal is to enable live, read-only access to shared reference data across databases in a secure and manageable way.

Prerequisites
Before diving into the setup, ensure the following:
1. Azure SQL Databases: Both the source and target databases should be Azure SQL Databases.
2. Same Region: Ensure both databases are in the same Azure region to minimize latency.
3. User Credentials: A SQL user say (ext_table_user) must exist in the source database with the necessary permissions to access the target table.

Why Create Separate user in the Source Database?

The ext_table_user is a SQL user in the source database that the target database uses to authenticate and access data. This user must have the necessary permissions (e.g., SELECT) on the target table.

Creating a dedicated user for external access enhances security by allowing you to:

• Limit Permissions: Grant only the necessary permissions to this user.
• Monitor Access: Track and audit activities performed by this user.
• Revoke Access Easily: If needed, you can revoke this user’s access without affecting other users.

Fow these steps to how to grant access to Azure SQL database.

How to Setup Cross Database Access in Azure SQL

Follow the below quick steps to setup cross database access in Azure SQL database.

Step 1: Create a Master Key in the Target Database

The master key is essential for encrypting credentials used in external data sources.

-- Execute in the target database

CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'YourStrongPasswordHere';

Skip this if SELECT * FROM sys.symmetric_keys WHERE name = '##MS_DatabaseMasterKey##' already returns a row.

Step 2: Create a Database Scoped Credential

This credential will be used to authenticate against the source database.

-- Execute in the target database

CREATE DATABASE SCOPED CREDENTIAL ext_table_user
WITH IDENTITY = 'ext_table_user',
     SECRET = 'ActualPasswordOfExtTableUser';

Step 3: Define the External Data Source

This step establishes a connection to the source database.

-- Execute in the target database

CREATE EXTERNAL DATA SOURCE LinkedAzureSQLDB
WITH (
    TYPE = RDBMS,
    LOCATION = 'source-server.database.windows.net',
    DATABASE_NAME = 'source-db',
    CREDENTIAL = ext_table_user
);

Ensure: Replace 'source-server.database.windows.net' and 'source-db' with your actual server and database names.

Step 4: Create the External Table

Now, define the external table in the target database to reference the table in the source database.

-- Execute in the target database

CREATE EXTERNAL TABLE dbo.external_table (
    [ID] NUMERIC(18, 0) NOT NULL,
    [list_id] INT NOT NULL,
    [type_id] INT NOT NULL,
    [description] VARCHAR(50) NULL
)
WITH (
    DATA_SOURCE = LinkedAzureSQLDB,
    SCHEMA_NAME = 'dbo',
    OBJECT_NAME = 'external_table'
);

Note: The schema and object names should match those in the source database.

Step 5: Query the External Table

With the setup complete, you can now query the external table as if it were local:

-- Execute in the target database

SELECT TOP 10 * FROM dbo.external_table;

Performance Tip: Monitor query performance, as cross-database queries may have higher latency.

Pro tips:
1. Use Managed Identities: For enhanced security, consider using Azure Managed Identities instead of SQL authentication.
2. Monitor Performance: Regularly monitor the performance of cross-database queries to ensure they meet your application’s requirements.
3. Secure Credentials: Store sensitive information, like passwords, securely using Azure Key Vault.
4. Elastic query feature in Azure SQL is still in preview.

By following this guide, you can efficiently set up cross databse query in Azure SQL, enabling seamless cross-database querying while maintaining security and performance.

See more

The post Cross Database Query in Azure SQL Database appeared first on AzureOps.

You write the query, run it, catch a missing condition, tweak the join, test again—repeat until it’s right. SSMS 21 brings Copilot into that loop.

TL;DR:

• SSMS 21 now includes built-in Copilot—no extensions. Just require connection to your Azure OpenAI deployment (like GPT-4o).
• Works only with SQL Server connections (not SSIS, SSAS, or SSRS).
• Runs in read-only mode—won’t execute any DML or modify data.
• Can generate queries, fix broken SQL, and suggest optimizations.
• Supports prompts like: “fix this query,” “optimize for performance,” or “generate insert script.”
• Ideal for SQL developers who want help writing, debugging, or learning better query patterns—without giving up control.

Installation: Copilot Comes Built-in

The installer for SSMS 21 clearly shows that Copilot is not an afterthought—it’s front and center. Once you launch the installer, you’ll notice the “Copilot Integration” checkbox enabled by default. This means no manual setup is required post-install. It’s part of the core experience now.

First Run: Setting Up Your Azure OpenAI Keys

Once SSMS launches, you’ll be prompted to connect Copilot to your Azure OpenAI setup.

In the first prompt, you’ll be asked to provide:

• Azure OpenAI service endpoint
• Azure OpenAI deployment
• Azure OpenAI API key

If you haven’t already created this in Azure, it’s a quick process: deploy a model, generate a key, and you’re good to go. we see how this information is pasted into SSMS.

It’s a one-time step, and once entered, Copilot starts responding instantly within SSMS.

To get an Azure OpenAI deployment name for SSMS Copilit, we need Gen AI model. That’s the engine powering your Copilot. GPT-4o is optimized for speed and real-time interaction, making it well-suited for something as dynamic as SQL writing and analysis.

If you already use GPT-based models in Azure, you can reuse the same deployment. If not, create a new one and pick GPT-4o or other depending on availability.

Which Connections Are Supported?

As of now, Copilot in SSMS works only with SQL Server connections—whether it’s a local database or Azure SQL.

It does not support:

• SQL Server Integration Services (SSIS)
• SQL Server Analysis Services (SSAS)
• SQL Server Reporting Services (SSRS)

So if you’re connected to an SSIS or SSAS instance, you won’t see Copilot suggestions. It’s focused purely on Transact-SQL (T-SQL) scenarios for now.

Read-Only Mode – No Live Data Changes

Copilot works in a read-only mode inside SSMS for GOOD.

It will not execute updates, deletes, or any data-modifying queries on its own. It doesn’t have permissions to alter databases or run DML commands. You remain in full control.

That said, it can generate update, insert, and delete queries for you, which you can review, tweak, and run manually. This helps reduce the risk of accidental changes while still speeding up development.

Quick Demo: Fixing a Broken Query

I pasted a malformed SELECT query into SSMS. Copilot immediately offered a fix and even explained what was wrong.

Few Prompt Examples: Result Accuracy

In below example, we asked Copilot to generate a complex query involving filters, aggregates, and joins. The result was accurate and ready to run with minimal edits.

the same query was further optimized by Copilot after being prompted with: “Make it efficient.” It introduced indexing suggestions and added a WITH (NOLOCK) hint—something even experienced developers can forget.

In another example, we asked Copilot another question on a different query context (another database connection), and it just worked fine.

These examples show that Copilot isn’t just copying boilerplate SQL—it understands context and best practices.

Should You Use This?

If your daily work involves writing or reading SQL, then yes—this is probably the most useful update to SSMS in years.

Here’s what it helps with:

• Writing long queries faster
• Understanding joins and filters
• Fixing syntax issues
• Learning better SQL patterns
• Generating stored procedure templates
• Drafting DML scripts (but not running them automatically)

What It Won’t Do (Yet)

Let’s be clear:

• It won’t connect to SSIS/SSAS/SSRS
• It won’t execute anything that modifies data
• It won’t automatically run DDL/ DML code it suggests

This is intentional, for safety and audit reasons. Think of it as a smart assistant—not a trigger-happy automation engine.

A Note on Privacy and Control

Copilot integrates directly with your Azure environment, so you control what it accesses and shares. It processes your prompts and data securely within your subscription. According to Microsoft, it doesn’t store any query logs outside your environment, which helps meet compliance requirements for enterprise users.

SSMS 21 doesn’t just add Copilot—it integrates it naturally into the flow of writing, editing, and reviewing SQL. It’s fast, context-aware, and doesn’t get in the way.

Pro tips:
1. While it’s good to have Copilot like features in SSMS 21, it seems to have lost some very important features like Import Export Wizard for databases like Azure SQL.
2. Learn how to build data agent in Microsoft Fabric.

See more

The post SSMS 21 Copilot: Here’s How It Works appeared first on AzureOps.

Why Use Managed Identity with Azure SQL?

Storing passwords in configuration files is risky. Managed Identity solves this problem by:

• Eliminating Secrets → No passwords, no connection strings with credentials.
• Centralized Identity Management → Uses Azure AD authentication for consistency.
• Least Privilege Principle → Assigns only the required SQL permissions.
• Automatic Rotation → No need to update or rotate credentials manually.

This makes it the recommended authentication method when connecting Azure Functions to SQL Database.

Prerequisites:
Before implementing, ensure you have:
1. An Azure subscription.
2. An Azure SQL Database already created.
3. An Azure Function App (Python runtime).
4. Azure CLI or Portal access.

This guide assumes you already have a basic understanding of creating an Azure Function. If you need a refresher, please refer to our previous article on How to Create a Python Azure Function.

Let’s now see how to connect Python Azure Function to Azure SQL database using managed identity authentication. In this example we will fetch data from a public API and securely write it to an Azure SQL Database table..

Step 1: Set Up Your Azure SQL Database

Creating WeatherData Table

In this guide, we’ll simulate fetching weather data from an API and storing it in Azure SQL.

Run the following SQL script inside your Azure SQL database:

CREATE TABLE WeatherData (
    Id INT PRIMARY KEY IDENTITY(1,1),
    City VARCHAR(100),
    Temperature DECIMAL(5,2),
    Humidity INT,
    ForecastDate DATETIME
);

Step 2: Enabling Managed Identity for Azure Function

INavigate to your Function App in Azure Portal.

Under Settings → Identity, enable System Assigned Managed Identity.

Azure will create an identity in Microsoft Entra representing your Function.

Step 3: Grant the Managed Identity Access to Azure SQL

Configuring Azure Entra Admin

To allow Managed Identity to authenticate, assign an Azure Entra admin for your SQL server.

1. Go to Azure Portal → SQL Server → Active Entra Admin.
2. Assign an Azure Entra user/group as the admin.
3. Save changes.

Grant the permissions

Now, connect to to your Azure SQL Database using the Entra ID authentication. You need to grant the Managed Identity permission to write to the database. Use the following T-SQL commands, replacing <your-azure-function-name> with the name of your function:

CREATE USER [your-azure-function-name] FROM EXTERNAL PROVIDER;
ALTER ROLE db_datawriter ADD MEMBER [your-azure-function-name];

• CREATE USER creates the identity inside SQL.
• ALTER ROLE gives write permissions to insert/update/delete.

Learn how to grant access to Azure SQL databases in this post.

Step 4: Configure Application Settings

In the Function App Environment variables, define following variables.

• SqlConnectionInfo → Server=<your-sql-server>.database.windows.net;Database=<your-db>;
• WEATHER_API_URL → https://api.open-meteo.com/v1/forecast?latitude=60.17&longitude=24.94&hourly=temperature_2m,relative_humidity_2m
• TIMER_SCHEDULE → 0 * * * * * (Runs every hour).

Step 5: Writing Python Azure Function Code

Now, it’s time to write the code that will perform the data fetching and insertion. For this example, we’ll use a Python HTTP-triggered function and the requests and pyodbc libraries. You will also need the azure-identity package to handle the Managed Identity authentication.

1. Installing Required Packages

Add to requirements.txt:

azure-functions
requests
pypyodbc

2. Write the function code (__init__.py):

import azure.functions as func
import logging, os, requests, pypyodbc

app = func.FunctionApp()

@app.timer_trigger(schedule="%TIMER_SCHEDULE%", arg_name="myTimer")
def load_weather_data_to_sql(myTimer: func.TimerRequest):
    if myTimer.past_due:
        logging.info("The timer is past due!")

    try:
        url = os.getenv("WEATHER_API_URL")
        response = requests.get(url)
        data = response.json().get("current", {})
        
        temperature = data.get("temperature_2m")
        humidity = data.get("relative_humidity_2m")
        city = "Berlin"

        conn = get_sql_connection()
        cursor = conn.cursor()
        cursor.execute(
            "INSERT INTO WeatherData (City, Temperature, Humidity, ForecastDate) VALUES (?, ?, ?, GETDATE())",
            (city, temperature, humidity)
        )
        conn.commit()
        cursor.close()
        conn.close()
        logging.info("Weather data inserted successfully")

    except Exception as e:
        logging.error(f"Error: {e}")

def get_sql_connection():
    conn_info = os.getenv("SqlConnectionInfo")
    parts = dict(item.split("=",1) for item in conn_info.split(";") if "=" in item)
    server, db = parts.get("Server"), parts.get("Database")
# Depending on the python version, select appropriate ODBC Driver. For latest version 3.12, use ODBC Driver 18. For previous versions, try ODBC Driver 17.
    conn_str = f"DRIVER={{ODBC Driver 18 for SQL Server}};Server={server};Database={db};Authentication=ActiveDirectoryMsi;"
    return pypyodbc.connect(conn_str)

Step 6: Testing the Integration

Running Function Locally

• Use Azure Functions Core Tools.
• Set local.settings.json with required env variables.
• Test the API call and SQL insertion.

Deploying to Azure

• Push code to Azure Function App.
• Check logs in Application Insights or Log Stream.
• Common Issues and Troubleshooting.

Best Practices for Secure and Scalable Integration

• Use least privilege for SQL roles.
• Store configs in Azure Key Vault.
• Monitor using Azure Monitor + Application Insights.
• Implement retry logic for transient failures.

FAQs

Conclusion

By following these steps, you’ve securely connected a Python Azure Function to an Azure SQL Database using Managed Identity. This approach avoids hardcoding credentials, ensures automatic rotation, and aligns with cloud security best practices.

With this foundation, you can extend the Function to handle more APIs, perform ETL pipelines, or automate reporting tasks, all while staying secure and compliant.

Pro tips:
1. Always prefer pypyodbc in Functions to avoid dependency issues.
2. Use retry policies when inserting data into SQL.
3. Keep Functions lightweight, offload heavy processing to Azure Data Factory or Microsoft Fabric.
4. If scaling out, validate SQL performance with connection pooling.
5. Learn how to connect to Azure SQL from Azure Data Factory using managed identity.

See more

The post Connect Python Azure Function to Azure SQL Using Managed Identity appeared first on AzureOps.

Assign a New Admin via Azure Portal

1. Go to the Azure Portal.
2. Navigate to Azure Analysis Services.
3. Find the AAS server where your model was deployed.
4. Under Settings, click Administrators.
5. Add a new administrator (Global Admins or Subscription Owners can do this).
6. Save the changes.
7. Delete the inactive admin account if any from the admin list.

With the addition of a new admin account, the Azure Analysis Services model missing issue is resolved.

See more

The post Azure Analysis Services Model Missing appeared first on AzureOps.

Terraform Environment Setup

Databricks is a cloud-based tool built on Apache Spark used in machine learning, big data analytics, and data engineering to enable AI model deployment, fast data processing, and collaboration. Its scalable Delta Lake infrastructure supports its efficient, secure data management.

To manage Databricks infrastructure with Terraform, you have to install and configure some of the systems; the first step is installing Terraform and the Databricks provider. Download Terraform from the official website and ensure it is available in the system’s path, and then initialize a Terraform working directory by creating a new directory and running :

terraform init

Authentication in Databricks is configured using the credentials from cloud providers. A service principal is used to set up authentication for Azure:

provider "databricks" {
 host  = "https://<your-databricks-instance>"
 azure_client_id     = var.azure_client_id
 azure_client_secret = var.azure_client_secret
 azure_tenant_id     = var.azure_tenant_id
}

An Access key or service accounts will be required for AWS and GCP

Databricks Infrastructure with Terraform Setup

Terraform allows configuration using HCL (Hashicorp Configuration Language). and configurations are made in the primary configuration file, main.tf, declares a Databricks workspace:

resource "databricks_workspace" "example" {
 name          = "example-workspace"
 resource_group = "example-rg"
 location      = "East US"
}

Configuring clusters within a workspace requires some important setup, like versioning, autoscaling, and specific nodes:

resource "databricks_cluster" "example" {
 cluster_name            = "example-cluster"
 spark_version           = "12.0.x-scala2.12"
 node_type_id            = "Standard_D3_v2"
 autotermination_minutes = 20
 autoscale {
   min_workers = 2
   max_workers = 8
 }
}

Access Control and Permissions

Terraform will be able to control user roles and access control lists using the Databricks permission control user and group access

resource "databricks_group" "data_scientists" {
 display_name = "Data Scientists"
}

resource "databricks_user" "analyst" {
 user_name  = "analyst@example.com"
 groups     = [databricks_group.data_scientists.id]
}

Integrating Terraform with a cloud IAM system like Microsoft Entra Id ensures authentication and access policies are in line with enterprise governance best practices.

Deploying Databricks Jobs with Terraform

Databricks jobs are used to automate batch and streaming workloads; the databricks_job resources define the job execution parameters:

resource "databricks_job" "example_job" {
 name = "Example Job"
 new_cluster {
   spark_version = "12.0.x-scala2.12"
   node_type_id  = "Standard_D3_v2"
   num_workers   = 4
 }
 notebook_task {
   notebook_path = "/Shared/example_notebook"
 }
}

Scheduling jobs and dependencies are managed within Terraform, allowing seamless automation of recurring tasks.

Versioning with Terraform

Terraform uses a state file to track all deployed resources to ensure collaborations and prevent conflicts, so remote state storage must be configured.

terraform {
 backend "azurerm" {
   resource_group_name  = "terraform-state-rg"
   storage_account_name = "tfstateaccount"
   container_name       = "tfstate"
   key                  = "databricks.tfstate"
 }
}

This ensures changes made to infrastructure are managed safely using terraform plan and terraform apply, preventing unintended changes.

Automating Deployment with CI/CD

To ensure infrastructure with consistency, Terraform needs to integrate with CI/CD; a Github action pipeline can be used for automated deployments:

name: Terraform Deployment
on: [push]
jobs:
 deploy:
   runs-on: ubuntu-latest
   steps:
     - uses: actions/checkout@v3
     - name: Setup Terraform
       uses: hashicorp/setup-terraform@v1
     - name: Terraform Init
       run: terraform init
     - name: Terraform Apply
       run: terraform apply -auto-approve

This pipeline can also be configured in GitLab CI/CD or Azure DevOps for streamlined deployments.

Monitoring and Scaling Databricks infrastructure

Databricks environments require monitoring for performance and cost efficiency; Terraform can be used to configure monitoring solutions:

resource "databricks_instance_profile" "monitoring" {
 instance_profile_arn = "arn:aws:iam::123456789012:instance-profile/DatabricksMonitor"
}
Autoscaling is enabled on the clusters to optimize resource utilization without over-provisioning:
autoscale {
 min_workers = 4
 max_workers = 16
}

Errors and debugging Terraform Deployments

Some common Terraform errors include authentication failures and state mismatches. Provider authentication issues can be resolved by verifying permissions and credentials. State mismatches often require manual state adjustments:

terraform state rm databricks_cluster.example
terraform apply

Quota restrictions and API rate limits must be handled by adjusting Terraform configurations to include retries and back-off mechanisms.

Databricks Deployments Optimization.

More resources like Unity Catalog for data governance can be managed with Terraform:

resource "databricks_metastore" "unity" {
 name = "example-metastore"
}

MLflow tracking servers can also be set up to manage ML models and notebooks through Git integration, ensuring streamlined workflows.

This tutorial has shown how to automate Databricks infrastructure with Terraform, including infra provisioning, access control, job automation, CI/CD integration, and monitoring, but more optimizations like cost management, security best practices, and additional Databricks features can be included.

Pro tips:
1. Follow this guide to know how to manage secret scopes in databricks.

See more

The post Automate Databricks Infrastructure as Code with Terraform appeared first on AzureOps.

Prerequisites:
1. Visual Studio with data tools installed on your system.
2. An Azure SQL Database set up and accessible.
3. A DACPAC file ready for deployment.

Create a DACPAC File

In this post, we cover the process of creating a DACPAC file, a critical artifact that encapsulates the schema and objects of a SQL Server database. Learn how to build and validate your DACPAC files to ensure seamless deployments.

Open a DACPAC File in Visual Studio

Discover how to open and inspect a DACPAC file in Visual Studio. This guide provides details of how to open DACPAC file in Visual Studio.

Deploy DACPAC to Azure SQL Database from Visual Studio

Follow the below steps to deploy the DACPAC project from visual Studio. You can follow the same below steps to deploy database project to any SQL Server database or Azure Managed Instance. Make sure to select correct target platform and provide applicable connection details.

1. Open Visual Studio

1. Launch Visual Studio and open your solution or project.
2. Ensure you have the SQL Server Data Tools (SSDT) installed.

2. Change target platform of database project

1. Right-click on database project and click properties.
2. Change the Target platform to Microsoft Azure SQL Database as shown in the below image.

3. Publish the Database Project

Right-click on the project and select Publish from the context menu.

In Solution Explorer, locate your database project.

4. Configure the Deployment Target

1. In the Publish Database dialog, click Edit to specify the target database.

2. In the Edit Target Database Connection window:

• Select Azure SQL Database as the target.
• Enter the server name, authentication details, and select the target database.
• Test the connection to ensure it is valid.
• Click OK to confirm.

5. Review and Publish

1. Back in the Publish Database dialog, review the settings to ensure the correct DACPAC file and target database are selected.

2. Optionally, click Advanced to configure deployment options such as excluding certain schema changes as shown in the below image.

3. Click Publish to start the deployment process.

6. Monitor and Verify Deployment

1. Monitor the progress of dacpac publish in Visual Studio as shown in the below image.
2. Once the deployment is complete, open SQL Server Management Studio (SSMS) or use Visual Studio’s built-in tools to verify the schema changes in your Azure SQL Database.
3. Run test queries to confirm the database is functioning as expected.

Pro tips:
1. Refer to this post if you wish to Automate Azure SQL Database deployment using Azure DevOps CICD pipeline.
2. If you skip target platform change step during the deployment, you may get an error “A project which specifies SQL Server 2022 or Azure SQL Database Managed Instance as the target platform cannot be published to Microsoft Azure SQL Database v12.”. Refer to this post to learn more about it.
3. Test your DACPAC deployment on a local environment before pushing it to Azure.
4. Always take a backup of your target Azure SQL Database before deployment.
5. Use tools like SQL Compare to identify schema differences between the DACPAC and the target database.

FAQ

See more

The post Deploy DACPAC to Azure SQL Database appeared first on AzureOps.

Prerequisites:
1. Visual Studio Code installed on your machine.
2. Azure Functions Core Tools installed. This can be installed via Node.js or the npm install -g azure-functions-core-tools command.
3. Python Interpreter (Ensure Python 3.8 or later is installed, preferably the latest version). Install it from Microsoft Store.
4. Azure Account with an active subscription. If you don’t have one, create a free Azure account at Azure Free Account.
4. Azure Functions Extension for Visual Studio Code installed. Install this from the Extensions marketplace in Visual Studio Code.
5. Azure CLI (Command Line Interface), for deploying the function.

Follow the below steps to Create Python Azure function in Visual Studio Code

1. Open Visual Studio Code and on the left-hand side ribbon, Click Azure Icon. A menu at the bottom will pop up. Click on the Azure function icon and click the ‘Create Function’ menu item.

2. Select Python as a language as shown in the below image.

3. Select ‘Model V2’ as the Python programming model.

4. Select Python interpreter as the latest available version (3.11.9).

5. Select the type of trigger you want to use for this function. In our case, we will use the Time trigger. Provide a CRON Expression (e.g., 0 */5 * * * * for every 5 minutes) to schedule the timer trigger.

6. Once the above steps are complete, Visual Studio Code will generate a Python Azure Function template with a pre-defined structure. Open the __init__.py file and add the desired logic for your function.

import datetime
import logging

import azure.functions as func

def main(mytimer: func.TimerRequest) -> None:
    utc_timestamp = datetime.datetime.utcnow().replace(tzinfo=datetime.timezone.utc).isoformat()

    if mytimer.past_due:
        logging.warning('The timer is past due!')

    logging.info(f'Python timer trigger function ran at {utc_timestamp}')

Debugging Your Function Locally

1. Press F5 or click on the Run and Debug option in Visual Studio Code.
2. Azure Core Tools will emulate the function runtime locally, allowing you to test the functionality.

Deploying the Function to Azure

1. Click on the Azure Icon in Visual Studio Code.
2. Sign in to your Azure account if not already done.
3. Right-click on the Functions node and select Deploy to Function App.
4. Follow the prompts to create or choose an Azure Function App in your subscription.

We have seen how to create Python Azure function in Visual Studio Code.

See more

The post Create Python Azure function in Visual Studio Code appeared first on AzureOps.

Why JSON in SQL Server?

JSON allows applications to exchange structured data with simplicity and efficiency. SQL Server’s JSON support makes it easy to:

1. Store JSON data in table columns.
2. Query JSON data directly from the database.
3. Transform JSON data into relational formats.

The key functions provided by SQL Server for JSON manipulation include:

• OPENJSON(): Parses JSON text and returns objects in tabular format.
• JSON_VALUE(): Extracts a scalar value from a JSON string.
• JSON_QUERY(): Retrieves a JSON object or array from a JSON string.

How to Parse JSON in SQL Query

Let us see various examples of parsing JSON strings in SQL Server.

Parsing JSON with OPENJSON()

The OPENJSON() function is the most versatile tool for working with JSON in SQL Server. It can parse JSON text and return a result set that can be queried like a regular table.

Example 1: Basic JSON Parsing

Suppose you have the following JSON stored in a variable:

DECLARE @json NVARCHAR(MAX) =
    N'{
        "employees": [
            {"id": 1, "name": "John Doe", "role": "Manager"},
            {"id": 2, "name": "Jane Smith", "role": "Developer"}
        ]
    }';

To parse this JSON and extract the employee details, you can use OPENJSON() with a specified path:

SELECT id, name, role
FROM OPENJSON(@json, '$.employees')
WITH (
    id INT '$.id',
    name NVARCHAR(50) '$.name',
    role NVARCHAR(50) '$.role'
);

Output:

Explanation:

• '$' denotes the root of the JSON object.
• $.employees navigates to the employees array within the JSON.
• The WITH clause specifies the schema for the JSON fields to be extracted.

Extracting Values with JSON_VALUE()

The function is ideal if you need to extract individual values from a JSON string.

Example 2: Extract a Specific Value

Using the same JSON example above, you can retrieve the name of the first employee:

SELECT JSON_VALUE(@json, '$.employees[0].name') AS FirstEmployeeName;

Output:

Extracting Nested Objects with JSON_QUERY()

The JSON_QUERY() function is used to extract complex objects or arrays from a JSON string.

Example 3: Extract an Array

To extract the entire employees array:

SELECT JSON_QUERY(@json, '$.employees') AS EmployeesArray;

Combining JSON Functions in Queries

You can combine these functions to manipulate and query JSON data effectively.

Example 4: Parsing JSON Stored in a Table

Assume you have a table named EmployeeData with a column EmployeeJSON that stores JSON strings.

CREATE TABLE EmployeeData (
    Id INT IDENTITY PRIMARY KEY,
    EmployeeJSON NVARCHAR(MAX)
);

INSERT INTO EmployeeData (EmployeeJSON)
VALUES
(N'{"id": 1, "name": "John Doe", "role": "Manager"}'),
(N'{"id": 2, "name": "Jane Smith", "role": "Developer"}');

To extract and display employee details:

SELECT
    JSON_VALUE(EmployeeJSON, '$.id') AS EmployeeId,
    JSON_VALUE(EmployeeJSON, '$.name') AS EmployeeName,
    JSON_VALUE(EmployeeJSON, '$.role') AS EmployeeRole
FROM EmployeeData;

Output:

Parsing Complex JSON Objects

SQL Server can handle deeply nested JSON structures. Let’s consider a more complex example:

Example 5: Parsing Nested JSON with Arrays and Objects

Suppose you have a JSON structure with nested objects:

DECLARE @complexJson NVARCHAR(MAX) =
    N'{
        "company": {
            "name": "TechCorp",
            "employees": [
                {
                    "id": 1,
                    "name": "John Doe",
                    "details": {
                        "role": "Manager",
                        "salary": 75000
                    }
                },
                {
                    "id": 2,
                    "name": "Jane Smith",
                    "details": {
                        "role": "Developer",
                        "salary": 60000
                    }
                }
            ]
        }
    }';

To extract employee details including nested properties:

SELECT id, name, role, salary
FROM OPENJSON(@complexJson, '$.company.employees')
WITH (
    id INT '$.id',
    name NVARCHAR(50) '$.name',
    role NVARCHAR(50) '$.details.role',
    salary INT '$.details.salary'
);

Explanation:

• The path $.company.employees navigates to the employees array inside the company object.
• Nested properties like role and salary are accessed using the full JSON path ($.details.role and $.details.salary).

Output:

Pro tips:
1. SQL Server’s JSON functions make it incredibly simple to parse and manipulate JSON data without requiring additional tools or complex workarounds. Whether you’re working with JSON stored in variables or tables, these functions provide powerful options for extracting and querying data.
2. Understanding these features simplifies JSON handling and enhances your ability to work effectively with modern data formats. Start leveraging SQL Server’s JSON capabilities in your next project to streamline your data workflows.
3. SQL Server 2025 private preview is announced. Read more about it here.
4. If you want to improve performance of SQL Server database, you can find long running queries by following this post.

See more

The post Parse JSON in SQL Server appeared first on AzureOps.

What is VACUUM in the Delta table?

VACUUM empties all files from the table directory that are not managed by Delta and data files that are no longer in the latest state of the transaction log for the table and are older than a retention threshold.

How to use Databricks VACUUM on Databricks Delta tables

database_names_filter = "20_silver_zendesk_eg"
dbs = spark.sql(f"SHOW DATABASES LIKE '{database_names_filter}'").select("databaseName").collect()
dbs = [(row.databaseName) for row in dbs]
for database_name in dbs:
print(f"Found database: {database_name}, performing actions on all its tables..")
tables = spark.sql(f"SHOW TABLES FROM {database_name}").select("tableName").collect()
tables = [(row.tableName) for row in tables]
for table_name in tables:
print(f"performing vaccum on {table_name}")
spark.sql(f"ALTER TABLE {database_name}.{table_name} SET TBLPROPERTIES ('delta.logRetentionDuration'='interval 2 days', 'delta.deletedFileRetentionDuration'='interval 1 days')")
spark.sql(f"VACUUM {database_name}.{table_name}")
spark.sql(f"ANALYZE TABLE {database_name}.{table_name} COMPUTE STATISTICS")

If you run VACUUM on a Delta table, you lose the ability to time-travel back to a version older than the specified data retention period.

See more

The post Databricks VACUUM Command appeared first on AzureOps.