Automating Confluent ksqlDB deployments with version-controlled SQL migrations provides a repeatable way to promote stream, table, query, connector, and other ksqlDB changes through CI/CD. Confluent’s current ksql-migrations tool applies versioned SQL migration files and tracks the migration state in the target ksqlDB cluster.

Quick answer: Store ksqlDB migration files in Git, initialize the migration metadata once, keep credentials in CI/CD secrets, and run ksql-migrations apply from your deployment pipeline. The tool supports migration status and checksum validation, which makes it suitable for repeatable deployments.

Prerequisites:
1. A Confluent account with a Kafka cluster and ksqlDB cluster.
2. Confluent Platform installed locally using Docker if you want to run the tooling locally.
3. Confluent CLI installed locally when using the CLI-based setup steps.
4. Basic understanding of Confluent Platform and ksqlDB.

How ksqlDB migrations work

The migration project contains a ksql-migrations.properties file and a migrations directory containing versioned SQL files. The naming convention is V000001__Description.sql, followed by the next version number for subsequent changes.

ksql-migrations-project/
├── ksql-migrations.properties
└── migrations/
    ├── V000001__Initial_setup.sql
    ├── V000002__Create_orders.sql
    └── V000003__Add_order_description.sql

Create a ksqlDB migrations project

Create the project with new-project. The command creates the configuration file and the migrations directory.

ksql-migrations new-project ./ksql-migrations https://<ksqldb-endpoint>

For Confluent Cloud, configure the generated properties file with the ksqlDB API key and secret, along with the required migration settings. Confluent’s current documentation uses the following properties for Confluent Cloud connections.

ksql.server.url=https://<ksqldb-endpoint> ksql.auth.basic.username=<CCLOUD_KSQLDB_APIKEY> ksql.auth.basic.password=<CCLOUD_KSQLDB_APIKEY_SECRET> ksql.migrations.topic.replicas=3 ssl.alpn=true

Initialize migration metadata

Run the following once against the target ksqlDB cluster:

ksql-migrations --config-file ./ksql-migrations/ksql-migrations.properties initialize-metadata

This creates the migration metadata stream and table, including MIGRATION_EVENTS and MIGRATION_SCHEMA_VERSIONS, which are used to track applied migration versions.

Create versioned ksqlDB migrations

Migration files contain the ksqlDB statements that should be applied to the target cluster. You can create a blank migration using:

ksql-migrations --config-file ./ksql-migrations/ksql-migrations.properties create Add_order_description

This creates a file such as V000002__Add_order_description.sql. The current migrations tool supports statements including CREATE STREAM, CREATE TABLE, CREATE OR REPLACE, ALTER, DROP, INSERT INTO ... AS SELECT, connectors, custom types, and other documented ksqlDB statements.

Apply migrations

Before deploying, you can preview which migrations would be applied:

ksql-migrations --config-file ./ksql-migrations/ksql-migrations.properties apply --next --dry-run

When the deployment is ready, apply the next migration:

ksql-migrations --config-file ./ksql-migrations/ksql-migrations.properties apply --next

You can also use --all, --until, or a specific version when your deployment process requires a different migration scope.

Automate ksqlDB deployments with GitHub Actions

The original version of this article used a third-party GitHub repository as the deployment action. A more maintainable approach is to keep the migration SQL in your own repository and invoke the documented ksql-migrations CLI directly from the workflow.

<pre class="wp-block-syntaxhighlighter-code">name: Deploy ksqlDB migrations
on:
  push:
    branches:
      - main
    paths:
      - 'migrations/**'
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Run ksqlDB migrations
        env:
          KSQLDB_ENDPOINT: ${{ secrets.CONFLUENT_KSQLDB_ENDPOINT }}
          KSQLDB_API_KEY: ${{ secrets.CONFLUENT_KSQLDB_API_KEY }}
          KSQLDB_API_SECRET: ${{ secrets.CONFLUENT_KSQLDB_API_SECRET }}
        run: |
          docker run --rm             -v "${{ github.workspace }}/ksql-migrations:/share/ksql-migrations"             confluentinc/cp-ksqldb-server:8.3.1             ksqldb-server             ksql-migrations             --config-file /share/ksql-migrations/ksql-migrations.properties             apply --next</code></pre>

Keep the endpoint and credentials in GitHub Actions secrets rather than committing them to the repository. If you generate the properties file during the workflow, populate the sensitive values from those secrets.

Check migration status

Use info to see the current migration version and the state of applied and pending migrations:

ksql-migrations --config-file ./ksql-migrations/ksql-migrations.properties info

Use validate to compare the checksums stored in migration metadata with the migration files currently in the repository:

ksql-migrations --config-file ./ksql-migrations/ksql-migrations.properties validate

Note that checksum validation verifies migration metadata against local files; it does not independently verify that every stream, table, query, or connector in the cluster matches the SQL files.

Important CI/CD considerations

  • Do not run migrations concurrently. Confluent documents that simultaneous ksql-migrations apply executions are not supported. Serialize deployments to a given ksqlDB cluster.
  • Review migration failures carefully. A migration file containing multiple statements is not atomic; earlier statements may already have executed if a later statement fails.
  • Use dry runs during deployment validation. The --dry-run option shows which migration files would be applied without submitting the statements to ksqlDB.
  • Protect credentials. Store API keys and secrets in your CI/CD secret store.
  • Keep migration files immutable after deployment. Because the tool stores checksums, changing an already-applied migration file can cause validation failures.

Pro tips

Pro tips:
1. Keep one migration file per logical change where practical.
2. Use the V000001__Description.sql naming convention consistently.
3. Run validate as part of your deployment checks.
4. Serialize production migrations so two pipelines cannot modify the same ksqlDB cluster simultaneously.
5. Keep API keys and secrets outside Git and inject them at deployment time.

Conclusion

ksqlDB migrations provides a version-controlled deployment model for ksqlDB SQL changes. The current workflow is straightforward: create a migration project, initialize metadata, commit versioned SQL files to Git, and run ksql-migrations apply from a controlled CI/CD pipeline. Status and checksum validation provide additional safeguards for repeatable deployments.

Current documentation: Confluent ksqlDB Migrations Tool

Kunal Rathi

With over 15 years of experience in data engineering and analytics, I've assisted countless clients in gaining valuable insights from their data. As a dedicated supporter of Data, Cloud and DevOps, I'm excited to connect with individuals who share my passion for this field. If my work resonates with you, we can talk and collaborate.