Syntax

select UNIX_TIMESTAMP() as current_date from your_stream emit changes;

 The function UNIX_TIMESTAMP returns the BIGINT value representing the number of milliseconds since 1970-01-01T00:00:00 UTC.

Convert TIMESTAMP from BIGINT to DateTime in KSQL.

If you want to convert the millisecond value to a TIMESTAMP, use the FROM_UNIXTIME.

select FROM_UNIXTIME(UNIX_TIMESTAMP()) as current_date from your_stream emit changes;

FORMAT_TIMESTAMP() can also be used further to format timestamps in the required format. Read more about DateTimeFormater here.

select FROM FORMAT_TIMESTAMP(FROM_UNIXTIME(unix_timestamp)) as current_date from your_stream emit changes;

Pro tips:
1. unix_timestamp() function to get current date in ksql, when used in a persistent query, is evaluated only once when creating a persistent query.
2. If you’d like to automate the deployment of ksqlDB streams and tables, you can read more about it here.

See more

The post Current date in KSQL appeared first on AzureOps.

What is Data Streaming?

Data Streaming is a method in which input is produced continuously to perform transformations. The output is also retrieved as a constant data stream, also called setting data in motion.

What is Kafka Stream?

Kafka Streams is a library for building streaming applications that transform input Kafka topics into output Kafka topics. Kafka Streams (Kstreams) internally uses producer and consumer libraries. It is coupled with Kafka, and the API allows you to leverage the abilities of Kafka by achieving Data Parallelism, Fault-tolerance, low latency, and much more.

What is Spark Stream?

Spark stream is an extension of the core Spark API that provides scalable, high-throughput, fault-tolerant stream processing of live data streams. It allows real-time data processing from various sources like Kafka topics, Flume, Amazon Kinesis, etc. The processed data can be sink to file systems, databases, live dashboards, etc. 

This article describes the difference between streaming part of Spark vs Kafka.

Key difference between Kafka streaming and Spark streaming

Summary

Kafka Streams and Spark Streaming are potent tools for real-time data processing, but they have different strengths and weaknesses depending on the specific use case and requirements. All the above differences are based on my experiences and research and may not be accurate.

See more

The post Kafka Streaming vs Spark Streaming appeared first on AzureOps.

What is Confluent Cloud?

Confluent is a full-scale data streaming managed cloud service that enables us to access, store, and manage data as a continuous, real-time stream. Confluent Cloud provides ksqlDB, a streaming SQL engine for Kafka. It is an easy-to-use yet powerful interactive SQL interface for stream processing on Kafka,  In this article, we will see how to filter a Kafka stream by date using KSQL.

Print Kafka topic messages

Print all new messages from a topic.

PRINT 'name.of.the.topic'

Print a topic from the beginning

PRINT 'name.of.the.topic' from beginning;

Print the first 5 messages from a topic

PRINT 'name.of.the.topic' from beginning limit 5;

print every 100th message from the beginning, and stop after 5 messages.

PRINT  'name.of.the.topic' from beginning INTERVAL 100 limit 5;

Read more about it at ksql documentation.

Pro tips:
1. If you want to automate Confluent ksql deployments, you can refer to this post.
2. This post describes how to filter kafka stream by date.

The post Print Kafka Topic Messages using KSQL appeared first on AzureOps.

When we create a KSQL stream or table, we implicitly get the following pseudo columns in each stream or table. These columns are present by default in each stream/table, and we do not need to specify them in stream/table definitions.

Query ROWTIME of a particular KSQL stream or table record

The following query will show the ROWTIME of particular row(s) filtered by COLUMN_NAME.

SELECT TIMESTAMPTOSTRING(ROWTIME, 'yyyy-MM-dd HH:mm:ss.SSS') FROM SOME_STREAM
WHERE COLUMN_NAME = 'some_value' 
EMIT CHANGES;

Query to filter KSQL stream or table based on a date

The following query will filter the KSQL stream or table based on the specified date.

SELECT * FROM  SOME_STREAM 
WHERE TIMESTAMPTOSTRING(ROWTIME, 'yyyy-MM-dd') = 'yyyy-MM-dd'  --some date in the format <yyyy-MM-dd>
EMIT CHANGES;

You can format the ROWTIME as you want to apply the filter. Use auto.offset/reset = ‘Earliest’ to get data from the past.

Pro tips:
1. If you want to automate Confluent ksql deployments, you can refer to this post.

The post Filter a Kafka stream by date using KSQL appeared first on AzureOps.

Pre-requisites:
1. Confluent account with a confluent cluster and ksqlDB cluster.
2. Confluent platform installed locally using docker.
3. Confluent CLI installed locally.
4. A Basic understanding of the Confluent platform and KSQL.

Setup GitHub repository

First thing first!

1. Fork the git repository jzaralim/ksqldb-migrations-action. This repository has most of the things we need to implement CICD for ksqlDB.

2. Clone the above repository locally using any client tools like VS Code.

Create a Confluent ksqlDB cluster API key.

Login to confluent.io and navigate to the confluent cluster and ksqlDB cluster. Click on CLI instructions as shown in the below image. These instructions are specific to your selected Kafka and ksqlDB clusters, so you can copy and paste them into the Confluent CLI.

Follow the below steps to get the API key, API secret, and endpoint URL for the ksqlDB cluster.

1. Log in to the Confluent CLI,

confluent login

2. Set the context to your environment and cluster.

confluent environment use env-v7ymd0
confluent kafka cluster use lkc-0xyop2

3. Create an API key and secret for your ksqlDB cluster.

confluent api-key create --resource lksqld-pgx9ro

4. Get the ksqlDB cluster endpoint URL.

Save this information somewhere, as we will use them in the below steps.

Initialize ksql-migrations metadata

The confluent ksqldb-migrations tool uses MIGRATION_EVENTS & MIGRATION_SCHEMA_VERSIONS topics to maintain the deployment log for the deployments made through ksql-migrations. To create these topics in the confluent cluster, follow the below steps.

1. Connect to the local ksqlDB-server docker container using bash. If you are using windows bash, you can execute the below command.

docker-compose exec ksqldb-server bash

The ksql-migrations tool is available with all ksqlDB versions starting from ksqlDB 0.17 or Confluent Platform 6.2.

2. Execute the below command on the local ksqlDB-server container.

ksql-migrations new-project orders https://pksqld-zjyqd.westeurope.azure.confluent.cloud:443

Here, orders is the project’s name, and https://pksqlc-zjyqd.westeurope.azure.confluent.cloud:443 is the ksqldb cluster endpoint.

The above command will create order\migrations\ksql-migrations.properties file under the home directory in the ksqldb-server container as shown in the below image.

3. Edit the ksql-migrations.properties file with the following details. Values assigned to the variables are based on the name of the variables in the GitHub workflow file.

ksql.server.url=${CONFLUENT_KSQLDB_ENDPOINT}
ksql.migrations.topic.replicas=3
ssl.alpn=true
ksql.auth.basic.username=${CONFLUENT_KSQLDB_API_KEY} 
ksql.auth.basic.password=${CONFLUENT_KSQLDB_API_SECRET}
ksql.migrations.dir.override=${CONFLUENT_MIGRATION_DIR}

4. Now, run the below command to initiate migration metadata in the confluent cloud based on the configurations set in the above steps.

ksql-migrations initialize-metadata -c orders/ksql-migrations.properties

You should be able to see the below topic in the Confluent cloud.

Note: GitHub collaboration branch name and the workflow trigger branch name should be the same.

Set up GitHub secrets for GitHub actions.

Create GitHub secrets for the ksqlDB cluster endpoint, API key, and API secret, respectively named:

• CONFLUENT_KSQLDB_ENDPOINT
• CONFLUENT_KSQLDB_API_KEY
• CONFLUENT_KSQLDB_API_SECRET

You have retrieved these secrets in the above steps.

We are all set to automate Confluent ksqlDB deployments

It’s time to verify our setup. Start committing changes to the migrations folder in the GitHub repository. For this demo, we will commit the three migrations files inside the ksqldb-migrations-action Github repository.

1. After the initial commit, the migrate-cloud-ksqldb workflow will be triggered. It will deploy the initial three ksql migration versions to the Confluent cloud cluster.
2. After migration ORDER_EVENTS stream would look like in the below image.

Let’s say, we need to add a column DESCRIPTION to the ORDER_EVENTS stream.

1. Add a new version file V000004__add_column_to_order_events.sql.
2. Commit the new version file, triggering a Github action, as shown in the image below.

Pro tips:
1. The steps mentioned in this article must be done only once by a DevOps person, and individual developers need only to commit changes to the migrations folder.
2. Format of the migrations version file is Vxxxxxx__name_of_the_file.sql. That means ‘V’ followed by an incremental six digits number with leading 0s. E.g., V000012__new_streams.SQL.
3. It is advisable to add the IF NOT EXISTS clause in CREATE STREAM/TABLE ksql statements. And IF EXISTS in DROP STREAM/TABLE ksql statements. This will avoid failures of version files in case of object availability conflict in the ksqlDB cluster.
4. Learn how to filter kstream by rowtime using ksql .

The post Automate Confluent ksqlDB deployments with ksql-migrations appeared first on AzureOps.