Confluent

FEBRUARY 21, 2019

KSQL can be used in numerous real-time security detection and alerting tasks. By taking data from a tool such as Sysmon and streaming it into Kafka for processing in KSQL, you can rapidly detect suspicious behavior by looking for a process spawning a new process that makes an external network connection. FROM SYSMON_JOIN.

Windows 81

Windows Network Data Performance 81

Confluent

NOVEMBER 20, 2019

Today marks a new release of KSQL, one so significant that we’re giving it a new name: ksqlDB. Like KSQL, ksqlDB remains freely available and community licensed, and you can […].

Applications 111

Applications 111

Confluent

MARCH 26, 2019

We will cover Kafka clients, Kafka Streams applications, Kafka Connect pipelines and KSQL. On the other side, several consumers will subscribe to these events: A Kafka sink connector for JDBC will send tweets downstream to a Postgres database, and a KSQL application will process records further. Tracing for KSQL.

Applications 111

Applications Transportation Architecture Open Source 111

Confluent

MAY 13, 2019

Not only that, as we had previously promised, we are also introducing three of the most popular components of Confluent Platform as fully managed services: Confluent Schema Registry, the Kafka S3 sink connector, and Confluent KSQL. Fully managed Schema Registry, Kafka S3 sink connector, and KSQL in Confluent Cloud. per GB per month.

Cloud 100

Cloud Load Balancer Infrastructure Scalability 100

Confluent

JULY 10, 2019

KSQL user-defined functions. KSQL provides a nice collection of built-in SQL functions for use in functional transformation logic when doing stream processing, whether the need is scalar functions for working with data a row at a time or aggregate functions used for grouping multiple rows into one summary record of output.

Lambda 87

Lambda Analytics Testing Groups 87

Confluent

APRIL 18, 2019

Second, it provides key frameworks and APIs like Kafka Connect, KSQL and Kafka Streams to build an entirely new class of applications incorporating both real-time and stored data. BAADER processes and analyzes events with KSQL, then leverages connectors to visualize the results with Elasticsearch and Kibana.

IoT 83

IoT Industry Cloud Retail 83

Confluent

MAY 29, 2019

In part 1 , we discussed an event streaming architecture that we implemented for a customer using Apache Kafka ® , KSQL from Confluent, and Kafka Streams. Now in part 2, we’ll discuss the challenges we faced developing, building, and deploying the KSQL portion of our application and how we used Gradle to address them. KSQL primer.

Groups 95

Groups Windows Testing Microservices 95

Confluent

FEBRUARY 20, 2019

The Exciting Frontier of Custom KSQL Functions, by Mitch Seymour (Mailchimp). When to KSQL & When to Live the KStream, by Dani Traphagen (Confluent). A three-day course on KSQL and Kafka Streams starting on Wednesday, April 3. From Zero to Hero with Kafka Connect, by Robin Moffat (Confluent).

Course 66

Course Conference Training Groups 66

Apiumhub

FEBRUARY 24, 2022

The idea is to cover such important topics as: Event-Driven Architecture Deployment Open Application Model (OAM) Design for sustainability Micro frontends Design for portability Policy as Code Service Mesh Kafka + KSQL Centralized Git Workflows with Python Software architecture in a non-enterprise world Security architecture Software Architecture Diagrams (..)

Architecture 59

Architecture Software Software Engineering AWS 59

Confluent

DECEMBER 23, 2019

When KSQL was released, my first blog post about it showed how to use KSQL with Twitter data. Two years later, its successor ksqlDB was born, which we announced this […].

Data 28

Data How To Analytics 28

Confluent

MAY 15, 2019

So we chose to use the Confluent Platform and specifically KSQL to curate our streams of data. Enter KSQL…and Kafka Streams. As KSQL uses Kafka Streams under the hood—and both use Kafka topics and Kafka offset management as the interface between processes—the integration was seamless. KSQL in a DevOps world.

Analytics 91

Analytics Cloud Data System 91

Confluent

APRIL 10, 2019

Operators can manage those clusters, view topic data and schemas and run KSQL queries against data streams. First and foremost, Confluent Control Center can manage multi-datacenter Apache Kafka ® deployments, whether on prem or in the cloud. It’s not just a data swamp of hundreds of metrics—it is information you can act on.

Data 86

Data Disaster Recovery Metrics Applications 86

Confluent

MAY 8, 2019

In anything but the smallest deployment of Apache Kafka ® , there are often going to be multiple clusters of Kafka Connect and KSQL. Kafka Connect is used for building event streaming data pipelines between upstream and downstream systems with Kafka, and KSQL is used for building stream processing applications declared in a SQL-like language.

DevOps 87

DevOps Analytics Windows System Architecture 87

Confluent

AUGUST 6, 2019

One of KSQL’s most powerful features is allowing users to build their own KSQL functions for processing real-time streams of data. The previous blog post How to Build a UDF and/or UDAF in KSQL 5.0 discussed some key steps for building and deploying a custom KSQL UDF/UDAF. DarchetypeArtifactId=ksql-udf-quickstart.

Software Review 17

Software Review Testing Examples Systems Review 17

Confluent

MAY 1, 2019

That was the reason we created KSQL which has now become one of the most popular tools in the Kafka ecosystem. The underlying approach of continuously updated views on streaming data matched our vision of how developers build summarizations and joins on event streams with KSQL.

Infrastructure 21

Infrastructure Applications Data System 21

Confluent

OCTOBER 9, 2019

Based on the votes of Summit attendees from within the Kafka Summit mobile app, here are the top-rated talks: Building Stream Processing Applications with Apache Kafka Using KSQL by Robin Moffatt of Confluent. KSQL Performance Tuning for Fun and Profit by Nick Dearden of Confluent. Why Stop the World When You Can Change It?

Video 17

Video Hotels Serverless Software Engineering 17

Confluent

JULY 3, 2019

In order to achieve our targets, we’ll use pre-built connectors available in Confluent Hub to source data from RSS and Twitter feeds, KSQL to apply the necessary transformations and analytics, Google’s Natural Language API for sentiment scoring, Google BigQuery for data storage, and Google Data Studio for visual analytics.

Analysis 21

Analysis Data Sport Windows 21

Confluent

OCTOBER 4, 2019

Before beginning verification, some of you may be wondering about the rest of the Confluent Platform, such as Schema Registry, KSQL, Control Center, etc. For more, check out Kafka Tutorials and find full code examples using Kafka, Kafka Streams, and KSQL. Based on the steps you just went through, I suspect you already know the answer.

How To 16

How To Enterprise Examples Testing 16

Confluent

SEPTEMBER 26, 2019

Among these are Confluent Schema Registry , which ensures the right message structure, and KSQL for continuous stream processing on data streams, such as filtering, transformations, and aggregations using simple SQL commands. Kafka’s ecosystem also includes other valuable components, which are used in most mission-critical projects.

Analytics 21

Analytics Serverless Blockchain Microservices 21

Confluent

MARCH 13, 2019

This can be seen from the metrics: It can also be seen from inspecting the topic itself: ksql> LIST TOPICS; Kafka Topic | Registered | Partitions | Partition Replicas | Consumers | ConsumerGroups. You can use any consumer tool to examine the messages on the dead letter queue (we used KSQL above for this).

Software Review 111

Software Review Systems Review Technical Review Metrics 111

Confluent

AUGUST 13, 2019

We’re offering more options this year too, including Introductory Tutorial, Developer Learning, KSQL & Kafka Streams, Operations Training, and Advanced Kafka Optimization. Whether you’re new to Kafka or an experienced user, there will be a hands-on training likely to match your needs. Get it, PR? Okay, just workshopping here.).

Conference 19

Conference Training Recruiting Meeting 19

Confluent

JUNE 26, 2019

This blog post discusses why Apache Kafka became the de facto standard and backbone for microservice architectures—not just replacing other traditional middleware but also building the microservices themselves using DDD and Kafka-native APIs like Kafka Streams, KSQL, and Kafka Connect. Microservices.

Microservices 109

Microservices Software Review Technical Review Architecture 109

Confluent

APRIL 4, 2019

To see this equivalent branching functionality using KSQL, read the latest blog post from Robin Moffatt, KSQL: What’s New in 5.2. It describes how to the new KSQL capability CASE which defines predicates and assigns values to a field, after which separate KSQL queries can branch to different output streams based on that field.

Microservices 108

Microservices IoT Applications Architecture 108

Confluent

SEPTEMBER 23, 2019

If you’re in town early, consider signing up for the half-day tutorial on September 29th, or make a week out of it by sticking around for the developer, KSQL & Kafka Streams, operations, and adv.Kafka optimization training courses. There’s then two jam-packed days of talks, up until until 5:35 p.m. Look after yourself.

Conference 18

Conference How To System Architecture Architecture 18

Confluent

JUNE 7, 2019

Our data science team uses KSQL to experiment with raw or lifted streams to ultimately deploy new machine learning models ( using custom user-defined functions ) without writing a single line of Java code. The fork-tree is very similar to how blockchain clients resolve competing blocks. The API backend and SDK are both written in TypeScript.

Blockchain 92

Blockchain Data Technical Review Software Review 92

Confluent

MAY 22, 2019

Various options like KSQL and Single Message Transforms (SMT) are available at this stage. Start the KSQL server. We will be interacting with the data in Kafka via KSQL. Once the KSQL server starts, open a KSQL shell and list all the topics available. You should see the temperature and brightness topics.

IoT 97

IoT Google Cloud Scalability Microservices 97

Confluent

FEBRUARY 12, 2019

Now we only get the three tables from the demo schema: ksql> LIST TOPICS; Kafka Topic | Registered | Partitions | Partition Replicas | Consumers | ConsumerGroups - […] mysql-03-accounts | false | 1 | 1 | 0 | 0 mysql-03-customers | false | 1 | 1 | 0 | 0 mysql-03-transactions | false | 1 | 1 | 0 | 0 […].

Firewall 88

Firewall Data Examples How To 88

Confluent

APRIL 25, 2019

” In one example application, BAADER leverages KSQL and IoT data from cameras in their processing facilities to calculate how many times a machine is empty in five-minute rolling windows. With a connector from Confluent, they push these events to Elasticsearch and Kibana for visualizations.

IoT 79

IoT Cloud Data Technical Review 79

Confluent

AUGUST 29, 2019

Stream processing engines like KSQL furthermore give you the ability to manipulate all of this fluently. We know that Apache Kafka ® is great when you’re dealing with streams, allowing you to conveniently look at streams as tables. But what about when the relationships between items dominate your application?

Social 55

Social Network Applications Cloud 55

Confluent

SEPTEMBER 5, 2019

If this tutorial was helpful and you’re on the hunt for more on stream processing using Kafka Streams, KSQL, and Kafka, don’t forget to check out Kafka Tutorials. In this case, Spring Boot will pick up application-cloud.yaml configuration file that contains the connection to data in Confluent Cloud. SNAPSHOT.jar. Interested in more?

Applications 20

Applications How To Open Source Cloud 20

Confluent

APRIL 16, 2019

KSQL and Security: The Current State of Affairs, and Where It’s Headed by Victoria Xia of Confluent. Zen and the Art of Streaming Joins—The What, When and Why by Nick Dearden of Confluent. How To Use Kafka and Druid to Tame Your Router Data by Rachel Pedreschi of Imply Data. That makes New York a wrap!

Video 101

Video Microservices Internet Course 101

Confluent

OCTOBER 7, 2019

For more, check out Kafka Tutorials and find full code examples using Kafka, Kafka Streams, and KSQL. I’d also like to thank Viktor Gamov, my colleague at Confluent, for developing the sample application used in this tutorial, and Victoria Yu as well for making me appear more precise and interesting than I usually am.

Applications 18

Applications Software Review How To Microservices 18

Confluent

OCTOBER 16, 2019

With tools like KSQL and Kafka Connect, the concept of streaming ETL is made accessible to a much wider audience of developers and data engineers. All of this is done using KSQL, in several stages where necessary. This Kafka topic is then modelled as a table in KSQL and joined to the live stream of movement data.

Data 19

Data Training Analytics Storage 19

Confluent

JULY 31, 2019

If you are using more components of Confluent Platform than just Kafka, remember that Confluent Operator can deploy Kafka Connect , KSQL , Schema Registry , Auto Data Balancer , Control Center , and Replicator in addition to Kafka and ZooKeeper. Deploy on any platform, on premises or in the cloud.

Open Source 19

Open Source Authentication Cloud Resources 19

Confluent

JANUARY 22, 2020

I am pleased to announce the release of Confluent Platform 5.4. Like any new release of Confluent Platform, it’s packed with features. To make them easier to digest, I want […].

Storage 118

Storage 118

Confluent

APRIL 17, 2019

Part 3: Managing Multi-Cluster Kafka Connect and KSQL with Confluent Control Center. In this three-part blog series, we’ll walk you through some of those new capabilities in Control Center: Part 1: Managing and Evolving Schemas with Confluent Control Center. Part 2: Managing Kafka Configurations at Scale with Confluent Control Center.

DevOps 75

DevOps Policies Comparison Applications 75

Confluent

APRIL 2, 2019

Several components of Confluent Platform have always been available for free, such as KSQL, Schema Registry and REST Proxy, which are licensed under our Confluent Community License. KSQL adds new query expressions and gets GUI enhancements. KSQL has grown some features that we think you’ll find useful. librdkafka is now 1.0,

Cloud 21

Cloud Applications Groups Enterprise 21