Confluent

AUGUST 29, 2019

We know that Apache Kafka ® is great when you’re dealing with streams, allowing you to conveniently look at streams as tables. Stream processing engines like KSQL furthermore give you the ability to manipulate all of this fluently. The approach we’ll use works with any Kafka run though. 8, and so on.

Social 55

Social Network Applications Cloud 55

OpenCredo

FEBRUARY 20, 2019

In this blog post, I’ll cover the steps to easily set up a PKI with Vault from HashiCorp , and use it to secure a Kafka Cluster. Kafka is a distributed fault-tolerant, high-throughput and low-latency streaming platform for handling real-time data feeds. So in such an event, the security impact will be limited.

Authentication 73

Authentication Policies Tools Systems Review 73

Cloudera

OCTOBER 17, 2023

Recently, Confluent hosted Current 2023 (formerly Kafka summit) in San Jose on Sept 26th and 27th. With few conferences curating content specific to streaming developers, Current has historically been an important event for anyone trying to keep a pulse on what’s happening in the streaming space. Flink is here to stay.

Technical Review 78

Technical Review Generative AI Conference Open Source 78

Cloudera

MARCH 15, 2023

In this post, I will demonstrate how to use the Cloudera Data Platform (CDP) and its streaming solutions to set up reliable data exchange in modern applications between high-scale microservices, and ensure that the internal state will stay consistent even under the highest load. Introduction Many modern application designs are event-driven.

Data 91

Data Microservices Storage Applications 91

Instaclustr

JULY 7, 2019

Massively Scalable Geospatial Anomaly Detection with Apache Kafka and Cassandra. In this blog we discover that we’ve been trapped in Flatland, and encounter a Dimension of the Third Kind. However, given that we want the ability to use different length geohashes to find nearby events, just using the raw altitude won’t work.

3D 28

3D Systems Review Software Review Examples 28

Confluent

FEBRUARY 27, 2019

FaaS functions only solve the compute part, but where is data stored and managed, and how is it accessed? What is more, as the world adopts the event-driven streaming architecture, how does it fit with serverless? The key to event-first systems design is understanding that a series of events captures behavior.

Serverless 109

Serverless Lambda AWS Systems Review 109

Confluent

JULY 10, 2019

As discussed in part 2, I created a GitHub repository with Docker Compose functionality for starting a Kafka and Confluent Platform environment, as well as the code samples mentioned below. gradlew composeUp. KSQL user-defined functions. KSQL user-defined functions.

Lambda 86

Lambda Analytics Testing Groups 86

Confluent

APRIL 4, 2019

Event-driven architecture means just that: It’s all about the events. In a microservices architecture, events drive microservice actions. No event, no shoes, no service. In the most basic scenario, microservices that need to take action on a common stream of events all listen to that stream.

Microservices 108

Microservices IoT Applications Architecture 108

Confluent

MAY 15, 2019

These systems acted as both sources and targets for Oracle WMS Cloud and were represented by a host of different APIs, including relational tables, file-based interactions, SOAP and REST. This left us with only two data integration requirements to solve: relational tables and REST APIs. Streaming data into Oracle WMS Cloud.

Analytics 90

Analytics Cloud Data System 90

Cloudera

MAY 4, 2021

Event-driven and streaming architectures enable complex processing on market events as they happen, making them a natural fit for financial market applications. Flink SQL is a data processing language that enables rapid prototyping and development of event-driven and streaming applications. Streaming VWAP.

Marketing 79

Marketing Data Windows Groups 79

Instaclustr

AUGUST 11, 2021

In Part 1 of this two-part blog series we discovered that Debezium is not a new metallic element (but the name was inspired by the periodic table), but a distributed open source CDC technology, and that CDC (in this context) stands for Change Data Capture (not Centers for Disease Control or the Control Data Corporation).

Data 68

Data Systems Review Scalability Off-The-Shelf 68

Confluent

MAY 29, 2019

In part 1 , we discussed an event streaming architecture that we implemented for a customer using Apache Kafka ® , KSQL from Confluent, and Kafka Streams. In part 3, we’ll explore using Gradle to build and deploy KSQL user-defined functions (UDFs) and Kafka Streams microservices.

Groups 94

Groups Windows Testing Microservices 94

Cloudera

NOVEMBER 2, 2020

Ingest 100s of TB of network event data per day . real-time customer event data alongside CRM data; network sensor data alongside marketing campaign management data). The factors driving this trend are part technical, part business, and part cultural. Billion records are updated daily in aggregate tables.

Data 95

Data Analytics Storage Big Data 95

Confluent

FEBRUARY 13, 2019

Part 1 of this series discussed why you need to embrace event-first thinking, while this article builds a rationale for different styles of event-driven architectures and compares and contrasts scaling, persistence and runtime models. Event-driven architecture. Event-driven, reactive architecture.

Architecture 89

Architecture Programming Microservices Serverless 89

Instaclustr

AUGUST 4, 2021

1) In the last year the U.S. Centers for Disease Control and Prevention ( cdc.gov ) has certainly been in the forefront of the news, and their Apache Kafka Covid-19 pipeline was the inspiration for my ongoing Kafka Connect streaming data pipeline series. below) with the new silicon transistor. What is Change Data Capture?

Data 59

Data Open Source Analytics System 59

Confluent

MAY 9, 2019

Event-first thinking enables us to build a new atomic unit: the event. Four pillars of event streaming. Pillar 1 – Business function: Payment processing pipeline. Pillar 4 – Operational plane: Event logging, DLQs and automation. Parting thoughts and preparing for the future. Putting it all together.

Microservices 92

Microservices Metrics Systems Review Architecture 92

Confluent

JUNE 18, 2019

It involves many moving parts, from data preparation to building indexing and query pipelines. Distributed transactions are very hard to implement successfully, which is why we’ll introduce a log-inspired system such as Apache Kafka ®. Building an indexing pipeline at scale with Kafka Connect. Direct indexing.

Scalability 95

Scalability Architecture Machine Learning Artificial Inteligence 95

Confluent

MAY 22, 2019

In light of this, we’ll share an emerging machine-to-machine (M2M) architecture pattern in which MQTT, Apache Kafka ® , and Scylla all work together to provide an end-to-end IoT solution. MQTT Proxy + Apache Kafka (no MQTT broker). On the other hand, Apache Kafka may deal with high-velocity data ingestion but not M2M.

IoT 96

IoT Google Cloud Scalability Microservices 96

Bernd Rucker

APRIL 6, 2020

I want to answer all open questions in this blog post. Q: Why do you name events and functions in the BPMN model in the same style? If you want to keep the event names from your application, we could model: I do agree that this might be the most accurate way of modeling this situation. Unfortunately this post got a bit long?—?sorry,

Microservices 52

Microservices Architecture Technical Review Engineering 52

Confluent

APRIL 17, 2019

As we announced in Introducing Confluent Platform 5.2 , the latest release introduces many new features that enable you to build contextual event-driven applications. In this three-part blog series, we’ll walk you through some of those new capabilities in Control Center: Part 1: Managing and Evolving Schemas with Confluent Control Center.

DevOps 74

DevOps Policies Comparison Applications 74

Henrik Warne

JULY 27, 2019

There are three parts in the book: Foundations of Data Systems (chapters 1 – 4), Distributed Data (chapters 5 – 9), and Derived Data (chapters 10 – 12). I really like the mix of references – some are to computer science papers from the 1970s and onwards, and many are to various blog posts. Partitioning.

Systems Review 214

Systems Review Software Review Applications Data 214

Instaclustr

APRIL 20, 2023

How Do Kafka Concurrency and Throughput Work? Apache Kafka ® is a high-throughput, low-latency distributed streaming platform. It enables messages to be sent from multiple distributed producers via the distributed Kafka cluster and topics, to multiple distributed consumers. Is there a limit?

Scalability 57

Scalability Performance Software Review Groups 57

Instaclustr

OCTOBER 14, 2019

A complete rundown leading up to ApacheCon USA, starting a week before the final event. This blog provides a brief of things I learnt during the 4-day event, including a summary of talks I attended and presented. IoT, geospatial, ML, streaming, etc.), Core Technology Talks (Cassandra, Kafka). Source: Paul Brebner).

Technical Review 36

Technical Review Technology Report Open Source 36

Confluent

APRIL 17, 2019

As we announced in Introducing Confluent Platform 5.2 , the latest release introduces many new features that enable you to build contextual event-driven applications. In this three-part blog series, we’ll walk you through some of those new capabilities in Control Center: Part 1: Managing and Evolving Schemas with Confluent Control Center.

DevOps 40

DevOps Policies Comparison Applications 40

Confluent

SEPTEMBER 20, 2019

As a distributed system for collecting, storing, and processing data at scale, Apache Kafka ® comes with its own deployment complexities. To simplify all of this, different providers have emerged to offer Apache Kafka as a managed service. Before Confluent Cloud was announced , a managed service for Apache Kafka did not exist.

Software Review 21

Software Review Technical Review Storage Cloud 21

Confluent

JULY 3, 2019

France, Brazil, and the USA are the favourites, and this year Italy is present at the event for the first time in 20 years. There’s a lot of dedicated press coverage, as well as the standard social media excitement following any kind of big event. confluent-hub install jcustenborder/kafka-connect-twitter:latest.

Analysis 21

Analysis Data Sport Windows 21

O'Reilly Media - Ideas

JANUARY 25, 2024

Each graph is scaled so that the topic with the greatest usage is 1. Use of content about event-driven architecture is relatively small, but it’s up 40%. That change is important because event-driven architecture is a tool for designing large systems that have to ingest data from many different streams in real time.

Trends 116

Trends Technical Review Technology Artificial Inteligence 116

Instaclustr

OCTOBER 14, 2019

A complete rundown leading up to ApacheCon USA, starting a week before the final event. This blog provides a brief of things I learnt during the 4-day event, including a summary of talks I attended and presented. . Core Technology Talks (Cassandra, Kafka). Countdown begins: The week before ApacheCon. How good is that!

Technical Review 28

Technical Review Technology Report Open Source 28

Altexsoft

MAY 14, 2021

To understand Big Data, you need to get acquainted with its attributes known as the four V’s: Volume is what hides in the “big” part of Big Data. Just so you could grasp the scale, 1 petabyte is equal to 1,000,000 gigabytes. Now imagine that 1 petabyte contains 250,000 movies. Traditional approach.

Big Data 78

Big Data Analytics Tools Applications 78

Bernd Rucker

JULY 18, 2019

part 2 In Zeebe.io?—?a I revealed that Zeebe plays in the same league as e.g. Apache Kafka. Event sourcing Zeebe is based on ideas from event sourcing. This means that all changes to the workflow state are captured as events and these events are stored in an event log alongside commands.

Scalability 42

Scalability Data Center Data Engineering 42

Confluent

OCTOBER 24, 2019

I’ve written an event sourcing bank simulation in Clojure (a lisp build for Java virtual machines or JVMs) called open-bank-mark , which you are welcome to read about in my previous blog post explaining the story behind this open source example. Either way, both are accomplished with event sourcing. The bank application.

Banking 18

Banking Software Review Testing Applications 18

OpenCredo

JULY 30, 2019

This did however serve as the inspiration that has resulted in me writing this blog – essentially providing a brief recap on the key components of Kafka Connect, and then providing a step by step guide how these relate to developing a Source Connector. Kafka Connect Summary. Kafka Connect was added in the Kafka 0.9.0

Software Review 15

Software Review Systems Review Technical Review System 15

Cloudera

JUNE 16, 2022

real-time data streams) and generate immediate insights for faster decision making provides a competitive edge for organizations. . Organizations are increasingly building low-latency, data-driven applications, automations, and intelligence from real-time data streams. Faster data ingestion: streaming ingestion pipelines.

Data 88

Data Disaster Recovery Analytics Microservices 88

Cloudera

NOVEMBER 27, 2023

Flink Forward is a great opportunity to learn about the cutting edge of streaming and stream processing technologies. This blog is a summary of what we observed there for anyone who was unable to attend or just wants to stay on top of what’s happening in streaming. billion events/s. Takeaway No. Takeaway No.

Open Source 81

Open Source Conference Storage Architecture 81