Disaster Recovery for Multi-Datacenter Apache Kafka Deployments
Confluent
NOVEMBER 29, 2022
Apache Kafka multi-datacenter with Confluent Replicator, Schema Registry, timestamp preservation, consumer offset reset, failover and failback workflows.
confluent-replicator 
Monitoring Data Replication in Multi-Datacenter Apache Kafka Deployments
Confluent
APRIL 10, 2019
Instead of having many point-to-point connections between sites, the Confluent Platform provides an integrated event streaming architecture with frictionless data replication between sites. The key to the solution is Confluent Replicator, which enables frictionless data replication between sites.
Data 
84
84 Join 48,000+
Insiders
Sign Up for our Newsletter
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Trending Sources
Dawn of DevOps: Managing Apache Kafka Clusters at Scale with Confluent Control Center
Confluent
MAY 2, 2019
To be fair, a lot of things turn into significant burdens at scale, and it’s Confluent Control Center’s job to ease as many of them as possible. In Confluent Platform 5.2, Confluent Platform 5.2 Those statistical variations notwithstanding, Control Center has leveled up significantly in Confluent Platform 5.2.
DevOps 87
87 
Kafka Migration and Lessons Learned
Honeycomb
APRIL 8, 2021
This led us toward the Tiered Storage feature in Confluent Platform 6.0, The new hosts were replicating slower than expected. Both profiles below are as likely, and we possibly hit each one of them in succession: Just replicating data out of the m6g.large instances was over-saturating the host. Operational costs.
Systems Review 144
144 
Dawn of Kafka DevOps: Managing Kafka Clusters at Scale with Confluent Control Center
Confluent
MAY 2, 2019
To be fair, a lot of things turn into significant burdens at scale, and it’s Confluent Control Center’s job to ease as many of them as possible. In Confluent Platform 5.2, Confluent Platform 5.2 Those statistical variations notwithstanding, Control Center has leveled up significantly in Confluent Platform 5.2.
DevOps 57
57 
Integrating SAP Datasphere and Databricks Lakehouse for Unified Analytics
Perficient
MARCH 29, 2023
And this is just the technical aspects of replicating data from point A to point B, skipping over governance, MDM, CDC and all the other details around duplicate data. Most of these intermediate steps required custom code. Meanwhile, your business sponsors are waiting not-so-patiently for any results.
Analytics 81
81 
Review of Industrial IoT Solutions – Part I
Perficient
DECEMBER 2, 2022
Kafka also supports replication between sites so it’s a great option to move data between sites and to the cloud. If you’re running your own MQTT broker, you can use MQTT replication to the cloud in some instances (Mosquitto Mirroring, HiveMQ Replication), but a better option is to use Kafka and leverage the mirroring feature.
IoT 69
69 
Introduction to Kafka and MongoDB Integration
Perficient
MARCH 6, 2023
They manage the data replication and provide fault-tolerance. Confluent MongoDB Connector for Spark — MongoDB Spark Connector Happy Learning!! Topics: Topics are the categories or channels that messages are published to. They can have one or many producers and one or many consumers.
Scalability 64
64 
Scaling Kafka at Honeycomb
Honeycomb
NOVEMBER 30, 2021
There are many Kafka implementations available for self-hosting, including the pure Apache-licensed Apache Kafka , Confluent’s Community Edition distribution, Cloudera’s DataFlow product, and even RedPanda , a scratch backend rewrite in C++ that is client API compatible. to Confluent Community 5.3
AWS 145
145 
Confluent Cloud Schema Registry is Now Generally Available
Confluent
AUGUST 27, 2019
We are excited to announce the release of Confluent Cloud Schema Registry in general availability (GA), available in Confluent Cloud , our fully managed event streaming service based on Apache Kafka ®. Before we dive into Confluent Cloud Schema Registry, let’s recap what Confluent Schema Registry is and does.
Cloud 18
18 
Building a Real-Time, Event-Driven Stock Platform at Euronext
Confluent
OCTOBER 8, 2019
As the head of global customer marketing at Confluent, I tell people I have the best job. That’s why Euronext turned to Confluent. Using Confluent Platform, Euronext easily leveraged the power of Apache Kafka ® to implement a reliable, scalable persistence layer for market orders that supports millisecond latencies.
How to Use Schema Registry and Avro in Spring Boot Applications
Confluent
SEPTEMBER 5, 2019
Following on from How to Work with Apache Kafka in Your Spring Boot Application , which shows how to get started with Spring Boot and Apache Kafka ® , here I will demonstrate how to enable usage of Confluent Schema Registry and Avro serialization format in your Spring Boot applications. Confluent Platform 5.3 Confluent Platform 5.3
Applications 20
20 
How to Deploy Confluent Platform on Pivotal Container Service (PKS) with Confluent Operator
Confluent
OCTOBER 4, 2019
This tutorial describes how to set up an Apache Kafka ® cluster on Enterprise Pivotal Container Service (Enterprise PKS) using Confluent Operator , which allows you to deploy and run Confluent Platform at scale on virtually any Kubernetes platform, including Pivotal Container Service (PKS). Download and expand Confluent Platform (.tar
How To 15
15 
KSQL: What’s New in 5.2
Confluent
APRIL 3, 2019
Go and download Confluent Platform 5.2 Robin Moffatt is a developer advocate at Confluent, as well as an Oracle Groundbreaker Ambassador and ACE Director (alumnus). Not forgetting to mention…. BETWEEN is now supported as a predicate operator ( GitHub issue #1004 ). What are you waiting for? today and see what KSQL can do!
15 Things Every Apache Kafka Engineer Should Know About Confluent Replicator
Confluent
MARCH 17, 2020
Single-cluster deployments of Apache Kafka® are rare. Most medium to large deployments employ more than one Kafka cluster, and even the smallest use cases include development, testing, and production clusters. […].
Engineering 122
122 
CloudBank’s Journey from Mainframe to Streaming with Confluent Cloud
Confluent
MARCH 4, 2019
This case study is based off real, concrete war stories, and will detail how Confluent Cloud was a key enabler for their digital strategy, unfolding new ways to capitalize on their product and differentiate their business, by leveraging the power of Apache Kafka and the cloud at the same time. Perfect balance with Confluent Cloud.
Cloud 87
87 
SAP rounds out data warehouse cloud functionality, renamed Datasphere
CIO
MARCH 8, 2023
The new functions Datasphere offers over Data Warehouse Cloud include automated data cataloging, simplified data replication, and improved data modeling. Confluent will connect its cloud-native data streaming platform to Datasphere, making it possible to connect SAP and external applications in real time.
Cloud 130
130 
Built-In Multi-Region Replication with Confluent Platform 5.4-preview
Confluent
SEPTEMBER 16, 2019
This changes with the preview release of Confluent Platform 5.4, which includes multi-region replication built directly into Confluent Server. Now operators can choose to replicate data on a per-region basis, synchronously or asynchronously, per topic. Observers in Confluent Platform are effectively asynchronous replicas.
17 Ways to Mess Up Self-Managed Schema Registry
Confluent
MAY 28, 2019
In particular, using Confluent Schema Registry makes this really easy for developers to use schemas, and it is designed to be highly available. Confluent Replicator then copies the Kafka schemas topic from the primary cluster to the other cluster for backup. Since Confluent Platform 4.0, Other articles in this series.
Zeebe.io?—?a horizontally scalable distributed workflow engine
Bernd Rucker
JULY 17, 2019
Zeebe uses event sourcing and event streaming concepts as well as replicated append-only log. from Cockroach Labs , Confluent , MongoDB , Redis , Timescale ). Partitioning allows for scaling out. It is designed as reactive system according to the reactive manifesto. As a result, Zeebe is in the same class of systems like Apache Kafka.
Scalability 40
40 
From Apache Kafka to Amazon S3: Exactly Once
Confluent
APRIL 11, 2019
At Confluent, we see many of our customers are on AWS, and we’ve noticed that Amazon S3 plays a particularly significant role in AWS-based architectures. In March 2017, we released the Kafka Connect S3 connector as part of the Confluent Platform. Begin by starting ZooKeeper, Kafka and Connect: confluent start connect.
AWS 110
110 
The Good and the Bad of Apache Kafka Streaming Platform
Altexsoft
OCTOBER 21, 2022
It achieves high availability and fault tolerance through replication at the partition level. You may join a Kafka community on Reddit or a forum of Kafka experts on the Confluent website. Confluent , founded by Kafka’s authors. process data in real time and run streaming analytics. High availability and fault tolerance.
Creating an IoT-Based, Data-Driven Food Value Chain with Confluent Cloud
Confluent
APRIL 25, 2019
By capturing Internet of Things (IoT) event data from farm to fork with Apache Kafka ® and Confluent Cloud, BAADER is increasing its value as part of this chain, creating new business opportunities and enabling its partners to optimize their operations. “By “With Confluent Cloud, we get more than just a Kafka service.
IoT 78
78 
Introducing Confluent Platform 5.4
Confluent
JANUARY 22, 2020
I am pleased to announce the release of Confluent Platform 5.4. Like any new release of Confluent Platform, it’s packed with features. To make them easier to digest, I want […].
Storage 116
116 Storage 116
116 Scylla and Confluent Integration for IoT Deployments
Confluent
MAY 22, 2019
Kafka is enterprise ready, and has powerful features like high availability (HA) and replication on commodity hardware. Demo of Scylla and Confluent integration. To demonstrate this, we are using Confluent MQTT Proxy (part of the Confluent Enterprise package), which acts as a broker for all the sensors that are emitting the readings.
IoT 96
96 
Project Metamorphosis Month 5: Global Event Streaming in Confluent Cloud
Confluent
AUGUST 24, 2020
This is the fifth month of Project Metamorphosis: an initiative that addresses the manual toil of running Apache Kafka® by bringing the best characteristics of modern cloud-native data systems to […].
Cloud 90
90 
Enabling the Deployment of Event-Driven Architectures Everywhere Using Microsoft Azure and Confluent Cloud
Confluent
SEPTEMBER 3, 2020
Hybrid cloud architecture and accelerated cloud migrations are becoming the norm rather than the exception, as our increasingly digital world introduces certain challenges along the way, including modernizing existing application/architecture, […].
Architecture 90
90 
Conquering Hybrid Cloud with Replicated Event-Driven Architectures
Confluent
NOVEMBER 25, 2019
Potential advantages of hybrid cloud architectures include avoiding vendor lock-in, increasing system resilience, optimizing costs, and inducing price competition among cloud providers. Hybrid cloud architectures require the ability to securely […].
Architecture 97
97 
Automatic Observer Promotion Brings Fast and Safe Multi-Datacenter Failover with Confluent Platform 6.1
Confluent
FEBRUARY 11, 2021
Persisting data in multiple regions has become crucial for modern businesses: They need their mission-critical data to be protected from accidents and disasters. They can achieve this goal by running […].
Data 57
57 Data 57
57 
How Bolt Has Adopted Change Data Capture with Confluent Platform
Confluent
AUGUST 20, 2020
This article describes why Bolt, the leading European on-demand transportation platform that operates across ride-hailing, food delivery, and e-mobility sectors, has journeyed toward adopting change data capture (CDC). Adopting CDC […].
Data 64
64 
Introducing Confluent Platform 5.3
Confluent
JULY 31, 2019
Delivers the new Confluent Operator for cloud-native automation on Kubernetes, a redesigned Confluent Control Center user interface to simplify how you manage event streams, and a preview of Role-Based Access Control for enterprise-grade security. To solve these challenges, Confluent Platform 5.3 Secure the platform end to end.
Open Source 19
19 Testing Event-Driven Systems
Confluent
APRIL 24, 2019
The interface is implemented using Clojure but thanks to the Confluent ecosystem, the system under test can be anything from an unholy assortment of Rails applications, Kafka Connect jobs and “untestable” PL/Perl triggers, to the latest, greatest, highly replicated, fault-tolerant, Kafka-Streams-based self-distributing API.
Free Apache Kafka as a Service with Confluent Cloud
Confluent
SEPTEMBER 30, 2019
There is fantastic news if you’re just coming up to speed on Kafka: we are eliminating these challenges and lowering the entry barrier for Kafka by making Kafka serverless and offering Confluent Cloud for free*. Sign up on the Confluent Cloud landing page, and we’ll give you up to $50 USD off each month for the first three months.
Serverless 19
19 
The Rise of Managed Services for Apache Kafka
Confluent
SEPTEMBER 20, 2019
Luckily for on-premises scenarios, a myriad of deployment options are available, such as the Confluent Platform which can be deployed on bare metal, virtual machines, containers, etc. Before Confluent Cloud was announced , a managed service for Apache Kafka did not exist. But deployment is just the tip of the iceberg.
DBLog: A Generic Change-Data-Capture Framework
Netflix Tech
DECEMBER 17, 2019
For example by using MySQL’s binlog replication protocol, or PostgreSQL’s replication slots. For MySQL, we use shyiko/mysql-binlog-connector which implementing the binlog replication protocol in order to receive events from a MySQL host. For PostgreSQL, we are using replication slots with the wal2json plugin.
Data 84
84 
DBLog: A Generic Change-Data-Capture Framework
Netflix Tech
DECEMBER 17, 2019
For example by using MySQL’s binlog replication protocol, or PostgreSQL’s replication slots. We found that some solutions use advanced database features that are not transferable to other systems, such as: using MySQL’s blackhole engine or getting a consistent snapshot for dumps from the creation of a PostgreSQL replication slot.
Data 79
79 
Schema Validation with Confluent 5.4-preview
Confluent
SEPTEMBER 27, 2019
Robust data governance support through Schema Validation on write is now supported in Confluent Platform 5.4-preview. This gives operators a centralized location to enforce data format correctness within Confluent Platform. To allow Schema Validation on write, Confluent Server must be schema aware. Get started in five minutes.
Government 16
16 
Apache Kafka Data Access Semantics: Consumers and Membership
Confluent
MAY 7, 2019
The broker then waits until that specific __consumer_offsets topic’s partition data gets replicated to all its followers. In case the replication fails within the offsets.commit.timeout.ms Confluent Schema Registry: Electing a Schema Registry master, in case of a multi-node Schema Registry deployment. Managing membership.
Data 111
111 
Introducing Confluent Platform 5.2
Confluent
APRIL 2, 2019
Includes free forever Confluent Platform on a single Apache Kafka ® broker, improved Control Center functionality at scale and hybrid cloud streaming. We are very excited to announce the general availability of Confluent Platform 5.2, Confluent Platform 5.2 At Confluent, we believe in the power of free and open software.
Cloud 21
21 
Why AWS PrivateLink Is Not Recommended With Kafka or Cassandra
Instaclustr
JULY 28, 2021
Kafka uses partitions that can have copies (dependent on replication factor) to increase durability and availability, and allow Kafka to failover to a broker with a replica of the partition if that broker fails. While it has been found that it’s possible to connect Kafka and PrivateLink through Confluent, this comes with a lot of limitations.
AWS 68
68 
Streaming JSON Data Into PostgreSQL Using Open Source Kafka Sink Connectors (Pipeline Series Part 6)
Instaclustr
JULY 22, 2021
Before we can experiment with streaming data out of Kafka into PostgreSQL, we need to replicate the mechanism we used in the earlier blogs to get the NOAA tidal data into it, using a Kafka REST source connector as described in section 5 of this blog. Source: Shutterstock). License Apache Camel JDBC Kafka Sink Connector Sink only available.
Open Source 92
92 KSQL UDFs and UDAFs Made Easy
Confluent
AUGUST 6, 2019
Now with Confluent Platform 5.3.0, Confluent Platform 5.3.0 Next, add the Maven repositories from Confluent to your ~/.m2/settings.xml Confluent releases --> <repository> <id>confluent</id> <url>[link] </repository> <!-- replication-factor 1. topic api_logs.
Journey to Event Driven – Part 2: Programming Models for the Event-Driven Architecture
Confluent
FEBRUARY 13, 2019
I’m limited to space so will briefly mention Confluent Schema Registry , data evolution (schema evolution), change data capture (data virtualization), event collaboration and coordination primitives , streaming patterns, evolutionary architectures, disposable architectures and event tracing (correlation IDs). Other articles in this series.
Architecture 89
89 Incremental Cooperative Rebalancing in Apache Kafka: Why Stop the World When You Can Change It?
Confluent
SEPTEMBER 24, 2019
and Confluent Platform 5.3 To reflect a more realistic scenario, data records were consumed from a Kafka cluster in Confluent Cloud, which was located in the same region as the Kafka Connect cluster. Konstantine Karantasis is a software engineer at Confluent. Implementation in Kafka Connect – Connect tasks are the new threads.
Let's personalize your content