OpenCredo

NOVEMBER 22, 2023

Here at OpenCredo we love projects that are based around Kafka and/or Data/Platform Engineering; in one of our recent projects, we created an open data lake using Kafka, Flink, Nessie and Iceberg. The first part of this blog is related to the Flink and S3 infra design. This makes it considerably easier to manage jobs and tasks.

Data 59

Data AWS Policies Compliance 59

Cloudera

JULY 15, 2021

This blog post provides an overview of best practice for the design and deployment of clusters incorporating hardware and operating system configuration, along with guidance for networking and security as well as integration with existing enterprise infrastructure. Traditional data clusters for workloads not ready for cloud.

Architecture 97

Architecture Cloud Data Technical Advisors 97

Cloudera

AUGUST 10, 2021

x customers to upgrade in-place to CDP PvC Base without having to create a new cluster. Cloudera Manager enhancements for greater efficiency and hardened platform security. We expand on this feature later in this blog. With the release of CDP Private Cloud (PvC) Base 7.1.7, please review the Release Summary.

Cloud 97

Cloud Policies Systems Review Groups 97

Cloudera

JULY 21, 2022

We want Cloudera customers that rely on Apache Druid to know that their clusters are secure and supported by the Cloudera partner ecosystem. Rill relieves that burden with a managed service offering or Druid monitoring for existing clusters. The Druid native Kafka indexing service features: Pull-based ingestion.

Metrics 84

Metrics Data Technical Review Analytics 84

OpenCredo

JUNE 29, 2023

Recently one of our fintech customers approached us with the interesting task of designing and delivering a Kafka Backup solution for their AWS managed Kafka ( MSK ) cluster. We do not want to add or change anything on the source MSK cluster. Let’s take a look first at the presented functional requirements.

Backup 52

Backup AWS Storage Testing 52

Instaclustr

AUGUST 6, 2021

Why Use the Kafka REST Proxy. Apache Kafka is known best as a powerful, open source message streaming and queueing solution. Used across more than 80% of the Fortune 500, Kafka is ubiquitous in supporting event-driven architectures. Read Now: Apache Kafka Architecture – The complete guide.

Architecture 65

Architecture Software Review Open Source Performance 65

Cloudera

JANUARY 26, 2021

Has your organization considered upgrading from Hortonworks Data Flow (HDF) to Cloudera Flow Management (CFM) , but thought the migration process would be too disruptive to your mission critical dataflows? Here we explore three common use cases where a CFM cluster can assume an HDF cluster’s dataflows with minimal to no downtime.

Load Balancer 76

Load Balancer Groups Examples Hardware 76

Confluent

JUNE 12, 2019

Confluent Cloud, a fully managed event cloud-native streaming service that extends the value of Apache Kafka ® , is simple, resilient, secure, and performant, allowing you to focus on what is important—building contextual event-driven applications, not infrastructure. Next, click on the cluster name whose configuration you want.

Cloud 93

Cloud How To Examples Infrastructure 93

Cloudera

MARCH 22, 2021

These workloads cover the entire data lifecycle and are managed from a central multi-cloud Cloudera Control Plane. It is the most secure deployment option, but this prevents direct access to their resources from the public internet and makes it difficult for their users to access the UIs and APIs in SDX and DataHub clusters.

Network 98

Network Cloud Load Balancer Internet 98

Netflix Tech

JANUARY 13, 2022

This blog will explore these two systems and how they perform auto-diagnosis and remediation across our Big Data Platform and Real-time infrastructure. These steps launch jobs on clusters running Apache Spark and Presto via Genie. And we can’t discount the productivity impact it causes on data platform users.

Data 95

Data Systems Review Big Data Software Review 95

Confluent

APRIL 10, 2019

Applications can publish streams of data to a self-hosted on-prem cluster, replicate them to another on-prem cluster or to different cloud providers, load them into data systems in the cloud and trigger cloud-native applications from those events. It’s not just a data swamp of hundreds of metrics—it is information you can act on.

Data 86

Data Disaster Recovery Metrics Applications 86

Cloudera

JUNE 15, 2021

By automating cluster deployment this way, you reduce the risk of misconfiguration, promote consistent deployments across multiple clusters in your environment, and help to deliver business value more quickly. . This blog will walk through how to deploy a Private Cloud Base cluster, with security, with a minimum of human interaction.

Cloud 82

Cloud AWS Azure Linux 82

Cloudera

DECEMBER 10, 2020

In the previous post, we talked about Kerberos authentication and explained how to configure a Kafka client to authenticate using Kerberos credentials. In this post we will look into how to configure a Kafka client to authenticate using LDAP, instead of Kerberos. We use the Kafka-console-consumer for all the examples below.

Authentication 52

Authentication How To Examples Network 52

Cloudera

MARCH 5, 2024

This marks a significant milestone for the platform: according to IDC, today about half of the world’s enterprise production data under management is on-prem. The platform is ready to address the complexities of managing highly sensitive, yet critical, company data while still extracting the most value from its use.

Data 87

Data Data Center Analytics Artificial Intelligence 87

Instaclustr

OCTOBER 18, 2021

In Part 6 and Part 7 of the pipeline series we took a different path in the pipe/tunnel and explored PostgreSQL and Apache Superset, mainly from a functional perspective—how can you get JSON data into PostgreSQL from Kafka Connect, and what does it look like in Superset. Comparing Elasticsearch and PostgreSQL Replication Settings.

Performance 84

Performance Comparison Systems Review Metrics 84

Cloudera

AUGUST 24, 2021

As a Software Engineer at Cloudera, Barnabas gets to experience rewarding work with emerging technologies like Apache Kafka. As a Software Engineer at Cloudera, Barnabas gets to experience rewarding work with emerging technologies like Apache Kafka. As he sees it, “Kafka is a famous and widely used project.

Software Engineering 96

Software Engineering Engineering Software Open Source 96

Cloudera

SEPTEMBER 30, 2021

Our recent blog discussed the four paths to get from legacy platforms to CDP Private Cloud Base. In this blog and accompanying video, we will deep dive into the mechanics of running an in-place upgrade from CDH5 or CDH6 to CDP Private Cloud Base. Finally we also recommend that you take a full backup of your cluster, including: RDBMS.

Cloud 69

Cloud Systems Review Backup Software Review 69

OpenCredo

FEBRUARY 20, 2019

Common tasks include issuing client and server certificates to trusted parties, managing certification renewal, distributing Certificate Authority (CA) trust chains and publishing certificate revocation lists ( CRL s). All these tasks require expertise and tools to make a PKI manageable and effective. The target architecture.

Authentication 73

Authentication Policies Tools Systems Review 73

Honeycomb

APRIL 1, 2021

A quick refresh: Honeycomb consists of our ingest workers [called “shepherd”], Kafka, query workers [“retriever”], and frontends [“poodle”]. We also run a “dogfood” copy of these services for analyzing telemetry from the production cluster.) . Graviton2 is a large part of how we do it. xlarge instances to m6gd.2xlarge

Architecture 134

Architecture Testing Continuous Integration Performance 134

Cloudera

SEPTEMBER 9, 2020

This blog post will present a simple “hello world” kind of example on how to get data that is stored in S3 indexed and served by an Apache Solr service hosted in a Data Discovery and Exploration cluster in CDP. We will only cover AWS and S3 environments in this blog. We will only cover AWS and S3 environments in this blog.

Data 81

Data How To AWS Resources 81

Cloudera

JANUARY 19, 2022

Run on your desktop or cloud node, connecting to Kafka or other sources/sinks via API calls to their respective clusters. Save your SQL, UDF’s, etc into files (perhaps in a source code repository) and run/manage it via REST on production versions of CSA (again via API calls). We plan future blog posts on this workflow.

Analytics 86

Analytics Cloud Architecture Testing 86

Mobilunity

APRIL 29, 2023

From this article, you’ll learn which solutions use Netflix and Disney and watch the AWS Kinesis vs Kafka “battle.“ However, the list of top data streaming solutions is much shorter: Kafka and Kinesis. For instance, Netflix uses Apache Kafka , and Disney+ uses Kinesis. But which service wins the AWS Kafka vs Kinesis battle?

AWS 52

AWS Technical Review Systems Review Software Review 52

Cloudera

JUNE 28, 2022

In a previous blog of this series, Turning Streams Into Data Products , we talked about the increased need for reducing the latency between data generation/ingestion and producing analytical results and insights from this data. This blog will be published in two parts. This is what we call the first-mile problem. The use case.

Analytics 81

Analytics Machine Learning Artificial Inteligence Cloud 81

InnovationM

MAY 14, 2023

Platforms like Hadoop Distributed File System (HDFS) or cloud-based storage solutions such as Amazon S3 and Azure Data Lake Storage offer fault-tolerant and scalable storage capabilities across clusters of machines. These tools streamline data flow, enable real-time data ingestion, and ensure data quality and metadata management.

Data 52

Data Storage Construction Business Intelligence 52

Cloudera

AUGUST 10, 2021

This article assumes that you have a CDP Private Cloud Base cluster 7.1.5 As businesses look to scale-out storage, they need a storage layer that is performant, reliable and scalable. CDP integrates its existing Shared Data Experience (SDX) with Ozone for an easy transition, so you can begin utilizing object storage on-prem. With CDP 7.1.4

Storage 103

Storage Policies Architecture Data 103

Instaclustr

SEPTEMBER 15, 2021

With the release of AWS’s Graviton2 (ARM) instances and gp3 disks, we at Instaclustr saw an opportunity to further improve the quality of our Managed Apache Kafka offering. All of our previous offerings have used Java 11 (OpenJDK) to run the Kafka service on x86 instances. Graviton2/gp3 vs. x86/gp2 Benchmarking.

AWS 41

AWS Performance Architecture Network 41

Cloudera

FEBRUARY 9, 2021

This ‘need for speed’ drives a rethink on building a more modern data warehouse solution, one that balances speed with platform cost management, performance, and reliability. Today, more and more customers are moving workloads to the public cloud for business agility where cost-saving and management are key considerations.

Cloud 67

Cloud Data Storage Analysis 67

Netflix Tech

OCTOBER 15, 2019

Part I: Overview Andreas Andreakis , Falguni Jhaveri , Ioannis Papapanagiotou , Mark Cho , Poorna Reddy , Tongliang Liu Overview It is a commonly observed pattern for applications to utilize multiple datastores where each is used to serve a specific need such as storing the canonical form of data (MySQL etc.), caching (Memcached etc.),

Data 77

Data Microservices Transportation Systems Review 77

Netflix Tech

AUGUST 30, 2021

The Partner Infrastructure team at Netflix provides solutions to support these two significant efforts by enabling device management at scale. Together, they form the Device Management Platform, which is the infrastructural foundation for Netflix Test Studio (NTS). In this blog post, we will focus on the latter feature set.

Software Review 85

Software Review Systems Review Windows Testing 85

Cloudera

NOVEMBER 16, 2021

With the general availability of Cloudera DataFlow for the Public Cloud (CDF-PC) , our customers can now self-serve deployments of Apache NiFi data flows on Kubernetes clusters in a cost effective way providing auto scaling, resource isolation and monitoring with KPI-based alerting. Functions as a Service. Event driven use cases.

Serverless 105

Serverless Lambda Microservices KPI 105

Instaclustr

MARCH 12, 2019

In the previous blog , we deployed the Anomalia Machina application in a production-like Kubernetes environment. In this blog, we test it out and see many anomalies it can detect at scale on small Kafka and Cassandra Instaclustr production clusters. Kubernetes Resource Management (Replicas, Pods, Nodes).

Scalability 31

Scalability Metrics Resources AWS 31

Datavail

JANUARY 5, 2023

AWS had a lot to say about its analytics and data management capabilities this year at re:Invent 2022. Real-time streaming ingestion for Amazon Kinesis Data Streams and Amazon Managed Streaming for Apache Kafka is now generally available for Redshift. We have the highlights from the event in this handy summary. AWS Glue 4.0

AWS 52

AWS Analytics Disaster Recovery Serverless 52

Instaclustr

JUNE 4, 2019

At Instaclustr, we constantly push ourselves to test the limits of the performance and scalability of the open source technologies our Managed Platform offers. Million events per dollar spent (considering cost for infrastructure, managed service, anomaly detection application R&D and ongoing maintenance). That’s enough of bragging!

Scalability 29

Scalability Open Source AWS Applications 29

Cloudera

APRIL 20, 2021

IT teams tried solving the problem by adding more clusters but noticed the rising cost for infrastructure and struggled to hire the right talent to manage them. Streaming Analytics is a type of data analysis that processes data streams for real-time analytics. The data is often structured and doesn’t vary much.

Analytics 93

Analytics Retail Architecture Government 93

Cloudera

OCTOBER 28, 2020

Cloudera services logs offer a breadth of information to assist in cluster maintenance; from assisting in security checks, auditing tasks, and validation for performance tuning and testing tasks – to name a few. . This is where Cloudera Flow Management (CFM) can be utilised. x cluster that has both Kerberos and TLS enabled.

Examples 64

Examples Groups Performance Applications 64

Linux Academy

JUNE 28, 2019

Apache Kafka Deep Dive. Course Description: Apache Kafka is a publish/subscribe messaging system with many advanced configurations. You’ll be able to build your own Kafka cluster using Linux Academy servers, which will help your understanding and provide you with real-world examples. Difficulty Level: Intermediate.

Course 60

Course DevOps Training Linux 60

Confluent

FEBRUARY 6, 2019

The blog posts How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka and Using Apache Kafka to Drive Cutting-Edge Machine Learning describe the benefits of leveraging the Apache Kafka ® ecosystem as a central, scalable and mission-critical nervous system.

Machine Learning 107

Machine Learning Artificial Inteligence Scalability Data Engineering 107