Cloudera

MAY 5, 2021

Apache YuniKorn (Incubating) has just released 0.10.0 ( release announcement ). As part of this release, a new feature called Gang Scheduling has become available. By leveraging the Gang Scheduling feature, Spark jobs scheduling on Kubernetes becomes more efficient. What is Gang Scheduling?

Resources 135

Resources Groups Policies Applications 135

Cloudera

SEPTEMBER 2, 2021

That’s why turning to traditional resource scheduling is not sufficient. That’s why turning to traditional resource scheduling is not sufficient. When building CDE, we integrated with Apache YuniKorn which offers rich scheduling capabilities on Kubernetes. . fixed sized clusters).

Data Engineering 81

Data Engineering Performance Engineering Data 81

Cloudera

DECEMBER 21, 2021

Since the release of Cloudera Data Engineering (CDE) more than a year ago , our number one goal was operationalizing Spark pipelines at scale with first class tooling designed to streamline automation and observability. We also introduced Apache Airflow on Kubernetes as the next generation orchestration service.

Data Engineering 115

Data Engineering Technical Review Software Review Engineering 115

Cloudera

OCTOBER 14, 2020

Why choose K8s for Apache Spark. Apache Spark unifies batch processing, real-time processing, stream analytics, machine learning, and interactive query in one-platform. Support for multiple Spark versions, Python versions, and version-controlled containers on the shared K8s clusters for both faster iteration and stable production.

Policies 112

Policies Resources Systems Review Technical Review 112