Software:Apache YuniKorn

Apache Yunikorn

Developer(s)	Apache Software Foundation
Repository	Yunikorn Repository
Written in	Go
Operating system	Cross-platform
License	Apache License 2.0
Website	yunikorn.apache.org

Apache YuniKorn is an open-source, lightweight resource scheduler that supports large-scale, multi-tenant, and cloud-native environments. Apache YuniKorn also adapting different ResourceManager YuniKorn's architecture design implementations like Apache Hadoop YARN  Its primary purpose was to replace Apache YARN. As for scheduling Apache Spark on Kubernetes and now it became the widely accepted method.

On a large scale, in multi-tenant settings on the one hand, and in dynamically created cloud-native environments on the other, YuniKorn facilitates efficient fine-grained resource sharing for different Spark workloads. By providing a solid foundation for a range of applications, including large-scale data transformation, analytics, and machine learning, YuniKorn enables Apache Spark to become an enterprise-grade fundamental platform for users.

History

In order to build a contemporary method for scheduling massive, distributed operations in a future where clouds predominate, Apache YuniKorn was initially created at Cloudera in 2019. The small group of individuals were searching for a truly solid solution to the scheduling and Kubernetes scheduling issue that they had seen. At that time, they also believed that Yarn and Kubernetes should be merged, therefore they decided on a single scheduling method. Apache incubation beginning on January 21, 2020, and the stable version 0.8.0 was published on May 4, 2020, nearly six months later.

Naming

The meaning of Yunikorn [Y YARN + K Kubernetes + Uni Unified] and the purpose of a scheduler support any container orchestration system both on-cloud and on-prem use cases.

Development

Apache YuniKorn is developed under the Apache License 2.0 by the Apache YuniKorn Community within the Apache Software Foundation. The project is driven by over 25 Apache committers

Overview

Apache YuniKorn is a resource scheduler that is designed for scheduling and managing larger dynamic applications on the container orchestration structure and became an Apache Top Level Project in March 2022.

The use of conventional scheduling techniques with big data tools has a number of shortcomings. Due to the absence of fine-grained control over autoscaling in the majority of resource schedulers, resource usage is often out of sync, autoscaling periods for both upscaling and downscaling are prolonged, and as a result, cloud costs are higher and throughput/performance are decreased.

Resource consumption is enhanced and autoscaling performance is boosted thanks to YuniKorn's Gang scheduling and bin-packing. YuniKorn simultaneously ran periodic Apache Spark tasks and saw almost a 2-fold increase in throughput, with a 2-fold decrease in average job runtime and a 3-fold decrease in scale-up and scale-down latencies for 200 nodes.

There are four components in YuniKorn:

• yunikorn-core
• yunikorn-web
• yunikorn-scheduler-interface
• yunikorn-k8shim

Releases

Version	Original release date	Latest version	Release date
0.8.0	2020-05-04	0.8.0	2020-05-04
0.9.0	2020-08-28	0.9.0	2020-08-28
0.10.0	2021-04-09	0.10.0	2021-04-09
0.11.0	2021-08-18	0.11.0	2021-08-18
0.12.1	2021-12-26	0.12.1	2021-12-26
0.12.2	2022-02-03	0.12.2	2022-02-03
1.0.0	2022-05-06	1.0.0	2022-05-06
1.1.0	2022-09-08	1.1.0	2022-09-08
{{{2}}}

External links

• Official website