Apache Mesos

Steven RogerJune 27, 2023

0 190 2 minutes read

Mesos is used by the same principles as Linux kernel at different levels of abstraction. The Apache Mesos kernel will perform in each machine having applications with APIs for resource management and scheduling across all data center and cloud environments. Apache Mesos is a free source cluster manager and also has workloads in a shared environment on dynamic resource sharing and isolation. Mesos is used for deployment and management of applications in large-scale clustering environments.

Mesos together have existing resources of that machines/nodes in a cluster into a single pool from which of various workloads that utilize. Also known as node abstraction, this removes the need to allocate specific machines for different workloads. Companies such Twitter, Xogito utilize Apache Mesos. It was originally developed at the University of California at Berkeley, Mesos will sit between the operating system and application layer and act as a data center kernel.

Mesos will keep the idle processes running the cluster-like memory. CPU file system, rack locality and I/O to keep them from interfering with each other, this type of isolation will allow Mesos to create a single, very large pool of resources to offer workloads. Apache Mesos uses master agent architecture in conjunction with the frameworks to manage and isolate resource requests. The agent demons run on each other cluster node. The applications frameworks like apache spark, chromos, apache aurora and Mesosphere.

Spark will offer Mesos-powered and standalone cluster computing. Marathon and Aurora are geared towards long-running services and also specialize in scaling services. Chronos will be distributed. Without frameworks in place, specifically, workloads could consume all the pooled resources. Mesos utilize the Apache zookeeper part of Hadoop that combines distributed process to make sure all the clients receive consistent data and also assure fault tolerance. The framework has at least two components: a scheduler and an executor. Schedulers registered in Mesos master have resources and executors to launch the command or program that executes tasks on the slaves. This master may offer resources to all frameworks. The scheduler selects the available resources to use. After a framework that accepts the resources offered by the master.

Table of Contents

Apache Mesos architecture:

Mesos consists of the main components of the master daemon that manages agent daemons executing on each cluster and Mesos frameworks that perform tasks on these agents. The masters highlight a fine-grained sharing of resources across frameworks by making resource offers. Each resource offer contains a list of <agent ID, resource:amount1, resource2:amount2,….>. The master determines how many resources to provide to each framework according to a given organizational policy, such as fair sharing or strict priority.

To assist a diverse set of policies, the master employs a modular architecture that has made it very easy to include new allocation of modules. A framework that is executing on Mesos has two components: a scheduler that registers with the master to be offered resources and an executor process that is launched on agent nodes to run the framework tasks. The masters determine the resources that are offered to each framework. The framework’s schedulers select the resources to be used. Suppose the framework will support offered resources, it transforms to Mesos a description of the task it wants to run them.

Why do we need Apache Mesos?

There are many resource managers that exist today, like Hadoop on-demand,batch scheduler, and VM scheduler, but the problem with Hadoop is that the workload is data locality, it was compromised because of the static partitioning of nodes, and the job holds nodes full duration of time utilization of the system was affected. These drawbacks are resolved in Mesos with fine-grained sharing and two-level scheduling.