DisResc: self-organized and flexible distributed resource cluster

Abstract

The idea of DisResc is to place workloads with different requirements on heterogeneous resources, while both the requirements and the resources are considered. The overall goals are to better utilise (heterogeneous) data centres, provide better user experiences by selecting the best matching hardware for a workload, and to allow different workload types like virtual machines, containers or HPC jobs side by side in one data centre. DisResc is the vision to build a cluster management software for cloud and hpc workloads, running in parallel on a heterogeneous physical data centre. DisResc enabled compute nodes should i) run virtual machines, containers or HPC jobs, ii) compile time-based behaviour profiles of their workloads, in order to iii) detect suboptimal situations like over/under utilization. The resource utilisation should consider multiple metrics like processor, memory, disk, and network utilisation. The cluster should communicate in a peer-based manner to agree on the best fitting placement of new workloads, and workloads of nodes in a suboptimal state. This non-centralised approach can remove single points for management and monitoring, which are usually found in Cloud and HPC clusters. Beside the distributed and stateful compute nodes, stateless gateway nodes allow user interactions with the cluster. The two main challenges and open questions of DisResc are first the compilation of a time-based behaviour profile out of monitoring data, and second to define a distributed consensus algorithm for determining a best fitting node for a behaviour profile.