Evaluation methodology for the CACTOS runtime and prediction toolkits: project deliverable D5.4

Abstract

Infrastructure as a Service (IaaS) cloud data centres enable customers to run arbitrary software systems on virtualised infrastructure. In contrast to Software or Platform as a Service approaches, customers do not need to adapt the design of their applications to be cloud-compatible. At the same time, they can benefit from easy scalability and pay-as-you-go models. Customers do not pay for dedicated physical machines. Rather, they are able to request Virtual Machines (VM) with varying characteristics, such as processing speed or memory size. Data centre providers can assign the VMs of multiple customers within their data centre to physical machines. If the VMs are deployed in a manner where the Quality of Service (QoS) of all customers is upheld, the data centre provider benefits from drastically larger economy of scale when compared to traditional one-customer-per-server hosting. The efficient utilisation of the underlying physical infrastructure including management and topology optimisation determines the costs and ultimately the business success for data centre operators. The CACTOS project develops an integrated solution for runtime monitoring, optimisation and prediction. The solution supports data centre providers in data centre management and planning. CACTOS consists of two toolkits: • The CACTOS Runtime Toolkit facilitates automated resource scheduling and optimisation for IaaS data centres. • The CACTOS Prediction Toolkit enables what-if analyses including effects caused by automated resource optimisation based on existing or planned data centre topologies. The CACTOS Runtime Toolkit collects data on a distributed data centre as input to scheduling and optimisation algorithms. Up-to-date load and topology measurements are essential for runtime monitoring, data collection and optimisation. The monitoring and data collection infrastructure introduces unavoidable load in the data centre. The benefit gained by using an automated monitoring and optimisation framework such as the CACTOS Runtime Toolkit strongly depends on the amount of this additional load. The CACTOS Prediction Toolkit requires resources to simulate the behaviour of a data centre. The size and complexity of the simulated data centre influences the feasibility of such a simulative analysis. If the simulative analysis takes a brief amount of time, the data centre planner can quickly account for the results of the simulation and adjust his plans accordingly. This document presents an evaluation methodology for the CACTOS Toolkits as established in (D5.1 Model Integration Method and Supporting Tooling) and (D5.2.1 CACTOS Toolkit Version 1). The evaluation focuses on performance and scalability of the CACTOS Runtime Toolkits. The evaluation approach is driven by the use-case specific requirements for the scientific computing use case of the University of Ulm (c.f. (D7.3.1 Validation Goals and Metrics), (D7.4.1 Validation and Result Analysis)) and Flexiant’s business analytics IaaS hosting use case. For an overview of the use cases, please refer to (D7.1 Scenario Requirements on Context-Aware Topology Optimisation and Simulation) and (D7.4.1 Validation and Result Analysis). The application of the evaluation methodology presented in this document will be outlined in (D5.5 Performance Evaluation of the CACTOS Toolkit on a Small Cloud Testbed). The use case brought into the project by PlayGen will be included in this evaluation. This document closely relates to the documents (D7.3.1 Validation Goals and Metrics) and (D7.4.1 Validation and Result Analysis). These two documents outline goals and results of a practical validation of the CACTOS Runtime Toolkit against the specific goals of each use case. Their focus is on an evaluation in small-scale testbeds and on use-case specific benefit analyses. This document outlines an evaluation methodology that is concerned with the applicability of CACTOS to different testbeds with respect to the performance of the CACTOS tools.