Hybrid Workflow Provisioning and Scheduling on Edge Cloud Computing Using a Gradient Descent Search Approach
Raed Alsurdeh, Bahman Javadi, Rodrigo N. Calheiros and Kenan M. Matawie
School of Computer, Data, and Mathematical Sciences
Western Sydney University
fr.alsurdeh, r.calheiros, k.matawie, b.javadig @ westernsydney edu au
Abstract: The dramatic growth of the Internet of Things (IoT) technology in many application domains, ranging from intelligent video surveillance, smart retail to the Internet-of-Vehicles brings new computation challenges for rationalized utilization of computing resources. IoT application models formulate the integration between stream and batch processing to achieve data analytics objectives. The integration refers to the hybrid workflow model. The main challenge of the hybrid model is achieving the quality of service requirements of the two computation models. In one hand, stream processing is latency-sensitive and subjects to constraints like stream rate and throughput, while batch processing is resource-intensive. In this work, we propose a two-stage hybrid workflow scheduling framework on edge cloud computing. In the first stage, we propose a resource estimation algorithm based on a linear optimization approach, the gradient descent search (GDS), and in the second stage, we propose a cluster-based provisioning and scheduling technique for hybrid workflows on heterogeneous edge cloud resources. This work provides a multi-objective optimization model for execution time and monetary cost under constraints of deadline and throughput. Results demonstrate the framework performance in controlling the execution of hybrid workflows by an efficient tuning for stream processing parameters, such as arrival rate and processing throughput. Under working constraints, the proposed scheduler provides significant improvement for large hybrid workflows in terms of execution time and monetary cost with an average of 8% and 35%, respectively.
Keywords: Hybrid workflows, stream processing, workflow scheduling, resource provisioning