Manara - Qatar Research Repository
10.1109_access.2021.3072609.pdf (1.26 MB)

Distributed Tree-Based Machine Learning for Short-Term Load Forecasting With Apache Spark

Download (1.26 MB)
journal contribution
submitted on 2023-08-29, 08:25 and posted on 2023-08-30, 12:17 authored by Ameema Zainab, Ali Ghrayeb, Haitham Abu-Rub, Shady S. Refaat, Othmane Bouhali

Machine learning algorithms have been intensively applied to perform load forecasting to obtain better accuracies as compared to traditional statistical methods. However, with the huge increase in data size, sophisticated models have to be created which require big data platforms. Optimal and effective use of the available computational resources can be attained by maximizing the effective utilization of the cluster nodes. Parallel computing is demanded to allow for optimal resource utilization in dealing with smart grid big data. In this paper, a master-slave parallel computing paradigm is utilized and experimented with for load forecasting in a multi-AMI environment. The paper proposes a concurrent job scheduling algorithm in a multi-energy data source environment using Apache Spark. An efficient resource utilization strategy is proposed for submitting multiple Spark jobs to reduce job completion time. The optimal value of clustering is used in this paper to cluster the data into groups to be able to reduce the computational time additionally. Multiple tree-based machine learning algorithms are tested with parallel computation to evaluate the performance with tunable parameters on a real-world dataset. One thousand distribution transformers' real data from Spain for three years are used to demonstrate the performance of the proposed methodology with a trade-off between accuracy and processing time.

Other Information

Published in: IEEE Access
See article on publisher's website:


Open Access funding provided by the Qatar National Library.



  • English



Publication Year

  • 2021

License statement

This Item is licensed under the Creative Commons Attribution 4.0 International License.

Institution affiliated with

  • Texas A&M University at Qatar
  • Hamad Bin Khalifa University
  • Qatar Computing Research Institute - HBKU