Aprimorando o desempenho da rede através da agregação de links em um ambiente de cluster Spark

Abstract

Communication among nodes in distributed processing clusters can become a significant bottleneck when executing large-scale workloads. This study investigates the application of the Multipath TCP (MPTCP) protocol to optimize communication in Apache Spark clusters, leveraging multiple network paths to increase data transfer rates and reduce latency. Furthermore, the impact of MPTCP has been analyzed on the execution of the FP-Growth algorithm for frequent pattern mining in large datasets. The proposed approach involves comparative experiments between conventional TCP and MPTCP, evaluating metrics such as data transfer rate, latency, and execution time. The results demonstrate that integrating MPTCP into Apache Spark can significantly enhance communication efficiency between cluster nodes, reducing bottlenecks and optimizing distributed processing performance.