MARS: Memory-Aware Metadata Management for Scalable Hadoop Systems

NONE 20260210124059831 wseas/wseas content-registration-form+ja@crossref.org content-registration-form WSEAS TRANSACTIONS ON COMPUTER RESEARCH 1991-8755 2415-1521 MARS: Memory-Aware Metadata Management for Scalable Hadoop Systems Youngmi Baek School of Smart Factory, Changshin University 262, Palyongro, Changwonsi, Gyeongsangsam-do, 51352 REPUBLIC OF KOREA Jung Kyu Park Department of Computer Engineering Daejin University 1007, Hoguk-ro, Pocheon-si, Gyeonggi-do, 11159 REPUBLIC OF KOREA Effective management of metadata is emerging as an increasingly important task in large-scale distributed storage systems. In particular, in the Hadoop HDFS environment, due to the structural characteristics of maintaining all file system metadata in a single NameNode memory, memory overload and performance degradation problems occur as the data scale increases. This study proposes a new metadata management framework called Memory-Aware Routing and Scaling (MARS) to address these limitations. MARS is designed to analyze the access frequency of metadata and the system memory state in real time, keep high-priority items in main memory, and distribute and store low-priority items in auxiliary storage. The proposed system adopts dynamic routing algorithms and priority-based distributed storage structures without relying on fixed cache size or static thresholds to maximize memory utilization and enable stable processing without performance degradation. As a result of the experiment, MARS reduced memory usage by about 42% compared to the existing HDFS. In addition, it was confirmed that the delay time of metadata query was improved by 35%, and the metadata loss rate in the context of memory shortage was also reduced by more than 60%. These results show that MARS can effectively enhance scalability and fault response capabilities without significantly changing the existing HDFS structure. This study presents a practical research direction for adaptive memory-based storage structures in an environment where data-driven applications are increasing. 02 10 2026 02 10 2026 208 18 https://creativecommons.org/licenses/by/4.0/deed.en_US 10.37394/232018.2026.14.18 https://wseas.com/journals/cr/2026/a365118-429.pdf A. Ahmed, R. Xi, M. Hou, S. A. Shah and S. Hameed, Harnessing Big Data Analytics for Healthcare: A Comprehensive Review of Frameworks, Implications, Applications, and Impacts, IEEE Access, Vol.11, 2023, pp. 112891–112928, doi: 10.1109/ACCESS.2023.3323574. M. S. Mahmud, J. Z. Huang, R. Ruby, A. Ngueilbaye and K. Wu, Approximate Clustering Ensemble Method for Big Data, IEEE Transactions on Big Data, Vol.9, No.4, 2023, pp. 1142-01155, doi: 10.1109/TBDATA.2023.3255003. J. K. Park, Y. Seo and J. Kim, A Flash-Based SSD Cache Management Scheme for High Performance Home Cloud Storage, IEEE Transactions on Consumer Electronics, Vol.65, No.3, 2019, pp. 418–425, doi: 10.1109/TCE.2019.2897686. H. Lin, J. Luo, J. Lim, Z. Sha, Z. Cai, and Y. Shi, Fast Online Reconstruction for SSD-Based RAID-5 Storage Systems, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, Vol.43, No.6, 2024, pp. 1886–1899, doi: 10.1109/TCAD.2023.3348033. G. Wu, Hadoop Processing Methods for Large Scale Video Data, IEEE Access, Vol.12, 2024, pp. 176247–176258, doi: 10.1109/ACCESS.2024.3505144. A. Kumar, N. Varshney, S. Bhatiya and K. U. Singh, Replication-Based Query Management for Resource Allocation Using Hadoop and MapReduce over Big Data, Big Data Mining and Analytics, Vol.6, No.4, 2023, pp. 465–477, doi: 10.26599/BDMA.2022.9020026. H. Liang, Z. Zhang, C. Hu, Y. Gong and D. Cheng, A Survey on Spatio-Temporal Big Data Analytics Ecosystem: Resource Management, Processing Platform, and Applications, IEEE Transactions on Big Data, Vol.10, No.2, 2024, pp. 174–193, doi: 10.1109/TBDATA.2023.3342619. S. Ghemawat, H. Gobioff, and S. Leung, The Google File System, ACM SIGOPS Operating Systems Review, Vol.3, No.5, 2003, pp. 29–43, doi: 10.1145/1165389.945450. D. T. T. Van, M. N. Khan, T. H. Afridi, I. Ullah, A. Alam and Y. -K. Lee, GDLL: A Scalable and Share Nothing Architecture Based Distributed Graph Neural Networks Framework, IEEE Access, Vol.10, 2022, pp. 21684–21700, doi: 10.1109/ACCESS.2022.3148126. D. Dahiphale, MapReduce for Graphs Processing: New Big Data Algorithm for 2-Edge Connected Components and Future Ideas, IEEE Access, Vol.11, 2023, pp. 54986–55001, doi: 10.1109/ACCESS.2023.3281266. Z. Cheng, W. Shen, W. Fang and J. C. -W. Lin, A Parallel High-Utility Itemset Mining Algorithm Based on Hadoop, Complex System Modeling and Simulation, Vol.3, No.1, 2023, pp. 47–58, doi: 10.23919/CSMS.2022.0023. S. Vengadeswaran, P. Dhavakumar and V. Viswanathan, GSHetero - Grouping and Heterogeneity-Aware Data Placement to Improve MapReduce Performance in Hadoop, IEEE Networking Letters, Vol.7, No.3, 2025, pp. 229–233, doi: 10.1109/LNET.2025.3586883. M. A. Ahmed, M. H. Khafagy, M. E. Shaheen and M. R. Kaseb, Dynamic Replication Policy on HDFS Based on Machine Learning Clustering, IEEE Access, Vol.11, 2023, pp. 18551-18559, doi: 10.1109/ACCESS.2023.3247190. M. Hajeer and D. Dasgupta, Handling Big Data Using a Data-Aware HDFS and Evolutionary Clustering Technique, IEEE Transactions on Big Data, Vol.5, No.2, 2019, pp. 134–147, doi: 10.1109/TBDATA.2017.2782785. D. Chen, Y. Chen, B.N. Brownlow, P.P. Kanjamala, C.A.G. Arredondo and B.L. Radspinner, Real-Time or Near Real-Time Persisting Daily Healthcare Data Into HDFS and ElasticSearch Index Inside a Big Data Platform, IEEE Transactions on Industrial Informatics, Vol.13, No.2, 2016, pp. 595–606, doi: 10.1109/TII.2016.2645606. C. Ma and Y. Chi, KNN Normalized Optimization and Platform Tuning Based on Hadoop, IEEE Access, Vol.10, 2022, pp. 81406–81433, doi: 10.1109/ACCESS.2022.3195872. H. Alshammari, J. Lee and H. Bajwa, 2Hadoop: Improving Hadoop Performance Using the Metadata of Related Jobs, IEEE Transactions on Cloud Computing, Vol.6, No.4, 2018, pp. 1031–1040, doi: 10.1109/TCC.2016.2535261. K. Neshatpour, M. Malik, A. Sasan, S. Rafatirad and H. Homayoun, Hardware Accelerated Mappers for Hadoop MapReduce Streaming, IEEE Transactions on Multi-Scale Computing Systems, Vol.4, No.4, 2018, pp. 734–748, doi: 10.1109/TMSCS.2018.2854787. D. M. Amaral, J. J. C. Gondim, R. De Oliveira Albuquerque, A. L. S. Orozco and L. J. G. Villalba, Hy-SAIL: Hyper-Scalability, Availability and Integrity Layer for Cloud Storage Systems, IEEE Access, Vol.7, 2019, pp. 90082–90093, doi: 10.1109/ACCESS.2019.2925735. J. Song, H. He, R. Thomas, Y. Bao and G. Yu, Haery: A Hadoop Based Query System on Accumulative and High-Dimensional Data Model for Big Data, IEEE Transactions on Knowledge and Data Engineering, Vol.32, No.7, 2019, pp. 1362–1377, doi: 10.1109/TKDE.2019.2904056. D. -M. Bui, S. Hussain, E. -N. Huh and S. Lee, Adaptive Replication Management in HDFS Based on Supervised Learning, IEEE Transactions on Knowledge and Data Engineering, Vol.28, No.6, 2016, pp. 1369–1382, doi: 10.1109/TKDE.2016.2523510.