Optimizing Large-Scale Data Migration for Cloud-Native Architectures

Abstract

Large-scale data migration is emerging as an essential need for businesses migrating from traditional on-premises systems to cloud-native solutions. However, the traditional data migration process has limitations in addressing contemporary needs for real-time scalability, schema adaptability, governance, and cost-effectiveness. In this paper, an artificial intelligence-based cloud-native data migration framework will be proposed that encompasses distributed pipes, meta-driven orchestration, quality checking, and validation. Structurally, this proposed work will be reinforced by novel cloud-native ETL frameworks [1], artificial intelligence-based automatic migration systems [2], and meta-governance systems [3] for better accuracy of cloud data migration, minimized downtime, and adherence to governance and regulatory compliance. Simulations and results verify that artificial intelligence-based schema mapping for cloud data migration, anomaly detection, and serverless parallelization [4] can dramatically reduce total migration time and enhance data consistency.