Toward Autonomous Data Engineering: An Architectural Framework and Verification Engine for Self-Healing Data Pipelines
Keywords:
Autonomous Data Engineering ; Agentic AI; Self Healing Pipelines, Data Infrastructure; Model Context Protocol(MCP) ; Data Contracts, Shadow Execution; Differential Profiling ; Data Lineage; Abstract Syntax Tree(AST) Verification; Transactional Lakehouses;Schema Drift Remediation; Automated Programme Repair(APR); Silent Data Corruption; Data EngineeringAbstract
When there are no semantic checks performed when applying repairs to data pipelines,autonomous remediation can create reliability and governance concerns. In this study, the authorspropose AutoDataEngine ,an architectural framework that separates autonomous repair generationfrom production state mutation using executable contracts ,statistical profiling ,AST verification, lineage aware checks ,gated promotion ,rollback and forensic logging
References
1)Afzal, H., & Huzaifa, M. (2026). AI-powered intrusion detection system for securing Internet of Things (IoT) smart home environments. In N. G. Nhu, A. Jaiswal, & T. M. Thanh (Eds.), Proceedings of International Conference on Computational Technologies for Research in Data Analytics: ICCTRDA 2025, Volume 4 (pp. 360–370). Springer. https://doi.org/10.1007/9783-032-25694-2_32
