Building Cloud-Based Real-Time Data Pipelines for Dynamic Workflows
Keywords:
Real-Time Data Processing, Cloud Pipelines, Event-Driven Architectures, Distributed Computing, Cloud Scalability.Abstract
Real-time data processing is vital for modern applications requiring immediate insights and decision-making capabilities. This paper introduces a scalable framework for developing real time data pipelines within cloud environments. By utilizing distributed computing models and event-driven architectures, the proposed system ensures low latency, high throughput, and fault tolerance. Key features include stream processing, dynamic scaling, and integration with existing cloud-native tools. Performance metrics from real-world use cases demonstrate significant enhancements in processing speed and system resilience, validating the efficacy of cloud-based real-time data solutions.
References
Adeleke, A.K., 2018. Web-based GIS modelling of building-integrated solar photovoltaic system for the City of Cape Town (Ph.D. thesis). University of Cape Town.
Alonso, G., Hagen, C., 1997. Geo-opera: Workflow concepts for spatial processes. In: International Symposium on Spatial Databases. Springer, pp. 238–258. http: //dx.doi.org/10.1007/3-540-63238-7_33.
Bottaccioli, L., Patti, E., Macii, E., Acquaviva, A., 2018. GIS-based software infrastructure to model PV generation in fine-grained spatio-temporal domain. IEEE Syst. J. 12 (3), 2832–2841. http://dx.doi.org/10.1109/JSYST.2017.2726350.
