Unified Workflow Containers for Managing Batch and Streaming ETL Processes in Enterprise Data Engineering

Abstract

Unified workflow containers establish a coherent execution framework for handling batch, streaming, and hybrid ETL processes within a single operational model, reducing the fragmentation traditionally seen in enterprise data engineering ecosystems. By combining container-level isolation with centralized metadata synchronization and adaptive scheduling, the approach delivers more predictable performance, stronger lineage transparency, and greater runtime stability across diverse workloads. The evaluation confirms that these pre-2019 architectural principles already embodied the foundations of modern unified data platforms, enabling smoother recovery, improved data freshness, and more consistent system behavior under fluctuating load conditions. As data volumes and real-time processing demands continue to grow, unified workflow containers provide a resilient and forward-compatible architecture for building scalable, high-efficiency ETL infrastructures.