Basic pipeline architecture

During each (critical) step data will be written to storage (which will be HA/redundant) to ensure no data will be lost when a critical failure occurs. The amount of data in memory will be limited as much as possible