Create an id/record for each batch uploaded into the pipeline.
id
record
Manage state inside the record
state
How are you measurign the pipeline?
How many units are going through the pipeline?
How long does it take to go through the pipeline?
Can you easily test/validate a single part of the pipeline?