Pipelines

  • Create an id/record for each batch uploaded into the pipeline.

  • Manage state inside the record

Monitoring/Alerts

  • How are you measurign the pipeline?

  • How many units are going through the pipeline?

  • How long does it take to go through the pipeline?

  • Can you easily test/validate a single part of the pipeline?