Pipelines

  • Create an id/record for each batch uploaded into the pipeline.
  • Manage state inside the record

#Monitoring/Alerts

  • How are you measurign the pipeline?
  • How many units are going through the pipeline?
  • How long does it take to go through the pipeline?
  • Can you easily test/validate a single part of the pipeline?