As something that should respond to and embrace regular change, pipelines should be treated as products rather than projects.
Pipelines need product managers to understand the pipelines’ current statuses and operability, and to prioritise the work.
When creating pipelines, we try to architect them in a way that allows reuse whilst also remaining lean in our implementation choices.
Adopting this mindset and these practices is essential to support continuous improvement and create feedback loops that rapidly expose problems and address user feedback.
Consider what will be the most frequent queries when specifying bucket names, table partitions, shards, and so on.