Principles
Pipelines bring data from important business sources. In many cases, reports and analysis that endure for a long time. Unless your business does not expect to alter how it operates, or there are no amendments to low-level processes, the data pipelines will always need to adapt to the changes in the fundamental processes, new IT, or the data itself.
This means that there should be multi-year funding to monitor and maintain the existing pipelines and provide headroom to add new ones, as well as support the analysis and retirement of old ones.
(See this Martin Fowler articles products-over-projects for a wider description of working in product-mode over project-mode.)
The data collected for a given problem or piece of analysis will nearly always be useful in answering other questions.
In many cases there are simple ways of achieving this. For example, there are usually a variety of places where data is stored in the pipeline. Raw ingested data might be useful for unanticipated purposes, and can often be made available to skilled users by providing them access to the landing zone.
Appropriate identity and access technologies, such as role-based access, can support re-use while permitting strict adherence to data-protection policies and regulations. The fundamental architecture can stay the same, with access being provided by adding or amending access roles and permissions to data buckets, databases or data warehouses.
Pipelines have a cadence which is driven by the need for decision-making and limited by the availability of source data. The developers and users of a pipeline should understand and recognise this as a well-defined unit of work - whether every few seconds, hourly, daily, monthly or event-driven.
In general, we recommend building pipelines around the use case rather than the data source. This will help ensure that business value is achieved early. In some cases, the same data source might be important to several use cases, each with different cadences and access rights. Understanding when to re-use parts of pipelines and when to create new ones is an important consideration. For example, faster pipelines can always be used for slower cadences, but it typically requires more effort to maintain and adapt them. It might be simpler to create a simpler batch pipeline to meet a new low-latency use case which is not expected to change substantially than focus on upgrading a fast-streaming pipe to meet the new requirements.
Experience report:
We have been helping the Office of National Statistics (ONS) to plan and manage the 2021 census of England and Wales. We have a variety of end users, which are defined as people who construct dashboards showing important management information, and people who need to manage large device estates. They need information from many ONS suppliers who provide that data in a variety of formats over different channels.
We built up our data pipeline architecture by starting with an initial high-priority use case from which we created the first thin slice from a single (if complex), data source to provide data for a dashboard. We iterated around this initial architecture, reusing key pieces of infrastructure (such as user groups and permissions), and data engineering patterns (e.g., retaining a common approach to ETL), and adding extra complexity - such as new ingestion methods, decryption and secrets management, as required by new use cases.
Taking a use case-based approach has enabled us to prioritise areas of highest value first, meaning we focus on what matters most for the business users, and dashboard makers, who rely on our pipelines.
We want to be able to amend our data pipelines in an agile fashion as the data environment and needs of the business change. So, just like any other piece of working software, continuous delivery practices should be adopted to enable continuous updates of data pipelines in production.
Data pipelines are a mix of code and infrastructure that can become confusing as they grow if care is not taken with the naming. Pipelines will include at least a set of databases, tables, attributes, buckets, roles, etc., and they should be named in a consistent way to facilitate understanding and maintenance of the pipelines, as well as make the data meaningful to the end users.
In many architectures, naming will directly affect how your data is partitioned, which in turn affects the speed of the search and retrieval of data.
