The whole data world is now talking about Data Mesh, a new approach to managing centralized data platforms.

Early adopters such as Zalando, Netflix, and many other data-savvy companies, are doing it, and the rest are following their example.

The Slack channel “Data Mesh Learning” grows faster than most other data-related sites.

The term Data Mesh was originally introduced by Zhamak Dehghani from ThoughtWorks. It has a lot to do with the fact that companies such as Netflix, Facebook, and Amazon are originally software startups.

As they’ve become masters of utilizing their data assets (that’s what makes them so successful!) they’ve incorporated app-building practices, such as microservices and agile development, into data management.

all domains, you’ll end up with business-driven information architecture.

The problems of the traditional approach

Data Mesh was developed to tackle the problems of a centrally managed data platform that’s been the paradigm for the past years. Most people are aware of how that works, but here’s a short recap.

The centrally managed basic architecture usually consists of a data lake for unstructured data (so-called big data) and a data warehouse for structured data, such as clients, customers, and products. The data is used for different purposes, such as BI, data analytics, AI/ML, and applications.

All of this is usually managed by a team or department of data engineers and other technical experts. The team is usually in a matrix structure, serving all departments and business units or domains.

This involves various technologies that we are not going to address here in more detail, as Data Mesh is not a new technology – it’s more of a design principle.

According to Dehghani, there are some fundamental challenges in the current, centralized approach.

First, most of the data is always produced in operational systems and applications such as ERPs or CRMs. There are dedicated people who are responsible for these domain-specific applications.

Currently, there is little to no communication between the operational application team and the data platform team. The former doesn’t even necessarily know that “their” data is being brought to a data platform in the first place.

Second, the data team, in turn, doesn’t really understand what’s going on in business applications.

This leads to a situation where none of the groups take ownership of the data as a whole. The outcome of this is that data might be of poor quality, or the quality issues are addressed too late in the process.

The data engineer team working on the centralized platform struggles to produce usable data for consumption as they don’t understand how it’s generated and what it actually means as there is no context to it: it’s just tables with weird names, rows, columns, or files.

How can you deal with data of, for example, customer leads, if you’re not aware of what a “lead” is and how it looks like in the marketing applications?

The Data Mesh concept aims to solve these problems. Next, introduction to the very basics of Data Mesh; there’s a lot more to it of course, but you’ll get the picture.

Data Mesh basics: Domain-centric data pipelines

Eric Evans introduced the theory of domain-driven design, providing a broad framework that emphasizes the meaning of vocabulary in business design. The idea is that applications should be based on domain expertise and language.

In Data Mesh, data pipelines should be built by business domains, meaning that one dedicated team should understand the whole data pipeline of a certain domain, from where it’s produced to how it’s consumed.

Let’s say a company builds automation for managing marketing leads. The dedicated team of that domain would be able to understand the domain (marketing) and it’s operational systems (CRM and other marketing applications). They, therefore, become competent in dealing with leads and other customer data.

This aims to solve the “ownership dilemma” that was addressed earlier and the fact that one must understand business in order to understand and utilize the data it produces.