4 minute read
Data Mesh: Solving the business plea for data access
New model empowers businesses to streamline data to users for enhanced decision-making
Every day, 2.5 quintillion (250 trillion) bytes of data are created. While that's a staggering amount, it's likely to more than double by 2026! As the “dataverse” grows, many companies recognize the need for data intelligence, and it’s leading them to take a data-first approach. They understand the increasing demand on data accessibility to drive growth, and with it, the complexity of managing that data.
Unfortunately, organizations struggle with enabling employees to find and consume the right data to make informed and timely decisions. In fact, more than 80% of data that companies create is not used because of challenges with latency and costs of moving data across environments1.
Operational and organizational issues hamper companies from getting full value with their data. Overcoming such challenges requires them to pivot how they manage data. For example, instead of your central IT team trying to maintain a backlog of company requests, data would go directly to business domains—such as the product and development teams—because they know best what’s in their data and what they want to do with it. This is how the concept of Data Mesh works, it focuses on organizational principles to improve how data is delivered.
Data Mesh: 20 years in the making
Simply put, data mesh is a new strategic approach to data management. It brings order to an organization’s data, helping deliver that data more efficiently to business leaders so they can make better decisions. In more technical terms, data mesh is a set of principles for designing a modern data architecture.
To better understand data mesh and help you decide whether it’s a viable approach for your business, let’s look back in time. Roughly 20 years ago, companies used Enterprise Data Warehouses (central repositories) to manage data as an asset. The idea was to create a single location for data to be stored that could be used to drive business decision-making. This required time and money up front. Plus, the warehouses could not meet storage or management needs, and were unable to support AI or machine learning.
Fast-forward 10 years and we reach the Big Data era. The main concept was to quickly integrate large datasets into a company’s data lake or data warehouse to be analyzed. The data lake developed from the need for more storage and better management of data. While data lakes removed the upfront costs and time-consuming development work to store and access the data, they were complex and time consuming to maintain.
Today, it’s common for companies to have multiple disparate data silos. This also presents challenges, however, when leveraging data as an enterprise asset including slow data-driven decision-making and poor collaboration across teams. Data mesh is a way companies could address this issue.
Why is Data Mesh important?
Think of data mesh as a kind of central nervous system for your organization's data. Just like your central nervous system is the body's processing center, data mesh for your company is like a network to exchange data about the business.
Data users can easily access and query data where it lives without first moving it to a data lake or data warehouse. Business teams are responsible for their data and resulting data products. As such, data mesh has a heavy emphasis on self service, allowing business teams to efficiently manage and utilize the data.
“Data mesh has solved the business plea for data access, something that organizations struggled with for many years,” said Kenneth Viciana, vice president of Global Data & Analytics Products at TSYS, a Global Payments company. “The decentralized strategy delivers data to the right people in the organization in real time, reducing costs and providing faster analysis. The result is an increase in the validity of data-driven insights that allows businesses to make better data-based decisions.”
A central nervous system for data
- Reduces costs: By not having to move data across multiple environments, businesses can shift away from a manual process to self-service business access.
- Provides faster analysis: Since data is decentralized, it goes straight to the domain owner—reducing the lag time between capturing and analyzing data—for quicker analysis and faster insights.
- Enhances scalability: By giving data ownership to business teams, it solves the need to make changes to complex data pipelines as companies grow operations and increase the amount and sources of data.
Putting data into the right hands
With data mesh, data is accessible to business users, which decentralizes the data and allows for a self-service infrastructure. This approach avoids potential bottlenecks when trying to integrate data into a data warehouse or data lake. Teams can now generate, collect and distribute data as needed in what amounts to a “data ownership” model.
No enterprise data team has ownership of the organization’s data assets. Instead, these assignments land in business domains. For instance, customer relationship management data is often managed in Salesforce by an operations team, and the data mesh would assign operations as the business domain owner of this data set.
Opportunities also present challenges
As you have learned, data is being delivered more efficiently to the business and its users with data mesh, but does it provide the right insights? Despite the need for business access to data being met, data mesh comes with challenges. These challenges stand in the way of consistent, accurate insights that could lead to improper business decisions.
For one, there is a need for creating consistency across the business in how teams manage the data. To handle this situation, organizations generally have an Enterprise Data Governance Team that provides business units with frameworks and standards that promote alignment.
Secondly, data mesh requires a strong federated data governance approach. Cross domain analytics and the potential to duplicate data across different domains could affect resource utilization and data management.
While data mesh may be the first step in solving the business data access need, there is more to solve to ensure delivery of the right insights to make smart business decisions.
1McKinsey & Company, August 2022, McKinsey Technology Trends Outlook 2022