What is Data Mesh?
A data mesh architecture is a decentralized data storing approach that enables domain teams to use the storage technologies of their own choice. The primary component of a Data mesh architecture is a processing engine which is capable of performing cross-domain data analysis.
Typical data mech processing engine should be capable of processing the data resides in various sources. Few example data storage options can be:
- Relational data in SQL Server.
- Delta files in Azure Data Lake Storage Gen2
- Images and Videos in ADLS Gen2
- PDF reports/logs/Text stored in Blob.
- Streaming data in Kafka/Event-hub
Data Mesh Using HDInsight Trino
In recent years, many organizations successfully use modern architectural and analytical patterns that combine data warehousing technologies and more recent big data technologies.
Microsoft is committed to provide best possible options to customers and chosen Trino for one of the options for Data Mesh architecture.
Trino is a highly parallel and scalable query engine. It supports federated Query over wide range of data sources.
Azure HDInsight provides AKS based Trino managed service which can be used for developing the Data mesh architecture. Its PaaS form of service supporting key features like AutoScale, Security management, Library Management and In-Place Upgrade. HDInsight Trino supports various storage formats like ORC, Delta and etc.
Typical Data mesh architecture using HDInsight Trino looks as below.
This is reference architecture. Trino can support much larger variety of sources and visualization tools.
Feel free to try out the boundless capabilities with Trino, taking a giant leap to build data mesh.
Here is an interesting talk, on data mesh (https://youtu.be/ePr-iVQ5ri4?si=rK-VG3gyIp8jPkF2)
We are super excited to get you started, lets get to how?
- Signup today - https://aka.ms/starthdionaks
- Read our documentation - https://aka.ms/hdionaks-docs
- Join our community, share an idea or share your success story - https://aka.ms/hdionakscommunity
- Have a question on how to migrate or want to discuss a usecase - https://aka.ms/askhdinsight