The Russian invasion of Ukraine and the COVID-19 pandemic have led to a state of uncertainty, demanding operational excellence and the rapid detection of new business ventures from organizations. These recent developments and the accompanying need for resilience have fueled organizations’ ambition to become more data-driven. While investments in data, analytics, and AI are increasing, the results remain sobering. Most organizations have realized the importance of data but only pay lip service to the acceleration of their data journey. However, although much has been achieved, more remains to be done. In a Harvard Business Review article, Thomas C. Redman states,
Data science, broadly defined, has been around for a long time. But the failure rates of big data projects in general and AI projects in particular remain disturbingly high. And despite the hype (e.g., “data is the new oil”), companies have yet to cite the contributions of data science to their bottom lines.
In our previous articles, we discussed the biggest barrier to the ambition of organizations to become data-driven—culture—and described the critical role of business users and how concepts such as data mesh promote the perception of data as a product, as well as its domain-based organization-wide governance (see The Challenge of Becoming a Data-Driven Company). Data-driven companies have deeply rooted the idea of data as a product in their organizational culture. Their goal is to democratize data and give power to business departments to govern their data and provide consumable data products to other units. This cultural change is further accelerated by self-service big data and analytics, as well as AI, since individuals now consume information and data whenever and however they want to. While data mesh describes the organizational perspective toward modern data architectures, a data fabric encompasses the technological aspects. In this article, we discuss how the emerging concept of data fabric helps organizations democratize organization-wide data.
What Is a Data Fabric?
A data fabric describes a modern data architecture with the required capabilities that encompass composable technologies and provide services across hybrid multi-cloud environments. In simple terms, a data fabric is a net that spans multiple data sources and applies machine learning to provide access and meaning to distributed data. While companies realize the challenges and shortcomings of monolithic data lakes as a single source of truth, a data fabric enables the management of the data where it resides. The core engine of a data fabric is metadata. Gathering, analyzing, and enriching metadata, paired with the ability to automate these processes powered by machine learning, allows for the analysis of the underlying data without the need to move and transform it right away. A data fabric is not provided by a single vendor or solution; it is a composable, flexible, and scalable architecture.
Data Democratization
There are several benefits of a data fabric, including enhancing companies’ architectures through increases in efficiency, making companies more scalable, enabling better integration, and helping companies gain more control and agility. However, the ultimate goal of a data fabric is to maximize the value of data and accelerate digital transformation through data democratization. This means giving power to business departments to govern their data and provide consumable data products to other units. Before data fabric, the gap between data and business users as data consumers had been artificially closed by expert data teams. Without their intervention, business users were unable to use, understand, or apply data.
How to Build a Data Fabric
There are several pathways to building a data fabric. The right pathway depends on the organization’s previous architecture decisions. These decisions determine the selection of the capabilities to start with. In any case, do not get overwhelmed with the complexity of the task; start small. With a case-by-case approach, you ensure faster results and keep the motivation level of your team high. Ideally, you can start with the most advanced business department to serve as an example of democratized data. Next, you may need to collect all sorts of metadata and give meaning to them. While metadata might not have much value in their passive form, their collection, analysis, and enrichment lead to active metadata with enormous potential. Applying machine learning to automate these tasks will ultimately lead to a full-fledged data fabric.
Why Now?
The key to overcoming the challenge of data culture is to empower business users and thus democratize data. Other areas of IT management have adopted a more business-oriented approach for decades. For example, operating systems are aligned with business capabilities, thus easily justifying their need and existence. Similarly, efforts in agile IT management focus on products with the end user in mind. Data-driven companies understand data as a product in their organizational culture. All of this has given rise to the emerging concept of a data fabric. Advancements in metadata management have further accelerated the transition toward modern architectures governed across distributed data sources.
