Appian Data Fabric

Let’s define data fabric: It’s an architecture layer and tool set that connects data across disparate systems to create a unified view.
‍As a virtualised data layer, it enables you to access data without migrating it from where it lives - whether in a data lake or data warehouse, relational database, enterprise resource planning (ERP) system like SAP, CRM like Salesforce, or SaaS application.

The data may be on-premises, in a cloud service, or in multi-cloud environments.

What is a data fabric used for?

With the proliferation of digital technologies and tools, enterprises have exponentially increased their amount of data. And it has become increasingly fragmented across disparate applications in the organisation.

Over time, new data management practices have emerged for managing these complex data issues, including data warehouse, data lake, data lakehouse, and data mesh. But for most modern businesses with complex data structures, they each have their own shortcomings for managing data efficiently - from increased or shifted technical debt to long and costly data migrations to reduced data integrity and data security concerns.To overcome these enterprise data management challenges, what many organisations need is a data fabric.

Data fabric plays a key role in a modern process automation platform that optimises complex business processes end to end. That’s crucial as you seek to scale automation across the enterprise to achieve holistic improvement, not just isolated wins.

Data fabric connects your enterprise.

Data management approaches

As mentioned above, there are multiple strategies for managing data across the enterprise. Let’s take a look at some of these approaches to understand why they don’t meet the needs of today’s fast-moving enterprise.

Data warehouse, data lake, and data lakehouse

The primary goal of these approaches is to collect data in a single repository - not connect it.

Data warehouses store structured operational data for analysis. Data lakes store large sets of unstructured data, to be cleaned and analysed later.

Data lakehouses combine the data lake’s flexibility in type of data with the data warehouse’s high-quality data components.

Data warehouses, lakes, and lakehouses are great tools for analytics and operational reporting. But they still require you to lift all the data out of your siloed systems and load it into a new system (the warehouse, lake, or lakehouse). This means more maintenance and technical debt of yet another siloed system.

While data warehouses focus on collecting data, data fabric connects it.

What’s more, these repositories only suit analytical work that uses historical data. They don’t support transactional systems that require real-time data, such as CRM applications.

Data mesh
‍
In contrast, data fabric and data mesh architecture design takes a different approach. They focus on connecting directly to the data sources rather than extracting all of your data. This allows you to access real-time data and avoid timely and costly migration projects.However, data mesh and data fabric solve this data connection problem differently. Data mesh uses complex API integrations across microservices to stitch together systems across the enterprise. With data mesh, while you avoid a lot of data engineering work, you trade it for additional software development efforts dealing with the APIs.

Read our related article. Data fabric vs. data mesh vs. data lake

Data Fabric

What makes data fabric unique is its ability to create a virtualisation layer on top of data sets, removing the need for the complex API and coding work that a data mesh data warehouse, data lake, or data lakehouse requires. This gives teams added speed and agility to do data analysis, data modeling, and digital transformation work.

Why use a data fabric? Why is it important?

A data fabric architecture lets you quickly connect data across enterprise silos. But many data management and integrated tools out there claim to do this. So why use a data fabric, specifically?

A key benefit of using a data fabric is that it covers both transactional and analytical systems.

A data fabric thrives in situations where data is constantly changing, such as applications that involve partner data-sharing. Because the data is virtualised and connected directly to the source systems, you can easily read/write to those systems. No migration work is required. Your team gains real-time data for real-time insights into your organization. This single source of data gives you a complete view of your business - a holy grail that organizations have chased for years in search of better business outcomes.

De-silos your data by connecting disparate data sets - without hordes of database specialists.

Improves decision-making by giving users a complete view of data.

Reduces development time and costs by connecting data directly from where it lives, eliminating time-intensive and costly data migrations.

Increases agility by letting you easily change and update your organisation’s data models. Because the virtual data layer sits on top of the data, you don’t need to do complex maintenance work and you can quickly add, delete, and relate sources together as business needs change.

Provides a simplified data modeling experience that democratises data analysis, giving access not only to skilled data engineers and developers but also to line-of-business employees who need data-driven insights to reach their business objectives faster.omplete view of data.

Reduces security and compliance risks by giving IT a centralised picture showing who can view, update, and delete specific data sets.

Data Fabric Examples

What does a data fabric look like in real life? Data fabric technology can be used across an entire organisation for many different use cases, as they can connect a wide range of data sets. Let’s look at just a couple of examples of how an organisation would use a data fabric to connect disparate data sources across the enterprise to improve visibility and efficiency.

Data fabric in service request management

Many enterprises have a business unit that deals with maintenance and service to their products (machines, utilities, etc.) To streamline efficiency, they need an application to manage all aspects of the request process, so the business unit looks to build a service request management application.But typically the data they need to access, update, and take action on is spread out across the organization. The parts inventory lives in an ERP system, the customer’s equipment lives in a homegrown relational database, and the customer information sits in their CRM, for example.In order to properly handle these service requests, the business needs to connect all three of their disparate systems. Migrating them into a single bucket would take too much time and effort. Plus this data is constantly changing so it would be stale by the time it got to the business users.Instead, the organization uses a data fabric architecture to connect directly to each system while leaving the data in its current place. This breaks down silos and minimizes the data design work needed. The result is real time data and operational insights throughout your service request application, where users can read and write directly to each source as if it were local data.

Data fabric in vendor onboarding

Many organisations have to manage the process of onboarding vendors, whether they are contracted workers, materials suppliers, etc. In this example, let's say an organisation wants to manage the granting of intellectual property (IP) to third parties based on their contracts with the organisation. This will require merging data across brand intellectual property stores, contract information, and CRM data. You’d also need to keep it up-to-date as users take their respective actions. And in this case, you would definitely need to secure certain intellectual property, depending on what the third party’s contract provides them.

A data fabric platform connects all three of these sources to let you access real-time information and secure your data across multiple systems of record with row-level security. This lets you reference your CRM to determine if certain rows of data in your IP database should be visible to the contractor. That’s the power of building a data fabric.