Progress Software Completes Acquisition of Domo's AI and Data Platform Business. Learn more

 Explore All Inisghts
Insights Article

The Evolution of Metadata 
Management

As a data architect, you probably already understand the importance of metadata management. By defining and leveraging information about the data itself (descriptions, definitions, relationships, etc.) you can achieve better data quality, governance, unity, reliability and security, among other benefits.

In 2021, Gartner pronounced that traditional metadata practices were now “insufficient” to meet the needs of the enterprise. In its Market Guide for Active Metadata Management, it heralded the transformational capabilities of “active metadata.” This evolved type of metadata powers some of the most innovative data architectures, from data fabrics and digital twins to semantic knowledge graphs, and enables powerful new business use cases.

Read on for an overview of this metadata evolution.

Passive vs. Active Metadata

To grasp the concept of active metadata, it helps to contrast it with traditional, passive metadata. If your data is a digital file, then the (passive) metadata is the author, file type, date created or modified, and so on. While these types of metadata fields are basic and static, they still provide some elements of context and meaning to data.

For example, at a life sciences company, standardized metadata fields for clinical trials (e.g., agent dose, administered time, disease recurrence type, etc.) help facilitate data that is consistent, shareable and compliant with regulations. This gives pharmaceutical team members greater transparency into data, facilitating better communication, deeper insights and faster results

What Is Active Metadata?

What makes metadata “active” rather than static or passive? It changes as the data changes. More specifically, it immediately reflects a change in state, its use or its intended audience of the associated data as it moves through the information supply chain. Active metadata creates new meaning for existing data, making it more insightful and actionable to improve business operations.

Think of this metadata as a backpack attached to your data, which travels everywhere your data goes. Inside that backpack is everything you know about the data—for example, its meaning, relationships with other data elements, process journey, governance record of how it changed over time—which means that as data flows through the supply chain, you’re continually adding new items to the backpack. All of this is then available for use in analytics, process management and for contextual applications.

Here’s a summary of metadata’s evolution from passive to semantic, active and augmented:

Metadata TypeDescriptionExamples
Passive metadataInformation about dataSystem-applied dates, creator, source
Semantic metadataDescribes the meaning of dataTopics, product, geography, audience, concepts and relationships
Active and augmented metadataIntelligent and dynamic dataFacts, status, PII, protective and retention markers, data orchestration, ML analytics

To take advantage of active metadata, you typically deploy machine learning (ML) and/or business-oriented AI algorithms to automate metadata management. An AI/ML-powered data platform or tool can continuously collect, create and analyze metadata, then trigger automated actions or proactive recommendations, such as:

  • Detecting missing or incorrect data, then correcting the issue
  • Protecting or masking data when it contains personal information
  • Adding context to dashboards to help leaders make more informed decisions

Most architectures have metadata management (active or otherwise) as a distinct layer in the data stack. But this approach is suboptimal. Metadata IS data (a different type of data, but data nonetheless) and it should be addressable as data from a single data platform without compromise. Separate layers suggest custom integration between different vendor products with all the consequential risk of versioning, compatibility and loss of capability.

In summary, effective metadata management tightly coupled with its associated data eliminates the effect of both data and knowledge silos, delivers data agility and accelerates insightful business decision-making.

Metadata Challenges for the Enterprise

Here’s what’s holding back most organizations from leveraging the full power of their metadata:

  • It is passive only, with rigid structures and schema that are rarely well maintained
  • It is input manually—a time-consuming process that restricts the size of the metadata set, limits the ability to add or change metadata in real time, or is system-generated and rife with errors, as there is no applied context
  • It is stored separately from the data itself, eliminating the possibility of actively updating it to reflect how its related data changes or how it was used, preventing joined-up analytics
  • It requires a third-party tool, adding further complexity to the system when integrating the metadata layer 

These limitations severely hamper an organization’s data agility—defined as the ability to quickly and easily make changes to how information is interpreted and acted upon, for any purpose, at any scale.

Put Metadata at the Center of Your Strategy

How can you achieve data agility? By putting metadata at the center of your data strategy. This means connecting active data, metadata and meaning, and keeping data and everything that's known about it together at all times.

That’s the power of Progress® MarkLogic® data management and Progress® Semaphore® semantic AI platforms.

  • Couple data with its metadata to establish a single data resource
  • Create and manage active metadata, allowing you to integrate, store, manage, govern, contextualize and surface data regardless of format, schema, or type
  • Employ semantic AI to synthesize, enrich, extract and harmonize all types of metadata
Uniform Security Governed Data Model Reference Data Metadata AS IS DATA INGESTION ACCESS CURATED DATA

Build More Accurate and Trusted AI Applications

See how Progress® Data Platform combines knowledge graphs, multimodel data and semantic search to power enterprise AI.