DATA MANAGEMENT: TRANSFORMING DATA INTO USEFUL INFORMATION

oil-pipeline-picture

COLLECT, PREPARE, ANALYZE, AND DELIVER

Data is a key component for an increasing number of use cases, to the point where a use case-centric approach no longer meets the needs for flexibility and time-to–market.

Transforming raw data into gold data tailored to your business

Aggregate then cluster according to each objective: analytics, operations, ML, and data science…

Consume customized, reliable, and traceable data in dedicated spaces.

Data is a key component for an increasing number of use cases, to the point where a use case-centric approach no longer meets the needs for flexibility and time-to-market. A shift towards a more open approach is necessary: providing consumable data to different stakeholders without presupposing use cases, which evolve over time. Data becomes a product whose consumption is the responsibility of the services that produce it. 

In this context, it is no longer sufficient to simply circulate data, but to give it meaning that adds value. Hence the need for a dedicated ecosystem, combining integration and platform tools with functions for formatting and quality control.

This ecosystem has introduced the notion of a data pipeline to encompass all the steps from collection to sharing of data, directly consumed by operational teams. 

Data Management
Data Management

Data is a key component for an increasing number of use cases, to the point where a use case-centric approach no longer meets the needs for flexibility and time-to-market. A shift towards a more open approach is necessary: providing consumable data to different stakeholders without presupposing use cases, which evolve over time. Data becomes a product whose consumption is the responsibility of the services that produce it. 

In this context, it is no longer sufficient to simply circulate data, but to give it meaning that adds value. Hence the need for a dedicated ecosystem, combining integration and platform tools with functions for formatting and quality control.

This ecosystem has introduced the notion of a data pipeline to encompass all the steps from collection to sharing of data, directly consumed by operational teams. 

However, there is no single architecture that meets all requirements.

At least three elements are key: 

  • Type of data: For example, reference data can be managed specifically in Master Data Management (MDM) systems. Similarly, real-time individual data is handled differently than population data, which requires sequential processing over time.
  • Data volumes: For example, the number of asset records is vastly different from the volume of transactions. Moreover, B2B services are not comparable to in-store sales of thousands of different items.
Data Quality
  • Refresh requirements: Analytical needs often rely on cold data, while customer journey orchestration depends on event-driven logic using hot data.

For businesses, data management has become a crucial challenge, combining business expectations, scalability, security, and cost control. The pooling of data management tools based on their roles, within a data platformization, allows mastering the technology, building a 360° view, and enhancing operational efficiency to provide successful user experiences, both internally and externally. This contributes to performance while offering the necessary flexibility in the short and medium term. 

Traitement des données en entonnoir
Data Processing
Data Pipelines

DATA PROCESSING

Data management sits between data governance, which defines the principles (roles, rules, etc.), and the architecture that supports its implementation. This can be grouped under the term data processing, with the following principles: 

This can be grouped under the term data processing, with the following principles: 

  • Include transformation functions:
  • Make raw data queryable (via SQL, Python, etc.), regardless of its initial format: structured, semi-structured, or unstructured.
  • Align metadata with the company's glossary, as it appears in the Data Catalog.
  • Ensure data quality: remove noise, standardize, apply business rules, deduplicate, enrich with external providers, etc.
  • Store data to facilitate access and processing:
  • Use storage media and formats appropriate to the expected volumes and availability.
  • Adopt a model that facilitates access and query performance.
  • Partition by domain or workspace to optimize compute performance.
  • Make data available for different objectives:
  • Analytics: Prepare data and create metrics used for reporting, dashboards, and any form of visualization.
  • Operational: Create specific data for each type of actor and make it accessible, including allowing them to build their own metadata, such as complementary indicators.
  • ML and Data Science: Provide as broad a data set as possible to enable experts to achieve the best certainty rate.
  • Data sharing: Offer data in a queryable format, with a marketplace logic defining who has access to what.

DATA CONTROL AND SECURITY

Once data pipelines are in place, additional needs arise to control data transformation and ensure secure sharing.

Among these needs, at least three are worth mentioning: 

  • Observability: As systems become more complex, maintaining their performance and reliability becomes a challenge. This is where monitoring and observability capabilities play a crucial role. Neglecting this aspect exposes the company to multiple failures and their potential impacts.
  • Lineage: This involves improving data knowledge to better analyze and control the impacts, whether intended or not, while meeting a regulatory objective: being able to justify the origin of every piece of data.
  • Anonymization, pseudonymization, data masking: Legal constraints (particularly on sensitive data) reinforce the need to control who has the right to see what, and at what level of granularity.
Data Lineage

Our experts will support you in the success of your data projects.

Ch
Genève

Rue de l'Arquebuse, 22
CH1204, Geneva

contacts@arhis.com
Get Directions
Fr
Paris

61 Boulevard Haussmann,
75008 Paris

contacts@arhis.com
Get Directions
Fr
Lyon

94 Avenue Felix Faure,
69003 Lyon

contacts@arhis.com
Get Directions
Fr
Toulouse

32 Rue Riquet,
31000 Toulouse

contacts@arhis.com
Get Directions
Fr
Bordeaux

39 Avenue du Médoc,
33320 Eysines

contacts@arhis.com
Get directions
es
Valencia

Gran Vía del Marqués del Turia, 49
46005 Valencia

contacts@arhis.com
Get Directions
RU
Saint Petesburg

Ulitsa Radischcheva, 39
Sankt-Peterburg, 191123

contacts@arhis.com
Get Directions

Copyright 2020 | All rights Reserved | ARHIS | Web created by OR Design

Data Quality

A lack of data quality undermines the efficiency of internal processes but also impacts performance analysis. As data is increasingly exposed and shared, data quality also affects the customer experience and exposes the company to the risk of penalties for non-compliance with regulatory requirements. 

Thus, quality is part of overall governance, and its improvement must consider the following challenges: 

  • Managing roles throughout the data lifecycle, particularly identifying who owns and manages it at each stage.
  • Measuring quality based on various criteria: completeness, consistency, uniqueness, integrity, obsolescence, etc.
  • Defining appropriate remediation strategies: either addressing quality issues at the point of collection or entry (“a priori”), or after a longer period requiring potentially human analysis (“a posteriori”).
  • Integrating enrichment processes: cross-referencing information with other company data, using open data sources, or calling upon services from specialized providers.

Technologies to improve data quality are directly linked to these challenges: 

  • Measuring source and dataset quality through Data Profiling solutions.
  • Control at the point of entry via APIs, and post-entry validation and enrichment workflows.
  • Connections to other databases, especially in a service mode with DaaS providers (RNVP, Duns & Bradstreet, GS1, etc.).
  • Implementation of cleaning, normalization, validation rules, including assigning a quality score where appropriate.

A major evolution in tools is the widespread adoption of low-code/no-code approaches, allowing business users and data analysts to define and simulate quality rules during their design and configuration. 

Data Processing

Data Processing encompasses all operations that transform raw data into useful information. 

The initial step of data collection has become more complex with the increasing number of source types: databases, applications, but also social media, IoT sensors, etc. However, it remains possible to pool resources through the “API-ization” of sources and middleware tools. 

Data preparation combines quality control with data organization and structuring. This process can occur at all stages, from ingesting raw data to customizing it for specific domains or stakeholders. 

Storage solutions are typically organized in multiple layers to facilitate engineering and analysis. Persisted data must be organized to meet business objectives. Depending on these objectives, different types of datasets, complemented by views or virtualizations, should be offered within a customized architecture. 

Data analysis leads to the preparation of data for visualization solutions but also unfolds through more innovative approaches like machine learning and artificial intelligence. 

Finally, the marketplace concept has emerged to structure data sharing through the clustering of storage spaces and fine-tuned access rights management. 

Data Pipelines

A data pipeline differs from a simple interface in that it combines several steps within an automated process. It can incorporate multiple technologies for ingestion, transformation, storage, and data sharing. 

The adoption of the cloud has enabled businesses to benefit from unprecedented scalability and elasticity for their data pipelines, allowing dynamic adaptation to volume fluctuations through on-demand compute and storage resources. 

The challenge of handling large volumes must be reconciled with a strong focus on real-time data processing and streaming, particularly to provide quality service to operational teams. This has led to the emergence of technological solutions that handle large-scale messaging streams. 

To be secure and reliable, these pipelines must be managed within a robust technical governance framework: access management, observability, cost control, etc. 

Data Lineage

Data lineage serves several purposes: 

  • Better understanding of data.
  • Compliance by being able to justify specific data.
  • Evolving the information system: if an element changes or is replaced, what are the impacts?

It’s not just about tracking the origin of a piece of information but also about controlling the governance that has been applied to it. This enables: 

  • Evaluating data quality to increase the confidence level of an analysis.
  • Managing operational performance to serve customers, whether by strictly complying with regulations or improving service quality.
  • Anticipating ecosystem changes to enhance performance or invest in new activities.

The implementation of pipelines has made this challenge more complex, particularly with the use of the cloud, and justifies the need for dedicated tools. In this context, it becomes increasingly interesting to integrate these modules into broader data governance platforms.