WHAT IS A DATA PLATFORM?

stacks-of-multi-coloured-shipping-containers-2023

UNLOCKING THE POTENTIAL OF YOUR DATA

Data is often siloed within business applications. The Data Platform enables the valorization of this data by connecting it across different domains. Data truly transforms into “assets,” making the company “data-centric.”

An ecosystem that prioritizes access to rich and trustworthy data.

Data mastery: security, observability, and compliance.

Data as an asset to optimize operations and develop new practices.

Data is often siloed within business applications. The Data Platform allows for the valorization of this data by connecting it across different domains. Data truly transforms into “assets,” making the company “data-centric.”

Platformization, based on a centralized architecture, is built around workspaces and enables moving beyond the logic of business “use cases,” empowering everyone to achieve their goals. Data becomes a reliable and available “product.”

The platform ensures data mastery:

  • Data reliability
  • 360° data visibility
  • Data accessibility and sharing

But also governs it:

  • Monitoring and observability of data
  • Data traceability and security
  • Compliance with regulatory requirements: GDPR, etc.
Functional-architecture-of-an-enterprise-data-platform
Data sharing and marketplace
Functional-architecture-of-an-enterprise-data-platform
Data sharing and marketplace

Platformization, based on a centralized architecture, is built around workspaces and enables moving beyond the logic of business “use cases,” empowering everyone to achieve their goals. Data becomes a reliable and available “product.”

The platform ensures data mastery:

  • Data reliability
  • 360° data visibility
  • Data accessibility and sharing

But also governs it :

  • Monitoring and observability of data
  • Data traceability and security
  • Compliance with regulatory requirements: GDPR, etc.
Database Floating Squares 3 layers
Warehouse and Workspace
Machine Learning

3 DATA LAYERS

The platform acts as a hub that ingests data at the entry point and propagates it at the output after transformation. Traditionally, there are 3 layers in the data processing workflow:

  1. Storage, also known as the Bronze layer. The choice of a dedicated layer for data storage, separate from processing, allows for storing data in its raw format. This approach offers maximum flexibility regarding types of ingestion (streaming or batch) and scalability. Additionally, this space is secured at the access level, usually reserved for technical managers.
  2. Processing, or Silver layer. This contains a validated and enriched version of the data with a high level of trust. It is obtained through transformation operations, joins, aggregations, and formatting, performed using processing-oriented frameworks (e.g., parallel processing). The data is queryable using common languages like SQL, Python, etc.
  3. Data presentation layer, or Gold layer. Here, the data must be usable information for the expected purposes of the platform: analytics, ML, or data as a product. End users interact with the data using their query language or through interfaces and applications.
  4.  

Data Centricity

The Data Platform aims to centralize all of the company’s data: reference, transactional, and behavioral. A number of automated pipelines and processes are implemented to:

  • Adhere to company formats (BOM, pivot format) and, more generally, adopt a catalog logic.
  • Consolidate transactions and behavioral data around business objects.
  • Define autonomous and specific datasets, workspaces, or warehouses, maintained and managed by expert teams in their respective fields.
  • Calculate dedicated indicators within these workspaces to combine expertise, performance, and accountability of end users.
  • Precisely manage data access rights and establish a “FinOps” framework to secure and hold accountable for data consumption.
engineers working on data center
Data Mesh
Data platform organization
AI and Data Governance
Generative AI

Data Warehouse, Machine Learning, and Digital Fronts

A Data Platform is an integrated framework that aims to combine the functionalities of a data lake, data warehouse, and data hub. It addresses analytical needs, Data Science, and operational services.

  • The Data Platform replaces traditional data warehouses by adding scalability and the capacity to handle different data formats, including semi-structured data (JSON, etc.) and even unstructured data. Specific processes are applied to prepare data for Data Visualization tools.
  • Data Platforms integrate a space and functionalities for Data Science and offer shareable Machine Learning models that can be adjusted (tuned) according to the company's needs.
  • These platforms are open to business users through dedicated workspaces and to external users, supporting the development of digital fronts or integration with marketplaces, allowing for the valorization of company data (or benefiting from data or developments offered by others).

Discover our Data content!

How to build a Data-Centric architecture?

Our client interviews

Our ambition is to implement innovative solutions that address your business imperatives.

Ch
Genève

Rue de l'Arquebuse, 22
CH1204, Geneva

contacts@arhis.com
Get Directions
Fr
Paris

61 Boulevard Haussmann,
75008 Paris

contacts@arhis.com
Get Directions
Fr
Lyon

94 Avenue Felix Faure,
69003 Lyon

contacts@arhis.com
Get Directions
Fr
Toulouse

32 Rue Riquet,
31000 Toulouse

contacts@arhis.com
Get Directions
Fr
Bordeaux

39 Avenue du Médoc,
33320 Eysines

contacts@arhis.com
Get directions
es
Valencia

Gran Vía del Marqués del Turia, 49
46005 Valencia

contacts@arhis.com
Get Directions
RU
Saint Petesburg

Ulitsa Radischcheva, 39
Sankt-Peterburg, 191123

contacts@arhis.com
Get Directions

Copyright 2020 | All rights Reserved | ARHIS | Web created by OR Design

Data sharing and marketplace

Once the best version of the data is centralized, sharing this data must be specifically designed as a service. This involves associating data consumption tools with strict governance of access. The establishment of a marketplace is a response that satisfies both the need for internal collaboration and the publication, even monetization, of data to external stakeholders.

The marketplace can also be seen as one of the outcomes of the Data Mesh logic, allowing data to be presented as products.

Warehouse and Workspace

Should we oppose warehouse and workspace? Beyond semantic disputes, we can distinguish three different needs:

  • Centralizing operational data storage from various sources for analysis and reporting to support decision-making. This is referred to as a warehouse. These solutions are optimized to execute complex analytical queries on large amounts of historical data, using SQL or Python. Typical users of these systems are Data Analysts.
  • Providing a flexible and collaborative environment where users can prepare, explore, and visualize data, often relying on various Data Science tools such as notebooks. This is called a workspace. It is the preferred space for Data Scientists, who test and refine their ideas, improve their models, and ensure that proposed solutions are both robust and effective.
  • The Data Mesh logic introduces a third concept: a dedicated space for certain stakeholders (e.g., marketing or logistics). This space relies on so-called “enterprise” data, but domain experts add their own metadata and indicators tailored to their needs, along with external data, leading to more sophisticated frameworks that integrate business functionalities. These features run directly on centralized data, eliminating the need for additional synchronization to another application dataset.

Machine Learning

Machine Learning (ML) aims to uncover an understanding of data, such as links and segmentations, and to transform this information into predictive models that will serve as the foundation for commercial policies, regulations, etc. ML is organized within a storage and services environment that benefits from Cloud Computing platforms and their performance in storage and computation.

Based on a Big Data-type infrastructure environment, resource provisioning, etc., ML is structured around several steps:

  • Preparing the data: cleaning, quality control, and formatting, organized around metadata to ensure reliable results.
  • Using self-service models (potentially paid within marketplaces), corresponding to generic use cases aligned with business needs.
  • Focusing on operational results and modifying the algorithms of pre-existing models to adjust them to the company’s datasets.

Data Mesh

The Data Mesh is not a technology but rather an organization and architecture around data. It intertwines accountability (engaging business units to take ownership of data availability and quality) and management (empowering data producers on different processes). These two conditions enable data to be transformed into “products.”

However, this organization must be framed within a controlled enterprise architecture to ensure data availability, security, and cost management. The goal is thus to overcome the bottleneck of an organization where IT plays a role to define:

  • Infrastructures: solutions, tools, and architectures validated by urbanization managers, and shared as much as possible to facilitate their operation.
  • Boundaries: defining enterprise data vs. business-specific data vs. external data necessary for a business. The objective is to manage the data made available to internal or external consumers, notably through marketplaces.
  • Roles and service contracts: governance and FinOps.

AI and Data Governance

AI is known for its contributions to data analysis, modeling, and presentation, particularly through generative AI. The power of AI also manifests in:

  • Metadata management: automating the cataloging and indexing of metadata. AI also helps map attributes from new sources to a target format and significantly accelerates interface development.
  • Rule management: proposing quality rules based on source data or other datasets. It also plays a role in deduplication, particularly by suggesting matching rules based on the analyzed population.

Generative AI

The transformations brought by generative AI do not stem from a new way of thinking about data: it also relies on AI algorithms based on a massive data stack. However, generative AI, through the use of a Large Language Model (LLM), tends to change and simplify the way of working:

  • We no longer navigate: we converse.
  • We no longer query: we interrogate via prompts.
  • Thus,
  • The business user gains insights on performance, trends of a product or service.
  • The analyst perceives the patterns behind extractions and receives recommendations on quality rules.
  • The Data Engineer requests datasets, calculations, and retro-documents their code.

The challenge for companies is that generative AI must operate on their internal data. Beyond data preparation, it is crucial to maintain data confidentiality while using a market LLM. Several options exist to achieve this, particularly by leveraging various software providers. The final outcome of this subject remains to be written.