What is a middleware?

3. Middleware and data quality: transferring data or also transforming it?

Choices diverge when determining where to add intelligence: in the middleware, business applications, or a Data Platform. This choice depends on the company’s strategies, the capabilities of the concerned applications, and internal organization around these challenges.

4. Service buses: up to the data hub?

The concept of a Bus rationalizes communication and data exchange between sources and consumers (which can be the same successively) through exchange patterns that are either point-to-point or publish-subscribe.

Management rules, API calls to providers for data enrichment, etc., can be added. The data service bus thus approaches the notion of a hub.

5. The two major real-time technologies: synchronous (API) and asynchronous (streaming/messaging) are, in the end, more complementary than competitive:

Web APIs allow exposing data for consuming applications, for internal needs, or for externally-facing fronts. The main limitation is the high availability requirement.

Note: This type of API often applies more restrictive security policies in terms of usage and authorization to guarantee data access confidentiality and maintain API service levels.

Streaming, given its near-zero latency, meets most real-time requirements. Streaming is the preferred solution in a ‘data pipeline’ usage.

Managing acknowledgments is a crucial requirement to ensure all information is transmitted. It requires monitoring and observability functions.  

For large-scale processing, solutions like Kafka complement the setup.