Ingesting and preparing data help organizations to deploy automated data pipelines in an event-driven architecture. In other spaces where continuous data is produced, such as sensor metrics, temperature, and vibrations can be imported directly into an Apache Kafka topic in AMQ Streams for example.

Data engineers can create pipelines to help automate workflows and business processes.

The Data Acquisition stage in the solution pattern is presented in the diagram below:

In a first phase, when a new product needs to be added in the catalogue, its imagery is pushed to the system (Device to Edge1). The data is kept in local object storage (S3) while waiting to complete the image collection.

In a second phase, when the imagery is collected in full, an edge integration application (Camel) packages all the data and sends it over the wire, to the main data centre (Central) where a second integration (Camel) unpacks the images and pushes them to a local S3 bucket.

When the training data upload to S3 completes, a coordination signal is sent via Kafka (Streams) to announce new training data is available, making other systems to react.

The second stage is focussed on two important tasks:

There is considerable manual labour involved into the activities above listed. However, once the team of data scientists produce the desired implementation to generate the AI/ML models, it can be injected as part of an automated pipeline, along with other automated steps.

The same is true for the pre-processing tasks to refine the raw training data. More and more organisations seek to implement autonomous refinement and labelling processes which can be added to the chain of automated steps in the pipeline.

The solution pattern implements this stage as a fully automated workflow, initiated by the Kafka signal (Streams) triggered upon the reception of new training data, as illustrated in the diagram below:

The Kafka (Streams) event (signal) announces new training data is available. The Camel integration consumes the event and triggers the automated pipeline.

The pipeline obtains all the training data from the S3 bucket and starts training the new AI/ML model. When the training completes, it uploads the new version to a model repository.

The third stage is primarily focussed on including all the DevOps and MLOps processes necessary to deliver the applications and upgrades the near edge needs to continuously evolve.

Here, all the best cloud-native practices apply. To know more, follow the link of the resources listed below:

Smart applications, powered by the AI/ML models, need to be designed and implemented along with their automated delivery mechanisms.

The demo provided in this Solution Pattern already includes all the applications and integration systems. The focus is set on showcasing the automated pipeline responsible for producing new model versions, which are uploaded to the platform’s object storage system (S3), keeping duplicates in a model repository.

The diagram below illustrates the stage:

When the pipeline produces new model versions it pushes a copy into an edge-dedicated S3 bucket. An integration system (Camel) on the near edge environment monitors the bucket and when a new model version is available it pulls it and hot-deploys it in the model server.

The last stage involves customers and users generating live traffic that interacts with the platform and invokes AI/ML inferences against the Model server.

Other systems are also at play in orchestrated workflows to deliver end-to-end business use cases and provide responses to customers.

The demo also includes a monitoring view that provides more in-depth insight into the systems involved in each stage.

The Solution Pattern connects (Near) Edges to a Core Data Centre with Red Hat Service Interconnect.

Red Hat Service Interconnect simplifies application connectivity across the hybrid cloud. Unlike traditional means of interconnectivity (such as VPNs combined with complex firewall rules), development teams can easily create interconnections without elevated privileges and deliver protected links without compromising the organization’s security or data.

Applications and services across your environments can communicate with each other using Red Hat Service Interconnect as if they were all running in the same site. This connectivity can be maintained even as applications are migrated between environments.

In the Solution Pattern, when Edge systems interact with Core systems, they don’t need to configure remote endpoints. Service Interconnect bridges the connectivity between the two sites and exposes remote Core services as local ones to Edge, as illustrated in the image below.

Edge computing shifts computing power away from core data-centers and distributes it closer to users and data sources—often across a large number of locations, providing faster response times, more reliable services, and a better application experience back to users.

The Red Hat® AMQ streams component is a massively scalable, distributed, and high-performance data streaming platform based on the Apache Kafka project. It offers a distributed backbone that allows microservices and other applications to share data with high throughput and low latency.

The latest AMQ Streams release introduces the new UseKRaft feature gate. This feature gate provides a way to deploy a Kafka cluster in the KRaft (Kafka Raft metadata) mode without ZooKeeper. This feature gate is currently in an experimental stage, but it can be used for development and testing of AMQ Streams and Apache Kafka.