Deploying Edge AI Across Heterogeneous Hardware: From Compatibility to Controlled Composition

How this article fits the broader architecture

This article extends the Edge Appliance architecture I have been developing across the series rather than replacing it. It Started With a Kubernetes Upgrade introduced the problem created when Infrastructure, Platform, and Application evolve on different lifecycles. What Is a Product Version? then treated the Product Version as a validated composition rather than a synonym for an application release. Configuration Is Part of the Product added configuration to that product boundary, and From Product Version to Product Package moved from logical composition to the deliverable that carries it.

The Edge AI case adds two new software concerns, the AI Runtime Layer and the LLM Layer, and makes the physical hardware configuration an explicit compatibility boundary. The underlying architectural principle remains the same: independently evolving concerns should keep their own version and ownership boundaries, while the Product Version records what was validated together.

The capitalized terms in this article are intentional architectural terms. Their definitions are maintained in the living Edge Appliance Glossary so that Product Version, Layer, Hardware, Product Manifest, and related concepts can stay consistent as the architecture evolves.

When hardware becomes part of the release model

Consider an industrial assistant that runs locally in a factory. Operators use it to query equipment information, summarize local events, and receive guidance without relying on a continuous cloud connection. The product team wants to ship the same Product Version to two device families. One is based on an x86 processor with one accelerator stack; the other is ARM-based, uses a different accelerator, and has a different memory profile. 

From the user’s perspective, these systems should represent the same release. They should expose the same product behavior, the same application workflows, and the same LLM-backed capabilities. But the path from hardware to application is not the same. The two targets may require different driver families, different runtime releases, different platform components, and different packaging or compilation of artifacts. Containerization does not erase those relationships. The application may be portable, while the software needed to make the accelerator usable is not. 

A useful way to reason about the product is therefore to separate the software into layers and keep the physical hardware outside those layers. In the model used here, the Application Layer contains the product-specific functionality. The LLM Layer identifies the single LLM carried by the Product Version. The AI Runtime Layer executes or serves that LLM. The Platform Layer provides shared operational capabilities. The Infrastructure Layer enables the physical system through the operating system, kernel, drivers, firmware dependencies, and similar low-level components. Beneath them sits Hardware: the physical configuration that constrains which lower-layer combinations are possible. 

This full-stack view is not unique to this architecture. Recent work on industrial Edge AI similarly argues that deployment must be treated as a systems problem spanning hardware and BSP/OS, runtime and acceleration, application/inference, and validation rather than as late-stage model packaging. The product-release question explored here starts from the same systems reality, but asks a different question: how do we preserve one product identity when that full stack cannot be identical everywhere?

Figure 1. The product layers used in this article.

This distinction matters because Hardware is not a lifecycle environment such as development, staging, or production. It is a physical compatibility boundary. An x86 system with one accelerator and an ARM system with another may both operate in the production lifecycle environment, yet they can require different validated software compositions. Treating Hardware as an explicit part of the release model makes that constraint visible instead of hiding it inside deployment logic.

What defines the Product Version

Once multiple hardware-specific compositions are allowed, Product Version can no longer mean “one immutable stack of files.” It needs a stronger semantic definition. In this model, the Product Version represents a functional capability contract: the user-visible behavior that the release promises, together with two logical invariants, the Application Layer Version and the LLM identity and version. If Product Version 4.0 contains Application 7.2 and LLM X 3.1, every supported hardware configuration for Product Version 4.0 must use that same Application Version and that same LLM identity and version.

This creates a useful identity boundary. A change to the Application Version, to the LLM identity or version, or to the functional capability contract changes the logical product and therefore requires a new Product Version. A change that only alters the composition used for a given hardware configuration belongs below the Product Version boundary. That distinction is what allows a driver patch, a different runtime build, or a newly supported accelerator to evolve independently without weakening the meaning of the Product Version itself.

The word logical is important. Fixing the version does not require the physical files to be byte-identical on every target. The same Application Version may be packaged differently or built for a different architecture. The same LLM may require a different executable representation for another accelerator. A runtime may compile target-specific kernels. A model may be quantized to fit the compute and memory envelope of an edge device.

For the scope of this article, quantization is treated as a target-specific representation of the same LLM version when it is derived from the same source model and still satisfies the Product Version’s validation criteria. That is a deliberate architecture decision, not a universal rule for how every organization should version models. The important point is that logical identity and artifact identity are separate concerns. The Product Version tells us which Application and LLM the product contains; the hardware-specific release metadata tells us which exact artifacts were validated and delivered.

Once target-specific transformation is allowed, provenance becomes part of reproducibility. It is not enough to record that a file represents LLM X 3.1. The release metadata should be able to establish which source model it was derived from, which transformation or compilation process produced it, the parameters that materially affect the result, and the immutable digest of the artifact that actually passed validation. Otherwise two files can carry the same logical version label while representing different, untraceable artifacts.

Hardware can still change non-functional behavior substantially. Throughput, latency, power consumption, token-generation rate, maximum context under memory pressure, and concurrent workload capacity may all differ between hardware configurations. The goal is not to make those numbers identical; it is to define the minimum operational envelope a validated composition must satisfy before it can represent the Product Version. Depending on the product, validation gates may therefore include a memory ceiling, a latency threshold, startup time, power budget, or minimum throughput. Detailed SLO design remains outside the scope of this article, but the principle is important: functional identity does not imply performance equality, and hardware variability does not remove the need for release criteria.

I am also intentionally not allowing a different LLM on different hardware configurations. The moment Hardware A can use one LLM and Hardware B another, the release model must answer a harder question: what evidence proves that two different models implement the same product capability? That requires a richer discussion of model evaluation, behavioral equivalence, quality thresholds, and drift. Keeping one LLM identity and version fixed lets us isolate the hardware-composition problem first.

One Product Version, several hardware-specific compositions

Product Version does not need one universal lower-layer stack. It needs one validated composition for each supported hardware configuration.

Figure 2. One Product Version can map to different lower-layer compositions while preserving the same Application Version and LLM identity/version.

In the running example, Product Version 4.0 keeps Application 7.2 and LLM X 3.1 fixed, but the lower layers change for concrete engineering reasons. Hardware A may run successfully with AI Runtime 1.8, Platform 3.4, and Infrastructure 2.1. Hardware B may need AI Runtime 2.0 because support for its accelerator enters there; that runtime may in turn require a newer driver or kernel baseline in Infrastructure 2.3 and a newer Platform release for integration. The LLM may also be represented by a different compiled or quantized artifact while remaining logically LLM X 3.1. This is not arbitrary stack divergence. It is a dependency chain induced by the hardware.

That distinction avoids two common failure modes. The first is artificial version fragmentation, where every hardware-specific lower-layer change forces a new Product Version even though the customer-visible product has not changed. The second is hidden variability, where the Product Version stays stable but deployment logic silently picks whatever combination appears to fit the device. The first makes product versioning noisy and semantically weak. The second makes releases difficult to reproduce and almost impossible to reason about after the fact.

A controlled-composition model keeps the stable identity at the Product Version level and makes target-specific variability explicit below it. The question is no longer “what stack is Product Version 4.0?” It becomes “what validated composition supports Product Version 4.0 on Hardware A, and what composition supports it on Hardware B?”

Compatibility, validation, and release selection are different decisions

Once compositions are explicit, compatibility can be treated as a property of the complete system rather than of individual components. A released Infrastructure version may be valid. A Platform version may be supported. An AI Runtime may be generally available. None of those facts proves that the combination works together on a particular hardware configuration with the pinned Application and LLM.

A candidate composition is therefore only a proposal. Compatibility evaluation checks it against known relationships: hardware-vendor compatibility matrices, runtime support tables, platform constraints, driver requirements, and rules maintained by the engineering organization. A candidate that satisfies those relationships becomes a Compatible Composition. These rules encode what is already known and prevent obviously invalid combinations from consuming validation capacity.

Those relationships should themselves be treated as versioned engineering knowledge. A runtime may support an accelerator only above a particular driver level; a Platform release may require a kernel feature that is unavailable on an older Infrastructure release; a board revision may reduce the memory available to the AI Runtime. When such constraints live only in deployment scripts or in engineers’ heads, composition becomes trial and error. When they are explicit, the release system can treat compatibility evaluation as a constrained selection problem: first eliminate combinations that violate known requirements, then spend validation capacity on the candidates that remain.

That changes the shape of CI/CD. Once Hardware becomes a compatibility dimension, the pipeline is no longer one linear build-test-deploy path. It behaves more like a matrix over Product Version × hardware configuration. Each row may resolve different lower-layer releases and target-specific artifacts, but every row must pass the same product-level release contract. Current Edge AI MLOps guidance describes the same operational pattern: build hardware-specific variants in the pipeline and validate them on representative physical hardware rather than discovering incompatibilities after rollout.

Compatibility, however, is not proof. Edge AI adds a second layer of uncertainty because the product must be validated as a system. A Compatible Composition can satisfy every support table and still fail the Product Version’s functional contract. The model may load but produce unacceptable behavior. An accelerator-specific execution path may introduce numerical differences. An application integration may expose a runtime-specific edge case. Validation must therefore test the concrete composition, including the exact target artifacts, on the hardware configuration for which it will be released. A Compatible Composition whose exact artifacts pass that validation becomes a Validated Composition.

For AI workloads, “validated” should mean more than successful deployment or inference startup. Validation must prove both the functional contract and any required operational envelope on representative hardware. Depending on the product, that can include behavioral regression checks, latency, memory footprint, power draw, token generation rate, or concurrency. Small hardware-dependent output differences can be acceptable, but only within predefined tolerances and acceptance criteria. Compatibility says that a composition is plausible; validation says that this exact artifact set, on this hardware, passed the release contract.

The final step is selection. Engineering may discover several compatible candidates and may even validate more than one successfully. That does not mean deployment should choose among them dynamically. The release process should select one validated composition for that hardware configuration. The decision about what is shipped belongs in release metadata, not in deployment-time heuristics.

Figure 3. Controlled composition separates candidate generation, compatibility evaluation, exact artifact resolution, hardware-specific validation, and release selection.

This separation is important operationally. Compatibility answers what could work. Validation answers what has been demonstrated to work. Release selection answers what the product actually ships. Combining those stages into one opaque deployment process makes failures harder to diagnose and releases harder to reproduce.

Product Manifests make the selected composition explicit

Once a composition has been selected, it needs a durable representation. I use the terms Product Manifest and Product Manifest List for that purpose. In the earlier Edge Appliance architecture, the Product Manifest was the point at which a Product Version’s composition became explicit. Heterogeneous hardware adds another dimension to that idea: the composition must now be resolved for a specific hardware configuration. A Product Manifest therefore describes the selected composition for one Product Version on one hardware configuration. The singular is intentional: one Product Manifest describes one hardware-specific composition, while the Product Manifest List groups the set of Product Manifests that belong to the Product Version. Structurally, this is similar to an OCI image index pointing to platform-specific image manifests. The Product Manifest identifies the selected Layer Versions and, critically, the immutable identities of the concrete artifacts that were validated.

Figure 4. A Product Manifest List maps one Product Version to one immutable, hardware-specific manifest revision per hardware configuration. Logical Application and LLM versions remain fixed while concrete artifact digests may differ.

The artifact identity is necessary because version labels alone are not sufficient for reproducibility. Application 7.2 may have an ARM build and an x86 build. LLM X 3.1 may have two target-specific representations. Both are still logically Application 7.2 and LLM X 3.1, but a release needs to know exactly which artifacts were tested on a particular target. Content digests or equivalent immutable identifiers provide that precision.

Within one published Product Manifest List revision, resolution is deterministic:

(Product Version, Hardware) -> exactly one Product Manifest revision

That rule removes ambiguity from delivery. It also creates a clean boundary between engineering exploration and release selection. Engineering can test many candidates. The released manifest records one selected answer.

The structure resembles an OCI image index at a conceptual level. A logical reference can lead to architecture-specific concrete content, while immutable digests preserve exact identity. The analogy is useful because it separates a human-facing logical name from reproducible content. It should not be stretched further than that: OCI does not define this product-level architecture, and an Edge AI product is more than a container image.

Other delivery systems solve pieces of the same problem at different scopes. AWS IoT Greengrass can select different manifests from platform requirements; Mender carries compatibility metadata and dependency information to prevent deployment to incompatible device types; OCM describes versioned sets of resources and component references. None of these is equivalent to the Product Manifest model proposed here. What they share is the same mature delivery principle: target-specific differences belong in explicit metadata and should be resolved before deployment, not guessed on the device.

A second consequence appears when the product evolves. If Product Version 4.0 can receive a lower-layer patch or gain support for a new hardware configuration without changing its Application Version, LLM, or functional contract, then “4.0” describes a family of released compositions rather than one eternal set of files. Exact reproduction therefore requires the Product Manifest List revision, or an immutable digest of that list, in addition to the Product Version.

Lifecycle becomes part of the architecture

Once target-specific manifests are first-class release objects, lifecycle behavior can be defined explicitly instead of being left to deployment convention. A device first needs to identify its Hardware deterministically from physical characteristics that matter to compatibility. The exact discovery mechanism is implementation-specific, but the architectural rule is not: the same supported physical configuration must resolve consistently to the same Hardware identity. If no matching manifest exists in the selected Product Manifest List revision, delivery should fail closed. Choosing the “closest” supported device would reintroduce unvalidated composition at deployment time.

Lower-layer patches should also preserve the distinction between product identity and released composition. If Hardware A needs a new driver, Infrastructure build, Platform patch, or AI Runtime patch, but the Application Version, LLM identity/version, and functional contract stay unchanged, the Product Version does not need to change. Instead, the release creates a new immutable Product Manifest revision for that target and a new Product Manifest List revision that points to it.

Rollback follows the same principle. A reliable rollback does not reconstruct an old state by taking an old Product Version label and resolving today’s dependencies. It selects a previously validated manifest revision with the original artifact identities. Reproducibility is a property of the exact composition, not of the version label alone.

New hardware can be added in the same way. A later hardware configuration must go through compatibility evaluation and validation against the same Application, LLM, and functional contract. If it passes, a new manifest is created and a new Product Manifest List revision includes it. Earlier list revisions remain immutable.

Figure 5. Product Version 4.0 can remain the logical release identity while immutable Product Manifest List revisions capture patches and newly supported hardware configurations over time.

This revision model gives the product two kinds of stability at once. The Product Version remains meaningful to users because it changes when the logical product changes. The manifest revisions remain meaningful to engineering because they identify exactly what was released at a specific point in time.

The Platform Engineering responsibility

The architecture becomes useful only when Platform Engineering can turn it into a repeatable release process. That responsibility is broader than deployment automation. The platform needs mechanisms to consume compatibility knowledge, expand the hardware matrix, resolve exact artifacts, execute hardware-specific validation, record the selected composition, and make rollback or later reproduction possible. The result is not merely a deployment pipeline; it is a release factory that turns product intent plus hardware constraints into reproducible evidence.

This is also where ownership boundaries matter. Product engineering defines the functional capability contract and decides which Application and LLM belong to the Product Version. Layer owners publish` the versions, artifacts, and compatibility information for the layers they control. Platform Engineering provides the mechanisms that compose those inputs into a release, evaluate constraints, execute validation, and preserve the evidence. Treating the platform as the owner of every compatibility decision would simply centralize domain knowledge in the wrong place; its role is to make that knowledge executable and auditable.

In other words, the platform is not simply moving binaries onto devices. It is operationalizing the relationship between product identity and hardware-specific composition. The output of that process is evidence: which Product Version was delivered, on which hardware configuration, with which Layer releases and artifact digests, and against which validation results.

The transport mechanism comes later. A selected Product Manifest could be delivered through GitOps, an OTA mechanism, or an air-gapped Product Package. Those mechanisms answer how the release reaches the device. They should consume an already selected and validated composition rather than re-running composition logic during deployment.

This is especially important at the edge, where devices can remain in service for years, hardware families evolve independently of application releases, and deployment may happen in constrained or air-gapped environments. The cost of ambiguity grows over time. Six months after a release, “it was Product Version 4.0” is not enough information to diagnose a problem if 4.0 has had three manifest-list revisions and several hardware-specific compositions. A controlled-composition model preserves both the product-level identity and the operational evidence needed to understand what actually ran.

From “can it run?” to “what exactly are we shipping?”

Edge AI discussions often begin with a compatibility question: can this LLM run on this device? That is a useful engineering question, but it is not yet a release model.

A product organization must be able to answer a stronger set of questions. Which Application Version and LLM define this Product Version? Which lower-layer composition is valid for each hardware configuration? Which exact artifacts were validated? Which composition did the release process select? Which immutable revision can be reproduced or rolled back later?

The answer does not require forcing one universal software stack onto every device, nor does it require creating a new Product Version for every hardware-specific change. It requires separating logical product identity from hardware-specific composition, making the variability explicit, validating the complete composition, and recording the exact result.

That is the move from compatibility to controlled composition. It turns heterogeneous hardware from a deployment-time surprise into a release-time decision, and it gives Platform Engineering a concrete mechanism for shipping the same Edge AI product across different hardware without losing determinism or reproducibility.

References and further reading

It Started With a Kubernetes Upgrade: Series origin: why independently evolving Infrastructure, Platform, and Application lifecycles turn an edge appliance into a product-lifecycle problem. 

Configuration Is Part of the Product: Extends Defines Product Version as more than an application release and develops the validated-composition model used here. 

Configuration Is Part of the Product: Extends the product boundary to configuration and separates product-owned configuration from target-environment configuration. 

From Product Version to Product Package: Moves from logical Product Version to the deliverable Product Package, Product Manifest, layer packages, and lifecycle responsibilities. 

Edge Appliance Glossary: Living definitions for the architectural terminology used throughout the series. 

OCI Image Index Specification: External structural reference for mapping a logical reference to platform-specific manifests. 

AWS IoT Greengrass Component Recipe Reference: Platform-specific manifests with artifacts and lifecycle definitions. 

Mender Artifact: Compatibility metadata, Provides/Depends relationships, and payload integrity. 

Open Component Model Component Descriptor: Versioned resources, sources, references, and signatures. 

Edge AI Deployment Beyond Models: A BSP-Aware Systems Framework for Industrial 

Embedded Platforms: A systems-oriented view of Edge AI deployment across hardware, BSP/OS, runtime, application, and validation. 

TinyMLOps: Operational Challenges for Widespread Edge AI Adoption: Research on the operational and lifecycle challenges introduced by distributed Edge AI deployments.