Edge AI means running an artificial intelligence model directly on the device that produces the data (camera, sensor, robot, phone, machine) instead of shipping that data to a server in the cloud. You gain latency, offline operation and privacy; you give up compute power and easy updates. And no, it has nothing to do with the Microsoft Edge browser, even though Google mixes the two up in its suggestions.
The decision comes down to five criteria: definition, how it works, hardware for a first prototype, then the grid for choosing cloud or device, product by product.
Edge AI: definition and scope
📍 The principle in one sentence. The model runs where the data is born, and the answer comes back in milliseconds, network or no network.
Inference leaves the cloud
Red Hat defines edge AI as the use of AI combined with edge computing to process data at or near the point where it is collected. The classic example: image recognition running on the camera itself, with no round trip to a datacenter. IBM adds that companies adopt it to cut latency, tighten security and lower costs.
What stays in the cloud is training. Red Hat puts it plainly: edge AI "could not exist without cloud computing," because devices at the edge have neither the compute power nor the data volume to train a deep model. The cloud trains, compresses, deploys and monitors; the device runs.
Edge AI, edge computing, embedded AI: three terms, one principle
Edge computing means compute moved close to the source, AI or not. Edge AI is the specific case where that compute is an AI model. Embedded AI puts the emphasis on the hardware: the model lives inside an object, often a microcontroller or a dedicated board. Microsoft Edge is a browser, and "Microsoft Edge AI" refers to Copilot inside it. Unrelated.
How edge AI works
⚙️ Two stages. You train at the center, you run at the edge. In between, you compress.
Train at the center, run at the edge
The cycle Red Hat describes has four steps. Training in the cloud, on the data sent back from devices. Deployment of an optimized version of the model onto resource-constrained hardware. Synchronization: the device processes locally and sends back whatever is useful for retraining. Monitoring: the cloud platform tracks the fleet, spots drift and plans maintenance.
An article in The Conversation gives the order of magnitude that justifies all of this: sending large volumes of IoT data to the cloud adds "anywhere from a few hundred milliseconds to several seconds" of delay depending on the network and the volume. For a smart meter, that is fine. For a robot avoiding a person, it is not.
Compressing a model: quantization, distillation, pruning
Three techniques shrink a model that is too heavy for a board running on a few watts. Quantization replaces floating-point numbers with shorter integers. Pruning removes connections that do nothing. Distillation trains a small model to imitate a large one. Red Hat cites the first two among the tools cloud services provide to prepare a model for the edge. Runtimes such as Google's LiteRT (formerly TensorFlow Lite) or ONNX Runtime then execute that compressed model on the board's CPU, GPU or accelerator.
Benefits and limits of edge AI
⚖️ Four gains, three costs. Vendor glossaries list the gains. Here are the costs too.
Latency, offline, privacy, cost at scale
Red Hat lists five benefits: lower energy consumption, lower bandwidth, privacy (sensitive data never leaves the device), security (only the essentials get transmitted) and scalability. The Conversation makes the privacy point with a vivid example: in a connected home, the data from a meter or a light reveals when the place is occupied; processing it locally keeps it from being exposed.
A cloud inference costs you on every request, forever. An on-device inference costs you once, in the price of the board. At ten devices, the cloud wins. At ten thousand, the math flips.
Limited compute, updates, physical security
Compute: a 25-watt board does not run the same model as a server, and Lenovo, in its own glossary, lists the "limits of Edge AI Computing" right alongside its strengths. Updates: a model fixed in the cloud is fixed for everyone; across ten thousand devices, you need a deployment mechanism, versioning and rollbacks. Physical security: the device sits at the customer's site, exposed to theft, teardown and model extraction.
Real-world applications
🏭 Four domains where the millisecond and the sensitive data decide.
Industrial vision and robotics
Quality control on the line, defect detection, meter reading, mobile robot navigation: all tasks where the image cannot wait on a server. Ultralytics documents deploying vision models on NVIDIA Jetson, the most widely used board for this case. In a robot, edge AI is the perception and reflex layer; heavy planning can live elsewhere.
Healthcare, mobility, retail
IBM cites smartphones, traffic updates in autonomous vehicles and connected devices. In healthcare, edge AI processes the image or the signal on the device to protect patient privacy and shorten time to intervention. In retail, in-store video analytics runs locally for the same reason.
LLMs on the device?
Yes, in stripped-down form. LiteRT advertises support for on-device generation, and a French-language Reddit thread compares the "best LLM models at the edge" in late 2025. A compact language model on a phone or a Jetson board handles simple commands offline. For long-form reasoning, the cloud is still the rule.
Hardware and tools to get started
🔌 Three tiers of board, three tools, one prototype budget.
Jetson, Coral, Raspberry Pi: three tiers
| Board | Maker | What for | Price reference |
|---|---|---|---|
| Jetson Orin Nano Super Developer Kit | NVIDIA | vision, robotics, small generative models; onboard GPU, 67 TOPS claimed | 249 dollars, price announced by NVIDIA on 17 December 2024 (down from 499 dollars) |
| Coral Dev Board | inference of compressed models on an Edge TPU accelerator, low power draw | price on coral.ai and at resellers | |
| Raspberry Pi | Raspberry Pi | prototypes, sensors, lightweight models on CPU or an accelerator module | list price on raspberrypi.com |
The Jetson Orin Nano has become the default entry point for vision and robotics since the December 2024 price cut, announced by NVIDIA on its developer blog along with a 1.7x performance gain on generative inference. Coral and Raspberry Pi remain relevant for smaller models and tighter energy budgets.
Edge Impulse, LiteRT, ONNX Runtime
Edge Impulse is an end-to-end platform for collecting sensor data, training, compressing and deploying across the board ecosystem; Qualcomm announced its acquisition in March 2025. LiteRT is Google's runtime, the successor to TensorFlow Lite, deployed on billions of devices according to Google. ONNX Runtime is the open runtime that executes a model exported to ONNX format from PyTorch or another framework. Pick a board, then the runtime its documentation recommends.
What a first prototype costs
A 249-dollar Jetson board, a camera, a pre-trained vision model (Ultralytics provides them) and one engineer-week for the collect, train, export, deploy chain. The first measurable result, frames per second and error rate on your own use case, lands before the end of week two.
When a founder should choose edge AI
🧮 Five criteria, three verdicts. The grid vendor glossaries leave out, because they are selling you the board.
Five decision criteria
- Latency. Does the answer have to arrive in under 100 milliseconds? A robot, a vehicle, a machine tool: yes. A dashboard: no.
- Connectivity. Will the device sometimes be offline? A field, a basement, an industrial site, a medical practice: often.
- Privacy. Can the raw data leave the site? Patient images, the inside of a home, a customer's production line: often not.
- Unit cost at scale. How many devices, how many inferences a day, over how many years? Multiply, then compare against the price of a board.
- Model updates. How often does your model change? Every week: the cloud is simpler. Every quarter: the edge holds up.
Required, desirable, unnecessary: three verdicts per product
- Required when criterion 1 or 2 is true. Mobile robot, bedside medical device, farm machinery, standalone security camera.
- Desirable when criterion 3 or 4 dominates. In-store video analytics, industrial sensors at volume, a mobile app that processes the user's images.
- Unnecessary when none of the five applies. A B2B SaaS processing documents, an internal tool, a text assistant: the cloud wins, and the edge only adds complication.
When the verdict is "desirable," the lowest-risk call is to start in the cloud with an architecture that lets you migrate inference later: the model exported to ONNX or LiteRT from day one, the device sending its data back, the cost per inference measured. The rest of the Tech hub covers the other layers, and if you are building a hardware product, you can apply to swanbase.
The embedded compute layer of Physical AI
In the swanbase pillar on Physical AI, the stack of a system that acts in the physical world starts with "sensors and embedded compute." Edge AI is that layer. It runs perception and reflexes on the machine, while world models and heavy planning can live elsewhere. French humanoid robots illustrate the point: Calvin perceives and balances onboard, in milliseconds.
Edge AI FAQ
What is the difference between edge AI and edge computing?
Edge computing brings compute closer to the source of the data, whatever that compute is. Edge AI is the case where that compute is inference from an AI model. All edge AI is edge computing; the reverse is not true.
Does edge AI replace the cloud?
No. The cloud trains models, compresses them, deploys them and monitors the devices. The edge runs them. As Red Hat sums it up: edge AI could not exist without the cloud.
Do you need a GPU for edge AI?
Not always. A microcontroller can run an anomaly detection model on a sensor; an accelerator like the Coral Edge TPU is enough for image classification; an embedded GPU like the Jetson becomes useful for high-resolution vision, robotics and small generative models.
