Validating a vehicle Human-Machine Interface (HMI) traditionally depends on physical head units. That hardware is expensive and limited in supply, which makes it a scheduling bottleneck for a distributed engineering team. The objective here was to remove that dependency: deliver the infotainment stack to a browser tab — an interactive screen, bidirectional audio, and adb access — so any engineer can reach a device from a URL.
This is achievable with an Android Automotive OS (AAOS) emulator running in the cloud; Cuttlefish streams display and audio to the browser over WebRTC. Two problems separate a working proof of concept from a team-wide capability: the media path (specifically, audio delivery) and provisioning (serving on-demand instances without incurring bare-metal cost around the clock). This article covers both.
Why the host must be bare metal
Cuttlefish runs a full Android system image in a virtual machine and requires KVM. Hardware virtualization on AWS is exposed only on .metal instance types; without /dev/kvm the emulator will not start. This constraint drives both cost and architecture — a single metal host runs at several USD per hour — so the design is dominated by one question: how to avoid paying for idle capacity.
Root cause: WebRTC media travels over UDP
The initial access model followed a conservative default: no public IP, with the emulator console reached through an SSH-over-SSM proxy. Display rendering and input worked; audio did not, in either direction, and the failure was deterministic rather than intermittent.
The root cause was structural, not a misconfiguration. WebRTC negotiates media as UDP/RTP, because real-time audio and video cannot tolerate the retransmission and head-of-line blocking inherent to TCP. An SSH-SOCKS tunnel over SSM forwards TCP only. Signaling therefore completed and the session appeared healthy, but the negotiated media stream had no transport to the browser. Audio was not degraded — it had no viable path.
Generalized takeaway: when WebRTC is in scope, the first design question is the media transport path — specifically, how UDP reaches the client. Signaling, the operator console, and TLS all run over TCP and can mask a missing media path.
The resolution inverted the access model while preserving the security posture. The host is assigned a public IP so UDP media reaches the browser directly, and inbound access is constrained at the security group to an explicit list of source CIDRs. Critically, the allow-list is mandatory: an empty value fails terraform apply at validation time, making accidental exposure structurally impossible.
| Control | Function |
|---|---|
| Public IP with direct UDP route | Delivers WebRTC media (bidirectional audio and video) to the browser |
| Security group scoped to specific CIDRs | Restricts access to trusted source addresses; never 0.0.0.0/0 |
| Non-empty allow-list validation | Fail-closed by construction: without source ranges, the apply is rejected |
| Let's Encrypt certificate (DNS-01) | Trusted certificate on the operator console — no browser warning, and a secure context for microphone capture |
With the media path in place, audio was delivered in both directions. Verification is defined against a measurable criterion rather than subjective listening: headless Chromium (Playwright) reads the peer connection's getStats(), and both packetsReceived and totalAudioEnergy must increase. Rising packet counts with near-zero energy indicate silence originating in the guest (mute or volume state), not a transport fault — a distinct diagnostic path on the AAOS framework side.
Scaling from a single host to on-demand provisioning
A single working emulator is straightforward; serving the full team without manual console operations is not. In the target design, terraform apply alone yields a ready device — a zero-touch boot in which a first-boot systemd unit fetches the latest custom AAOS image from S3 and launches the instance. The substantive improvement was a self-service API layered on top:
- REST API with two endpoints:
POST /emulatorsinitiates provisioning and returns anidplus a status URL;GET /emulators/{id}returns status and, once ready, the connection bundle. Both are API-key gated. - Step Functions as the orchestrator, using the Standard (not Express) workflow type — a cold metal boot takes roughly 16 minutes, which exceeds the 5-minute Express limit. Separate workflows handle provisioning, teardown, and reinstall.
- CodeBuild executes the underlying
terraform apply/destroy, so infrastructure is created and removed programmatically per workspace. - SES sends a "device ready" notification with the connection link, removing the need to poll the console.
Bin-packing and slot-allocation concurrency
Provisioning one metal host per user is not cost-viable. Instead, multiple emulators (CVDs) are bin-packed onto a shared host, and a new host is created only when no free slots remain. This introduces a concurrency hazard: slot allocation is a read-modify-write over instance tags (read used slots, select a free one), so two concurrent requests can select the same slot, and two "no host yet" requests can both trigger creation of the same workspace.
A single DynamoDB table makes both operations atomic through conditional writes:
- Slot claim:
PutItemwithConditionExpression = attribute_not_exists(lock_id)— exactly one concurrent caller succeeds; the others retry against the next slot. - Host lock: an equivalent conditional write per workspace, so concurrent host-creation requests cannot collide on shared Terraform state.
This mirrors the reasoning behind formal guarantees in autoscaling work: rather than assuming collisions are unlikely, the system enforces an invariant — a single guaranteed winner, with deterministic back-off for the remainder.
Cost control
Because bare-metal capacity is the dominant cost, scheduling governs spend (EventBridge Scheduler):
| Mechanism | Behaviour |
|---|---|
| Weekday nightly stop | Stop rather than terminate — the booted image and /etc/letsencrypt persist on EBS, so restart is fast; instances remain stopped over weekends |
| Optional morning warm-up | Pre-starts the host ahead of working hours; a request would otherwise start it on demand |
| Hourly reaper | Revokes expired ephemeral security-group rules — access is granted per session, not permanently |
| Architecture selection | x86_64 (96 vCPU / 192 GB) or arm64 on Graviton — approximately $2.3 vs $4 per hour, given a compatible AAOS image |
Generalizable takeaways
- Transport protocol is an architectural decision, not an implementation detail. The conservative tunnel was secure but incompatible with real-time media; meeting the UDP requirement outweighed the convenience of routing everything through SSM.
- A public IP and a strong security posture are not mutually exclusive, provided fail-closed behaviour is enforced in code (an empty allow-list is a hard error) rather than left to operator discipline.
- A shared, high-cost resource is a concurrency problem. Bin-packing without atomic locking is a race; DynamoDB conditional writes resolve it without a dedicated coordination service.
- Verification should be programmatic. Subjective confirmation is not a criterion; increasing
packetsReceivedandtotalAudioEnergyin a headless browser is.
End to end — from AAOS images through the WebRTC media path to self-service provisioning — the platform is defined in Terraform and operated by a small set of Lambda functions. The design principle is consistent throughout: correctness enforced by construction — fail-closed network access, atomic slot allocation — and validated by measurement rather than assumption.