Architecting
systems at scale.

Vehicles connect through a clustered MQTT broker behind a load balancer, landing on arbitrary nodes. A subscriber on one node may need a message published on another, so every message is replicated across the whole cluster.

That fan-out grows with node count, so past a point adding nodes stops buying throughput. Commands were also modelled as jobs an API polled for completion — paying the replication cost and a polling round trip.

Every vehicle’s topics are namespaced by its own identifier, and no vehicle ever subscribes to another vehicle’s topics.

So for those topics the replication is work that can never pay off. The cluster was spending nearly all its capacity distributing messages to nodes that provably had no interest in them.

On connect, an Orleans grain representing the vehicle is activated on the node holding that connection. The distributed grain directory lets any service address a vehicle by id and be routed straight to the owning node, with no broadcast.

The grain relays down the socket it already holds. Delivery lands in tens of milliseconds, first attempt about 90% of the time; the rest fall back to polling.

At MarqMetrix I had built an emulator of our spectrometers and the APIs they exposed, so a partner could download it and be writing code in minutes — no shipping hardware, no remote rig, and far fewer security and stability problems.

The unlock was a question rather than code. They were integrating over an MQTT library, so I asked whether the library itself actually mattered to them. It did not — so we shipped a .NET package that spoke to the emulator directly.

Six months of stalled integration became one week. They flew a developer to Seattle to validate against real hardware; it passed in two hours.

I went into Volkswagen knowing physical cars would not scale. They could get us vehicles, but the quote was six months — and a fleet is still shared, scarce and in one building, so access serialises: developers wait, and automated testing barely happens because it needs a car too.

Pure local emulation could not answer it either. You cannot emulate a whole vehicle's hardware on a laptop, and autonomous-driving and sensor work needs a world — something to drive through, collide with, and read sensors against.

A global service where you spawn a vehicle in your own region from a CLI — the same VW.AC command line wired into the build and test pipelines. Spawning dedicates micro-VMs emulating the compute ECUs and the services running on them, networked privately to each other the way the in-vehicle network is.

The vehicle is also placed in a 3D world, so sensors and collisions produce real data. A UI lets you pull a handle, unlock a door or switch on lights — and those actions propagate messages across the ECUs exactly as a real car would.

A vehicle arrives in five minutes instead of six months, and costs what the VMs cost — not a production vehicle, let alone one with a test harness built into it.

Sales teams uploaded license inventory spreadsheets, all of them in the same few days at month end. Every row was one software license; all anyone wanted was a count per product.

The app was cookie-cutter: web front end, .NET middle tier, SQL back end, with the business logic exhaustively encoded as stored procedures. Uploads went through the middle tier, held whole in memory, then pushed to blob storage — so the progress bar finished and users still waited. Then every row of every workbook was bulk-inserted into one shared table and summarized in SQL.

I identified SQL as the bottleneck. The PM’s fix was to scale the database SKU up for month end and back down after — which in practice means copying the database, with the replication lag and cutover risk that brings.

Plan B was to leave it on the expensive SKU permanently, on the reasoning that it was Microsoft’s cloud and Microsoft’s project, so the cost wasn’t real. I had to explain that every project carries its own cloud budget. Neither plan touched the actual problem.

Upload straight to blob storage and take the middle tier out of the data path entirely. A new blob raises an event onto the service bus. A pool of background workers picks up the event and streams the workbook.

Because only the summary mattered, the workers tally each license as the row goes by and write a single summary record at the end. The raw rows never enter SQL, and the shared table disappears — the data was per-upload all along.

The first implementation leaned on elaborate LINQ queries and took about 24 hours to chew through ~150 MB. A second developer got it to 12 hours.

Both shared the same root flaw: load everything into memory, then do clever parsing on top. The work scaled with total input size, so the only way out was a bigger machine.

Pick one file as the lead. Read a fixed window of it — on the order of 10,000 rows — into a dictionary keyed on the join column. Then stream every other file start to finish once, completing any record that matches the window.

Advance to the next window in the lead file and repeat. When the lead file is exhausted, every combination has been seen.

52 seconds for the same 150 MB — marginally above what it cost to read and write that much data on the drives of the day. The compute bottleneck was gone; what was left was the disk.

Memory is now bounded by the window, not the input. The same code handles multi-gigabyte files that would have taken the earlier versions off a cliff.