Fleet platform

Each taxi publishes its position, battery and status over MQTT. A Node-RED flow drops readings outside the service area, readings with a repeated timestamp and readings where nothing moved. It then tags each one with one of nine zones. The simulator corrupts 6% of its output on purpose, so the filter has real work to do.

22 Aug 21:01:47 - [info] [debug:ingest stats]
{
  in: 58000,
  out: 54650,
  dropped: 3350,
  badRange: 1250,
  dupTs: 1028,
  unchanged: 1072,
  dropPct: 5.8,
[...]
--- state service, the subscriber side ---
{"vehicles":2000,"idle":2000,"zones":{"z1-0":255,"z2-1":246,"z0-2":223,"z0-0":218,"z2-0":225,"z2-2":204,"z0-1":202,"z1-2":214,"z1-1":213},"counters":{"events":758685,"claims":281214,"claimConflicts":19602,"noCandidate":0}}
--- state service before the run, for the delta ---
{"vehicles":2000,"idle":932,"zones":{"z1-0":248,"z2-1":255,"z0-2":222,"z0-0":218,"z2-0":225,"z2-2":199,"z0-1":207,"z1-2":214,"z1-1":212},"counters":{"events":704035,"claims":281214,"claimConflicts":19602,"noCandidate":0}}

58,000 readings went in and 54,650 came out. The state service counted 758,685 minus 704,035 events, which is the same 54,650. The code was fine; the first attempts disagreed because of how I measured. Node-RED only reports every ten seconds, so the run now restarts it first and waits 25 seconds after the last reading before it reads either side.

A load harness sends ride requests in five stages, from 200 a second up to 3,000 and back down to 800. Each request goes on a queue. A dispatch worker takes eight at a time, asks the state service for idle cars nearby and claims one. A claim can lose to another worker, and then it tries the next car on its list. Across the two runs below that happened on about one claim in twelve: 27,429 lost claims out of 321,340.

I ran the ramp twice, once with a single worker and once with an autoscaler free to run up to six.

stage  asked   actual  sent   matched  unmatched  p50    p95    p99    max    matched/s  mean m  p95 m
1      200/s   198.1   4000   4000     0          19     32     35     40     198.1      238     497
2      800/s   792.7   16000  14643    1357       25     43     48     53     725.4      1205    6519
3      1600/s  1584.2  32000  18531    13469      27     46     52     58     917.4      939     4091
4      3000/s  2969.9  90000  29017    60983      27     45     50     55     957.5      1048    5046
5      800/s   787.7   16000  14024    1976       24     40     48     65     690.4      1032    4797

sent 158000  matched 80215  unmatched 77785  never answered 0  arrived after their stage 560

Both runs flatten near 950 matches a second, and both leave about 61,000 requests without a car at the 3,000 stage. p95 latency held at 45 ms there and no request went unanswered, so the worker kept up with everything it was given. Each match holds a car for 300 ms, and the simulator then keeps it on a trip for half a second to a second. I read the limit as free cars running out, though my counters could not tell an empty candidate list apart from a car another worker claimed first. Either way it was not the matching service, which is the part I spent the project scaling. The autoscaler never went past three workers.

I ran the same script again on 27 Oct 2026, from a copy of the repo, without Postgres, on a laptop busy with other work. Ingest came out identical, because the simulator is seeded. Matching peaked at 968.6 a second with one worker and 971.2 autoscaled. Same ceiling.

Each match also sends a trip to a trip service, which writes to Postgres 16 in batches: once a second, at most 500 rows. The matcher was producing about 960 a second. On 24 Aug I ran the ramp once more and watched the writer instead.

--- trip service counters, one line every 5 s ---
trips written=0  dropped=0  buffered=0
[...]
trips written=45980  dropped=0  buffered=29695
trips written=48480  dropped=0  buffered=30783
trips written=50980  dropped=0  buffered=29003
trips written=53480  dropped=0  buffered=26503
[...]
trips written=78480  dropped=0  buffered=1503
trips written=79983  dropped=0  buffered=0

--- rows actually in postgres ---
79983

The buffer peaked at 30,783 rows and took about a minute to clear after the load stopped. The worker matched 79,983 rides, the trip service wrote 79,983 rows, and the table held 79,983. The rows arrived late, but they all arrived. The cap is not Postgres: the 500 is a constant in my own flush loop.

I moved the part the project is about. The ride queue became SQS, and the dispatch worker and state service became Fargate tasks on ARM64 at a quarter of a vCPU. The state service was seeded from a snapshot of the fleet. Ingest, the broker, the monitor and Postgres stayed on my laptop.

Two CloudWatch alarms watch the number of rides waiting. Above 200 for a minute, a step policy adds dispatch tasks, more of them the further over it is. Below 50 for two minutes, another removes two. The count stays between one and six.

The run asked for 25 requests a second for 90 s, then 200 for 240 s, then 25 for 240 s. That is far below the 3,000 a second on the laptop, on purpose. It is there to show the policy working, not to measure speed, so I never put its numbers beside the local ones.

[...]
    {
        "start": "2026-08-26T02:31:42.015000+10:00",
        "cause": "monitor alarm sit314-backlog-high in state ALARM triggered policy sit314-out",
        "status": "Successfully set desired count to 6. Change successfully fulfilled by ecs."
    },
    {
        "start": "2026-08-26T02:29:42.043000+10:00",
        "cause": "monitor alarm sit314-backlog-high in state ALARM triggered policy sit314-out",
        "status": "Successfully set desired count to 4. Change successfully fulfilled by ecs."
    },
[...]

stage  asked   actual  sent   matched  unmatched  mean m  p95 m
1      25/s    29.6    2700   2700     0          262     563
2      200/s   198     48000  28027    19973      3764    11118
3      25/s    29.6    7200   6171     1029       535     1655

sent 57900  matched 36898  unmatched 21002  never answered 0  arrived after their stage 15360

The task count went 1, 4, 6, then back to 4, 2 and 1, and AWS records the alarm and policy behind every step. 36,898 of 57,900 requests were matched. Almost all of the 21,002 misses are in the 200-a-second stage, and fleet size explains them on its own: a match holds a car for 20 s here, so 198 a second needs about 3,960 cars, and there were 2,000. That points the same way as the laptop runs. My poller saw a peak of 19,187 waiting, while CloudWatch's highest one-minute average was 15,071. The alarm reads the average, which is part of why scaling out lags.

Each of three demo vehicles has its own X.509 certificate, under one IoT policy scoped by thing name. To test it, I pointed v0000's certificate at v0001's topic. IoT Core does not send back an error, it just drops the connection, so the proof is in AWS's own log. The state service has no task role, accepts traffic only from the dispatch security group, and wants a bearer token that reaches the container as a Secrets Manager ARN.

$ aws logs filter-log-events --log-group-name AWSIotLogsV2 --start-time <this run>
  Publish-In   AUTHORIZATION_FAILURE  clientId=v0000    fleet/vehicle/v0001/telemetry  x4

2026-08-25T20:56:22 SEC_PROBE GET /stats with no bearer header -> HTTP 401 {"ok":false,"error":"unauthorized"}
2026-08-25T20:56:22 SEC_PROBE GET /stats with the bearer header -> HTTP 200 {"vehicles":2000, [...]

Three certificates and not 2,000 is a real gap. The simulator drives the whole fleet over one connection at about 1,000 messages a second, and IoT Core allows roughly 100 per connection, so a real fleet needs a connection and a certificate per car. Mosquitto, which stayed local, has no TLS and no device identity at all.

The worker takes eight requests at once, then lets each one race for its own nearest car. As an extension I tried a batched hill climb adapted from a 2017 DiDi paper. It collects requests for 250 ms and assigns cars to the whole batch together.

Across the loaded stages, mean pickup distance fell from 1,068 m to 814 m, about a quarter. The cost was latency: p95 went from 51 ms to 112 ms. The full comparison is in the post Batched assignment in a driverless-taxi fleet.

Move to AWS early. I left it until the end, so the experiments ran on two shapes of infrastructure instead of one. I also scaled the wrong thing: the matcher never became the bottleneck, and the fleet most likely did.

Two limits are my own doing. The trip writer's 500 rows a second is a constant I hard-coded. And the scaling policy reads raw queue depth. Once enough tasks keep up, the depth sits near zero however busy the system is, so the low alarm could remove tasks that are still needed. On this run the scale-in came after the load had dropped, so I never saw it, but I would scale on the arrival rate instead.