The Good Tech Companies - How We Rebuilt Photon's Shared iMessage Routing to Handle 10M+ Messages a Day

Episode Date: July 8, 2026

This story was originally published on HackerNoon at: https://hackernoon.com/how-we-rebuilt-photons-shared-imessage-routing-to-handle-10m-messages-a-day. How we rebuilt Photon's shared iMessage routing to handle 10M+ messages a day — migrating Bun to Node, fixing a memory leak, and adding a Postgres event log. Check more stories related to undefined at: https://hackernoon.com/c/undefined. You can also check exclusive content about #imessage, #messaging-app, #photon-imessage-api, #imessage-ai-agent-platform, #imessage-routing, #message-scaling, #postgres, #good-company, and more. This story was written by: @photonhq. Learn more about this writer by checking @photonhq's about page, and for more stories, please visit hackernoon.com. TL;DR: We rebuilt Photon's shared iMessage routing to handle 10M+ messages a day reliably. We migrated the runtime from Bun to Node.js to unlock OpenTelemetry tracing and kill silent gRPC failures, fixed a Promise.race memory leak that ballooned pods to 5 GB, corrected a binding-cache bug that silently dropped messages, and replaced per-request fan-out with a durable Postgres event log — deleting 5,207 lines of coordination code along the way.

This audio is presented by Hacker Noon, where anyone can learn anything about any technology. How we rebuilt Photon's shared iM-M-plus messaging to handle 10M-plus messages a day by Photon. TLDR. We rebuilt photons shared iM-M-M-plus messaging to handle 10M-plus messages a day reliably. We migrated the runtime from Bun to Node, JS to unlock open telemetry tracing and kill silent GRPC failures, fixed a memory leak that ballooned Podstow 5 gigabytes, corrected a binding cache bug that silently dropped messages, Andre placed per request fan out with a durable Postgres event log, deleting 5,207 lines of coordination code along the way. How We Rebuilt Photon's shared

iM message routing to handle 10 M plus messages a day. Scaling iMessage routing to more than 10 million messages a day broke almost every assumption baked into our original architecture. This is the story of how we rebuilt photons shared iMessage routing layer, migrating from from Bun to Node.js fixing a memory leak, replacing a fragile fan-out design with a durable Postgres event log, and finally making the whole system observable in production. The short version, Photons-free and pro users share iMessage phone numbers, so every message routes through a central proxy that resolves ownership at runtime. Outbound is straightforward, inbound is not. At 10M-plus messages per day across thousands of concurrent users, the legacy fan-out architecture was

re-deriving event ownership on every delivery, coordinating through Redis, and running on a runtime, Bun, that made the whole system effectively unobservable in production. We migrated the runtime from Bun to Node to fix silent GRPC connection failures that were killing long-lived streams without any error logs, and to unlock end-to-end distributed tracing. We then replaced the per-request fan out with a durable Postgres backed event log that resolves ownership once it ingest, and deleted approximately 5,200 lines of legacy coordination code. The binding cache that Gates' inbound delivery was silently dropping messages when upstream services returned transient errors, a subtle bug that looked identical to, no binding exists. Quote dot, the result, a system that is simpler, fully observable end-to-end,

and durable by default. Business users on dedicated numbers were never affected. This was purely about making the shared number path reliable at scale. The routing problem behind our reliability complaints, If you've used Photon's freer pro tier in the last few months, there's a good chance you hit connection drops, delayed messages, or outright missing replies. We heard you in support tickets on social media in Discord. The complaints were real and they all traced back to the same place. Photon gives every business customer a dedicated iMessage number with its own back end. That's the easy path, one number, one customer, one system.

No routing ambiguity. Free and pro users are different. We share iMessage phone numbers across a pool and every message, inbound or outbound, passes through a central proxy service that figures out who it belongs to. That routing layer was the source of every stability issue. Outbound is simple. You have the project ID and the target phone number, so you can deterministically look up which photon number to send from. One lookup, one send.

Inbound is where things get hard. An inbound message arrives with two pieces of information. The photon phone number it was sent to and the sender's phone number. From that pair, you have to resolve which project owns that conversation, find the active stream the client is listening on, and deliver the event. For a deeper look at how we message works under the hood, we've written a full technical overview. Every inbound message requires this resolution, and it has to happen fast. We're reprocessing over 10 million messages a day on the shared number service alone. At that scale, the routing layer is the product.

If resolution is slow, messages feel laggy. If it's wrong, messages vanish. If the system can't tell you what a shepening inside it, you're flying blind when things break. All three of those things were happening, how the legacy system worked, and why it stopped being enough. The original architecture was a fan-out model. Each proxy replica opened live GRPC streams to every iMessage relay instance in the fleet, received raw events from all of them, and resolved ownership per event, per stream, per replica. This worked when the fleet was small and the user, count was modest. But it has a structural problem. It re-derives event ownership on every delivery.

Catch-up replaying missed events when a client reconnects, auto-reply detection, I ask anyone actually receiving this conversation, and subscribe, streaming new events in real time, all depend on walking live relay state instead of reading from a single source of truth. Coordination lived in Redis, Auto-reply claims, presence heartbeats, global catch-up caps, all Redis-backed, all stateful in ways that made failure modes subtle and hard to reproduce. And the whole thing ran on bun. The um that scaling couldn't fix. The first crisis was the service falling over entirely. Each proxy pod was ballooning to two to five gigabytes of heap and eventually crashing. An um loop we were masking by running approximately 32 replicas. Scaling bought time, but it also multiplied the problem. Every replica ran the

background event listener against all 20 iMessage relay instances, so more pods meant more aggregate fan more connections and more memory pressure. A heap snapshot on a relatively small pod, 73 subscriptions, 287 megabytes, told the story to 2 million objects in a linked list, each retaining a delivered event, message content, timestamps, the full payload. The reactions were growing without bound. The root cause was a classic leak in the stream merger. The proxy merges approximately 20 back-end GRPC streams per client subscription using. Each iteration re-es every pending promise. The 19 losers keep their promise object, so a quiet relay instances accumulates one reaction,

holding the full event, for every event delivered by any instance. Forever, the rapper had the same bug, Racing or used re-D it on every tick. The fix was a leak-free merge primitive. Each source gets exactly one reaction that pushes onto a ready cue and wakes the consumer via a single recreated promise. No re-of-pending promises, no unbounded reaction chains. Per source overhead became constant. After the fix, per pod steady-state heap dropped from gigabytes back to approximately 90 megabytes. We scaled the replica count back down. The 32-pod fleet had been compensating for the leak, not for actual load. With the service no longer crashing, we could

turn to the deeper problems. The observability gap. Why we couldn't debug production. Bun was fast, to develop on. But in production, it was a black box, in more waste on one. The most immediate problem was GRPC. Our proxy holds long-lived GRPC streams to every iMessage relay instance in the fleet. Under Bun, these connections would silently die, no error, no log, no event. The stream just stopped delivering. We've been bitten by silent connection failures on macOS before, so we knew how expensive and invisible network bug can be. We'd see symptoms, missing messages, stale state, but nothing in our logs to explain them. Moving to Node resolved this immediately.

Nodes GRPC stack handles long-lived connections reliably and surfaces errors when they occur. The silent connection failures vanished overnight. The deeper problem was observability. Open telemetries instrumentation for HTT Panned GRPC, the same tracing approach we rely on across spectrum, depends on hooking into nodes and internals. doesn't expose these. That meant our calls into upstream services, eligibility checks, opt-in flows, the cloud APIW host Postgres pool was the known root cause of our P99 latency spikes, eight seconds, were completely untraced. We knew the database was slow. We couldn't prove where or why

from the proxies perspective. Every outbound fetch was an opaque blob of time inside or. No client spans, no trace propagation, no way to correlate a slow inbound delivery with the specific upstream query that caused it. This wasn't a minor inconvenience. It meant every production incident was a guessing game. Is it the proxy? Is it the cloud service? Is it the database? Is it a specific replica? We had monitoring, but it was monitoring the wrong layer. Step one. Switch the runtime. We migrated from Bun to Node to Node as the runtime, keeping Bun as the package manager and test runner. The core motivation was unlocking. Nodes is built on Mundici, and Hotel can hoikinto it via to auto instrument every outbound HTTP call.

On Bun, this is on Yoab. The migration itself was surgical. The repo had no build step. Bun ran directly. Node can't do that with TypeScript enums in the generated protobah files, so we added S-bill-backed transpilation, as a runtime loader. Telemetry preloads, previously handled by, moved to Node's flags. The Docker file switched the runtime stage from to while keeping the dependency install stage on Bun.

Tests stayed on, the 79 test suite passed unchanged and gated the release through the same C, CD pipeline we've written about before. The payoff was immediate. With node running, we registered undisciato instrumentation and set a so headers flowed into upstream services. For the first time, a single trace could show. Proxy receives inbound event right pointing arrow resolves binding via HTTP call right pointing arrow upstream service hits postgress right pointing arrow query takes force second's right pointing arrow that's your latency. One trace, end to end, the 8-second P99 was no longer a rumor. It was a specific query on a specific service, visible from the proxy's own spans. The silent monitoring failure, switching to node exposed a second problem we didn't know we had.

The OTLP exporter was using HTTP transport. On nodes' undici stack, Keep Alive sockets can half close when the collector restarts or the refuses a batch. When that happens, every subsequent export attempt reusing that socket silently fails. No error, no retry, no log. The exporter just stops working. We discovered that two out of three production replicas, including the busiest pod, were completely invisible to our monitoring. They were running, serving traffic, processing messages, but no traces, no logs, no metrics were reaching the collector. If an incident concentrated on one of those dark pods, we'd never know. The fix was switching the OTLP exporter from HTT to GRPC. The library owns its channel life cycle and transparently

reconnects after a broken connection. A blip cant permanently wedge the export pipeline. We also hardened the shutdown path. The telemetry flush was previously unbounded, so if the collector was unreachable at Sigderm, the process would own. We added a five-second race with so shutdown is always bounded. The binding cache bug that silently dropped messages, while building the new inbound pipeline, we found a subtle, high-impact bug in the binding cache. The binding cache answers a simple question. Is this sender eligible to receive messages on this photon number? It calls an upstream service, cash is the result, and uses it for the duration of the TTL. The problem, the cache treated every negative response the same way. A definitive. This sender is not provisioned, HTTTTP 4004-420 seconds,

and a transient. The upstream service is down. HTTP 5XX, network timeout, both got cached as. For up to the full cash TTL, every inbound message for that sender would be silently dropped, not because the sender wasn't eligible, but because the cash remembered a transient failure as a permanent answer. The inverse was also broken. Definitive negatives, a sender that genuinely isn't provisioned, weren't cached long enough, so a backlog of messages from an in provisioned sender would re-hit the upstream surface on every retry, 148,000 calls per day in one incident, creating a poison loop that degraded performance for everyone. The fix was a three-state binding result. Positive, cash normally, with a reason, definitive negative, cash at full TTL, and

never cached, always retried. Simple in concept, but it required threading the distinction through every caller in the resolution path. The real fix, a durable inbound event log, the Finding cash fix stopped the bleeding, but the fan-out architecture was still fundamentally fragile. Every delivery re-derived ownership. Every reconnection re-walked live Mac State. Every, is anyone listening? Question depended on presence heartbeats and Redis coordination. We replaced it with a durable, replayable post-gress event log.

The core insight. Resolve ownership once, at ingest time, and persist the result. Everything downstream, catch-up, subscribe, auto-reply, time-out detection, reads from the log instead of re-deriving from live state. How the new pipeline works in jest, a least-gated writer per iMessage Relay Instance Multiplexes catch U-PANs subscribed streams into a sequence-ordered feed, resolves ownership inline, with retry and circuit-breaking for transient failures,

and batch writes events into a Postgres table. Watermark heartbeats track progress. Lag metrics make stalls visible. Catch-up, when a client reconnects and asks for missed events, The system reads directly from the log, a fenced, ordered page scan of that project's event slice. No more replaying the relay. Gap fencing, using snapshots, PG bounce or safe, ensures reads don't return uncommitted rows. Bidirectional cursor translation lets legacy clients with old format cursorses migrate transparently. Subscribe. A shared per pod tail polar fans resolved rows out to bounded per subscriber cues.

Stalled subscribers are shed with after a grace. period, back pressure is explicit, not silent. Live streams start at the fenced project head, so there's no gap between catch up and subscribe. Auto reply and timeout detection. The sweeper, a singleton redrive loop, handles pending row retries, retention deletes, and delivery timeout scanning. Instead of relying on presence heartbeats to answer, is anyone receiving this? It looks at? If the cursor is stale, the subscriber isn't consuming, and the system can trigger auto reply. The auto reply claim store moved from Redis to Postgres, eliminating one more piece of external coordination state. We migrated ingest, catch up, subscribe, auto reply, and timeout detection onto

the log incrementally, validating each layer in staging before enabling it in production, and once everything was stable, cut over fully. The pipeline runs unconditionally, and the entire legacy fan out, merge, Redis coordination sub-system is deleted. The final result. 7 lines added, 5,207 deleted, 42 files change. The system gaw smaller and more capable at the same time. What we gave up, nothing is free. Here's what the new architecture costs. Write amplification.

Every inbound event is written to Postgres. At our volume, that's meaningful, tens of millions of rows per day, with retention deletes running on a schedule. The previous system held no persistent state for event delivery, so storage cost was zero. Postgres is a critical dependency. The log makes Postgres load-bearing for inbound delivery, not just for configuration and metadata. We mitigated this with an explicit read replica pool, and by making the catch-up path replica aware, but a Postgres outage now directly impacts message delivery. Complexity in the ingest path, lease gated riders, circuit breakers, watermark heartbeats,

gap fencing, these are non-trivial components. The operational surface moved from many replicas independently fan out to a coordinated ingest pipeline with lease arbitration. The former was simple and unreliable. The latter is complex and durable. We judged these trade-offs worthwhile, persistent state that you can query, replay, and monitor beats ephemeral state that you can only hope was delivered correctly. Lessons. Your runtime is a tracing decision. We chose Bun for developer velocity. We didn't realize that choice made production unobservable until we'd scaled past the point where, add more logging, was sufficient. If your system processes millions of events a day, the runtime's compatibility with your observability stack isn't a nice to have,

its load-bearing infrastructure. Cash semantics are failure semantics. The binding cash bug was invisible precisely because it looked like correct behavior, the cache returned a value, the system acted on it, and the message was dropped. The only signal was subsence. Messages that should have arrived but didn't. Three state results, yes, definitely. No, I don't know, are more work to implement, but they're the only way to cash safely in a distributed system where upstream failures are routine. Resolve once, persist, read many. The fan-out model's fundamental mistake was Ray deriving ownership on every delivery. That's fine when you have 10 users. Add in million messages a day, every derivation is a chance

for inconsistency, a latency tax, and an operational blind spot. Persisting the resolution at ingest time turned every downstream consumer into a simple log reader. Try it yourself. These changes are live now. If you tried Photon's free tier before and hit reliability issues, connection drops, missing messages, delayed replies. That's the system we just rebuilt. The shared iMessage service is faster, more stable, and fully observable. We are confident enough in it to invite you back. Get started for free at Photon. Codes. Build your first agent with the Photon docs. Need a dedicated number, our business plans give you your own iMessage number with a separated back end, zero shared routing. We're weighing your options first. We compared every way to build an

iMessage agent in 2026. Questions or feedback, reach us on Discord or on our contact page. Related reading Frontier agent interaction on iMessage, a technical overview, how iMessage actually works under the hood. We found a ticking time bomb in MacOS TCP networking, another deep production reliability fix. traces for reliable spectrum messaging, the open telemetry tracing approach behind this rebuild. Astrolabe. Managing Mac fleets at scale. How we run the relay fleet this routing depends on. See, CD in the age of AI, the pipeline that shipped this migration. How to build an iMessage agent in 2026. Every approach compared, the options, side by side. Thank you for listening to this Hackernoon story, read by artificial intelligence. Visit Hackernoon.com to read

Read, write, learn and publish.