Ingest pipeline
Atlas takes in about 40 60 million events a day from the mobile clients and the edge workers. This document is the design of the path an event takes between arriving and being queryable. accept and queryable, and the back pressure that keeps it there.
Goals
- Never drop a well-formed event. A backlog is recoverable; a gap is not.
- Bound the tail. p99 from accept to queryable under 90 60 seconds.
- Keep the operational surface small enough for one on-call engineer.
An earlier draft promised exactly-once delivery from the edge to the warehouse.
Stages
An event passes through four stages. Each one is idempotent and keyed on the event id, so a replay of any prefix is safe.