Module 2: cereal & msgq

Predict

controlsd needs to know the current speed and steering angle, which card computes. plannerd needs to know what the camera model saw, which modeld computes. None of these processes import each other or call a function across process boundaries — a Python import can't reach into a different OS process. Before reading on: what's your best guess for how one process's output becomes visible to another with a completely separate memory space?

Run

Start a live source of messages: the demo route, replayed over the same bus a live system publishes on.

./tools/replay/replay --demo --no-vipc -c 3 -s 45 &

Then, in the same venv, run a plain subscriber script against it — no manager, no other sunnypilot process running:

import time
import cereal.messaging as messaging

sm = messaging.SubMaster(['carState', 'modelV2', 'longitudinalPlan', 'radarState', 'selfdriveState', 'carControl'])

deadline = time.monotonic() + 15
seen = set()
while time.monotonic() < deadline and len(seen) < 6:
    sm.update(100)
    for k in sm.data.keys():
        if sm.updated[k] and k not in seen:
            seen.add(k)
            print(f"=== {k} (valid={sm.valid[k]}) ===")

That's the entire subscribing mechanism: name the six messages you want, poll sm.update(), and check sm.updated[k] to catch each one exactly once. The rest of the real script is per-message field formatting (which field to print for each of the six types) with no new pub/sub concept in it — full listing on the Source page if you want it.

Observe

=== carState (valid=True) ===
  vEgo=27.209 m/s  gasPressed=False  cruiseState.speed=33.528  standstill=False

=== modelV2 (valid=True) ===
  frameId=983  laneLineProbs=[0.56, 0.922, 0.393, 0.027]  leadsV3[0].prob=0.115  action.desiredAcceleration=0.306

=== selfdriveState (valid=True) ===
  state=enabled  enabled=True  active=True  alertType=''

(cross-check: Log page, lines 1-8; full six-message capture there too)

Six named messages, published by processes this script never imported and has no other relationship to, arrived at a plain 30-line script with no special privileges. sm.updated['carState'] is True exactly once, the moment that field's message is fresh; on later sm.update() calls the cached value stays readable via sm['carState'] but updated goes back to False until the next one arrives, which is what lets the loop above print each message exactly once.

Explain

Two separate things share the name "cereal" in this codebase, and conflating them is the usual point of confusion:

1. The schema. Every message name (carState, modelV2, ...) is a typed struct defined in a .capnp schema file under cereal/, compiled by Cap'n Proto (an existing open-source serialization format cereal is built on, not sunnypilot's own invention) into Python and C++ bindings. The schema is what makes sm['carState'].vEgo a typed float access rather than a raw byte offset you'd have to remember.

2. The transport. cereal.messaging's SubMaster/PubMaster classes are a thin Python (and C++) API over msgq: named blocks of shared memory that any process on the same machine can attach to directly, without going through the kernel's normal socket or file APIs. This is why sm.update(100) above returned data from replay, a completely separate process, with no sockets, ports, or files involved: replay's PubMaster and this script's SubMaster both memory-map the same named region.

Alongside the schema, cereal/services.py gives every named message a small amount of metadata:

_services: dict[str, tuple] = {
  # service: (should_log, frequency, qlog decimation (optional))
  # note: the "EncodeIdx" packets will still be in the log
  "gyroscope": (True, 104., 104),
  "accelerometer": (True, 104., 104),
  "temperatureSensor": (True, 2., 200),
  "deviceState": (True, 2., 1),
  "touch": (True, 20., 1),
  "can": (True, 100., 2053, QueueSize.BIG),  # decimation gives ~3 msgs in a full segment
  "controlsState": (True, 100., 10, QueueSize.MEDIUM),

(cereal/services.py lines 22-31.) 81 services are registered this way in total; one more of interest a little further down in the same dict:

  "modelV2": (True, 20., None, QueueSize.BIG),

(cereal/services.py line 72.) The frequency is a nominal rate (can at 100 Hz, modelV2 at 20 Hz), and decimation controls how much of a high-rate service actually survives into the small "quick log" (qlog) that's kept for every drive versus the full log kept only briefly: can's 2053 means roughly 1 in 2053 CAN messages makes it into the qlog, which is why the comment notes "~3 msgs in a full segment" — the qlog is a sampled index into a drive, not a full recording.

Diagram: everyone talks to the bus, not to each other

dump_msgs.py is drawn as just another subscriber, at the same level as controlsd and loggerd — because that's exactly what it is. No process in this diagram has a direct connection to any other; every arrow terminates at the bus.

One instant, source vs. reconstruction

Notice replay --demo never ran modeld, radard, plannerd, or controlsd at all. What this script saw as modelV2 and carControl are not being recomputed live from the camera; they are the exact bytes those processes wrote during the original recorded drive, played back through the same bus at (approximately) their original relative timing. replay's own description is blunt about this: it "replays all messages logged during the use of openpilot" so that "analyz[ing] and visualiz[ing] system behavior" looks the same as if it were live — but it is a replay of a past decision, not a new one. That distinction matters for Module 3, which traces a single one of those replayed instants across five of these six messages to see how one becomes the next.