Contents
Services
The Pipeline
Restream Data Path
Live Preview
Event Status and Sessions
Operation Modes
Services
The platform is a set of independently-deployed services, each doing one job. They fall into three
groups, in roughly the order footage and data actually move through them:
Main chain — recording a match and getting it uploaded:
- rtmp-ingest — a MediaMTX server that receives the incoming RTMP stream and records it to
disk. It also serves the HLS live preview and gives in-cluster readers the stream over RTSP. Its
metrics aren’t exposed across the cluster, because the series are labelled with stream keys. - stream-auth — authorizes incoming streams against stream keys, and measures stream health
(codec, keyframe interval, bitrate). It also authorizes reads of a stream, and allows only
three kinds: the key-free live preview over HLS, in-cluster RTSP, and RTMP carrying the internal
stream read secret. Every other read is refused, including any public RTMP pull. A publish is
accepted only inside the event’s publish window — from one day before its start time through its
end time — and only to a non-archived event belonging to the stream key’s own client; anything
else is refused with “No active event”. - first-poller — polls the FIRST Events APIs for schedules and match results, and is what
actually notices a match has ended. - clipper — cuts per-match clips from the recordings.
- s3-mover — moves finished clips to object storage.
- uploader — uploads clips to YouTube (subject to Operation Mode and quota — see
Operation Modes). - blue-alliance — posts FRC results/video links back to The Blue Alliance.
- slack-notifier — sends event notifications to Slack; it’s fed from any stage above that needs
to report a success or failure.
Side paths — driven by the same event configuration, but independent of the recording chain:
- restream-pusher — pushes the live restream toward YouTube. A reconcile loop drives each push
from its desired state. It reads the source over RTMP using an internal stream read secret; RTSP
is used only as the fallback for VP8 video, which needs re-encode (an RTSP read drops AAC audio,
so audio is re-encoded on that path). In YouTube-channel mode it finds the destination through
today’s broadcast (see Restream Data Path); the platform reuses one YouTube ingest
stream per stream-key/channel pair, which can serve up to three broadcasts. If the deployment
has no read secret, every push fails with “The restream worker has no internal stream read
secret configured (…), so it can’t read the source — ask an administrator to deploy it”. - signage-controller — drives event signage displays.
Control plane:
- api-server — the API and auth backend the web dashboard and OBS plugin talk to. It also carries
out manual admin/operator actions (retries, re-clips, force-releases) by placing jobs on the same
queues below. - web-frontend — the dashboard UI. Its nginx also proxies the live preview (see
Live Preview).
The OBS plugin talks to api-server’s device-auth endpoints; it isn’t a separate deployed service.
These twelve names are also exactly the deployments the admin Status page and restart action know
about — see
Monitoring and Troubleshooting.
The Pipeline
Work moves between services through eight named job queues (Redis lists, not a Kubernetes concept),
each corresponding to one pipeline stage:
| Queue | Fed by | Consumed by |
|---|---|---|
queue:clip-jobs | first-poller, at match end | clipper |
queue:s3-move-jobs | clipper | s3-mover |
queue:upload-jobs | s3-mover, unless Operation Mode defers the clip | uploader |
queue:upload-deferred | s3-mover, when Operation Mode defers a clip; uploader, when YouTube quota would be exceeded | uploader, on release |
queue:tba-jobs | uploader, after a successful FRC upload | blue-alliance |
queue:slack-notifications | multiple services, on success/failure | slack-notifier |
queue:restream-jobs | the Livestream Control Center | restream-pusher |
queue:signage-jobs | event/sign configuration changes | signage-controller |
rtmp-ingest only receives and records the stream — it never touches a queue itself; first-poller
is what actually notices a match has ended and starts the pipeline. One other thing worth knowing:
if a match is replayed before its original clip has uploaded, first-poller removes (not adds) that
clip’s stale jobs from queue:clip-jobs, queue:s3-move-jobs, queue:upload-jobs, and
queue:upload-deferred, so the superseded footage doesn’t consume upload quota — on the Jobs page
this shows up as a job disappearing, not a new one appearing.
Roughly, the main recording/upload chain runs:
OBS/encoder → rtmp-ingest (+ stream-auth) → recordings on disk
first-poller (at match end) → queue:clip-jobs → clipper → queue:s3-move-jobs → s3-mover
→ queue:upload-jobs → uploader → queue:tba-jobs → blue-alliance
queue:slack-notifications ← any stage above → slack-notifier
Side paths:
Livestream Control Center → queue:restream-jobs → restream-pusher
event/sign configuration change → queue:signage-jobs → signage-controller
rtmp-ingest --RTMP (internal read secret; RTSP only for VP8)--> restream-pusher
--RTMP--> YouTube ingest (today's broadcast)
browser → web-frontend nginx (preview-check) → rtmp-ingest HLS preview
A backlog in any queue usually means its consumer is stuck, crash-looping,
or scaled down too far — see
Monitoring and Troubleshooting for how to check queue depth and
System Settings Reference for adjusting worker replica counts.
Restream Data Path
Reading the source. restream-pusher reads the event’s stream back from rtmp-ingest over RTMP,
presenting the internal stream read secret. RTSP is used only when the source video is VP8, which
can only be pushed with Restream Re-encode Enabled on; an RTSP read loses AAC audio, so on that
path the audio is always re-encoded.
Finding the destination. In YouTube-channel mode the push goes to today’s broadcast: the
current, non-replaced broadcast for today’s date in the event’s timezone. There is no fallback to
the event’s stream key or channel — with no broadcast for today, Start Push is refused. Once a push
starts, it stays pinned to that broadcast for its whole life, even past midnight; replacing or
deleting the broadcast doesn’t move a running push (see
Restream and Push Problems).
In manual-key mode the push goes straight to the restream key’s RTMP destination.
Holding. If OBS drops, the push keeps YouTube’s broadcast open with a generated standby slate
(a black frame with silence) so YouTube doesn’t end the broadcast for lack of data, and re-probes
the source every 3 seconds. It resumes by itself when the source returns, and gives up after
Restream Hold Timeout (s) (restream_hold_timeout_s, default 600). The same timeout bounds how
long a push waits for a source that never started.
Limited Audio and Playback Checks
The platform warns when OBS sends no audio track and when YouTube hasn’t started playback a minute
after going live (see
Monitoring and Troubleshooting),
but nothing checks that audio actually reaches YouTube. A green status means YouTube is
receiving. Confirm audio and video with an unlisted test
broadcast — see
Pre-Event Testing and Test Broadcasts.
Live Preview
The live preview in the Control Center is rtmp-ingest’s HLS output, proxied by web-frontend’s nginx.
api-server issues a signed preview token that names the stream key by its id (never its value) and
is valid for one to two hours. Before proxying a preview request, nginx checks the token with
api-server; nginx keeps redirects relative, so the preview works behind TLS.
Who may preview is set by Stream Preview (stream_preview_enabled) on the Ingest tab. When it’s
on, any member of the owning client, Readonly included, can preview. When it’s off, only system
admins can; everyone else sees ”🔒 Live preview turned off by system administrators”.
Event Status and Sessions
An event’s status moves through:
scheduled— nothing is coming in.pre_live— the encoder is publishing before the start time; shown as “Pre-event test”. A
pre-event test doesn’t restream, except to the event’s setup-day test video on the day before
the event.live— at the start time, a watcher (every 30 seconds) promotes a pre-event test to live,
with no reconnect and no split in the recording.in_progress, thencompleted.
A feed that ends before the start time returns the event to scheduled. Stuck states heal
themselves: a pre_live event with no session for over 2 minutes goes back to scheduled, and an
orphaned live event with no session for over 2 minutes moves to in_progress. An in_progress
event whose stream window ended more than the Event Auto-Complete grace ago (120 minutes by
default), with no recording session open, moves to completed; a live event is never completed
this way, so a running feed is never cut. Reopen takes a completed event back to
in_progress, and is refused for any other status.
When the stream ends, the session and the event are closed together. A late “stream ended” from an
OBS connection that has since been replaced can’t close the newer session. Watchdog baselines (such
as the recording-health check) are kept per session.
Operation Modes
A system-wide Operation Mode setting — Record Only (record_only), Lite Mode
(lite_mode), or Full Power (full_power) — is shown as a banner in both the admin and client
dashboards. It’s easy to assume this gates recording itself; it doesn’t. What it actually gates,
precisely:
| Mode | Recording & clipping | YouTube upload | Playlists | Blue Alliance video submission |
|---|---|---|---|---|
record_only (default) | Runs normally | Deferred (queued, not sent) | — | Never |
lite_mode | Runs normally | Sent | Skipped | Sent |
full_power | Runs normally | Sent | Created/managed | Sent |
In every mode, ingest, recording, and clipping happen the same way — a clip file is always produced
and moved to storage regardless of Operation Mode. What changes is only whether the resulting
upload job is actually sent to YouTube’s API. In record_only, that job sits in
queue:upload-deferred until either the mode changes or a system admin force-releases it — see
Monitoring and Troubleshooting.
An unrecognised mode value is treated as record_only. Test-event uploads also follow the mode:
they upload only when it isn’t record_only (see
How Test Matches Are Scheduled).