FIRST Chesapeake Knowledgebase

3. Monitoring and Troubleshooting

11 min read

Contents

Status Page
Logs
Jobs
Health-Check Banners
Restream and Push Problems
What Clients Don’t See
YouTube Channel Re-Authentication

Status Page

The admin Status page is a live view of the whole cluster — start here first when something
seems off. It shows:

  • Live Events — events happening today, which are live right now, and aggregate stream metrics
    (average bitrate, ingress/egress transfer rate) pulled from the same per-stream measurements
    stream-auth records for health checking.
  • Services — one card per deployment (the twelve listed in
    Architecture Overview), each showing a health state (healthy / degraded / down / unknown),
    desired vs. ready pod counts, and per-pod detail: restart count, age, phase, and — when a pod is
    unhealthy — a reason like CrashLoopBackOff, OOMKilled, or ImagePullBackOff. Each card has a
    Restart button.
  • Cluster Resources — aggregate and per-node CPU/memory usage, pod counts, and node conditions
    (Ready, and pressure flags for memory/disk/PID/network). This needs metrics-server installed;
    without it, this section shows “Metrics unavailable.”
  • Storage — recordings (NFS) and object storage (S3-compatible) usage and clip counts, database
    size, and Redis memory usage, plus a table of every PVC with its capacity and used/available
    bytes.
  • Queue Depths — the length of each of the eight job queues described in
    Architecture Overview.
  • Worker Scaling — for each background worker: current pods (ready/desired), its configured
    min–max replica range (see
    System Settings Reference), its queue depth, and a utilization bar that turns red once a
    worker is over 80% of its max range with a nonzero queue — the visual cue that it’s time to raise
    the max, not just wait.

rtmp-ingest’s own MediaMTX metrics aren’t exposed across the cluster, because the series are
labelled with stream keys; the stream figures on this page come from stream-auth’s measurements.

Logs

The admin Logs page streams pod logs directly from the cluster (with a live-tail option), plus a
pod-events feed (starts, restarts, scheduling failures) — no separate tooling needed to see what a
service is doing. This is system-admin-only; there’s no client-facing equivalent at any role, since
pod logs can span every Client’s activity.

Stream keys are scrubbed from every line before it reaches the page: an ingest path live/<key>
is shown as live/<redacted>. stream-auth doesn’t log refused publish paths at all.

Don't Share MediaMTX Connection Listings

MediaMTX’s own connection list (outside this page) can contain credentials. Don’t paste it into
tickets, chat or anywhere else.

Jobs

The admin Jobs page shows queue depth for each pipeline stage (see
Architecture Overview), plus these actions:

  • Pause All on queue:upload-jobs — “Pause all uploads?” moves every waiting upload job to the
    deferred queue.
  • Release Paused and Release Mode Deferred on queue:upload-deferred (shown as Upload
    Deferred    ), each appearing only when that kind of deferred upload is waiting.
  • Clear on a queue, confirmed with “Clear queue?”.
  • Requeue All / Clear All for failed clips, uploads and Blue Alliance submissions in bulk,
    plus a per-item Requeue.
  • Force Release on a single deferred upload. This takes two confirmations: first “Force release
    upload?” (answer Continue), then a “Type RELEASE to confirm” dialog whose field reads “Type
    RELEASE (all caps)” and whose button is Force Release. Anything other than RELEASE, or
    cancelling, gives “Force release cancelled” — “Nothing was released.”

Needs Review is counted as its own stage here alongside pending/processing/failed, so clips held
for being too short don’t read as a pipeline failure — see
Clips, Uploads and Signage
for approving or discarding them. A backlog in pending/processing/failed usually means a downstream
service is stuck, crash-looping, or scaled down too far. “YouTube uploads paused” here means either
the system-wide quota limit or the current Operation Mode — see
System Settings Reference.

Unlike the Status and Logs pages, Jobs isn’t backed by Kubernetes at all — the queues themselves are
Redis lists, and every action on this page (pause and release, clear, bulk requeue/clear,
force-release a deferred upload) goes through the dashboard, not a cluster command.

Health-Check Banners

The same health-warning banners operators see (see
Troubleshooting) come from explicit checks the platform runs against each live
stream, only while its restream mode isn’t “none”:

  • Incompatible video codec (error) — the incoming video codec isn’t one YouTube’s ingest
    accepts (H.264, AV1, HEVC, or VP9). The banner names the actual codec detected and suggests
    switching encoders or enabling re-encoding.
  • Incompatible audio codec (error) — “Incoming audio codec is codec, but YouTube’s ingest
    requires AAC. Restreaming will likely fail — switch your encoder to AAC, or enable Restream
    Re-encode in Settings.”
  • Keyframe interval too long (warning) — only checked if keyframe checking is turned on (see
    System Settings Reference, off by default). YouTube requires a keyframe interval of 4
    seconds or less (2s recommended); the banner reports the measured interval, or “at least Ns” if
    the platform’s 15-second measurement window didn’t catch two full keyframes.
  • YouTube-reported ingest issues — for broadcasts using the Livestream Control Center path,
    YouTube’s own reported ingest health issues are surfaced verbatim, prefixed “YouTube reports:”.

Enabling Restream Re-encode Enabled (see
System Settings Reference) skips the whole ingest-compatibility check —
the codec checks and the keyframe-interval check — since a re-encoded stream is normalized
regardless of what came in.

Two things are not banners:

  • Holding. When OBS drops during a push, operators see only the amber Holding — standby slate
    on YouTube     row, with “Resumes automatically when OBS reconnects · gives up in m:ss”. No
    health banner appears.

  • YouTube healthy but not playing. YouTube can report the ingest as healthy (“excellent”) and
    still never play the stream; at the 2026-09-26 Icebreaker the cause was missing audio. Two
    status lines, in the Control Center and the OBS dock, now point at it rather than a banner:

    • “No audio track from OBS — YouTube may not start playback without audio”, on the incoming
      stream’s status when OBS sends video with no audio track.
    • “YouTube hasn’t started playback after time. It is receiving our stream (ingest health:
      health) but not playing it; viewers see a spinner. Check the preview in YouTube Studio.”, once
      today’s broadcast has been starting on YouTube for more than 60 seconds. If YouTube isn’t
      receiving the push either, it reads ”…, and it isn’t receiving our stream. Check that the push
      is running.”

    Neither proves viewers hear audio: nothing checks that audio actually leaves
    in the push. Still confirm audio and video by watching an unlisted test broadcast (see
    Pre-Event Testing and Test Broadcasts).

Source Codec Rules

  • A source with no video (audio only) is never pushed; the push errors.
  • Copy mode (re-encode off): H.264, AV1, HEVC and VP9 video all pass through to YouTube
    unchanged, including AV1, HEVC and VP9 sent by OBS’s Enhanced RTMP output. VP8 is refused, with
    “Source video codec VP8 can’t be stream-copied to YouTube’s Enhanced RTMP ingest — enable
    re-encoding to push this source”. Audio passes through unchanged, so it must be AAC —
    the audio banner above warns otherwise.
  • Re-encode on: video is re-encoded within its own family — H.264 → H.264, AV1 → AV1, VP9 and
    VP8 → VP9 — and HEVC → H.264, because the image has no HEVC encoder. Audio is always re-encoded
    to AAC at 128 kbps / 44.1 kHz.
  • Re-encode skips the whole ingest-compatibility check, keyframe check included.

If an operator reports a warning you can’t explain from the Operators troubleshooting page, check
the Status and Logs pages for the specific service next — usually rtmp-ingest or restream-pusher,
depending on the warning.

Restream and Push Problems

These push errors and refusals are ones a system admin can act on:

  • “The restream worker has no internal stream read secret configured (…), so it can’t read the
    source — ask an administrator to deploy it”     — every push fails until the deployment has its
    internal stream read secret. Deployment details are in the kube-match-splitter repository’s
    DEPLOYMENT.md, for people with repository access.
  • ”🔒 Turned off by system administrators” on Start Push and Go Live — Restream
    Integration Enabled     is off (see
    System Settings Reference).
  • Pinned-broadcast errors. A push stays on the broadcast it started with, so changes to that
    broadcast surface as push errors:
    • “This push’s YouTube broadcast was deleted — Stop and Start Push to use today’s broadcast”
    • “This push’s broadcast was replaced — Stop and Start Push to switch to the new one”
    • “The YouTube broadcast for date has ended (status: status) and can’t receive video again
      — replace it with a new broadcast on the event page to push again”
    • “The YouTube broadcast for date was created on a different YouTube channel than the event is
      now set to — set the event back to that channel, or delete that broadcast and create one on
      the new channel”
  • Failing pushes. When the source is still connected but the push to YouTube keeps stopping,
    operators see a red Push failing — retrying with the reason, the attempt number and the next
    wait (“Attempt N of 5 failed; trying again in Ns”). Waits are 5, 15, 30 and 60 seconds, and a
    push that runs for three minutes starts the count again. After five failures in a row the push
    errors with the reason and “Gave up after 5 attempts in a row — detail” and waits for an
    operator’s Start Push. The reasons are:
    • “YouTube closed the connection after Ns (source video codec codec)” — YouTube accepted the
      connection and then dropped it. Check the source codec against Source Codec Rules and the
      broadcast in YouTube Studio.
    • “The push process stopped after Ns (source video codec codec)” — the push ended for another
      reason; check the restream-pusher log.
    • “The ingest server isn’t handing over the source’s video (codec codec), so there is nothing
      to send to YouTube” — the ingest side isn’t serving that codec; check rtmp-ingest.
  • Hold give-ups, after Restream Hold Timeout (s):
    • ”… — Source did not return within Ns — giving up” (OBS dropped and didn’t come back)
    • “Source never started publishing within the configured hold timeout — giving up” (the push
      started before OBS ever connected)
  • Stale live events. An orphaned live event with no session heals itself to in_progress
    after 2 minutes, and an in_progress event completes itself once its stream window has ended
    (see Event Auto-Complete in
    System Settings Reference).
    There’s no manual force-close; Reopen is only for completed events.

For a stream YouTube reports as healthy but that won’t play, walk through
YouTube Shows Healthy but Won’t Play.

What Clients Don’t See

File internals — storage paths (efs_path, s3_key) and the processing worker (worker_id) —
are shown to system admins only. Client members, Client Admin included, never see raw error
text for clips or uploads. They get one of a fixed set of reasons instead, and you see the raw
text in the same place:

What Client members seeWhat it stands for
”Shorter than the scheduled match; check before approving”A clip held because it came out shorter than the match by more than the clipping tolerance
”Held for review after an automatic check”A clip held for any other reason
”Clip processing failed”A failed clip
”A processing attempt failed”An error left on a clip in any other status
”No YouTube channel is set up for this event”The event has no upload channel
”The YouTube channel needs to be reconnected”The channel has no usable credentials
”YouTube’s daily upload quota was reached”The upload hit the daily quota
”Replaced by a replayed match”The clip was superseded by a replay
”The clip file could not be read for upload”The clip’s file couldn’t be fetched from storage
”YouTube rejected the upload”YouTube’s API returned an error
”An earlier attempt failed; it will be retried”A queued or running upload that failed before
”The upload failed”Any other upload error

Impersonating a member shows you these reasons too, not the raw text. Readonly viewers see
“The push failed” instead of the raw push error, and no stream remote address.

So when a Client reports an error, look the item up on the admin side, without impersonating, to
see the real cause.

YouTube Channel Re-Authentication

A linked YouTube channel that fails to refresh its access token is flagged as needing
re-authentication; uploads and restreams using it will fail until an admin re-links it (see
YouTube and Slack Integrations). This is a per-Client action — a system admin can see that a
channel is flagged, but only that Client’s admin can complete the re-link.

After a failed token refresh the channel isn’t retried automatically; it stays flagged until a
Client Admin re-links it, and re-linking clears the flag by itself.

Backlinks