Camera Uptime Monitoring

[jmealo]

Describe what you are trying to accomplish and why in non technical terms
I want to be able to see if some of my cameras are flakey, but I only care about the video steam. Answering ICMP ping or an HTTP request doesn't tell me whether the camera stream is up in Frigate.

Describe the solution you'd like
I would like an uptime percentage for each camera, as well as knowing the longest occurrence of the stream going away, along with the total time the stream was gone. If all of the streams go away, it can/could be inferred that it's a network issue.

Describe alternatives you've considered
I'm used UptimeKuma to monitor things, however, the events when an HTTP probe or ICMP ping fail don't always correlate with a loss in the stream in Frigate.

Additional context
I would be happy to help contribute to this feature.

[NickM-27]

Frigate 0.17 already has a dedicated topic for camera health status. This would be quite complicated as it would involve keeping track of separate roles (detect vs record). In general it seems like a lot to keep track of as users would want different behavior, for example:

• Persisting between Frigate restarts
• Storing the reason in logs for a camera going down
• Which camera streams went down

IMO this seems like something that would be better done outside of Frigate where the user can customize the behavior for their exact needs more readily.

[NickM-27]

And for the primary use case of this request, we are going to be creating a "known problems" feature that would include alerts on cameras which have repeatedly gone down / been unreliable, which is a more direct way to solve that issue.

[jmealo]

@NickM-27: Thanks for the quick reply. I hadn't thought of checking the MQTT. I'll see if that's sufficient and/or extend it to be, and then post back with a solution for monitoring outside of Frigate, if I go that route.

I do think that it'd be nice to know that from within the Frigate UI (or Home Assistant), however, any functionality that can exist outside of Frigate, likely should.

So to recap:

• Monitor this using MQTT externally (extend if necessary)
• Use the prometheus endpoint at /api/stats/prometheus (extend if necessary)
• Review ffmpeg log parsing, make sure it's capturing anything we'd be interested in
• Consider packaging this as a Home Assistant add-on (or, if it's just primitives, possibly a template?)
• UptimeKuma already supports MQTT, it could be extended to serve this use case if it cannot already.

I suppose in Home Assistant, if we map MQTT to sensors, and create a sensor group, that could also detect/infer a network outage using native HA features.

I'd be interested in contributing to "Known problems" I have a few flakey cameras (one wired and infrequent, one wireless and frequent) at present that would be perfect for real world testing.

[strb173]

I think using frigate_camera_fps as a gauge metric is less reliable than using a Prometheus counter. Counters must only increase, so if processing stops or stat collection breaks, the counter simply stops moving — making hangs and collection failures much more visible. Because they accumulate every event, they can also catch intermittent drops that might otherwise be missed between scrapes.

Counters will reset to zero when the camera / ffmpeg process restarts, which actually makes it easier to detect when a camera has gone down — metrics like camera_frames_processed_total or detection_frames_processed_total will repeatedly reset instead of continuing to grow. FPS can still be derived using something like rate(camera_frames_processed_total[5m]).

I’ve even seen a case where a stable FPS value was still being reported after the ffmpeg process had hung. That situation would have been caught automatically with a counter, though I haven’t been able to reproduce the issue.

[NickM-27]

To be clear my recommendation was to use the new health status in 0.17.

[strb173]

That's fair, perhaps there are stats improvements that could be expanded in the future. I just happen to use prometheus/alert-manager and detecting some issues is tricky with the current set of metrics.

@jmealo you can use the mqtt monitor in UptimeKuma to monitor the topic frigate/<camera_name>//status, for the value "online" instead of your http or icmp probe.

[YBonline]

@jmealo

• Consider packaging this as a Home Assistant add-on (or, if it's just primitives, possibly a template?)

If you monitor the Home Assistant stream status for unavailable, it will already show you when the stream is unavailable. I have a camera that occasionally misbehaves and needs a reboot, and I have an automation in HA that cycles the POE power when the stream goes to unavailable for 2 minutes. HA Templates should be able to do what you want.

[fancygaphtrn]

FWIW, I use go2rtc stream api to monitor cameras. For me it works very well. I use rest sensors in home assistant and parse the json from http://xxx.xxx:1984/api/streams

Example sensors:
`rest:

• resource: http://xxxx.xxx:1984/api/streams
  scan_interval: 10
  sensor:
  • name: go2rtc consumers NVR Cam 1
    value_template: >-
    {% set camera = "nvr_cam_1" %}
    {% if camera in value_json %}
    {% if value_json[camera].consumers is list %}
    {{ value_json[camera].consumers | selectattr("user_agent", 'defined') | selectattr("user_agent", "search", "^FFmpeg Frigate") | list | count }}
    {% else %}0{% endif %}
    {% else %}0{% endif %}
  • name: go2rtc producers NVR Cam 1
    value_template: >-
    {% set camera = "nvr_cam_1" %}
    {% if camera in value_json %}
    {% if value_json[camera].producers is list %}
    {{ value_json[camera].producers | selectattr("user_agent", 'defined') | selectattr("user_agent", "search", "go2rtc") | list | count }}
    {% else %}0{% endif %}
    {% else %}0{% endif %}
    `
    It will then give you nice graphs of cameras