SiteSense: a video agent for site wellbeing, built in a day

On Saturday 17 October I spent the day at the Real-Time Video Agents Hack in London, run by tokens& as part of the VAST Builders Challenge, with NVIDIA, VAST Data, CoreWeave and Weights & Biases behind it. The brief was simple to say and hard to do: build an agent that understands video and acts on it. We had about seven hours.

Our team built SiteSense, a prototype for worker wellbeing on construction sites. The idea came from a real gap. Sites already have cameras, but nobody watches the footage unless something goes wrong. Meanwhile the early signs that someone is struggling, moving more slowly, favouring one side, taking more pauses on a task they usually do easily, are visible on camera long before anyone mentions them.

So the question we asked was: can a video agent notice that kind of change in a worker, compared with that same worker earlier, and suggest that a supervisor checks in? Not diagnose anything. Just notice, and ask a human to look.

How the pipeline works. My teammate built the core evidence pipeline. YOLO finds the people in each frame, ByteTrack follows each person through the clip, and the system cuts short windows of video around each tracked person. NVIDIA Cosmos turns each window into an embedding, so the system can find earlier footage of the same person doing the same task. Then a Cosmos reasoning model looks at the earlier clip and the current one side by side, and returns structured notes: what visibly changed, what else could explain it, and what it can't tell.

Tools of the day. We wrote the code in Cursor, working directly on the virtual machine the organisers gave each team, which was already connected to the shared model services running on CoreWeave GPUs. We also used Grok along the way.

What I built. I built the live layer that turns that pipeline into something a site manager could actually use: a progress view so you can see each stage working, alert bubbles and counters as findings come in, a single-camera Live page that replays footage as if it were streaming, and a Feeds command centre with a grid of camera feeds, a site map and a findings panel. I also added two modes: Detect, which shows what the models see, and Compare, which puts the earlier and current clips side by side for a reviewer.

I also used my own agent framework, Alfred, throughout the day. It researched the problem space, sourced stock footage that matched our construction-site scenario, and wrote and reviewed code alongside us. In a seven-hour build that matters: the bottleneck stops being typing and becomes deciding what to build next.

The most important feature is the human. It's easy to build a demo that says "this worker is fatigued". We deliberately didn't. A model can misread what it sees, a similarity score is not a health score, and a camera can't know someone's circumstances. So SiteSense never sends alerts to management on its own. It suggests a check-in, shows the evidence, and a person reviews the footage and decides. The first clip of a worker only sets a baseline and makes no claim. If there's no fair earlier clip to compare against, the system says so and stays quiet. If the model returns something malformed, the job fails instead of pretending it worked.

That design isn't caution for its own sake. It's what makes a system like this usable at all. Nobody will trust a tool that confidently labels people. People might trust a tool that says "here's what changed, here's what it might mean, please take a look".

What it is, and what it isn't. SiteSense is a one-day prototype. The demo ran on a short simulated recording, not months of real site footage. It shows that the pieces work together and that the review workflow makes sense. It doesn't prove anything about wellbeing, fatigue or identity, and turning it into something real would need repeated recordings with trustworthy timestamps, proper consent, and identities confirmed by people rather than guessed by models.

What I took away. Video agents are now practical to build quickly. Detection, tracking, embeddings and video reasoning are services you can wire together in an afternoon. The hard part isn't the models. It's deciding when the system should speak, when it should stay quiet, and how it hands over to a person. The interface carries the trust. Side-by-side evidence and plain notes did more for credibility than any accuracy number would have. Agent-assisted development changes what a small team can attempt in a day.

The code is on GitHub at github.com/afareed007/ConstructionWellbeing (my work is on the my-feature branch), and the event page is at luma.com/vastlondon. Thanks to my teammates and to the organisers for a great day.

If you're thinking about putting AI on top of footage, documents or any data where a wrong answer affects a real person, that human-review layer is exactly the work I do. Get in touch via the contact form below.