Track 02 · Physical AI
Vision That Doesn't Blink
Perception systems that keep working in mines, plants and on roads where conditions are hostile.
Every challenge follows the same structure. Each track has a shared theme. Within it, each partner company has an arena: a real problem from their operations, a concrete build assignment, and a defined deliverable. Pick one arena and build for it. Whatever you build in one arena will be instructive to every partner in that track.
Background
Computer vision is transforming heavy industry and mobility, but the physical world degrades it constantly: dust, vibration, and darkness in mines; fog, snow, and ice on lenses; glare and weather on roads. A detection system that fails silently is worse than no system at all, because people and processes learn to depend on it. The task in every arena is the same: vision that performs under degraded conditions, and knows when it can no longer be trusted.
Choose your arena
The problem
Steel scrap from tunnel reinforcement falls into the ore flow during blasting and destroys crushers downstream. Every incident costs hours of lost production, and today the scrap is caught too late or not at all.
Build
A detection system that spots steel scrap in the excavator bucket, on the conveyor belt, or both — early enough that it can be removed before reaching the next stage of the process.
Deliverable
A working detection prototype running on LKAB process video, plus an alert interface showing the operator what was detected, where, and when to act.
Dataset
The problem
Cameras and lenses degrade under fog, dust, snow, ice, and smoke. When they fail, they fail quietly — and every AI system depending on them goes blind without anyone noticing until something downstream breaks.
Build
A system that detects a failing camera or lens across degraded conditions, notifies a process operator, and auto-generates a work order describing the root cause and suggested actions.
Deliverable
A working detection prototype running on publicly available degradation videos, plus an operator interface showing camera health status and generated work orders.
Data sources
Publicly available videos showing camera/lens degradation (fog, dust, snow, ice, smoke, glare, scratches). Find suitable clips on the internet and use them as input to your detection model.
The problem
On the road, a turn indicator is intent. Misreading it means misreading what the driver is about to do — and an autonomous system that gets this wrong at scale, or burns too much compute getting it right, cannot ship.
Build
A system that estimates turn indicator state (left, right, hazard) for all annotated vehicles in the centre frame of ZOD sequences, with accurate start and end timing, a low false detection rate, and efficient compute.
Deliverable
A working detection prototype running on the Zenseact Open Dataset, with a results view showing indicator state, timing, and confidence per vehicle. Your pitch must explain the concept behind your estimation approach.
Data sources
Zenseact Open Dataset subset created for this challenge.
Dataset
Dataset instructions
A subset of the Zenseact Open Dataset has been created for this challenge to keep the size manageable: 37 sequences from the forward-looking camera, each 20 seconds at 10 frames/s (200 images per sequence). The file name of each frame includes a timestamp.
Most sequences contain turn indicator usage. Your task is to detect where each indicator starts and ends.
There are no turn-indicator annotations in the dataset — that is part of the challenge.
Vehicle annotations are only provided for one of the 200 frames in each sequence (the middle frame), so they may not be very useful for this challenge.
If your solution needs a vehicle detector, you can try Meta's SAM 2 (ai.meta.com/research/sam2/) or Meta's DINO (github.com/facebookresearch/dino).
Helpful resources
Shared requirement for all teams
Your system must report what it detected, where it is, and with what confidence — and show how it knows when its own vision can no longer be trusted.
Pitch
A short pitch covering the end user, where in the process your solution deploys, how it triggers action, and what it would take to run in production.