frames

Lays out the frames of a video span as one contact sheet, each tile labelled with its time.

Tern wrote this one. Tern is a Claude agent in my lab, and finished it 2026-11-04. It's public at https://github.com/mackenziebowes/frames, MIT licensed, and it runs with bun, ffmpeg, ffprobe and chromium.

This page is for people who build with agents and want their agent to see inside screen recordings, bug videos and demos.

Here's the problem. An agent reads images, and a video isn't one. frames <video> writes one PNG where every sampled frame is a tile, labelled under it with its time, like 0:32.00. So the agent looks at one image and knows when each thing on screen happened.

It's one sheet every time. More frames make smaller tiles, never a second sheet.

The coarse-to-fine loop

Run one sheet over the whole video.
Find the tiles where the picture changes in a way that matters, and read their times.
Run frames again with --from and --to around those times and a smaller --every.
Repeat until the tiles are close enough to say what happened and when.

A worked example

The video here is a 39-second screen recording of a 3D pond scene in the Godot game editor, cropped to the 3D view only - 1280 by 706 pixels, 30 frames a second.

The first command covers the whole video:

frames pond.mp4 --out pond-coarse.png

That gives 21 tiles, 4 columns by 6 rows, from 0:00.00 to 0:39.02 - one tile every 1.95 seconds, on a sheet of 2400 by 2226 pixels.

A contact sheet of 21 tiles from a 39-second recording of a 3D pond, each labelled with its time from 0:00.00 to 0:39.02. The camera starts above the pond and flies over trees, skims low over the water, then turns away so the pond sits at the top of the tile with grass and blue sky below at 0:31.25 and 0:33.21, and returns over the whole pond at 0:35.16.

The camera starts above the pond and flies over trees, then goes low over the water from 0:21 to 0:29, then lifts to a view where the pond is small at the top of the tile and grass and blue sky fill the rest, at 0:31.25 and 0:33.21, then comes back over the whole pond at 0:35.16 and settles at the water's edge by 0:37.11.

The tiles at 0:31.25 and 0:33.21 show the pond shrinking to the top edge, and the camera is back over it at 0:35.16. I want to know what happens in between, so the second run narrows to 0:29 to 0:35:

frames pond.mp4 --from 0:29 --to 0:35 --every 0.5 --out pond-zoom.png

13 tiles, 3 columns by 5 rows, one every 0.50 seconds, on a sheet of 2394 by 2400 pixels.

A contact sheet of 13 tiles from 0:29.00 to 0:35.00, one every half second. The camera is low over the water at 0:29.00, rises, and by 0:31.50 the pond has slid to the top edge with flat green ground and blue sky below. Tiles from 0:32.00 to 0:33.50 show mostly empty ground and sky. At 0:34.00 the whole pond and its trees are back in view.

At 0:29.00 to 0:30.50 the camera is low over the water and then rises. At 0:31.50 the pond has slid to the top edge and the rest of the tile is flat green ground and blue sky. The tiles at 0:32.00, 0:32.50, 0:33.00 and 0:33.50 show mostly empty ground and sky, with the pond small or cut off at the top. At 0:34.00 the whole pond and its trees are back in view.

So the answer to "what is on screen between 0:32 and 0:33.5" - the camera has flown off the pond and is looking at empty ground and sky.

How the tool began

At 13:39 on 2026-11-04 I recorded a screen video where something froze around the 31st to 34th second. A coarse sheet located the span, and a second sheet with 91 tiles - one per frame at 30 fps, covering 0:31.00 to 0:34.00 - showed a dialog appearing at about 0:32.30, gone by 0:33.00, and a panel empty from 0:33.07. That recording shows my agents' panes, so it's not published.

Options

--from A and --to B set the span. --every STEP sets the sampling step. --out sheet.png sets the output file. --width PX and --height PX set the sheet size. --force allows overwriting an existing output file.

Limits and safety rules

At most 600 tiles per sheet.
A sheet of at most 16 million pixels - the default is 2400 by 2400.
--out must end in .png and must not exist unless --force is given.
ffmpeg may open only plain files, so a playlist named x.mp4 cannot pull in other files or URLs.
A run keeps everything in one temp folder and removes it on exit or Ctrl-C.

Testing

This is a small, single-person test, and what follows is Tern's self-report of a run on 2026-11-04:

A normal run on the 1:03 to 1:10 span of my video produced a sheet and left no temp folder behind.
Two runs at once both produced correct sheets.
--out ~/.bashrc, --out x.txt and an --out in a missing folder were all refused - an existing file was refused until --force.
A sheet size of 8000 by 7000 was refused.
A run cut short with SIGINT left no temp folder and no running ffmpeg.
Not tested: ffmpeg reading past the end of a video, and a hostile playlist file. Those are covered by the code only.
Kestrel, another Claude agent, read the code and found faults. They were fixed in commit dadc439, and Kestrel passed the fixes.

Install

git clone https://github.com/mackenziebowes/frames
ln -s "$PWD/frames/frames.ts" ~/.local/bin/frames

It needs ffmpeg, ffprobe and chromium on the path, and bun to run it. MIT licensed.

Commands

frames: frames <video> [--from A] [--to B] [--every STEP] [--out sheet.png] [--width PX] [--height PX] [--force]: one PNG of the span, never a second sheet; read a video coarse to fine by narrowing --from and --to and shrinking --every.