Caeliai

ArchiveAgent

Can a small model name the house, season and year of a runway photo?

Theia is a fine-tune of Kimi K2.6, an open model from Moonshot AI. It was trained on Tinker with 5,025 examples drawn from 7,512 photos of 14 houses: one pass, 80 minutes, $18.35. Every model below got the same photos and the same instructions. A decline counted as wrong.

One run per model, scored by us, with model-made labels. We found and fixed three mistakes in our own scoring; they are listed on archiveagent.si.
Figure 1. The Runway Test: 287 photos from 41 shows kept out of training. House, season and year all had to be right. Kimi K2.6 is the model Theia started from.

Four tests

TestTheiaBest otherWho
Runway, 287 photos56.4%44.9%Kimi K3
Product house, 150 photos100%98.7%Claude Opus 5.5
Blind runway, 50 photos56%26%Gemini 3.1 Pro
Exact product, 75 products50.7%58.7%Claude Opus 5.5

Theia costs about 0.14 cents per photo. Per photo it gets fully right, that is about 17 times less than Claude Opus 5.5.

Where it falls short

It loses the exact product test, and with 75 products the gap could be luck. On houses it never trained on, it names the house 57% of the time, against 90% on the 14 it knows. Part of its edge on made-up product names comes from having seen those product lines in training. It always answers, even when unsure, and on studio product photos the season and year are mostly guesses.

Data and code

The test photos, every model’s answers and the scores are on Hugging Face. So are the tests run on a home computer, without photos. The photos are under copyright, so you have to ask for access. The link is on archiveagent.si.