Research · second study
I tried three small models at reading a typed request. Ordinary rules did better.
Media lets me type what I want done to a picture, such as “fit inside 800 x 600 and make it black and white”. I assumed a small AI model would read a sentence like that better than a list of hand-written patterns. On 60 requests the patterns got 59 right and the best model got 48.
Run on 6 October 2026, with a follow-up on 7 October 2026.
One thing to keep in mind. Every request in this study was written by an AI model, not typed by a person. That makes the rules look better than they are likely to be. This is about a prototype that is not published: Media runs on my own computer and there is nothing to download.
What I tried
Something has to turn a typed request into settings: a size, whether to crop, whether to make it black and white. I tried two ways of doing it.
- Ordinary rules: hand-written patterns for the words people use, with no model.
- A small model of the kind used for search, called an embedding model, with a little training on top. I tried three, the largest about eight times the size of the smallest.
Code read the numbers either way, so a model was never asked for a width or a height. It only had to say yes or no three times: resize, crop, black and white. I ran each once, on 60 requests that neither the rules nor the models had seen.
What happened
The rules got the whole request right 59 times out of 60. The three models managed 47 or 48, and more often set a change nobody had asked for. The largest model did no better than the smallest.
The one request the rules got wrong, every model got right: “cut to 1080 x 1350 exactly and remove colour”. Nobody had written a rule for “cut to”. A model can be right about wording that no one thought of, and rules cannot.
Why I think the models lost
There was little left for a model to do. Once code has read the numbers, most requests are settled by a handful of words: “crop”, “black and white”, “wide”.
Nearly right three times is not right. Each model got each yes or no answer right about nine times in ten. A request needs all three right at once, and that is how nine in ten becomes 47 or 48 out of 60.
I chose the wrong kind of model. Looking back, an embedding model was the wrong kind of tool for this job, and I could have seen that before running it. It is built to say how alike two pieces of text are, which suits search. This job turns on a few exact words.
The next day I tried a different kind of model
This one was a small chat-style model, Qwen3 4B, in a 2.5 GB file. I did not let it write anything. Code reads every number and time in the request first, and the model fills in a form by choosing among them. Then code checks that every value was used.
On the same 60 picture requests it got 56, against 59 for the rules. On new requests it got 28 of 30 for recordings, 22 of 24 for video and 28 of 30 for pictures. So the first result was about the kind of model I picked, not about models.
Two things did not change. For pictures the rules in this study were still ahead, by one request on the new set and three on the old, and they need no download. A wrong answer from a model can be the opposite of the request. Two smaller versions of the same model read “chop it to the last 90 secs” as taking the last 90 seconds off.
What I do now
The rules won this test, and I have still stopped using them. They had grown to more than twenty patterns for three changes, and I did not want to write a list like that for every kind of file.
Media now reads only the plainest requests with a few rules, such as “720p” or “crop to 800 x 600”. Any other wording goes to the form-filling model. With no model set up, nothing is filled in and I choose the settings by hand. Either way I check a written plan before anything is made.
That lasted two days. On 9 October I took the model out of the app I use. Once the page showed example requests, typing one was no easier than pressing it, so I put those sizes on the form itself. Nine smaller models tried on the same requests did not come close to the 2.5 GB one, so it was that or none. Media’s page has the figures.
What this does not tell you
- How the rules or the models do on requests typed by people. None was in any of these runs.
- How often any of this is right in general. These are counts from single runs on 24 to 60 requests.
- Anything about open-ended jobs, such as finding a document from a description. This was one small, closed task, and a model may be the right tool elsewhere.
- Anything about energy. I did not measure it.
The detail
Everything behind the account above, for anyone who wants to check it.
The numbers
Media as it was before this study got 18 of the 60, because it picked one task for a whole request and could not hold two changes.
| What was counted | Media before | Rules, no model | BGE-small as already tuned for Media | BGE-small as published | EmbeddingGemma 2, text only |
|---|---|---|---|---|---|
| Complete suggestions, of 60 | 18 | 59 | 48 | 47 | 48 |
| One change, of 20 | 11 | 20 | 15 | 14 | 16 |
| Two changes together, of 20 | 0 | 19 | 16 | 17 | 17 |
| Awkward requests, of 20 | 7 | 20 | 17 | 16 | 15 |
| Requests given a change nobody asked for | 3 | 1 | 6 | 9 | 6 |
| Requests missing a change that was asked for | 40 | 1 | 7 | 6 | 7 |
| Answers matching, of 60: resize, crop, black and white | not applicable | 59, 59, 60 | 53, 58, 52 | 54, 54, 53 | 54, 53, 55 |
Earlier the same day I tried two models with no training at all, on 18 requests. EmbeddingGemma 2, comparing each request with a few fixed sentences, was right on 3. GLiNER2.5 Small, a model that pulls named things out of text, was right on 6. Neither was usable that way.
| What was counted | Rules, no model | BGE-small as already tuned for Media | BGE-small as published | EmbeddingGemma 2, text only |
|---|---|---|---|---|
| Size of the model | none | 33 million parameters | 33 million parameters | 271 million parameters for the text part |
| Download | none | none, already on my computer | 133 MB | 1.5 GB |
| Peak memory while reading requests | not measured | about 0.6 GB | about 0.6 GB | about 2.4 GB |
| Time to turn one request into numbers | under a thousandth of a second for the whole reading | about 2 thousandths of a second | about 2 thousandths of a second | about 10 thousandths of a second |
| Its own software environment | no | no | no | yes, 0.9 GB |
| Kind of request | Requests | A few plain rules alone | The same rules, then the model | What Media used before |
|---|---|---|---|---|
| Sound recordings | 30 | 17 | 28 | No suggestions |
| Video | 24 | 10 | 22 | An older small model: 7 |
| Pictures | 30 | 3 | 28 | The rules from this study: 29 |
More on why the models lost
The rules got each of the three answers right 59 or 60 times. Each model got each answer right in 52 to 58 of the 60 requests.
The mistakes were scattered. 21 of the 60 requests were got wrong by at least one model, and only 4 by all three. The same model added black and white where nobody asked for it and missed it where somebody did. There was no single gap to mend.
I tested one explanation after the result was known. The training requests came from two sources written in different styles, and only one of them had any black and white in it, so a model could have learnt the style. I retrained on that one source alone and scored the same 60 requests: 46, 47 and 49. The mix of styles was not the cause.
The largest model, EmbeddingGemma 2, cost a 1.5 GB download, about four times the memory and its own software environment, and scored the same as the smallest.
All three models failed the same four requests, and each turns on a few words. Two give a crop as permission: “exactly 1920 by 1080? Cutting some off the top and bottom is fine.” One asks for a crop that must keep everything. One gives a number and says it could be either side. My reading is that a model which turns the whole sentence into one set of numbers moves very little for words like these, while a rule looks for exactly them. I did not test that directly.
The next day’s model got three of those four right. Size mattered for it: on the recording requests, two smaller versions gave 25 and 26 of 30 and each filled in a wrong part at least once, while the 4 billion one gave 28 and filled in none. The few plain rules in front of it read little on their own: 8 of the 30 recording requests, 3 of the 24 for video and 2 of the 30 for pictures. With the model behind them they did not change a single total. The model took about a second a request on my computer.
How the test was set up
- The task: read one typed request about one picture and say which of three changes it asks for (make it smaller, crop it from the centre to a given size, make it black and white), with the numbers, and list anything it asks for that the tool does not do.
- Every route used the same code to read the numbers and to list what was not done. A route only had to give three yes or no answers: resize, crop, black and white.
- Rules: ordinary patterns for the words people use, including “don’t” and requests that pull two ways. No model.
- Models: three embedding models, each left as downloaded, with three small classifiers trained on top, one for each answer. They were trained on 636 invented requests: 406 from earlier work on Media and 230 written for this study.
- The test: 60 requests none of the routes had seen. 20 ask for one change, 20 for two together, and 20 are awkward on purpose: something said not to be done, a request that contradicts itself, or something the tool does not do.
- All wording was written by AI models, not typed by me. Two separate AI writers produced the 60 test requests without seeing the rules. The rules and the training requests were written by a third.
- The rules and the training recipe were fixed, and their files recorded, before the 60 requests were opened. Each route ran once and nothing was changed afterwards.
- A suggestion counted as complete only if the size, the crop choice and black and white all matched the answer written beforehand, and everything not done was shown.
- Run on an Apple M1 Pro with 32 GB of memory and macOS 27.0.1. The models ran on the processor with network access refused.
Everything that was not measured
- The rules’ 59 is flattering. The cards, the rules and the training requests were written by one AI assistant, and the test requests by two others from the same family of models. Their wording is likely closer to what the rules expect than a person’s would be.
- No request typed by a person has been through any route.
- I did not check the 60 written answers myself. They were read against the rules before the run and none was changed. Four were judgement calls.
- The models were not adjusted for this task beyond the small classifiers. Nobody retrained an embedding model. No chatbot-style model was tried in this run; one was tried the next day, and that is described above.
- The classifiers were trained on 636 requests. More, or better, examples might have closed some of the gap. That was not tested.
- One run. Variation between repeated runs was not measured.
- English requests about still images only.
- Energy was not measured.
Record source and dates
Copied on 2026-10-06 from docs/evidence-media-compound-suggestions-2026-10-06.md in dogwonder/kettle-cups at f91af3d4ea92b8f31b870edec71e2fdc331cee3f. This page is a reviewed copy of that evidence note. Building the site runs no model.
Models: BAAI/bge-small-en-v1.5 at revision 5c38ec7, google/embeddinggemma-2 at revision 914f7f8 and fastino/gliner2.5-small-v1 at revision 7132dc4. The requests, the rules and the scripts are in the repository. Model files and full outputs stay on my computer.
The 7 October runs are from docs/evidence-media-compound-suggestions-2026-10-06.md in dogwonder/kettle-cups at 15206c61eaaa7d7e27cd7a33b1e83049866fedbd. Model: Qwen3-4B-Instruct-2507, 4-bit, from unsloth/Qwen3-4B-Instruct-2507-GGUF at revision a06e946, with the 0.6B and 1.7B files of the same family beside it.