Research · third study
I tried a legal AI model at writing complaint letters. It kept my facts straight, and I am not building the tool.
I have written a fair few complaint letters to a housing association, some of them for neighbours. They are private, and they all have much the same shape. I expected a small AI model trained on legal text, running on my own computer, to draft them better than a general model of similar size. It did. I am still not building anything from it.
Run on 12 October 2026.
One thing to keep in mind. There is no complaints tool, and nothing to download. This was four letters to one housing association, from rough notes that an AI assistant wrote from the finished letters. I said which draft I preferred. I never said that one was good enough to send.
What I tried
Two small models that write, both running on my own computer:
- A legal one, Saul 7B, in a 4.4 GB file. It was published in March 2024.
- A general one, Qwen3 4B, in a 2.5 GB file. It is the model Media used for a few days.
Each got the same rough notes for the same four letters, and the same instruction: use only the facts in the notes, and add no date, name, reference number or law of your own. A short script with no model in it then listed anything in a draft that was not in the notes. I read the two drafts of each letter without knowing which model had written which.
What happened
The legal model added nothing. The general model added three things to one letter: two dates and the name of a department, none of them in my notes. I preferred the legal model’s draft for three letters of the four.
I liked those drafts because they read like a letter: plain paragraphs, then a list of what I wanted. So I ran both models again and asked for exactly that. The legal model again added nothing. The general model made two claims that my notes did not make. One was that a complaint had been stuck for over a month, which was not true.
What the models got wrong
The one that did as it was told made things up. The general model followed the layout I asked for both times. Where my notes gave a reference number with no date, it supplied a date.
The one that kept to my notes did not do as it was told. Asked for a bulleted list, the legal model used bullets in none of its four drafts, and one came back as eighteen numbered points. It also left more out. In one letter it dropped what the complaint was really about.
Neither could be trusted with the shape of a letter. In the second run the general model left the greeting and the sign-off off all four letters.
Both bent small things. A phone call that took thirty minutes became thirty minutes on hold. Something raised at a meeting became something that happened at the meeting. The script cannot see changes like these, because every word in them was in my notes. They were found by reading.
The legal model’s law is two years old. I told both models to add no law, and neither did, so the legal model did not win by knowing any. I looked for a newer one of its kind and found none. Its makers have since published only much larger models. The rest, going by their names and the few descriptions I read, are small chatbots for one country’s law. Anything it did say about the law would be at least two years out of date, and would need checking.
Why I am not building it
I compared two models before asking the question that mattered: why would anyone use this?
Writing a letter from notes is what people use a chatbot for. I expect a chatbot would write a better letter than either of these, though I did not test one. Nobody, including me, is likely to pick a tool whose model is not very good, or out of date, or both, just because it runs on their own computer.
That is a trap I keep walking towards: treating local as the answer every time. Here the case for it is thin. The letter goes to the housing association anyway. What needs keeping private is other people’s details, and Share Copy already makes a copy with names and reference numbers left out, before anything is pasted into a chat model.
So there is no complaints tool. I closed the plan for one on the day I opened it, and deleted the legal model.
What I keep
The useful part of this was the check, not either model. A few lines of ordinary code found the invented dates in seconds. Reading each draft against the notes found the rest.
That is what I take from it. Whatever writes the draft, these models or a chatbot, it can add a date, stretch a fact or leave out the point, and it reads just as smoothly when it does. So I check every date, amount, reference number and name against what I gave it. If it quotes a law, I check that against the law as it stands today.
I have not yet run that check on a chatbot’s draft of a real complaint. That is the next thing to try.
What this does not tell you
- Whether either model’s draft was good enough to send.
- How a chatbot does on the same notes. I did not run one.
- How either model does from notes I typed myself. An AI assistant wrote these from letters that were already finished, which is the easy direction.
- Whether legal training is why the legal model kept to my notes. I did not run a general model of the same size and age.
- Anything about energy. I did not measure it.
The detail
Everything behind the account above, for anyone who wants to check it.
The numbers
| What was counted | Saul 7B, the legal model | Qwen3 4B, the general model |
|---|---|---|
| Facts added that were not in my notes, letter by letter | 0, 0, 0, 0 | 3, 0, 0, 0 |
| Laws named that were not in my notes | 0 | 0 |
| Drafts with a blank left to fill in, of 4 | 1 | 3 |
| The draft I preferred, of 4 | 3 | 1 |
| Seconds for one letter | 19 to 35 | 16 to 26 |
| What was counted | Saul 7B, the legal model | Qwen3 4B, the general model |
|---|---|---|
| Plain paragraphs followed by a bulleted list, of 4 | 0 | 4 |
| Any list of outcomes at the end, of 4 | 2 | 4 |
| A greeting and a sign-off, of 4 | 4 | 0 |
| Claims that were not in my notes | 0 | 2 |
| Drafts with a blank left to fill in, of 4 | 2 | 0 |
A blank is a place the model left for me to fill in, such as “[Your Name]”. In the second run the script found nothing in either model’s drafts. The two claims were found by reading.
How the test was set up
- The task: turn rough notes about one complaint into a letter, using only the facts in the notes.
- Four letters that had already been sent to one housing association. Two I wrote and two were written by neighbours.
- An AI assistant, Claude, wrote the rough notes from the finished letters: what happened, with dates and reference numbers, and what the writer wanted. I did not write the notes.
- The legal model: Saul 7B Instruct, version 1, published by Equall in March 2024, in a 4.4 GB file made by someone else from the original.
- The general model: Qwen3 4B Instruct 2507, in a 2.5 GB file, the model Media used for a few days.
- Both ran on an Apple M1 Pro with 32 GB of memory. Each model had one go at each letter in each pass, with the same wording.
- First pass: I asked for a dated timeline, the issues as a numbered list and the requests as a numbered list. Second pass: I asked for plain paragraphs followed by a bulleted list of outcomes.
- Both times the models were told to add no date, amount, reference number, name, address, law or policy that was not in the notes.
- A short script with no model in it listed every date, number, reference and name in a draft that was not in the notes, and anything that looked like a law. Claude read each list and counted what had really been added.
- For the first pass I read the two drafts of each letter without knowing which model wrote which, and said which I preferred. I have not read the second-pass drafts that way.
Everything this does not show
- Whether any draft was good enough to send. I said which of two I preferred and never marked one as usable.
- How either model does from notes typed by a person. The notes were written by an AI assistant from finished letters, so every fact was already there and in order.
- How a chatbot does on the same notes. None was run.
- Whether training on legal text is why Saul kept to the notes. A general model of the same size and age was not run.
- Anything about other kinds of complaint. All four letters went to one housing association.
- How often either model adds something. These are counts from one go at four letters, twice.
- Whether the script finds everything. In the second pass it found nothing and reading found two claims.
- Whether either model gets the law right. Both were told to add none, and neither did.
- Energy was not measured.
Record source and dates
Copied on 2026-10-12 from the two result comments of 12 October 2026 on issue 76 in dogwonder/kettle-cups. This page is a reviewed copy of those counts. Building the site runs no model.
The letters, the notes and the drafts stay on my computer. Only the counts above are published, and no letter is quoted. Models: Equall/Saul-7B-Instruct-v1, run from the file Saul-Instruct-v1.Q4_K_M.gguf in MaziyarPanahi/Saul-Instruct-v1-GGUF, and Qwen3-4B-Instruct-2507 as already set up for Media.