Can an LLM play Redactle?
I gave a pile of language models the same redacted Wikipedia puzzles. The cheap, fast models did better than I expected.
Redactle gives you a Wikipedia article with nearly every word hidden. You guess words, reveal their matches, and keep going until you can identify the title.
I was curious whether a language model could do the same thing. I started pasting redacted articles into chats from different providers and got wildly different answers. Some models spotted the subject immediately. Others confidently wandered in the wrong direction.
That was enough to nerd-snipe me. I built a small solver and a repeatable benchmark so the models could play under the same rules.
The test
Each model received the same redacted article and the same basic tools available to a player: guess a word, inspect what it revealed, and eventually guess the title. The test articles were kept out of the public results so the benchmark did not reveal future answers.
I tracked more than whether a model eventually solved the puzzle:
- how many puzzles it solved
- whether it needed hints
- how many guesses it used
- how long it took
- how much the run cost
The first proper batch used 12 selected articles. It was small enough to run across many model configurations, but large enough that one lucky title guess did not decide the leaderboard.
The models can play
The short answer is yes. Good models can use the shape of an article, repeated phrases, dates, measurements, and the pattern of successful guesses to narrow down a subject. Watching that happen feels surprisingly close to watching a strong human player.
The winner was not what I expected. Gemini 3.7 Flash at low reasoning solved all 12 puzzles in the lead test while also being the cheapest and fastest configuration I tried. Gemini and Grok generally performed better than the OpenAI and Anthropic models I had expected to lead.
More reasoning did not reliably mean better play either. Increasing the reasoning setting could make a model slower and more expensive without improving its guesses. Redactle rewards useful uncertainty and quick information-gathering, not just longer internal deliberation.
The original experiment cost $22.04
That covered the original evaluation runs across the models and reasoning settings on the leaderboard. It is not a serious research budget, but it was enough to turn a vague question into a result I could inspect and reproduce.
What Redactle is testing
This is not a general intelligence test. It is a peculiar mix of broad knowledge, pattern recognition, search strategy, and knowing which question will reveal the most useful information next.
That is also why it makes a good model test. A model cannot get very far by writing a polished paragraph about the puzzle. It has to make a sequence of concrete decisions, learn from each result, and recover when its first theory is wrong.
You can explore the current results on the live Redactle LLM leaderboard.