Applied Research Note · Hebrew NLP · Cryptic Crosswords
🧩 Try the interactive solver — נסו את עוזר הפתירה
Abstract
We assemble a corpus of 362 Hebrew logic crosswords (8,249 clue-answer-explanation triples) by scraping a community solutions site and transcribing paywalled newspaper images, then build a solver that couples large-language-model wordplay reasoning with hard grid constraints, a 129k-word lexicon, corpus retrieval, and best-of-N consensus. On blind evaluation the grid layer is airtight (every answer is length-valid), and cross-run agreement is a 64%-reliable correctness signal. Mid-project, a run scored 96 percent on a puzzle where every prior version scored 4 to 25. The implausibility triggered an audit, which found the cause in our own retrieval tool: the lexicon had been built from a corpus that included the held-out answers, so pattern lookups were returning the gold answer ranked above ordinary words. Those results are retracted. After sealing the leak, clean accuracy fell to 14 to 25 percent, meaning the entire apparent gain from four iterations of tooling was contamination. The same audit then exposed a second defect, enumerations transcribed in reverse word order on six puzzles including both evaluation puzzles. Correcting both, two independently audited runs reach 43 percent each and merge to 64 percent, against a 25 percent baseline. We report the retraction, the detection method, and the two things that survive it: correct word boundaries, and the homograph collapse of unvocalized Hebrew. This page documents the method and opens every remaining failure for human correction.
A Hebrew logic crossword clue is a definition welded to a piece of wordplay, with no punctuation to tell you where one stops and the other starts. The answer is written into the grid without spaces or final-letter forms. The יורם הרועה puzzle in Haaretz is the genre's hardest: dense punning, heavy reliance on Israeli politics, Bible, and pop music, and clues contributed by a named community of setters. Our target: solve at least 80% of a puzzle's clues, each with a defensible explanation, without ever consulting the answers.
Two facts shape the whole design. Answers and crowd explanations live on one site; the clue text lives only inside newspaper JPEGs on another. The dataset is fundamentally a join of the two, keyed by date, clue number, and direction.
We scrape answers and crowd explanations for the main setter (52 weeks) and five simpler setters used as a tactic-learning corpus (310 weeks) from the community site. Clue text and grid geometry are recovered from the newspaper images: article bodies are paywalled to anonymous scrapers, so images are located through an authenticated browser session and pulled from the public image CDN. Each image is transcribed by a vision model and validated against the answers, the enumeration sum of every clue must equal the answer's letter count, which catches transcription slips and even a few typos in the crowd-sourced answers themselves.
Mining the crowd explanations gives an empirical mechanism distribution. Charade (concatenating parts) is the default; anagrams are near-mechanically detectable because the fodder appears verbatim in the clue; double definitions cluster in the short slots. Container, reversal, and homophone fill most of the rest.
A single-letter or acronym decoding appears inside roughly a quarter of all explanations, the setter's signature device. Contributor credits of the form (עפ"י X) are the name of the reader who sent the clue, not part of the wordplay, an early false lead we corrected from data.
The solver is an LLM working a fixed protocol, augmented with four tools. It reads only the clues, the grid, and a data-derived playbook, never the answers. To be explicit: no model was trained or fine-tuned. The LLM is a frozen commercial model; all learning in this project happens around it, in the corpus, the deterministic tools (lexicon, substitution table, homograph index), the playbook, and the evaluation loop.
Answers are placed into the real grid: exactly one letter per white cell, dark cells empty, and every crossing must agree. This alone eliminated the entire class of length errors, and it turns each unsolved clue into a letter pattern that filters candidates. As a user reviewing this work put it: build the grid first, confirm one letter per cell, and always move to the clue that already has the most letters.
A 129,574-word Hebrew lexicon (hspell plus all corpus answers) answers two queries: which real words match a crossing pattern, and which real words are anagrams of a fodder window. Retrieval pulls the most similar solved clues and their explanations to identify a mechanism. Finally, N independent solves are merged: an answer that two or more runs agree on becomes a trusted anchor, and the grid is filled anchors-first.
Blind, per-puzzle, exact-match on the normalized answer. The loop is the finding as much as any single number.
| Iteration | Idea | Result | |
|---|---|---|---|
| v2 | Careful, definition-first, grid | 25% / 57% | baseline |
| v3 | Lexicon tools, anagram-first | 4% / 14% | regressed |
| v4 | Best-of-3 consensus + discipline | 39% | retracted |
| v5 | Fact lookup + confidence-weighted consensus | 61% / 18% | retracted |
| v6 | Culture lexicon + grid discipline | 39-96% | retracted |
| v7 | Same tools, leak sealed | 14-21% | clean |
| v8 | Leak sealed + enumerations corrected | 43% / 64% merged | clean, audited |
Rows v3 through v6 are struck from the record. See section 6.
v3 is the instructive failure: an anagram existing is not evidence it is the answer (a multi-word answer has many real-word orderings), and one over-confident anchor poisons the whole grid. The bottleneck is confidence calibration, not candidate generation.
v4 makes the point positively. Merging independent runs beat every individual run, and agreement is a strong signal:
Only about 40% of clues reach agreement. Early on, every solver agent named the same wall independently:
the long culture clues resisted the blind toolset. I could not crack them blind without verifiable culture lookups.
So v5 permitted controlled fact lookup (who sang X, a politician's full name) while still forbidding the answers site and any verbatim clue search. It appeared to work: culture clues that had resisted every blind run started falling, and the agents correctly discarded any result that looked like a crossword answer key. The best single run reached 61%. That figure is retracted; see section 6.
The wall appeared to move rather than fall. On the harder puzzle v5 dropped to 18%, below the careful baseline. In hindsight the variance itself was the tell: runs of one configuration ranged from 18 to 61 percent because they differed in how heavily they leaned on a leaking tool, not in how well they reasoned. Fact lookup answers "what is this reference"; it cannot manufacture a foothold when a puzzle's core has no easy way in. The binding constraint is no longer mechanics or knowledge. It is bootstrappability: whether enough clues are individually crackable to start the crossing cascade. That is a property of the puzzle, and it varies week to week.
A v6 run returned 27 of 28 clues correct on 2026-06-05, a puzzle where every earlier version had scored between 4 and 25 percent. Nothing about the method justified a jump of that size, so the result was audited before it was reported.
The solver's transcript was clean. It had visited no solutions site, read no answers file, opened no puzzle image. The contamination was upstream, in the retrieval tool we built. lexicon.py assembled its wordlist from the same answer corpus that produced the evaluation set, so all 28 held-out answers sat in the lexicon at a priority that ranked them above ordinary dictionary words. Every pattern query on an unsolved slot returned the gold answer near the top of the list. The instruction that made it worse was ours: v6 told solvers to always run pattern lookups on long unsolved slots, which pointed them directly at the leak.
The fix blocks every dev and eval answer from the corpus and culture tiers of the lexicon. Ordinary words that happen to be answers stay, as they would in any real solver's dictionary, which leaves 8 or 9 of 28 per puzzle: honest coverage rather than leakage. Re-measured afterwards, accuracy is 14 to 21 percent, at or below the pre-tool baseline of 25 percent. Four iterations of apparent progress were contamination.
Two details make this worth writing down. First, the file-access rules were obeyed perfectly and the leak happened anyway, because a retrieval tool built from the same corpus as the evaluation set is a leak vector no access rule can catch. Audit the tools, not only the agent. Second, the detection signal was not a security check but a statistical one: identical prompts producing 39, 50, and 96 percent is not what capability looks like. Real skill does not triple between runs of the same configuration.
Retracting the result was not the end of it. Sealing the leak dropped accuracy to 14 to 21 percent, below the pre-tool baseline, which said plainly that four iterations of work had produced nothing. But the same scrutiny that caught the leak also caught the defect below, and with both fixed the clean, audited figure is 43 percent per run and 64 percent merged. The audit cost one impressive number and returned a real one.
Chasing the same result turned up a second defect. Multi-word enumerations are printed inside right-to-left text, and the transcription had preserved their visual order rather than the answer's word order on 6 of 50 puzzles, including both evaluation puzzles. Solvers were being told the last word's length where they needed the first. An automatic check now scores each orientation by whether the resulting split yields real Hebrew words, and the affected puzzles have been corrected.
One device came through the audit intact, and it is the most Hebrew-specific thing here. Written Hebrew omits vowels, so distinct words collapse into one letter sequence. שרה is simultaneously she sings, a female minister, and the name Sarah. רב is a rabbi, the word for many, and a surname. פרס is a prize, Persia, and a former prime minister. The setter builds clues on precisely these collisions, and a solver that can enumerate a token's senses can read the clue's definition half in the sense the surface hides.
Indexing the corpus against a dictionary, a culture list, and person-name components yields 4,949 ambiguous tokens, of which 55 follow the שרה pattern exactly: a role or verb that is also a personal name. The corpus confirms the setter exploits them. גפן appears as an answer in different weeks explained two different ways, once as the surname of a well-known musician, once as wordplay on the word for garden.
The practical rule, now the first step on every clue: when a clue names a profession, a role, or a title, suspect it is standing in for a person's name, or the reverse. Clean runs cracked several clues through this route with no lookup and no leak. It is a small, real gain, and it was suggested by a reader looking at the word שרה and asking whether the list of such collisions had been built.
Every scored clue is below, failures first. Expand any clue to see what each solver produced, the gold answer, the crowd explanation, and the source image, then record a correction: a fixed transcription, a fixed gold answer where the crowd made a typo, or a verdict that the solver was in fact right under an accepted alias. Corrections persist in your browser and export as JSON to feed back into the dataset.
The pipeline is deterministic end to end: scrape, transcribe with letter-count validation, build the grid and verify numbering against the printed clues, distill the playbook, solve blind, score, analyze, repeat. The solver toolkit (grid validator, lexicon, retrieval, consensus merge) and the evaluation harness are plain scripts. The honest read: 80% on יורם הרועה is not reached. Clean, audited accuracy is 43 percent per run and 64 percent merged, against a 25 percent baseline. That gain came from two corrections, not from the tooling that appeared to be working: fixing reversed word boundaries, and exploiting the homograph collapse. The easier setters, which lack that wall, are the realistic place to demonstrate the harness clearing the bar first.