mecellem-legal-bench: Do models apply Turkish law, or just cite it?

• A rule-only rubric misses a lot. Of the 446 answers our first rubric marked fully right on the law, 133 still skip a step, usually applying the rule to the facts or stating the result.
• Application is the weakest step. Across thirteen models, the median share of application criteria met is 72.2%, against 91.7% for identifying the issue.
• Being right every time is rare. The best model answers all three attempts fully right on 42.9% of questions.
• Open data, open scorer. The questions, the rubrics and the scripts that run and grade a model are on Hugging Face.

Ask a model how much of the tax on a company car bought through finance leasing a business may deduct, and it will usually find the right tax ruling and quote the cap. For 2024 that is 690,000 TL of special consumption tax and VAT. That is correct. The business also needs to know what happens to the amount above the cap and what to do about it in its tax return, and that is the part a lawyer gets paid for.

Most ways of scoring a legal answer can't tell those two answers apart. A multiple-choice exam only checks the letter. A rubric that asks whether the right article is cited gives full marks to an answer that cites it and stops there. We wanted something that reads the whole answer, so we built mecellem-legal-bench. Mecellem is the name of NewMind AI's legal AI work, and one of the retrieval services the models used carries it too. The benchmark itself is open: any model can be run on it and scored the same way.

It is an open set of 42 Turkish legal practice questions. Each answer is graded point by point on how it frames the issue, which rule it relies on, how it applies that rule and what it concludes. The questions are the kind a practising lawyer gets on an ordinary day, such as whether a deposit can be six months' rent, whether a VAT exemption applies, or whether the Competition Board has ruled on some conduct. A model answers with a written note, and a judge model goes through that note one criterion at a time.

Other jurisdictions already have legal benchmarks that go well beyond multiple choice. LEXam draws on 340 law-school exams in English and German, and gives each open question guidance on the reasoning expected of it, from spotting the issue to recalling and applying the rule (Fan et al., ICLR 2026). LexRubric grades Chinese legal answers against 12,337 expert-written atomic criteria (arXiv 2606.09389). GreekBarBench grades written bar-exam answers with a rubric checked against two lawyers (Findings of EMNLP 2025). For Turkish law, the benchmarks we found measure other things.

None of them asks a model to write the answer a client would receive and then checks that answer for the issue, the provision, the application to the facts and the conclusion. Even LegalBench, the largest legal reasoning benchmark in English, tests those steps separately, and its authors point this out themselves: "LegalBench evaluates IRAC abilities independently, while law exams and other legal work requires lawyers to generate outputs which follow IRAC in a multi-hop matter [sic] (i.e., each aspect is applied to the same fact pattern)" (Guha et al., 2023, §4.1). We grade all four steps together, in one answer, on the facts of one question.

1. From citations to IRAC

IRAC stands for Issue, Rule, Application and Conclusion. You find the issue the facts raise, name the rule that governs it, apply that rule to these particular facts, and say where that leaves the client. It is how legal writing is taught and graded in common-law schools, and LegalBench organises legal reasoning the same way: "a common framework for executing this type of legal reasoning is the Issue, Rule, Application and Conclusion (IRAC) framework. In this framework, legal reasoning decomposes into four sequential steps" (Guha et al., 2023, §3.1, citing [Stockmeyer, Legal Reasoning? It's All About IRAC, 2021] and [Ellsworth, "Legal reasoning", The Cambridge Handbook of Thinking and Reasoning, 2005]). Turkish law is a civil-law system, where the same reasoning goes by other names, the legal syllogism or, in the German tradition, subsumption under the norm. Benchmarks in other civil-law systems build on those, LAiW on the legal syllogism (arXiv 2310.05620) and BenGER on subsumption (arXiv 2605.28183). We kept the IRAC names because legal-AI evaluation uses them most, but the steps are the ones a Turkish legal opinion goes through anyway.

Each step leans on the one before. Misread the issue and you cite the wrong rule. Cite the right rule without applying it and the client's question is still open. Apply it and never conclude, and they are no better off than before they asked.

We didn't start out with IRAC in mind, though. When we went back and labelled the 279 criteria of our first rubric on the released questions one by one, 64% turned out to be about the rule. Issue had 4%, application 7% and conclusion 11%, with another 11% on hygiene and 4% on which tool the model had called. Only 3 of the 42 questions checked all four steps, and an answer passed a question only if it passed every single criterion.

So we rewrote the rubric. The new one has 367 criteria, 44% on the rule, 16% on the conclusion, 12% on application, 12% on the issue and 16% on hygiene. Every question now checks all four steps, and an answer gets credit for each criterion it meets.

Then we judged the same answers under both rubrics, with the same judge. Average scores barely moved. Of the 446 answers the first rubric had marked fully right on the law, 133, almost one in three, still skip a step, usually the application to the facts or the conclusion. The first rubric had no criterion for those steps. The IRAC rubric adds them without making the grading stricter overall.

One released question asks how the Competition Board decides whether a firm is dominant. One answer to it runs to 26,000 characters, lays out the criteria the Board uses and goes through six of its decisions, and it met everything the first rubric asked for. It never shows how any of those decisions defined the relevant market, though, and that is the step the dominance test actually turns on. The IRAC application criterion asks for exactly this, and the answer fails it.

2. Our questions

Every question comes with its criteria, and every criterion lists the elements a correct answer has to contain. Figure 3 shows one question in full.

Next to the four IRAC dimensions sits a fifth group we call hygiene. These criteria catch answers that look complete and aren't, because they cite law that doesn't exist or cite it wrongly, or because a correct result is buried under material that contradicts it.

In numbers, the 367 criteria hold 561 elements. 160 criteria are on the rule, 60 on the conclusion, 45 on application, 44 on the issue and 58 on hygiene. A question has between 4 and 13 criteria, nine at the median. Twenty-one questions also come with a gold source, the provision, decision or ruling that answers them, quoted word for word in 53 short passages from 35 provisions and decisions, each checked against the source text by machine. Twenty-three questions are rated medium, twelve hard, three sophisticated and four easy. All but two ask for research and a written answer, and the other two ask for an analysis of a set of decisions.

We didn't start with elements. Many of our early criteria asked the judge whether an answer was adequate, and when two judges scored the same answers from five models, they agreed on those at a kappa of only 0.50. Where a criterion was a numbered list of things the answer must contain, and the judge only had to check whether each one is there, the same two judges agreed at 0.86. Hygiene criteria, which ask whether anything is fabricated, were the hardest at 0.45. So in a pilot we rewrote 62% of the criteria as element lists. That removed all 21 pass conditions that had been left to the judge's discretion and grew the pilot from 54 to 77 criteria, and it didn't make the rubric harsher: on the same recorded answers, the fail rate went from 23% to 22%. Today a criterion passes only if every one of its elements is in the answer, and about twenty elements, such as an article number, a year or an amount, are checked by code instead of the judge. These measurements were made while the rubric was being rewritten, on an earlier version of the questions.

We wrote most of the questions backwards. We would start from something we had already verified, a provision, a Competition Board decision or a tax ruling in the retrieval sources, and write the question it answers. That makes a question hard to answer but easy to check, which is the asymmetry of verification (Wei, 2025) that BrowseComp is also built on (Wei et al., 2025). A question stayed only if a model needed the sources to answer it, so anything you could answer from general knowledge went out. Its answer had to be easy to check once found. And citing law that does not exist counts as a failure on its own, whatever else the answer gets right.

Some criteria were failed by every model. We sent those questions and criteria to lawyers for review. In each case the problem was in the rubric: a criterion asked for something the question never asked, and we corrected it.

When we moved to IRAC, we changed the criteria and left the questions alone, so the question texts are exactly the ones the models answered. The issue criteria also pin down what a question does not ask, so an answer gains nothing by wandering off the point. The distractor criteria, which catch a plausible wrong turn such as mistaking an exemption for a deduction, were kept, and the ones that named one of our tools now name the official source instead. Every criterion can be answered from the retrieval sources. The questions and their criteria were written by the NewMind AI research team.

3. How scoring works

Each question goes to the model in Turkish. The model runs inside our agent harness with our Turkish legal retrieval tools, decides for itself what to look up, and writes its own answer. We ask every question three times.

A judge model then reads each answer one criterion at a time. The judge is google/gemini-3.1-flash-lite and it works blind, which means it doesn't know which model wrote the answer and never sees the other answers or the other criteria. For each criterion it gets the question, the criterion with its elements, the reference material for that criterion, the answer, and the list of tools the model called. It returns PASS or FAIL and quotes the part of the answer it relied on. Before that, a deterministic check looks up every case, ruling and decision number the answer cites, both in what the retrieval tools returned and in the gold source. The judge is told which numbers turned up in neither, and is instructed not to pass a criterion on them.

An answer that isn't there fails every criterion. That includes an empty answer, an error message, and a turn that ran out of time. In this setting a missing answer is the model's result, and leaving it out would reward the models that answer less often.

The score has four parts.

pass^3 is the strictest part and also the noisiest, since one missed criterion in one of three answers fails the whole question. We kept it on purpose. A lawyer relying on a single answer needs that answer to be right every time, and an average won't tell you whether it is. Which retrieval tool a model chose isn't scored at all. Nobody outside can check it, so the score gives it no weight. Cost doesn't enter the score either, and the leaderboard shows each model's median time per answer instead.

Every model ran in the same harness, with the same retrieval tools and the same budget per question, 20 minutes per turn and 16,000 output tokens, reasoning included. That budget is enough for the longest answer any model wrote (about 12,000 tokens). It cut off an answer in 16 of roughly 1,600 attempts, 14 of them from one model, and we saw no reason to raise it, since longer reasoning has not been shown to buy accuracy and on several task types it lowers it (Gema et al., 2025; Ghosal et al., 2025).

Some attempts died for reasons that had nothing to do with the model, like a dropped connection. When an attempt's trace showed an infrastructure fault, we ran it once more with the same model and settings. Attempts that ran out of time, or ended without an answer for a reason we couldn't trace to the infrastructure, count as no answer. The models were run between 4 August and 7 October 2026, each through OpenRouter's default routing.

4. Judge accuracy

Since every number on the leaderboard comes from a judge model, we wanted to know how often that judge is wrong. We checked it against a reference set of 837 verdicts on answers to the released questions. Eight separate AI agents built that reference, each working through a share of the criteria with the gold sources in front of them, so what we report is agreement with a careful reference.

The first version of our judge passed answers that were never written. It saw the raw output of the retrieval tools next to the answer, and when a model had gathered the right documents but never got round to writing an answer, the judge read the documents and passed the criteria anyway. On the released questions it agreed with the reference on 83.4% of verdicts, and 121 of its 139 disagreements were passes the reference did not give.

Two changes fixed most of this. We stopped showing the judge the tool output, so it sees only the answer, and an answer that isn't there now fails every criterion without going to the judge at all. With both changes, agreement rises to 90.7% (759 of 837), and the remaining errors are more balanced, 50 passes the reference did not give against 28 fails. Re-run one at a time, either change alone comes within half a point of both together (90.2% and 90.6%), which makes sense, because both stop the judge from scoring an answer that was never written.

Hiding the tool output had a side effect. Some criteria asked whether a particular retrieval tool had been used, and now the judge had to guess. We first gave it the list of tools the model called, and later took those criteria away from the judge altogether. Which tool was called is now read from the execution record, and the public score gives it no weight. The judge that scored the board still sees that list, along with a flag on each cited case or decision number saying whether the deterministic check found it. These extras change almost nothing. On the released questions this configuration agrees with the reference on 90.6% of verdicts (758 of 837), against 90.7% without them, and the two differ on 23 verdicts. Its errors lean the same way, with 54 passes the reference did not give against 25 fails.

So when the judge is wrong, it is still more often too generous than too strict. A score on the board is more likely to be a little high than a little low, and because one judge scored every model, that holds for all of them alike.

These figures were measured on the first rubric's criteria; the IRAC criteria were scored by the same judge with the same prompt. In one attempt the judge failed three times to return a usable verdict on one criterion, and we left that criterion out of the attempt's score rather than count it either way.

5. Results

Thirteen models, 42 questions, three answers each.

Missing answers explain most of the bottom of the table. When glm-5.3 writes an answer, it meets 83.4% of the criteria, more than seven of the models ranked above it. But it wrote no answer in 47 of its 126 attempts, 33 of them after calling its tools and never getting to the final note, and that puts it last. The two grok models show the same thing more mildly. grok-4.7 hit the 20-minute limit in 25 attempts and grok-4.6 in 6, and on the attempts they did answer, grok-4.6 also meets more criteria (76.3% against 70.0%). So in our harness, the older model ranks above the newer one. That's why the table has a column for answered attempts only.

Few questions are answered fully right every time. The best model gets all three of its answers fully right on 42.9% of questions, while on 78.6% of questions at least one of the three is fully right. The distance between those two numbers is how much of what a model can do it doesn't do reliably.

We ran muse-spark-1.2 twice in the same environment, on 11 and 31 August. The second run scored 67.2, against 73.6 for the first. Its answers weren't measurably worse. Criteria met was 85.9% against 87.8%, and question by question the first run came out ahead on 18 and the second on 15, a split chance explains easily (sign test, p = 0.73). Almost all of the gap was pass^3, 21.4% against 38.1%, which accounts for 5.2 of the 6.4 points. Nine questions that passed in all three answers the first time didn't the second time, and on five of them a single criterion failed in a single answer. The leaderboard shows the first run.

Cost and score are only weakly related. At OpenRouter list prices on 8 October 2026, the three answers to a question cost anywhere from $0.03 (deepseek-v4-flash) to $2.54 (qwen3.8-max), and an attempt used on average between 118,000 and 403,000 tokens, input and output together. Across the thirteen models, cost and tokens track the score only weakly (Spearman's ρ = 0.26 and −0.10), and the share of criteria met on answered attempts no better (ρ = 0.23 and 0.24). deepseek-v4-pro costs $0.31 a question and ranks fourth, while qwen3.8-max costs $2.54 and ranks eleventh. These costs cover the model's own tokens, not the retrieval tools.

Application is the weakest step. The median share of criteria met is 91.7% for issue, 81.7% for rule, 77.6% for conclusion and 72.2% for application. It is the lowest for ten of the thirteen models, and conclusion is the lowest for the other three. Models are good at working out what is being asked. Turning the law into a result for these particular facts is where they struggle, and it is the same gap the first rubric missed.

5.1 Example: the company car question

The company car question from the opening asks how much of the special consumption tax and VAT on a passenger car acquired through finance leasing a business may deduct as an expense. Our first rubric checked it with three criteria, all of them on the rule: the article (Income Tax Law, article 40/1), the cap, and its scope. There was nothing on the issue, the application or the conclusion. The IRAC rubric adds one criterion for each, and its application criterion asks the answer to spell out what the cap means for this car, namely that the part up to the cap is deductible and the part above it is not.

Here are two answers from the same model, two of its three attempts at this question. Both find the same tax ruling, cite the same article and give the same cap, 115,000 TL in the law and 690,000 TL for 2024. The first rubric passed both on every criterion. The first answer goes one step further.

"…the part of the special consumption tax and VAT on the acquisition that exceeds this limit is treated as a non-deductible expense."

The second stops at the rule. It sets out the cap, puts the ruling's number and date in a table, and ends with this.

"…the special consumption tax and VAT paid at acquisition can be expensed within the limits stated above."

It never says that the amount above the cap has to be added back to taxable income as a non-deductible expense, which is the step a taxpayer actually has to take in the return. The IRAC rubric passes the first answer on all seven criteria and fails the second on application. The first rubric saw no difference between them.

The pattern holds across models. Of the 13 answers to this question that met every criterion of the first rubric, across the eleven models we compared under both rubrics, 5 never say what happens above the cap. That both answers come from one model is the pass^3 problem again, on a single question.

Excerpts translated from Turkish.

6. Using the benchmark

Here is what we are releasing.

• The questions: 42 questions in Turkish, exactly as the models received them, with client names replaced.
• The rubrics: 367 criteria, each with its rule, its elements and its IRAC dimension.
• The gold sources: for 21 of the questions, the provision or decision that answers them: its name and, in 53 short quotes, the passages that matter, word for word. The full documents are not included.
• The runner: run.py, which asks a model every question three times through any OpenAI-compatible API, with your own tools plugged in if you have them, under the same limits we used, and writes the answers in the format the scorer reads.
• The scorer: score.py, which needs only the Python standard library. score.py judge runs the blind judge one criterion at a time, and score.py report turns the verdicts into the leaderboard's score (criteria met, pass^3, grounding and attempts completed, with the same weights) plus pass@3, agreement and the I/R/A/C/H breakdown. Run on the leaderboard's own verdicts, it reproduces the leaderboard's numbers; the largest difference in any model's score is 0.2 points.
• A datasheet describing how the set was built, reviewed and masked.
• The leaderboard, a Hugging Face Space with every model's breakdown by area of law and by IRAC dimension.

All of it except the leaderboard is in the dataset. Data is under CC BY 4.0 and code under Apache 2.0. We are not releasing the models' answers themselves, only their scores.

What we can't release is the retrieval layer. Our models answered with our own Turkish legal retrieval tools, and those aren't public. To score your own model, run run.py with your own retrieval tools, or write the answers yourself in the same format, and judge them with score.py. If your harness already checks citations against its tool trace, you can pass those checks along. Each question names the retrieval service it was written for, and the release says what each service covers and where the same material is public, whether legislation, high-court decisions, Competition Board decisions or tax rulings. Our services stay in-house, but if you'd like your model run with them, get in touch. Just keep in mind that your scores measure your model with your retrieval, so they are comparable to ours in method, not in setting. The judge in score.py is the leaderboard's judge with our tool names taken out. It uses the same model, the same rules and the same citation check, without the list of in-house tools.

Benchmark data should never end up in training corpora. The canary string 93814075-e49c-44e4-9769-0c4ceccb5301 is in the datasheet.

7. Limitations

A score here describes a model working with our tools. Every model ran in our agent harness with our retrieval tools, under one time limit and one output budget, and a model that does well with other tools or with more time may rank differently.

Forty-two questions is a small set. Leaving out any one question moves no model by more than one place, but the areas are uneven, with 12 questions on obligations, lease and procedure, 11 on tax, 11 on company and commercial law and 8 on competition and regulation. Four areas are a long way from covering Turkish law.

The conditions weren't frozen. Models ran over nine weeks through OpenRouter's default routing, so the provider serving a given model wasn't fixed, and the same model, run twice twenty days apart, scored 6.4 points apart.

Stored tool outputs are truncated. Citations are checked against the full tool outputs when an answer is first judged, but a later re-judge only sees the stored, truncated outputs. We keep every citation that was verified at run time, yet in 7 of 1,638 attempts a citation verified at run time still counts as unverified.

And once questions are published, they leak. From the release on, future models may have seen these questions, and the canary string is there so a later evaluation can take that into account.

Next, we would like to add questions in the areas that are thin today.

Contact

The questions and rubrics were written by the NewMind AI research team. This post was written by Reyhan Bayraktar.

If you find a mistake in a question or a criterion, want your model on the leaderboard, or just have a question, open a discussion on the dataset page or write to us at [email protected].