In progress · pilot validated · no full-scale results yet
Does the language a model reasons in change its legal accuracy?
A factorial experiment over LegalBench: reasoning-language conditions × models × closed-label legal classification tasks.
Nineteen reasoning conditions (fourteen natural languages across nine families, plus formal logic, pseudocode, an emergent notation, a wildcard, and a no-chain-of-thought control) are run against nine LegalBench tasks on a set of low-cost cloud and local models, with the final answer always requested in English.
From the pilot
In the pilot, one model returned hidden reasoning on every sample, and in the Mandarin condition its visible reasoning was mostly Chinese while the hidden channel was mostly not. A model that thinks in a hidden channel is not reasoning in the requested language, which is why the design measures it.
Design
- Nine closed-label LegalBench tasks: hearsay, personal jurisdiction, contract NLI, unfair terms of service and others
- Identical seeded samples across every cell; temperature zero and no output cap where the API allows
- Hidden "thinking" switched off wherever possible, and measured and reported where it cannot be
- Fifty-six offline tests; a live pilot run completed end to end with zero errors