How the test was run
- The protocol was fixed before any run, on 19 October 2026: cases, instructions, versions compared and scoring rubric.
- The versions compared: Claude Opus 5.5 (in Claude Code) connected to Alpaca Law's database, with web search, and Doctrine's Assistant in "Raisonnement avancé" mode. In the same batch, the judges also scored an answer from Claude without Alpaca Law's database, which serves another comparison and is not published here; the judges' reports, published unedited, mention it.
- Doctrine received the same instruction and the same documents, as attachments, on 20 October 2026. It does not accept .md and .html files, so those were attached as .txt, with the same content.
- Blind: each answer was converted to text, without links or tool names, and labelled A, B or C at random; the second judge got different letters. The scores table gives the key.
- The judges are AI models (Claude Opus 5.5), two per case, with no answer key: they did their own research and checked the cited sources. A third judge was planned in case of a tie; there was none. Alpaca Law's database did not work for the judges, who checked on official and public sites instead.
Rubric and scores
- Scoring rubric (as given to the judges)
- All scores, with the letter each judge saw
Case 1: the furnished lease
- Instruction and documents: instruction · contrat-location.docx · objectifs.md
- Claude + Alpaca Law: revision-contrat.md · contrat-location_revise.docx
- Doctrine: answer.md
- Judge 1: report · text A · text C
- Judge 2: report · text B · text C
Case 2: the private loan
- Instruction and documents: instruction · explication.md · contrat.docx · Historique_Emails.html · Historique_Emails.txt · Recu_001 · Recu_002 · Recu_003 · Recu_004 · Recu_005 · Recu_006 · Recu_007 · Recu_008 · Recu_009 · Recu_010 · Recu_011
- Claude + Alpaca Law: analyse-assignation.md · assignation.md · assignation.docx
- Doctrine: answer.md
- Judge 1: report · text A · text C
- Judge 2: report · text B · text C