INDEPENDENT RESEARCH

Legal AI Citation Reliability Benchmark

An open, transparent benchmark measuring AI accuracy in legal citation generation. Methodology, sample prompts, and placeholder for future results. Conducted by DAS Group LLC.

Transparent methodology. All prompts, rubrics, and criteria are publicly documented.
View methodology → Test your own citations
This benchmark provides research information, not legal advice. Results are for informational purposes only and should not be relied upon for legal decision-making.

Benchmark Overview

The CiteClear Legal AI Citation Reliability Benchmark is an independent, open research effort to measure how accurately large language models generate legal citations.

Unlike many AI benchmarks that:

  • Focus on general factual accuracy rather than domain-specific performance
  • Use synthetic or academic prompts that don't reflect real-world usage
  • Keep their methodologies and prompts proprietary
  • Measure subjective or hard-to-verify outputs

Our benchmark is:

  • Legal-specific: Focused exclusively on legal citation accuracy
  • Realistic: Based on prompts that reflect actual legal research scenarios
  • Transparent: All prompts, rubrics, and criteria are publicly available
  • Objective: Citations are either verifiable or not; there's no subjective interpretation
  • Reproducible: Anyone can reproduce our results using our published methodology
  • Privacy-preserving: No confidential information is used in prompts or testing

Why Citation Accuracy Matters

Legal citations are the foundation of legal authority. A single fake citation can undermine an entire legal argument, damage credibility with courts and clients, and potentially violate ethical obligations. Unlike general factual errors, legal citation errors can have real legal consequences.

Benchmark Design Principles

Objectivity

Citations are either verifiable in authoritative sources or they're not. This binary outcome eliminates subjective judgment and allows for clear, reproducible scoring.

Transparency

All test prompts, scoring rubrics, and methodology are publicly documented. This allows independent verification and reproduction of our results.

Reproducibility

Anyone with access to the same AI models can reproduce our tests using our published prompts and methodology. Results should be comparable across runs.

Non-commercial

This is independent research, not a marketing effort. We don't sell benchmark data, reports, or access. Results are published freely.

Privacy-Preserving

All test prompts use publicly available information or hypothetical scenarios. No confidential client data is used in benchmarking.

Deterministic-First

Our verification methodology uses deterministic pattern matching before any AI-assisted review, matching our product's approach.

Benchmark Methodology

Step 1

Scope Definition

What we test: AI model ability to generate accurate, verifiable legal citations in response to legal research prompts.

What we measure:

  • Citation accuracy rate: Percentage of generated citations that are verifiable in authoritative sources
  • Hallucination rate: Percentage of generated citations that are completely fabricated
  • Malformation rate: Percentage of generated citations with structural errors
  • Precision: Of all citations generated, what percentage are accurate?
  • Recall: Of all relevant citations that should exist, what percentage does the AI generate?

What we don't test:

  • Legal analysis accuracy (beyond whether cited authorities exist)
  • Quote accuracy within AI output
  • Non-legal citations (except as they appear in legal contexts)
  • General factual accuracy outside of citation verification
  • Performance on confidential or proprietary information
Step 2

Prompt Design

Our prompts are designed to simulate real-world legal research scenarios while being reproducible and privacy-preserving.

Prompt Categories:

Case Law Retrieval

Example: "What are the leading cases on [legal topic] in the [jurisdiction]? Provide citations."

Purpose: Test ability to retrieve and cite relevant case law

Evaluation: Are the cited cases real and relevant to the topic?

Statutory Research

Example: "What federal statutes address [legal issue]? Provide citations with section numbers."

Purpose: Test ability to identify and cite relevant statutes

Evaluation: Are the cited statutes real and applicable?

Citation Formatting

Example: "Format this legal authority as a proper Bluebook citation: [case name, year, court]"

Purpose: Test ability to format citations correctly

Evaluation: Does the formatted citation follow proper Bluebook/ALWD rules?

Multi-Jurisdiction

Example: "What cases from any jurisdiction have addressed [emerging legal issue]? Provide citations."

Purpose: Test ability to identify relevant law across jurisdictions

Evaluation: Are the cited cases from the jurisdictions claimed?

Citation Correction

Example: "This citation appears to be malformed: [malformed citation]. Correct it or identify the error."

Purpose: Test ability to detect and correct citation errors

Evaluation: Does the correction fix the error or properly identify it?

Comparative Analysis

Example: "Compare the treatment of [legal issue] in [Jurisdiction A] vs. [Jurisdiction B]. Provide citations."

Purpose: Test ability to find and cite relevant authorities across jurisdictions

Evaluation: Are the cited authorities real and properly compared?

Prompt Constraints:

  • Public information only: No confidential client data
  • Reproducible: Same prompt can be run multiple times
  • Clear intent: Prompt clearly requests citations
  • No trick questions: We don't test for "gotcha" scenarios
  • Realistic scope: Prompts reflect actual research tasks
Step 3

Scoring Rubric

Each AI response is evaluated against our scoring rubric by trained human evaluators (currently CiteClear team members).

Citation-Level Scoring:

Category Score Definition Example
Perfect 3 Citation exists exactly as stated, all components correct Brown v. Board, 347 U.S. 483 (1954)
Minor Error 2 Citation exists but has minor formatting errors (extra space, wrong reporter abbreviation variant) Brown v. Board, 347 U. S. 483 (1954) (extra space)
Major Error 1 Citation exists but has major errors (wrong page, wrong year, wrong court) Brown v. Board, 347 U.S. 483 (1955) (wrong year)
Wrong Case 0 Citation exists but references a different case with that citation Smith v. Jones, 347 U.S. 483 (1954) (Brown is at that citation)
Malformed 0 Citation is structurally invalid, missing required components Brown v. Board (no reporter)
Hallucinated 0 Citation does not exist and never did Brown v. Board, 999 U.S. 123 (2099)
Unverifiable ? Unable to verify through available sources [Obscure foreign citation]

Response-Level Metrics:

  • Accuracy Rate: (Perfect + Minor Error) / Total Citations
  • Hallucination Rate: Hallucinated / Total Citations
  • Error Rate: (Major Error + Wrong Case + Malformed + Hallucinated) / Total Citations
  • Unverifiable Rate: Unverifiable / Total Citations

Prompt-Level Metrics:

  • Success Rate: Percentage of prompts with at least one perfect citation
  • Complete Failure Rate: Percentage of prompts with zero perfect or minor-error citations
  • Partial Success Rate: Percentage of prompts with some but not all citations accurate

Note on Unverifiable Citations: Citations marked as "unverifiable" are excluded from accuracy calculations but tracked separately. This may include foreign citations, very recent cases not yet indexed, or citations to specialized courts with limited coverage.

Step 4

Verification Process

Our verification process follows a deterministic-first approach that mirrors CiteClear's own methodology.

1

Deterministic Preflight

First, we run CiteClear's deterministic checks to flag obvious errors:

  • Impossible reporter numbers (e.g., 999 F.4th)
  • Non-existent courts (e.g., 13th Circuit)
  • Future dates
  • Fake keywords (Fake, Imaginary, Test)
  • Malformed structure

Result: Citations flagged as suspicious are marked for priority review.

2

Database Verification

For non-flagged citations, we verify through authoritative sources:

Primary: Westlaw, Lexis, Bloomberg Law
Secondary: Google Scholar, Justia, Casetext, Cornell LII
Official: Court websites, PACER

Requirement: A citation must be verified through at least one authoritative source to be scored as accurate.

3

Cross-Checking

We verify that:

  • The case name matches the citation
  • The court and jurisdiction are correct
  • The reporter, volume, and page are accurate
  • The year is consistent with the actual decision date
4

Scoring

Based on verification results, we assign scores according to the rubric above.

Double-check: All high-error responses (Complete Failure) are reviewed by at least two evaluators.

5

Documentation

We document:

  • The AI's original response
  • Our verification sources
  • The assigned score for each citation
  • Any notes on discrepancies or issues

Transparency: All documentation is available for independent review upon request.

Reproducibility Note

AI model outputs can vary between runs, even with the same prompt. We run each prompt multiple times when possible and note any variations in our documentation. Temperature and other sampling parameters are documented for each test.

Step 5

Sample Prompts

Below are sample prompts from our benchmark set. These prompts test various aspects of legal citation generation.

Set 1: Case Law Retrieval - Federal

Prompt 1.1 Case Law - Landmark
What are the five most important Supreme Court cases on the First Amendment? Provide proper Bluebook citations for each.
Difficulty: Easy
Expected Output: 5 real Supreme Court cases with proper citations
Evaluation Focus: Can the AI identify well-known cases and format citations correctly?
Prompt 1.2 Case Law - Specific Topic
What cases from the Ninth Circuit have addressed the Computer Fraud and Abuse Act in the past five years? Provide citations.
Difficulty: Medium
Expected Output: Multiple real 9th Circuit cases on CFAA
Evaluation Focus: Can the AI find recent, relevant circuit court cases?
Prompt 1.3 Case Law - Recent
What are the most recent Supreme Court decisions from the 2023-2024 term? Provide citations.
Difficulty: Medium
Expected Output: Recent Supreme Court cases with accurate citations
Evaluation Focus: Does the AI have up-to-date information and proper citation format?

Set 2: Statutory Research

Prompt 2.1 Statutes - Federal
What sections of the U.S. Code address wire fraud? Provide the statute citations.
Difficulty: Easy
Expected Output: 18 U.S.C. § 1343 and related sections
Evaluation Focus: Can the AI identify the correct statutory provisions?
Prompt 2.2 Statutes - State
What are the California statutes that govern non-compete agreements? Provide citations to the Business and Professions Code.
Difficulty: Medium
Expected Output: Cal. Bus. & Prof. Code §§ 16600-16607
Evaluation Focus: Can the AI identify correct state-specific statutes?
Prompt 2.3 Statutes - Cross-Reference
What federal statutes does 18 U.S.C. § 1030 (Computer Fraud and Abuse Act) cross-reference? Provide citations.
Difficulty: Hard
Expected Output: Related statutes in Title 18 and other titles
Evaluation Focus: Can the AI understand statutory relationships and find cross-references?

Set 3: Citation Formatting

Prompt 3.1 Formatting - Case
Format this as a proper Bluebook citation: Smith versus Jones, decided by the Ninth Circuit Court of Appeals in 2023, reported in volume 123 of the Federal Reporter 4th at page 456.
Difficulty: Easy
Expected Output: Smith v. Jones, 123 F.4th 456 (9th Cir. 2023)
Evaluation Focus: Can the AI properly format a case citation from natural language description?
Prompt 3.2 Formatting - Statute
How would you cite section 1030 of title 18 of the United States Code in Bluebook format?
Difficulty: Easy
Expected Output: 18 U.S.C. § 1030
Evaluation Focus: Can the AI format a simple statutory citation correctly?
Prompt 3.3 Formatting - Multiple
Convert these to proper Bluebook format: (1) Brown v Board of Education 347 US 483 1954, (2) 18 US Code section 1030, (3) In re Smith 200 F3d 500 3d Cir 2005
Difficulty: Medium
Expected Output: 3 properly formatted citations
Evaluation Focus: Can the AI fix malformed citations?

Set 4: Error Detection

Prompt 4.1 Error Detection - Obvious
Is this a valid citation? Fake v. Imaginary Corporation, 999 U.S. 123 (2099). Explain why or why not.
Difficulty: Easy
Expected Output: Identification that this is fake with reasons
Evaluation Focus: Can the AI detect obviously fabricated citations?
Prompt 4.2 Error Detection - Subtle
Is this citation properly formatted? Brown v. Board of Education, 347 U. S. 483 (1954). If not, what's wrong?
Difficulty: Medium
Expected Output: Identification of the Bluebook spacing issue
Evaluation Focus: Can the AI detect subtle formatting errors?
Prompt 4.3 Error Detection - Court
Is there an error in this citation? Smith v. Jones, 123 F.4th 456 (13th Cir. 2023). Explain any issues.
Difficulty: Easy
Expected Output: Identification that there is no 13th Circuit
Evaluation Focus: Can the AI detect non-existent courts?

Set 5: Multi-Jurisdiction

Prompt 5.1 Multi-Jurisdiction - Federalism
How do different federal circuits treat the issue of standing in data breach cases? Provide citations to representative cases from at least three different circuits.
Difficulty: Hard
Expected Output: Cases from 3+ different circuits on the topic
Evaluation Focus: Can the AI find and cite across different jurisdictions?
Prompt 5.2 Multi-Jurisdiction - State Comparison
Compare the approach to non-compete agreements in California vs. New York. Provide statutory citations for each state.
Difficulty: Medium
Expected Output: California and New York statutes on non-competes
Evaluation Focus: Can the AI provide correct state-specific statutes for comparison?
Prompt 5.3 Multi-Jurisdiction - International
What are the GDPR provisions that are similar to CCPA? Provide citations to both the GDPR and CCPA.
Difficulty: Hard
Expected Output: GDPR articles and CCPA statutory sections
Evaluation Focus: Can the AI cite both EU and California privacy laws correctly?

Note: These are sample prompts from our benchmark set. The full benchmark includes many more prompts across additional categories. All prompts are available in our public repository (link will be added when available).

Want to see results? While we're currently running tests internally, we plan to publish our findings publicly as we complete each benchmark cycle. Check back for results, or contact us if you'd like to collaborate on benchmarking efforts.

Frequently Asked Questions

What is the purpose of the CiteClear benchmark?

The CiteClear benchmark is designed to measure and publicly report on the reliability of AI-generated legal citations. Unlike proprietary benchmarks run by AI vendors themselves, our methodology is transparent, our prompts are published, and our results are independently verifiable. We aim to provide objective data that helps legal professionals understand both the capabilities and limitations of AI tools for legal research.

How does CiteClear's methodology differ from other AI benchmarking efforts?

CiteClear focuses specifically on legal citations rather than general factual accuracy. We use a deterministic-first approach that doesn't rely on the AI models we're evaluating. Our prompts are designed to test real-world legal research scenarios, not synthetic benchmarks. Most importantly, we publish our entire methodology, allowing anyone to reproduce or audit our work.

Can I contribute to or reproduce the CiteClear benchmark?

Yes. Our methodology and sample prompts are fully published. Researchers, journalists, and developers are encouraged to reproduce our tests, extend our methodology, or contribute new prompt sets. All our benchmark materials are available for independent use. Contact us at the email in our footer for collaboration opportunities or access to our full prompt library.

Why focus only on citations rather than full legal analysis?

Citations are the most verifiable component of legal writing. Unlike legal analysis, which requires subjective interpretation of complex legal principles, a citation either exists or it doesn't. This makes citations an ideal objective measure for AI reliability. Additionally, focusing on citations allows for privacy-preserving verification without sharing confidential client documents, which is essential for legal work.

Does CiteClear sell benchmark data or reports?

No. Our benchmark results are published freely on this page. We do not sell data, reports, or access to proprietary benchmark information. Our goal is transparency in AI legal citation reliability, not commercialization of benchmark data. All results are available to everyone at no cost.

How often will CiteClear update the benchmark results?

We plan to run benchmark cycles on a regular basis, particularly when new AI models are released or when existing models are significantly updated. We'll publish new results as they become available. The methodology will remain consistent across runs to ensure comparability, though we may add new prompt categories over time.

What AI models does CiteClear test?

We test widely available commercial AI models, including but not limited to: ChatGPT (various versions), Claude (various versions), Gemini, Perplexity, and other models that are accessible through standard APIs or consumer interfaces. We document which specific model versions we test in each benchmark cycle.

How does CiteClear handle model updates and version changes?

AI models are frequently updated, which can change their performance. For each benchmark run, we clearly document: 1) The specific model name and version, 2) The date of testing, 3) Any known model updates between runs. This allows for fair comparison across different benchmark cycles and helps track performance over time.

What about proprietary or specialized legal AI models?

Currently, our benchmark focuses on general-purpose AI models that are widely accessible. We may expand to include specialized legal AI tools in the future. If you're a legal AI vendor and would like your product included in our benchmark, please contact us. We maintain the same transparency requirements for all tested systems.

Can I use CiteClear's benchmark for my own research?

Yes. Our methodology and sample prompts are published for anyone to use. We only ask that you: 1) Cite CiteClear as the source of the methodology when publishing results, 2) Make your own results and methodology transparent, 3) Don't use our work for commercial benchmarking products without permission. Academic and journalistic use is encouraged.

Related Pages

AI Citation Checker for Lawyers

Lawyer-specific guidance on using CiteClear for AI-generated legal research.

How to Verify Case Citations

Step-by-step guide to manual citation verification.

Verify AI-Generated Citations

Comprehensive guide to AI citation verification.

Legal AI Citation Hallucination Examples

Real examples of AI-generated fake legal citations.

Fake Legal Citation Checker

Tool for detecting fake or hallucinated citations.

How to Check if ChatGPT Made Up Legal Citations

Focused guide on ChatGPT-specific citation issues.

Test AI citations yourself

While you wait for our benchmark results, use CiteClear's citation-only checker to test AI-generated citations yourself. Our deterministic approach provides immediate feedback on citation quality.

Check citations now →