The Stanford study suggests that advanced AI systems are beginning to demonstrate expert-level performance in fields requiring nuanced judgment and reasoning, challenging assumptions about which forms of professional expertise will remain uniquely human. (Source: Image by RR)

Law Professors Favor AI Responses Across Thousands of Comparisons

A new study led by Julian Nyarko at Stanford Law School found that law professors overwhelmingly preferred AI-generated responses to legal questions over answers written by their fellow professors. In a blind evaluation involving nearly 3,000 comparisons, AI-generated answers won approximately 75% of head-to-head matchups, suggesting that advanced language models are becoming remarkably effective at navigating complex, judgment-based domains.

The research, as noted in law.stanford.edu, focused on contract law, a discipline that requires nuanced reasoning rather than simple factual recall. Sixteen law professors developed realistic student questions, wrote answers themselves, and then evaluated responses without knowing whether they were generated by AI or another professor. Researchers found that AI systems consistently produced answers that legal educators viewed as more helpful, more complete, and more educationally effective than many human-generated alternatives.

Perhaps even more surprising was the quality and safety of the responses. Professors flagged AI-generated answers as potentially misleading or pedagogically harmful only 3.5% of the time, compared with 12% for responses written by their peers. The findings challenge long-standing assumptions that fields requiring judgment, ambiguity, and interpretation would remain insulated from major AI disruption for longer than technical or routine professions.

The researchers caution that the study does not imply AI should replace legal educators. Instead, it highlights the growing potential of AI as an educational tool capable of expanding access to high-quality guidance and tutoring. More broadly, the results suggest that AI systems are increasingly demonstrating competence in areas once thought to require uniquely human expertise, raising important questions about the future role of professionals across knowledge-intensive industries.

read more at law.standford.edu