From Hallucinations to Legislation: California Codifies AI Rules for Attorneys

Key Takeaways

• California's SB 574 establishes the nation's first statutory framework governing attorneys' use of generative AI.
• The central compliance question is no longer whether attorneys use AI but how they use it.
• Rather than regulating AI technology itself, SB 574 focuses on professional responsibility obligations, imposing requirements relating to verification, confidentiality, disclosure, citation review and attorney oversight.

California Becomes the First State To Statutorily Regulate Lawyers' Use of Generative AI

California has become the first state to expressly regulate attorneys' use of generative AI by statute. Signed into law on Sept. 30, SB 574 takes effect Jan. 1, 2027, and establishes new requirements governing attorneys' use of generative AI, including obligations relating to verification, confidentiality, disclosure, citation review and attorney oversight. The law also restricts arbitrators' use of generative AI in California proceedings. As a result, SB 574 applies to California-licensed attorneys, attorneys appearing before California courts, and arbitrators subject to California's arbitration procedures.

GenAI Is Broadly Defined

The law applies generally to "generative artificial intelligence," which is defined as "an artificial intelligence system that can generate derived synthetic content, including text, images, video, and audio that emulates the structure and characteristics of the system's training data." The definition would likely encompass widely used platforms such as ChatGPT, Claude, Gemini, Microsoft Copilot, and similar AI-powered drafting, research and content-generation tools.

The statute's broad definition may also raise questions about whether AI-enabled features embedded in legal research, document review and productivity tools fall within its scope.

New Obligations for Attorneys Using GenAI

SB 574 addresses several of the risks and concerns that have accompanied the increasing use of AI in legal practice, including hallucinated authorities, privilege concerns and the appropriate role of attorney oversight.

Among other things, attorneys using generative AI must now:

• Not "delegate the practice of law" to generative AI
• Take reasonable steps to verify AI-generated content
• Correct erroneous or hallucinated outputs
• Comply with new confidentiality requirements governing the use of nonpublic, personally identifying and confidential information
• Satisfy applicable disclosure obligations
• Personally verify citations included in California court filings

Of this list, one of the most consequential provisions concerns citations in all filings and other papers submitted to the California courts. In practical terms, attorneys remain responsible for ensuring that cited authorities exist and support the propositions for which they are offered, regardless of whether the citation originated with AI, another attorney, a legal research platform or another source.

Govern, Supervise, Defend

Although SB 574 is California's first statute specifically addressing attorneys' use of generative AI, its central premise is not new. ABA Formal Opinion 512 (2024) states that a lawyer's duties of competence, confidentiality, communication and supervision apply equally when using generative AI. Likewise, courts have repeatedly emphasized that attorneys remain responsible for the work product they submit, regardless of whether AI assisted in creating it.

Beyond these principles, however, SB 574 underscores a broader reality facing legal departments and law firms: Organizations may increasingly need to demonstrate that appropriate safeguards were followed when generative AI is used.

Recent disputes involving privilege, work-product protection, expert discovery and AI-generated content have focused not merely on whether AI was used but how it was used, what controls were in place and whether those controls can be demonstrated after the fact.

As AI adoption accelerates across legal practice, governance procedures, documentation protocols, and meaningful human oversight must evolve to ensure that AI-assisted work product can be explained, validated, and defended when challenged. Recent case law illustrates why.

• United States v. Heppner (S.D.N.Y. Feb. 2026): The court declined to apply privilege protections to communications with a public AI tool under the facts presented.
• Warner v. Gilbarco, Inc. (E.D. Mich. Feb. 2026): The court reached a different result in assessing work-product protections, finding a pro se plaintiff's use of a public AI tool did not waive work-product protection because the information had not been disclosed to an adversary or otherwise placed at substantial risk of reaching one, demonstrating that outcomes may depend heavily on the facts and the platform used.
• Conservation Law Foundation, Inc. v. Shell Oil Co. (D. Conn. May 2026): The court compelled discovery concerning an expert's use of generative AI, raising the possibility that prompts and outputs may become discoverable portions of an expert's methodology.

These decisions underscore a simple but important point: AI governance extends well beyond whether an organization permits the use of AI. The critical question is how that use is governed. Organizations using AI should be prepared to explain what tools are being used and for what purposes, how associated risks are managed, and what safeguards are in place. Accordingly, an effective AI policy should do much more than authorize AI use; it should establish a clear governance framework for its oversight and deployment.

As AI becomes increasingly integrated into legal and business workflows, organizations should evaluate whether appropriate governance structures are in place. This includes determining who approves AI tools, what information may be entered into those tools, how outputs are reviewed and validated, what safeguards exist for confidential information, and whether those controls can be demonstrated if challenged in litigation, discovery, regulatory inquiries or disciplinary proceedings.

Organizations should also revisit:

• AI governance policies
• Vendor agreements and confidentiality provisions
• Litigation and discovery protocols
• Expert engagement letters
• Attorney and employee training

Success in the age of AI will belong to organizations that pair innovation with accountability, supported by governance frameworks and documentation practices capable of withstanding scrutiny.

Conclusion

California may be the first state to enact attorney-specific legislation governing the use of generative AI, but it is unlikely to be the last. More importantly, SB 574 signals a broader evolution in AI regulation: the focus is shifting away from the technology itself and toward accountability for its use. The statute does not prohibit generative AI or attempt to regulate the underlying tools. Instead, it reinforces a principle that is likely to shape future legislative and regulatory efforts: professionals remain responsible for the accuracy, reliability, and integrity of work produced with AI assistance.

For legal departments and law firms, the lesson is clear: The question is no longer whether generative AI will be used in legal and business workflows but whether that use is properly governed, supervised and defensible. Organizations that establish thoughtful AI governance frameworks before they are called upon to explain them to courts, regulators or other stakeholders will be best positioned to realize AI's benefits while managing its risks.

BakerHostetler's White Collar, Investigations and Securities Enforcement and Litigation team and Intellectual Property team include experienced practitioners with deep knowledge of AI governance, compliance and risk management. If you have any questions about this alert or its implications for your business, please feel free to contact any of our experienced professionals.