The ethical risks of AI in law: Hallucinations, confidentiality, and client consent

This blog is an excerpt from AI for the Rest of Us, a book by Nextpoint and Tom O’Connor that serves as a comprehensive resource for attorneys navigating AI in the legal field. Click here to learn more and get your copy.

While lawyers have benefited from artificial intelligence (AI) for years, the recent explosion of “generative” AI (GenAI) has caused consternation and apprehension in the legal community alongside the hype and excitement. Attorneys are now grappling with critical questions: Are they ethically permitted to use GenAI tools? To what extent? Could these tools introduce ethical conflicts or practical confusion? But of course, there are no easy answers.

These questions (and more) have prompted some states to issue ethics opinions providing guidance and direction. The New York State Bar Association Task Force on Artificial Intelligence issued a "Report and Recommendations" in April 2024 that provided some excellent explanations and information. But in the end, it stated that the task force offered “no conclusions” and encouraged readers to take on “the mindset of ancient explorers” when it comes to GenAI: “Be cautious, be curious, be vigilant, and be brave.”

AI ethics at a glance

• AI ethics requires balance: As the NY State Bar suggests, when using AI tools lawyers should “be cautious, be curious, be vigilant, and be brave.”
• Hallucinations pose real risks: AI can generate convincing but completely fabricated legal citations and authorities, as seen in cases like Mata v. Avianca, where lawyers faced sanctions after relying on GenAI without further verification.
• Three key ethical duties apply: Legal AI use must satisfy competence (understanding the technology), confidentiality (protecting client information), and supervision (overseeing use of AI by junior and non-attorney personnel).
• Client disclosure is important: Most ethics opinions recommend informing clients about AI use, while some jurisdictions suggest getting explicit consent.
• Security concerns remain significant: Since most AI tools process data on external servers, lawyers must carefully evaluate privacy, security, and privilege risks.
• AI doesn’t replace attorney judgment: Despite advances, ethical obligations require lawyers to maintain oversight and responsibility for all work product.

Delirious visions of automation

The headline-grabbing risk of using GenAI in a legal context is the possibility of a “hallucination.” More specifically, when a lawyer or lay citizen relies on a GenAI tool to help them draft a legal document, there is a material risk that the tool could fabricate a case opinion or rule of law that appears to support the human user's argument. This is the equivalent of a reputable lawyer making up a case opinion to support an argument because they couldn’t find any case law in their favor. By any measure of the profession, this would be utterly unacceptable, resulting in a scathing tongue-wagging from a court at best, and career-blemishing disciplinary action at worst.

The most well-known occurrence of this hallucinatory behavior came to light in the summer of 2023 when the plaintiff's attorneys in the case Mata v. Avianca, 678 F. Supp. 3d 443 (S.D.N.Y. 2023) filed a response to a motion that “cited and quoted from purported judicial decisions that were said to be published in the Federal Reporter, the Federal Supplement and Westlaw,” but those judicial decisions did not exist. When opposing counsel and the court could not locate the referenced case decisions, the court took swift action to impose sanctions on the lawyers, stating that they “abandoned their responsibilities when they submitted non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT.”

While the Mata v. Avianca blunder has been discussed in many places and from many angles, the most important takeaway is that the technology is not to blame – it came down to inadequate lawyering. If these attorneys had tasked a young associate to write a brief, they would have expected the associate to only use actual, verifiable case opinions. Presumably, the managing lawyer also would have reviewed and vetted the work before filing it with the court. 

Unfortunately, the Mata v. Avianca case has not deterred other lawyers and citizens from relying solely on GenAI tools such as ChatGPT to write legal briefs and filings. Other lawyers have been sanctioned for much the same actions. Even more sadly, several pro se litigants have been caught in this trap, presumably because they were trying to use freely available tools instead of retaining professional lawyers.

Ethics opinions addressing lawyers using AI and GenAI

In addition to court proceedings where lawyers and litigants have been sanctioned for irresponsibly using GenAI to compose legal filings, there are several ethics opinions that provide guidance for lawyers regarding their use of AI.

One of the first was jointly issued in June 2024 by the Pennsylvania Bar Association and Philadelphia Bar Association. Other states, including California, Illinois, West Virginia, Florida, Missouri, and more, have offered similar guidance and information. On July 29, 2024, the American Bar Association issued Formal Opinion 512 on the use of “Generative Artificial Intelligence Tools,” which provided some general guidelines for legal professionals.

While the ethics opinions all include necessary definitions, explanations, and general education, they also address three main ethical themes based on the ABA Model Rules of Professional Conduct: Competence, Confidentiality, and Supervision.

Competence (ABA Model Rule 1.1)

The first ethical theme is competence, and more specifically, technical competence (but not “expertise”!). Model Rule 1.1 mandates that lawyers provide competent representation. This has historically referred to ensuring lawyers are up-to-speed on any recent changes in the areas of law where they practice so they can provide the most up-to-date and effective representation for their clients.

In 2012, Comment 8 to Model Rule 1.1 was amended to emphasize that “competent representation” also required lawyers to be competent in the “risks and benefits” associated with the technology that they use in their practice. This includes knowing how to use proper passwords and encryption to adequately protect client confidential information. It can also apply to using mobile devices for communication, ediscovery tools for document review, and accurate means to capture time so that bills and invoices can be prepared and sent in a timely manner.

As the Pennsylvania Joint Formal Opinion 2024-200 puts it: “Lawyers must be proficient in using technological tools to the same extent they are in employing traditional methods. Whether it is understanding how to navigate legal research databases, use ediscovery software, use their smartphones, use email, or otherwise safeguard client information in digital formats, lawyers are required to maintain competence across all technological means relevant to their practice.”

In today’s world, technology competence absolutely applies to the use of AI and GenAI, requiring legal professionals to understand the “risks and benefits” of those tools in a legal context. For example, the benefits of AI may allow for greater efficiencies or lower costs in legal work. But lawyers must also understand the risks of using GenAI tools, such as the possibility of hallucinations that generate false or inaccurate information.

As the same Pennsylvania opinion warns: “Rather than focus on whether a lawyer’s choice of specific legal arguments has merit, some lawyers have used Generative AI platforms without checking citations and legal arguments. In essence, the AI tool gives lawyers exactly what they were seeking, and the lawyers, having obtained positive results, fail to perform due diligence on those results.”

The competency mandate does not end there – it is an overarching requirement that affects almost all of the ethical themes related to AI and GenAI tools (including the next two).

Confidentiality (ABA Model Rule 1.6)

Perhaps there is no greater awareness about a lawyer’s duty to their client than the requirement that they keep client information confidential. Model Rule 1.6 states that it is the attorney’s duty not to reveal information relating to the representation of a client unless the client gives informed consent. The “risk” comes into play when lawyers provide AI tools with access to client confidential information, which could be used for further training in the language model. Many AI companies are becoming more transparent about where your data is stored and how it may or may not be used for training. But regardless, lawyers should ask the right questions to understand where data is being stored and how it is accessed. 

How are lawyers supposed to know where all this information is stored? It’s not easy, and in many cases the technology company may not provide sufficient answers. Does that mean lawyers cannot use these tools? Possibly – and that may depend on the risk tolerance of the lawyer and the client they represent. But a good rule of thumb that every lawyer should follow is that if there is a question or hesitation, then perhaps it is best to demur on the tool until such time that more transparent information can be obtained and verified.

Supervision (ABA Model Rules 5.1 and 5.3)

Lawyers also have an important and ongoing obligation to oversee the work of those acting on their behalf – including colleagues, employees, consultants, and outside vendors. This duty ensures that all conduct performed under their supervision complies with ethical obligations and applicable rules of professional conduct.

This supervisory obligation applies to lawyers overseeing personnel that might be using AI tools to accomplish the work required for the practice. This responsibility requires competence in the tools being used as well as assurance that the tools keep client information confidential.

Protecting confidential data in the world of AI

Maintaining client confidentiality has been a core responsibility for attorneys since the dawn of our legal system – long before the current deluge of digital evidence and data privacy concerns complicated the matter. Legal professionals have an ethical and legal obligation to protect the data entrusted to them, whether it’s stored on a laptop, the cloud, or an AI server.

While major AI companies promote their security measures and compliance standards, the complexity of AI systems creates new challenges for data transparency and accountability. Users often cannot verify exactly how their data is processed, stored, or potentially used for model improvement, even when companies provide assurances about data protection.

Some key considerations regarding data privacy and security include:

• How is the data protected?
• Is it encrypted in transit and at rest, and who controls the encryption keys?
• What are the vendor's data retention policies?
• Do any of the vendor's employees view prompts or submitted data during the law firm’s use of the product?
• What AI model is the vendor using?
• Is it open source or proprietary?
• Where is the model hosted?
• Will the firm’s data be used for training? If so, how?
• Does the model use law firm prompts or other information for training or customization?

Additional guidelines from state bar associations

While the three ethical themes of Competence, Confidentiality, and Supervision outline the primary focus of responsibilities for lawyers, particular states may present additional obligations or guidelines. Two key recommendations from the state level center on client confirmation and consent.

First, the State Bar of California issued “Practical Guidance for the Use of Generative AI in the Practice of Law,” recommending that lawyers inform and disclose to their clients if generative AI tools will be used as part of their representation. This is a general guideline, following an increasingly common premise that the client should be aware of the use of AI in their matter.

Second, the Florida Bar Board of Governors’ Review Committee on Professional Ethics suggested in Proposed Advisory Opinion 24-1 that lawyers not only inform clients that they intend to use GenAI tools, but also obtain informed consent from clients before utilizing such tools in their representation.

There may be additional states that require lawyers to disclose and/or obtain consent from clients, but a safe approach today would be for lawyers to be transparent about their intention to use GenAI tools and inform the client exactly how they intend to ensure the use is accurate and effective.

Remember: AI is not a replacement for Attorney Intelligence

We’ve often said that AI should be an acronym for Attorney Intelligence – otherwise, we’re just left with piles of math. It can be easy to lose sight of this, especially when we’re inundated with predictions claiming that advances in AI will lead to the end of the legal profession. A New York Times article from April 2023 highlighted how, "More than a decade ago, lawyers were singled out as an endangered occupational species, their livelihoods at risk from advances in artificial intelligence." But this didn’t happen. In fact, the growth of the legal profession outpaced the overall growth of the American workforce. Still, The Times asked, “Will the pessimists finally be right?” Our answer to this question is an emphatic NO.

The ABA Formal Opinion 512 on AI supports this belief by reinforcing the indispensable role of human judgment in the legal field:

“While GAI may be used as a springboard or foundation for legal work—for example, by generating an analysis on which a lawyer bases legal advice, or by generating a draft from which a lawyer produces a legal document—lawyers may not abdicate their responsibilities by relying solely on a GAI tool to perform tasks that call for the exercise of professional judgment. For example, lawyers may not leave it to GAI tools alone to offer legal advice to clients, negotiate clients’ claims, or perform other functions that require a lawyer’s personal judgment or participation. Competent representation presupposes that lawyers will exercise the requisite level of skill and judgment regarding all legal work. In short, regardless of the level of review the lawyer selects, the lawyer is fully responsible for the work on behalf of the client.”

No matter how exciting or advanced an AI tool may be, your ethical obligations require that you continue using your Attorney Intelligence to serve your clients.

Questions to consider for ethical and legal compliance with AI

Working through this checklist of questions will help you ensure your AI usage stays aligned with legal and ethical standards.

‍Data protection and privacy:

• Does your use of AI systems adhere to relevant data protection laws regarding client confidentiality and privacy? (e.g., the California Consumer Privacy Act or the European Union’s General Data Protection Regulation)
• Have you implemented proper safeguards to ensure data encryption and secure storage when using AI tools?

Transparency and explainability:

• Can AI-generated recommendations or decisions be easily explained to clients, stakeholders, and courts (if necessary)?
• Does the AI tool provide transparency into its decision-making process and the factors it considers when making predictions or suggestions?

Compliance with legal standards:

• Does the AI tool comply with industry-specific regulations (e.g., Sarbanes-Oxley for financial transactions, HIPAA for healthcare-related legal matters)?
• Are AI systems regularly updated to reflect new laws, regulations, or legal precedents that may impact your practice?

‍Informed consent:

• Have clients been informed about how AI is being used in their legal matters (e.g., contract review, case analysis)?
• Have clients consented to the use of AI in their legal proceedings?

Hear the authors talk AI ethics in practice

Want to go deeper on hallucinations, confidentiality, and keeping attorney judgment at the center of your AI use? Watch the on-demand author panel for Artificial Intelligence for the Rest of Us, where Tom O'Connor, Brett Burney, and Elizabeth Guthrie share practical strategies for adopting AI responsibly, evaluating tools and vendors, and avoiding the mistakes other lawyers have made in court.

{{wbn-ai-author-panel}}

Frequently asked questions

Can lawyers ethically use generative AI? Yes, as long as they meet their existing ethical obligations. ABA Formal Opinion 512 and a growing number of state bar opinions allow lawyers to use generative AI tools, provided they understand the technology, protect client confidentiality, supervise anyone using AI on their behalf, and verify the output. The lawyer remains fully responsible for any work product, regardless of whether AI helped create it.

What is an AI hallucination, and why does it matter for lawyers? An AI hallucination is output that sounds credible but is fabricated, such as a case citation, quote, or legal authority that doesn't exist. For lawyers, submitting hallucinated content to a court can lead to sanctions, disciplinary action, and lasting reputational damage. Every citation and legal argument generated by AI should be independently verified before it's filed or relied on.

What happened in Mata v. Avianca? In 2023, plaintiff's attorneys in Mata v. Avianca filed a brief citing judicial decisions that ChatGPT had invented. When neither opposing counsel nor the court could locate the cases, the Southern District of New York sanctioned the lawyers for submitting nonexistent opinions with fake quotes and citations. The key lesson is that the failure came from inadequate lawyering, not the technology itself.

What is ABA Formal Opinion 512? ABA Formal Opinion 512, issued on July 29, 2024, is the American Bar Association's formal ethics guidance on lawyers' use of generative AI tools. It applies the existing Model Rules of Professional Conduct to AI, with a focus on competence, confidentiality, client communication, supervision, and reasonable fees. It also makes clear that lawyers cannot hand off tasks requiring professional judgment to AI alone.

Which ABA Model Rules apply to lawyers using AI? The three primary rules are Model Rule 1.1 (competence), Model Rule 1.6 (confidentiality), and Model Rules 5.1 and 5.3 (supervision). Comment 8 to Rule 1.1, amended in 2012, requires lawyers to understand the benefits and risks of the technology they use, which now includes AI and generative AI tools.

Do lawyers have to tell clients they're using AI? It depends on the jurisdiction, but transparency is the safest approach. The State Bar of California recommends that lawyers disclose their use of generative AI to clients, and the Florida Bar goes further by advising lawyers to obtain informed consent in certain circumstances. Even where it isn't required, telling clients how you use AI and how you ensure its accuracy builds trust.

How can lawyers protect client confidentiality when using AI tools? Lawyers should vet AI vendors carefully before entering any client information. Key questions include whether data is encrypted in transit and at rest, who controls the encryption keys, what the vendor's retention policies are, whether vendor employees can view prompts, where the model is hosted, and whether client data is used for training. If the vendor can't give clear answers, it's best to hold off on using the tool for confidential matters.

Will AI replace lawyers? No. AI can serve as a starting point for research, analysis, and drafting, but ethics rules require lawyers to exercise their own professional judgment on all legal work. As ABA Formal Opinion 512 puts it, the lawyer is fully responsible for the work on behalf of the client, regardless of how much AI was involved.