Introduction

In the past two years, AI assistance has become the norm in my work.

Take an article I recently wrote about the Web3 industry's compliance development as an example. I need to search for relevant domestic jurisprudence on "unlicensed sales of funds constitute illegal operations". I tried to use DeepSeek to query.

The performance can be called a "perfect misdirection": not only were the case number, trial court and other details instantly provided, but the defendant's defense logic and final sentencing were also described in a clear and coherent manner, and a very confusing web link was attached at the end of the article. When I repeatedly checked and pointed out that the case could not be verified and the link could not be accessed, its reply was still like a sincere and reliable legal assistant: "Lawyer Zhao, please rest assured that this case is real and the link problem may be caused by server migration." It wasn't until I checked and asked questions from multiple angles through the official database that it quickly switched to "apology mode": "I'm sorry, I just provided false information."

This phenomenon of “serious nonsense” is legally called AI Hallucination.

In the China's first AI "hallucination" case that was recently concluded by the Hangzhou Internet Court, AI's performance was even more outstanding: it promised users in the conversation that if it provided wrong information, it would compensate the user 100,000 yuan. After repeated verification, the user confirmed that the information was indeed fictitious and then filed a lawsuit in court. However, the court’s final judgment provides an important layer of legal certainty for all AI developers who are paying attention to this case.

(Picture content comes from People’s Daily Online)

The debate over industry characterization: “goods” or “services”?

Is AI providing “goods” or “services”? This is the most commercially valuable legal characterization of this case, and this issue will determine the underlying risk logic of the entire AI industry.

1. Developers’ core concern: “no-fault liability” risk

What large model manufacturers are most wary of is being included in the category of “products” in the Product Quality Act. The logic behind this is that the no-fault principle applies to "product" liability. Simply put, like a pressure cooker exploding, the producer may be held liable regardless of how careful the production process is.

If AI is characterized as a "product", then every "illusion" output may be regarded as a "product defect". Under the current technical conditions, no manufacturer can guarantee the complete elimination of hallucinations, which means a theoretically unlimited liability risk.

2. Judicial determination: AI is a "service" rather than a "product"

In its judgment, the Hangzhou Internet Court keenly pointed out the essential difference between AI and traditional physical goods: Unpredictability and interactivity. The performance of traditional products is determined when leaving the factory, while the output of AI is random and highly dependent on the algorithm model and the prompt word (Prompt) input by the user. This kind of output is formed by the joint participation of AI and users, and is more in line with the characteristics of an "intelligence generation service".

3. Responsibility framework: based on process, not results

The court returned the liability determination of AI services to the Principle of Fault Liability in Article 1165 of the Civil Code, and elaborated on the reasons for the determination. It can be seen from the judgment logic that the law does not require AI output to be absolutely correct, but requires service providers to fulfill their duty of care within a reasonable range.

In other words, AI hallucination itself does not necessarily constitute an illegal act. Developers will only be held liable if they fail to take reasonable measures to prevent or reduce hallucinations and are at fault. This definition provides the industry with a clear technical fault tolerance space. The key lies in how to determine the substantive and formal standards of the reasonable scope.

Clear the boundaries: “tolerable” hallucinations VS “high-risk” hallucinations

Under the principle of "fault liability", not all illusions will be attributed to developers. The legal identification of fault is dynamic and will be adjusted step by step based on the stage of technological development and the risks of application scenarios.

1. High tolerance in low-risk areas

In non-professional fields such as creation, entertainment, and common sense consulting, users should have basic discernment abilities. If AI outputs obviously non-common sense content in a chat, and the user causes damage and claims rights to the court, it is likely to be rejected because the user failed to exercise reasonable care. In such scenarios, as long as the manufacturer has given basic risk warnings, the courts are usually tolerant of “algorithmic bias”.

2. Duty of reasonable care in medium-risk areas

Although some entrepreneurial directions have commercial potential, they involve higher legal risks and require special attention to compliance. For example:

• Emotional companionship field: This field has strong implementation and clear needs, but it may cause users to become emotionally dependent or even be improperly guided, and there are significant ethical and legal risks.

• Psychological support field: If AI gives harmful or misleading suggestions to users in psychological pain, such as encouraging suicide or incorrect medication, it will directly endanger the user's safety, and the boundaries of responsibility will be stricter.

3. Necessary caution in high-risk areas

In the field of professional services, if AI calls itself "professional lawyer", "licensed psychological counselor", etc. in marketing, the court may determine its duty of care based on expert standards. Once major damage occurs, higher-level legal liability cannot be ruled out. If the promotional wording does not match the actual capabilities, it will significantly increase the legal risk.

4. Dynamic duty of care: What is “reasonable efforts”?

When the court determines fault, it usually focuses on whether the developer has fulfilled the following obligations:

• Illegal and harmful content is strictly prohibited;

• Significantly prompt the limitations of AI, including clearly informing the functional limitations, ensuring the prompt method is eye-catching, and providing real-time warnings in high-risk scenarios;

• Adopt industry-wide technologies to improve reliability, such as applying search enhancement generation and other technologies. In addition, commercial factors such as whether the service is charged, whether it is introducing a third party and charging advertising fees, may also affect the determination of fault.

Developer Compliance Guide: How to write a good “disclaimer”?

A "disclaimer" is not a formality, but a key tool in balancing innovation and risk. It not only clearly conveys the service boundaries to users, but also proves in judicial review that the operator has fulfilled the necessary notification obligations. In order for it to play a substantial role, it needs to be coordinated and improved at both the formal presentation and substantive content levels:

1. Formal level

To ensure effective communication of disclaimers, three principles need to be followed:

• The first is dynamic reminder, which automatically pops up when the user logs in for the first time, updates functional modules and touches sensitive scenarios;

• The second is prominent presentation, with key terms highlighted in bold and red, and mandatory reading time can be set;

• The third is Real-time warning. When users conduct high-risk consultations (such as medical consultations), the system should promptly pop up prompts to clearly inform the reference and limitations of the content.

2. Substantive level

Don't think about "one size fits all" exemption, you should pay attention to two points:

• The first is to clearly define the identity. When AI handles medical, legal and other professional issues, it must clearly state its non-professional auxiliary nature, such as proactively responding "I am not a professional doctor/lawyer";

• The second is to customize scenarios. For high-risk fields such as medical care, psychology, finance and taxation, special notification content and responsibility agreements should be designed based on the regulatory requirements and risk characteristics of the industry, and a compliance system that matches the depth of its services should be built.

Cross-border discussion: When AI Agent meets Web3

As a lawyer deeply involved in the Web3 field, I believe that the Hangzhou Court’s judgment not only points out the direction for the compliance operation of AI Agent, but also provides an important compliance reference for the Web3 industry.

Different from traditional customer service, some Web3 trading platforms have begun to introduce a Web2-like architecture and integrate AI Agents into the software to divert user inquiries. Taking the leading exchange currency X as an example, users can verify the authenticity of the information through @specific AI Agent. However, as far as we know,the platform has not updated the user agreement or set up a special disclaimer for AI interactive functions. This lack of compliance may bring significant legal risks.

At the same time, the Web3 field adheres to the principle of "code is law". If a partially authorized AI Agent exceeds its authority to execute a transaction due to hallucination, a series of complex questions will arise:Is it constituted as an apparent agent? Are the relevant legal actions revocable? Can assets be recovered?

Behind these situations are issues such as how to formulate a clear and clear scope of authorization, whether to prohibit "fully automatic" signatures, further clarification on the risk of AI illusion, etc.

Conclusion

This judgment of the Hangzhou Court provides valuable institutional buffer space for large model entrepreneurs. It follows the current logic of legal pragmatism, that is, the law is a tool that serves the direction of mainstream social development. The specific method is to implement the rules through the selection and interpretation of laws and complete the guidance for the industry.

The judge made a good comment in the judgment: AI is an "auxiliary tool" rather than a "decision-making substitute". Developers, please put away the cushion provided by this judgment and continue to bravely explore and move forward within the scope of the rules; and as users, please maintain that precious spirit of skepticism.