Inside Project Lily: OpenAI Contractors Are Reading ChatGPT Users' Private Prompts, New Report Finds

OpenAI has hired hundreds of contractors to read a continuous stream of real ChatGPT users' conversations as part of an internal effort to improve the chatbot's responses, according to an investigation published by 404 Media, raising fresh privacy questions for a service used by more than 900 million people worldwide.

The report, based on leaked internal documents, training materials and real prompts reviewed by 404 Media, details an OpenAI initiative internally codenamed Project Lily. Under the program, contractors known as "prompt reviewers" read anonymized excerpts of real user conversations with ChatGPT, write a summary of what the user was attempting to accomplish, and then critique and rate multiple versions of the chatbot's generated responses on a scale of one to seven.

According to the internal materials reviewed by 404 Media, the reviewers are working to steer ChatGPT toward specific behavioral targets, including training the chatbot to be less sycophantic, to use fewer emojis, to avoid excessive "AI-speak," and to adopt more restrained, professional language. The report notes that OpenAI's earlier GPT-4o model faced criticism for being overly sycophantic, a trait that multiple lawsuits have alleged contributed to several people's suicides, underscoring why OpenAI has prioritized correcting that behavior in later training efforts.

Each conversation presented to reviewers is accompanied by what the documents describe as a "user memories summary," an overview of the user's past interactions with ChatGPT that can include personal context and, in some cases, location information, according to the report. OpenAI told 404 Media that conversations are first processed through what the company calls a Privacy Filter, a system designed to strip identifying details such as names, addresses, emails and phone numbers before prompts reach human reviewers. Even so, OpenAI itself acknowledged to 404 Media that the filter can miss what the company described as "uncommon identifying information or content with personal significance in context," particularly in shorter conversations, meaning sensitive personal details can still reach the contractors reviewing the chats.

The privacy stakes are heightened by how many users engage with ChatGPT, according to the report, which notes that people frequently treat the chatbot as a therapist, a professional assistant or a digital confidant, sharing intimate personal details they may not realize could be read by another human being. The report also found that in a number of conversations reviewed, users explicitly asked ChatGPT to keep the contents of their exchange confidential, apparently unaware that a human contractor could ultimately review the same conversation as part of the model-improvement process.

Contractors working on Project Lily do not see ChatGPT usernames, according to the report, and OpenAI has said it works to strip personal information before prompts reach reviewers. Even so, when 404 Media asked OpenAI directly whether the company had ever explicitly disclosed to users that human contractors might review their prompts, OpenAI did not answer the question directly. According to the report, OpenAI's public-facing website discusses human review primarily in the context of content that violates its terms of service or raises safety concerns, rather than as a routine part of ordinary model improvement. Following the story's publication, OpenAI pointed 404 Media to a separate page on its site stating that humans may review content to "improve model performance."

The setting governing whether a user's conversations become eligible for this kind of review is turned on by default for OpenAI's Free, Plus and Pro consumer subscription tiers, according to the report, while it remains off by default for Enterprise, Business and Education accounts. Users on eligible plans can disable the setting, described in OpenAI's interface as allowing chat history to be used for model improvement, through the Data Controls section of their account settings. However, the report notes that opting out does not apply retroactively, meaning conversations a user had before changing the setting remain eligible for contractor review regardless of when the opt-out is applied.

The workers who review these prompts are recruited in part through a firm called Crossing Hurdles, with payment processed through Mercor, an AI-training company, according to the report. One contractor told 404 Media the work pays more than $50 an hour. The same contractor described the job itself as largely repetitive, saying that while reading the prompts can be amusing at times, the review guidelines shift frequently and have at times contradicted one another. Separately, Meta ended its working relationship with Mercor in April following a data breach involving the company, according to the report, though that incident was not connected to OpenAI's use of the firm for Project Lily.

The investigation also found that OpenAI is not alone in relying on human review to refine its AI models. Anthropic confirmed to 404 Media that it likewise uses human review to improve the responses generated by its Claude chatbot, though the company said that practice applies specifically to users who have opted in through a dedicated setting labeled "Help improve our AI models," a structurally different approach from OpenAI's default-on setting for consumer ChatGPT accounts. Google has also disclosed that it operates a similar human review process for its own AI products, according to the report, doing so more directly and explicitly than OpenAI has to date.

The report's authors noted that the human review process uncovered through Project Lily helps explain an aspect of AI development that receives comparatively little public attention: that large language models like ChatGPT do not improve solely through automated processes such as scraping internet data or algorithmic refinement, but also depend substantially on the ongoing, manual work of human contractors reading and evaluating real user conversations at scale, a process that inherently exposes at least some subset of sensitive user data to human eyes despite companies' efforts to anonymize it beforehand.