Artificial intelligence can make public-facing writing longer, more varied and more sophisticated without making it more effective. A large natural experiment on Change.org found that access to an integrated AI writing tool substantially changed how petitions were written, but did not improve the engagement outcomes those petitions were trying to achieve.
The study, published in Nature Human Behaviour on 30 September 2026, analysed 1.5 million petitions written between 2022 and 2024. Researchers Isabel Corpus, Eric Gilbert, Allison Koenecke and Mor Naaman used the staggered introduction of Change.org’s AI drafting tool across countries to examine what happened when petition writers gained access to generative AI inside the platform itself.
A real-world test of AI-assisted civic writing
Generative AI is increasingly embedded directly into websites where people write product descriptions, social posts, applications and other public text. That makes an important question harder to avoid: does easier access to polished writing actually help people achieve what they want?
Change.org offered an unusually useful setting in which to investigate this. In 2023, the petition platform introduced an AI tool that could generate a petition draft from a short description supplied by a user. The tool was rolled out at different times in different countries. The United States, Great Britain and Canada received full access 11 weeks before Australia.
The researchers treated this staggered launch as a natural experiment. Their main analysis used a difference-in-differences design, comparing changes in the countries with AI access against changes in Australia during the period when Australia had not yet received the tool. The central comparison covered 2 October to 15 December 2023.
This design is stronger than simply comparing petitions that appear to have used AI with those that did not. People who choose to use AI may already differ from people who avoid it. Here, the researchers instead examined access created by the platform’s geographic rollout. They also tested whether trends were similar before the intervention and conducted robustness checks, including synthetic-control analyses using additional countries.
The writing changed quickly and substantially
The clearest effects appeared in the language itself. During the 11-week post-launch comparison period, AI access increased median petition length by an estimated 53.63 words, with a 95% confidence interval from 47.08 to 60.18 words. Before AI access, the median petition was about 110 words long. The estimated increase therefore represented roughly 49% more text.
Vocabulary also became more diverse. The Moving Average Type-Token Ratio, a measure of lexical diversity, increased by 0.049 units, about 6% from the pre-AI level of 0.8. The estimate was highly statistically significant.
At the same time, the median Flesch-Kincaid Grade Level rose by 1.49 grades, or approximately 19% from a pre-AI baseline around grade 8. In practical terms, the petitions became more linguistically complex as well as longer. The researchers also observed a shift in common wording. Simpler verbs that had been frequent before the AI rollout gave way more often to longer, more formal verbs.
The study’s sample for the principal time-window analysis was itself substantial. The pre-AI period covered 65 weeks and 237,020 petitions, the platform’s A/B testing period included 82,504 petitions, and the 11-week post-launch comparison included 42,111 petitions. The A/B testing period was excluded from the aggregate causal estimates because the researchers could not determine which users had access during those tests.
More polished petitions also became more alike
The changes were not limited to individual petitions. AI access also increased similarity across petitions, pointing to a broader homogenisation effect. The researchers found that petitions produced when the tool was available became more alike in their language and structure.
This matters because one of generative AI’s promises is that it can help people express their own ideas more effectively. If many users are channelled through the same model, prompt and interface, however, improvements in surface-level writing may come with a reduction in distinctiveness. For grassroots advocacy, local detail and an individual voice may be part of what makes a message persuasive.
The researchers estimated that more than half of petitions written during the differential-access period in countries with the integrated tool were AI-generated, based on a classifier developed specifically for the Change.org system. Importantly, the study defined treatment as access rather than detected use. That avoids relying on generic AI detectors as the basis for the causal comparison.
Better-looking text did not produce better outcomes
The central result was the gap between writing changes and practical outcomes. Despite becoming longer, more lexically diverse and more complex, petitions did not become more successful on the engagement measures examined by the researchers.
Access to AI did not improve the probability that a petition would receive at least one comment within 30 days. More strikingly, the share reaching at least 10 signatures declined by about 4.65 percentage points in the aggregate analysis. In other words, the measurable transformation in writing style did not translate into stronger early petition performance.
The researchers also examined repeat petition writers, providing a second way to test the pattern within people who had experience creating petitions both before and after AI access. Nature Human Behaviour reports that this analysis included 4,611 returning writers from treated countries. Their later petitions written with AI access were, on average, longer but had worse outcomes, reinforcing the platform-level findings.
The lack of improvement was not explained by a sudden flood of new users or a broad change in participation. Platform participation remained relatively stable following AI access. This helps separate the language effect from the possibility that the tool simply brought a different population of writers onto the platform.
Why polished writing may not be enough
The results challenge a simple assumption that better linguistic presentation automatically produces better real-world performance. Before the AI rollout, some of the language features changed by AI were associated with stronger petition outcomes. Once the tool made those characteristics easier to produce at scale, however, reproducing them did not create the expected benefit.
Several explanations are possible, but the study cannot definitively choose among them. AI-generated drafts may improve surface form without adding the personal knowledge, specificity or community context that makes advocacy compelling. Readers may react differently when writing appears generic or AI-assisted. Easier drafting could also alter how much ownership writers feel over a petition and how much effort they invest in promoting it after publication.
The finding therefore has implications beyond petitions. Platforms increasingly integrate AI into tasks where success depends not only on grammatical quality but also on authenticity, contextual knowledge, trust and follow-through. A tool can make the immediate output look stronger while leaving the larger human process unchanged.
Important limits to what the study shows
The natural experiment offers unusually strong real-world evidence, but it does not mean AI writing assistance is universally ineffective. The intervention involved one platform, one integrated model and prompt system, and a particular task. Results may differ for other writing tools, later models, different interfaces or settings where linguistic quality is more directly tied to success.
The treatment also measured access to the tool rather than confirmed use by every writer. Some people with access may not have used it, while some users in control settings could have used external AI tools. The difference-in-differences design reduces the threat from general off-platform AI use if it evolved similarly across countries, but it cannot observe every individual’s writing process.
Finally, signatures and comments capture only part of petition success. They do not establish whether a petition influenced a decision-maker, attracted media attention, changed policy or helped organise a community. The findings are strongest when interpreted as evidence about the measured engagement outcomes rather than every possible form of civic impact.
Even with those qualifications, the scale and quasi-experimental design make the study an important test of what happens when generative AI moves from a separate tool into the infrastructure of everyday online participation. The lesson is not that writing quality is irrelevant. It is that linguistic polish and practical effectiveness are not the same thing, and making polished language abundant can change the character of a platform without improving the outcomes users care about.
Source Information
Study: Introducing AI to an online petition platform changed outputs but not outcomes
Authors: Isabel Corpus, Eric Gilbert, Allison Koenecke and Mor Naaman
Journal: Nature Human Behaviour
Published: 30 September 2026
DOI: 10.1038/s41562-026-02580-8
