This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility: The change was obvious—but in the end, things stayed about the same. Cornell researchers took advantage of online petition website Change.org's gradual rollout of an artificial intelligence writing tool to gauge AI-enhanced language's impact on petition outcomes.
An analysis of 1.5 million petitions showed that the AI writing assistance altered the wording of petitions, increasing their homogeneity and length, but did not improve petition outcomes, according to a research team that included Mor Naaman, the Don and Mibs Follett Professor of Information Science at Cornell Tech, the Jacobs Technion-Cornell Institute at Cornell Tech and the Cornell Ann S. Bowers College of Computing and Information Science. "That was a surprise for us," said Isabel Corpus, a doctoral student in information science and lead author of "Introducing AI to an Online Petition Platform Changed Outputs but Not Outcomes," which was published Sept. 30 in Nature Human Behaviour.
"There is this existing research that shows AI-generated text is persuasive, it's believable and it can be creative, sometimes more so than people writing alone," Corpus said. "And then on top of that, we would expect that a platform would integrate a tool that would improve outcomes, so we were surprised that it didn't." In addition to Corpus and Naaman, co-authors included Allison Koenecke, assistant professor of information science at Cornell Tech and Cornell Bowers; and Eric Gilbert, professor in the School of Information at the University of Michigan. Naaman said he heard about Change.org's introduction of an AI writing assistant a few years ago from an employee of the site.
When they looked into it, Corpus discovered that the tool was being rolled out gradually in English-speaking countries—first in the U.S., Great Britain and Canada, and 11 weeks later in Australia. That staggered rollout allowed the team to measure AI's impact by comparing petitions written before and after the introduction of the tool across the different countries. The researchers hypothesized that, since all four countries followed similar trends in petition style and outcomes before the AI tool's release, differences in trends during and after the 11-week rollout period—Oct. 2 through Dec. 15, 2023—could be attributed to the AI tool.
Although he couldn't say for sure, Naaman suspects the AI writing assistant Change.org employed was trained, in part, on successful petitions the site hosted previously. "We saw in the data that those [AI-enhanced] petitions had the markers of what was successful in the past," he said, noting that Change.org was one of the first online platforms to directly employ an AI writing assistant. The team measured the effect of AI on lexical style using three main features: lexical diversity, by calculating the ratio of unique words to total words; readability, determined by measuring the number of words per sentence and syllables per word; and petition length, including text and title.
What stood out was the consistent effect on lexical features: Petitions written with AI's help were longer, with more complicated and varied words. "We also saw a lot of homogeneity in the language of the titles," Corpus said. "Petitions started calling for people to 'implement,' 'mandate' or 'urge,' whereas before we rarely saw that these words were used." Using two metrics to gauge petition effectiveness (the share of petitions that reach one comment in 30 days and the share that reach 10 signatures at the time of data collection), the researchers found that outcomes did not improve compared with pre-AI levels and in some cases worsened.
Extract — continue reading at the source.
