Training on viral junk posts dulls AI models' reasoning, Texas A&M study finds

Feeding an artificial intelligence model a steady diet of short, attention-grabbing social media posts can erode how well it reasons, according to research led by a Texas A&M University computer scientist.

Dr. Zhengzhong Tu, who teaches computer science and engineering at the university as an assistant professor, and his team borrowed the slang term "brain rot" for the effect. The university announced the work Monday and said the paper was accepted for presentation at this year's Conference on Language Modeling.

How the test worked

The researchers started with a million public posts on Twitter from 2010 that included engagement counts. They labeled a post as junk in two ways. One measure was popularity paired with brevity: posts with more than 500 combined likes, retweets and replies and fewer than 30 tokens, the word fragments AI models process.

The other was writing quality, with OpenAI's GPT-4o mini sorting out posts that leaned on all caps, hashtags, urgency or trivial topics. Three graduate students spot-checked those labels.

The team then trained four open models of different sizes and families, Llama3 8B Instruct and three Qwen models, on varying mixes of junk and higher-quality posts, and ran them through existing benchmarks that test reasoning, memory, personality and ethical norms.

What they found

The most common change was what the team calls thought-skipping: the models jumped to answers without working through the steps. Popularity of the training posts had the larger effect on reasoning, the release said, while post length mattered more for memory and comprehension. Some models also became more willing to answer harmful questions they had previously declined.

Retraining with better data or adjusting the models' responses did not fully restore performance, according to doctoral student Shuo Xing, a lead author. "It is very hard to recover the model's performance," Xing said.

Tu said the field has put too much weight on model design and computing resources. "We want to emphasize the importance of data," he said.

The researchers want to repeat the tests with posts from other platforms and with large commercial models such as ChatGPT and Gemini. They said that is not possible today because those companies do not disclose their training data, and Tu called for more open-source development so outside researchers can check data quality.

Sources

news.engineering.tamu.edu