The wordfreq Python library offers access to estimates of the frequency with which a word is used, in over 40 languages, based on data sources including social media, Wikipedia, news, books and web text.

According to a GitHub documentation for the project written by creator Robyn Speer, wordfreq will no longer be updated but its latest version will still be accessible.

Generative AI has “polluted the data”, Speer wrote in the documentation, saying: “I don't think anyone has reliable information about post-2021 language usage by humans.

“The open Web (via OSCAR) was one of wordfreq’s data sources. Now the Web at large is full of slop generated by large language models, written by no one to communicate nothing. Including this slop in the data skews the word frequencies.”

While there was previously “spam” in the wordfreq data sources, it was manageable and often identifiable, Speer noted. The arrival of large language models, however, has inserted generated text into the data. Speer wrote: “Large language models generate text that masquerades as real language with intention behind it, even though there is none, and their output crops up everywhere.”

Speer cited the example of the word ‘delve', which is overused by ChatGPT, according to analysis by Philip Shapira. 

Changes in the availability of online information from public sources, specifically Twitter (now X) and Reddit, also mean that data that was previously free is now “expensive” for open source purposes, cited by Speer as another reason for sunsetting the project.

The story was first reported by 404 Media.

Journalists, NGOs, and researchers use raw data from Twitter’s API to gather insights for research. The company ended free access to its API in February 2023, a few months after Elon Musk took over the business, prompting the Coalition for Independent Technology Research to warn that halting free access would disrupt research projects. Reddit also began restricting free access to its API in April 2023.