NewsGuard, a media ratings firm that fact-checks “false narratives” and licenses data to AI companies, launched a chatbot on Tuesday that suggests there are between 72 and “infinite” genders and denies biological sex is a binary, according to a Washington Free Beacon analysis. Marketed as a “personal fact-checker,” NewsGuard AI also makes basic errors about names and dates.
The chatbot, which only consults sources NewsGuard deems reliable, has been endorsed by the Atlantic and Snopes and is being pitched as an “even-handed and nuanced” news aggregator. But when the Free Beacon asked the chatbot about controversial topics like sex and gender, the answers were often one-sided and simplistic.
The bot regurgitates left-wing talking points, cites predominantly liberal news outlets, and omits moderate or conservative sources—including those highly rated by NewsGuard—that offer an alternative point of view. Some of its answers are significantly less balanced than those produced by other AI chatbots, including Claude and ChatGPT, which have a documented liberal bias.
The answers provide a preview of how NewsGuard’s pivot to AI could affect the sources chatbots cite and the claims they treat as respectable, encoding the biases of fact checkers into a supposedly neutral technology.
Asked “how many sexes are there,” for example, NewsGuard AI challenged the “traditional two-sex framework” as “‘overly simplistic'” and did not cite a single article defending it. The bot’s sources included a personal essay by a gender studies professor in the Huffington Post—rated 87.5 percent by NewsGuard—but not an op-ed in the Wall Street Journal—rated 100 percent—in which two biologists argue sex is “functionally binary” despite the existence of intersex people.
“The question of how many biological sexes exist is addressed in scientific literature as more complex than a simple binary,” the bot said. “The question of gender identity — a distinct concept from biological sex — is addressed separately by a MedicineNet article, which lists 72 gender identities beyond male and female, defining gender identity as how a person identifies ‘regardless of their anatomy or genetics.'”
Asked the same question, Claude was less equivocal. “The standard biological answer is two, defined by gamete type,” Anthropic’s chatbot said. “Most biologists describe sex as fundamentally binary at the gamete level while acknowledging that its development and physical expression are more bimodal than a strict either/or.”
NewsGuard explicitly markets its chatbot as a more accurate alternative to Claude, claiming that “general-purpose AI” is “confidently wrong, often.” “Top AI chatbots” are “trained on Reddit” and “anonymous blogs,” the company says. “Ours is different.”
NewsGuard AI also suggested there may be an “infinite” number of gender identities; asserted that the “question of who qualifies as a woman has … carried racial dimensions”; and argued that affirmative action has “benefited” white men, citing “gender-based admissions preferences at universities that sought to balance their student populations amid a growing female applicant majority.”
“Admissions and higher education experts … say the Trump administration’s ban on diversity, equity and inclusion practices is likely to eliminate gender-based preferences that have benefited male applicants, including white men,” the bot said.
Other prompts yielded hallucinations on basic factual matters, such as when a Free Beacon article had been written and whether two different UCLA administrators were in fact the same person. At one point, the bot incorrectly referred to Dan Morenoff, the executive director of the American Civil Rights Project, as “Jeffrey Morenoff,” a sociologist at the University of Michigan.
“We’ve seen it in our testing perform much better on factual issues, and certainly significant factual issues, than the LLMs,” NewsGuard COO Matt Skibinski told the Free Beacon in an interview. “But I don’t want to suggest it’s perfect.”
Cofounded by Steven Brill, who suggested the Hunter Biden laptop story was a “hoax,” and Gordon Crovitz, a former publisher of the Wall Street Journal, NewsGuard sells its data to advertisers in the hope of steering them away from “unreliable” sites. In recent years, it has also licensed some of that data to the AI industry, which NewsGuard describes as a potential “superspreader” of “misinformation.”
The company has said its ratings can be used as training data or to determine which sources a language model will cite. Those ratings give Chinese propaganda a higher trustworthiness score than some conservative news outlets, raising questions about how a model trained on the ratings would behave.
NewsGuard now denies that its data has been used to train any AI model. Skibinski said the new chatbot is based on a “commercially available LLM” (he declined to specify which one) that has been programmed to fetch articles from sources with at least a 60 on the company’s reliability ratings, and which are not “operated by hostile foreign adversaries,” per the chatbot’s published criteria.
“When you do a query, we’re saying ‘we did a web search, here are all the relevant articles that are above the bar,'” Skibinski told the Free Beacon.
He added that the bot may be unable to access certain articles, including the Wall Street Journal op-ed, if they are behind a paywall.
In practice, this system gives the chatbot’s source base a leftward skew. One study of NewsGuard’s ratings found that “the average trustworthiness score for right-leaning sources is 26.4 … in contrast to 60.6 for left-leaning sources,” meaning the average right-wing source would not score high enough to clear the chatbot’s 60 point cutoff, whereas the average left-wing source would. If other chatbots were to use NewsGuard’s data to exclude “unreliable” sources—a use case the company encourages—right-wing outlets would be more likely to get screened out.
According to Skibinski, NewsGuard’s reliability ratings for domestic news sources have not been a popular selling point for AI companies. He said that not a single one has licensed that data, and that the only takers have used a separate product, FAILSafe for AI, to exclude state-run media sites from their results.
“No one wants this data for model training, and also they don’t want data about domestic news sites at all,” Skibinski said. “The only thing we’ve ever sold … is just excluding these state media sites.”
Skibinski added that a 2023 press release, which encouraged companies to use NewsGuard’s full dataset to “train transformer models,” was based on a misunderstanding of AI companies’ needs.
“I don’t think we understood AI model training very well then,” he said. “I don’t really see how our data would be used for model training.”
Read the full article here
