Analysis by the Complexity Science Hub shows the misinformation watchdog has a high level of reliability across countries and provides stable ratings for news sources
A recent study evaluating the NewsGuard database, a leading media reliability rating service, has found no evidence supporting the allegation that NewsGuard is biased against conservative news outlets. Actually, the results suggest it’s unlikely that NewsGuard has an inherent bias in how it selects or rates right-leaning sources in the US, where trustworthiness is especially low.
“It seems unlikely that NewsGuard has an inherent bias against conservative sources, both in selecting and giving them lower ratings. Instead, the US media system is flooded with right-wing sources that tend to not adhere to professional editorial practices,” says first author Jula Lühring, from the Complexity Science Hub (CSH).
Stable Trustworthiness Over Time
Lühring and her colleagues analyzed NewsGuard’s trustworthiness ratings for more than 11,000 news sources in nine countries: United States, Great Britain, Italy, Canada, France, Germany, Austria, Australia, and New Zealand. The results showed that the ratings have remained consistent since 2022, with particularly stable coverage in the US, France, Italy, Germany, and Canada.
“NewsGuard selects the majority of news sites using web traffic data,” explains Lühring, also a PhD student at the University of Vienna. “We manually checked sources in the US, UK, and Germany, and found that the database misses almost no news sites with substantial traffic. The sites it does miss are not systematically biased towards any political ideology.”
Lowest Trustworthiness in the US
US-based news sources, however, consistently received lower trustworthiness scores compared to those from other nations—particularly for right-leaning sources, according to Lühring. “Smaller (hyper-)partisan sources tend to lack editorial practices and transparency measures. Since these are key criteria for NewsGuard, judging based on NewsGuard criteria results in an objectively lower overall trustworthiness,” explains the researcher.
“Our findings add to previous findings by Lin et al. (“High level of correspondence across different news domain quality rating sets”, published in 2023 in PNAS Nexus), who showed that trustworthiness ratings of news sources by NewsGuard align with ratings of other fact-checking endeavors,” adds co-author Jana Lasser, a professor at the University of Graz and associate faculty member at CSH.
- The average credibility of media in Germany (77%) and Austria (77%) is significantly higher than in the U.S. (57%) and also exceeds the ratings for Italy (71%) and France (65%).
- In Germany and Austria, news sources with a more left-leaning ideological orientation are considered nearly twice as trustworthy on average as right-leaning sources—a pattern also observed in the U.S. “Numerous studies confirm that misinformation is predominantly spread within right-wing circles. Psychological research shows that people with right-wing ideological views are more likely to believe misinformation. Additionally, analyses of social media and browser data demonstrate that false information circulates more frequently within right-wing groups,” says Lühring.
- News related to health topics receive lower credibility ratings across all countries. In Germany, their average credibility is just above 50%, while in the U.S., it is significantly lower. “This could also be related to the debates surrounding COVID-19 vaccinations,” explains Lühring.
- In Austria, Germany, France, and Italy, local news sources receive the highest credibility ratings and perform very well compared to other categories in all nine countries studied.
Timely Analysis Amid Political Tensions
“The evaluation is particularly timely given the upcoming change in the US government. The incoming Trump administration regulators and far-right Republicans in Congress are not only trying to hamper the work of misinformation researchers in the US, but also accusing NewsGuard to systematically censor conservative news sites,” points out co-author Hannah Metzler, a resident scientist at CSH.
Lasser also notes that misinformation and disinformation have become a global concern in recent years. “Tools to measure the trustworthiness of sources—like the NewsGuard database—are therefore crucial to quantify the spread of untrustworthy information in online environments.”
Navigating the Challenges of Misinformation Research
While the study confirmed NewsGuard’s general reliability, the researchers also caution about the limitations of binary trustworthiness labels. These limitations, they argue, could affect the validity of some misinformation research, and they offer recommendations for more nuanced approaches when using the NewsGuard database.
“We found that using a binary ‘trustworthy’ vs. ‘not trustworthy’ classification is prone to changes in the database over time, potentially leading to large variations in the measured prevalence of untrustworthy information. Therefore, if possible, the continuous point score that provides a trustworthiness assessment on a more fine-grained scale should be used,” explains Lasser.
- U.S.-based fact-checking organization
- Tracks the most-used news sites via web tracking and supplements smaller ones with a team of journalists and fact-checkers in each country
- Evaluates based on nine criteria (five for credibility, four for transparency), including correction standards and repeated misinformation – Link
- Rating scale from 100% (highly credible) to 0–59% (proceed with caution) – Link
- Covers various news categories (politics, health, local news, etc.)
Not freely accessible; requires a paid subscription
About the study
The paper “Best practices for source-based research on misinformation and news trustworthiness using NewsGuard,” by Jula Lühring, Hannah Metzler, Ruggero Lazzaroni, Apeksha Shetty, and Jana Lasser, was published in the Journal of Quantitative Description: Digital Media.
