(click to copy)

Publication

Best practices for source-based research on misinformation and news trustworthiness

Researchers need reliable and valid tools to identify cases of untrustworthy information when studying the spread of misinformation on digital platforms. A common approach is to assess the trustworthiness of sources rather than individual pieces of content.

One of the most widely used and comprehensive databases for source trustworthiness ratings is provided by NewsGuard. Since creating the database in 2019, NewsGuard has continually added new sources and reassessed existing ones.

While NewsGuard initially focused only on the US, the database has expanded to include sources from other countries, such as Germany and Italy. In addition to trustworthiness ratings, the NewsGuard database contains various contextual assessments of the sources, which are less often used in contemporary research on misinformation.

In this work, we provide an analysis of the content of the NewsGuard database, focusing on the temporal stability and completeness of its ratings across countries, as well as the usefulness of information on political orientation and topics for misinformation studies.

We find that the trustworthiness ratings are temporally stable and that coverage seems complete for the US and Germany. Information on the political orientation and topic labels of sources are relatively complete and can be valuable assets in characterizing sources beyond trustworthiness.

By evaluating the database over time and across countries, we identify potential pitfalls that threaten the validity of using NewsGuard as a tool for quantifying untrustworthy information.

Lastly, we provide recommendations for digital media research on how to avoid these pitfalls and discuss appropriate use cases for the NewsGuard database and source-level approaches in general.

J. Lühring, H. Metzler, R.M. Lazzaroni, A. Shetty, J. Lasser, Best practices for source-based research on misinformation and news trustworthiness, pre-print (2024).

 

Jula Lühring, PhD candidate at the Complexity Science Hub © CSH

Jula Lühring

Hannah Metzler, researcher at the Complexity Science Hub © Verena Ahne

Hannah Metzler

Apeksha Shetty, PhD Candidate at the Complexity Science Hub © Gulden Timur

Apeksha Shetty

portrait of Jana Lasser

Jana Lasser

0 Pages 0 Press 0 News 0 Events 0 Projects 0 Publications 0 Person 0 Visualisation 0 Art

Signup

CSH Newsletter

Choose your preference
   
Data Protection*