Abstract
Machine learning (ML) enabled classification models are becoming increasingly popular for tackling
the sheer volume and speed of online misinformation and other content that could be identified as
harmful. In building these models, data scientists need to take a stance on the legitimacy,
authoritativeness and objectivity of the sources of “truth” used for model training and testing. This
has political, ethical and epistemic implications which are rarely addressed in technical papers.
Despite (and due to) their reported high accuracy and performance, ML-driven moderation systems
have the potential to shape online public debate and create downstream negative impacts such as
undue censorship and the reinforcing of false beliefs. Using collaborative ethnography and
theoretical insights from social studies of science and expertise, we offer a critical analysis of the
process of building ML models for (mis)information classification: we identify a series of
algorithmic contingencies—key moments during model development that could lead to different
future outcomes, uncertainty and harmful effects as these tools are deployed by social media
platforms. We conclude by offering a tentative path toward reflexive and responsible development
of ML tools for moderating misinformation and other harmful content online.
