In Pursuit of Gen-AI Incidents: Alternative “Theories-In-Use” in Trust Infrastructures
Generative AI (GenAI) has rapidly become a general-purpose technology capable of producing content across varied formats, including text, images, sound, video, and code. The technology’s emergent properties are powerful yet the potential impact, both positive and negative, of this technology is fiercely debated across business and society. Many initiatives in government and civil society have begun to address societal concerns related to its responsible use.
We focus on initiatives concerning AI incidents. We use a well-accepted definition of incidents from the information systems literature: “…incidents are events that go against the desired course of actions and events as intended by certain users and stakeholders of the system (Tyre and Orlikowski 1994)” (Mehrizi et al., 2022). Among the plethora of documentation initiatives, we focus on four incident databases (OECD, AIAAIC, AIID, CyberPeace). The common objective of these sociotechnical infrastructures is to increase digital trust by rendering AI-mediated incidents visible. These databases function as transparency anchors into AI failures.
The databases share commonalities but also differ in their specific goals, values, stakeholders, sourcing and curation arrangements, and intended influence. None claims to be comprehensive or fully representative. Hence, it is important to understand, individually and collectively, which incidents become visible, which remain invisible, and how incident categories evolve over time. Those involved with databases acknowledge that many incidents are overlooked. For example, our analysis suggests that expectations expressed in global trust dialogues emphasize vulnerabilities that rarely appear in these databases, such as manipulation, epistemic injustice, subtle bias, and erosion of agency and attention. Yet much has already been learned from these databases. For example, published research leveraging them as data sources has made valuable contributions to the literature on responsible AI use.
Interestingly, there is little research on the incident databases themselves. We lack an understanding of the theories-in-use of digital trust in designing and maintaining the databases. These theories would address how incidents should be governed from the perspectives of society, industry, and citizens. Some databases view incidents as upstream
problems—issues in training data, AI algorithm design, risk management approaches, and policy instruments—where the focus is on pressuring the technology industry such as platform owners, and governmental policy makers to take preventative actions. Others see the problems as more emergent and heterogeneous, depending on the user and context; these incidents are potentially addressable downstream through user outreach and training and by supporting users with methods and tools to bound and contextualize response generation.
Beyond the upstream/downstream distinction, there are differences in expectations about the knowability and unknowability of incidents and harms. Typologies vary in how they link to accepted regulatory or legal definitions versus dynamically evolving with incidents. Standardized definitions are viewed as legitimate and facilitate trend analysis over time but may miss newly emerging vulnerabilities—and both their upstream and downstream consequences—for how GenAI-mediated content is sourced, manipulated, interpreted, governed, and trusted. These differences of upstream/downstream and knowability/unknowability suggest different theories-in-use regarding digital trust.
Accordingly we ask: How and why do these databases promote digital trust through their “theories-in-use”?
We will present our preliminary findings. We will share short cases on these four AI-related harm databases and summarize interviews with their founders or directors on how curation decisions are made, who gets to report incidents, and who audits them. Because the databases heavily rely on journalistic reporting of incidents, we also conducted interviews with investigative journalists and fact-checkers to understand how GenAI content appears in their workflows, what makes identification difficult, and what kinds of harms are reported, overlooked, and underreported. We interviewed journalism academics to understand how and to what extent GenAI has changed journalistic practices. Finally, the first author has been active in digital trust conventions and has conducted informal conversations and observations.
Our preliminary findings are important, as future theories and empirical research can further illuminate the consequences of these digital trust–building efforts.
Sirkka Jarvenpaa is a Professor and Endowed Chair of Information Systems at the McCombs School of Business, University of Texas at Austin, where she directs a Center for Business, Technology, and Law. She received her Ph.D. in Business Administration from University of Minnesota. Her research is focused on dispersed complex collaboration that span functions, organizations, geographies, cultures, technologies, etc. Dr. Jarvenpaa has studied collaborative projects in security related activities in global companies and in public sector. She is particularly known for her work in global virtual environments where fast response teams work on urgent and adhoc problems. She has conducted her research with funding from the National Science Foundation, Society for Information Management, and governmental research funds and foundations in Finland and Sweden. She has published her research in premier academic journals as well as practitioner journals.
We use cookies to analyze website traffic and optimize your website experience. By accepting our use of cookies, your data will be aggregated with all other user data.