July 24, 2024

expert reaction to AI models collapse when trained on recursively generated data

A study published in Nature looks at the collapse of AI models when trained on recursively generated data.

Dr Deepak Padmanabhan, Senior Lecturer, School of Electronics, Electrical Engineering and Computer Science, Queen’s University Belfast (QUB), said:, said:

” This research is very exciting, and illustrates – very clearly – one of the fundamental limitations of AI models as they are stand today. The empirical analyses and quantification of the phenomenon of model collapse will hopefully open the community’s eyes towards this issue, and other structural issues of this nature. It would be interesting for readers of this research to be aware of the broader political backdrop within which AI models are designed, trained and deployed. It is well known, primarily due to celebrated Austrian philosopher Ludwig Wittgenstein, that any representation tends to be reductive, in that it will exclude something. Wittgenstein was concerned about the reductivism in natural language (e.g., English, Irish) in building a representation of the world, but AI models build representations of languages which brings another layer of reductivism.

The question here is, what kind of things do AI models and their reductivism exclude. AI models predominantly operate using logics of the market, where the intent is to cater to as many users as possible. This is the philosophical position of utilitarianism, or offering the greatest good to the greatest number. This helps the owners of AI models – which are big tech firms such as OpenAI, Google and Facebook – increase their market presence. While utilitarianism has its upsides, the problem with it is that it tends to privilege mainstream perspectives and exclude fringe perspectives. For example, there may be many ways of getting hold of a book that one likes. One could go and visit the local library, or a village reading room, or a book re-seller, or even join a book swap club if one exists in the neighbourhood. The ways in which these institutions operate differ in different global contexts. However, there is one way of getting a book that works reasonably uniformly across all global contexts – that of procuring the book from the market, from a book retailer. It is not coincidental that ChatGPT and other language models may consistently suggest the market option dominantly when one asks them for ways of finding a book – market is the ‘mainstream’ institution, whereas other are diverse fringe modalities. By focusing more and more on the ‘mainstream’, complex AI models become more and more parochial, leading to model collapse. This is accelerated when there is a feedback-loop where the outputs of AI models percolate into the training set of the next generation of AI models, leading to a vicious cycle and accelerated model collapse. As an empirical work illustrating the size and shape of model collapse, the work will hopefully gather wider attention.

It is also interesting to note that we see this kind of ‘collapse’ of different kinds in the markets, though we seldom view it that way. During the initial years of mobile phones, there were a wide diversity, those with QWERTY keypads, joysticks, scrolling buttons and what not. By catering more and more to the mainstream consumer interests which market actors viewed as touch screen, markets have virtually made it impractical for a user to affordably get hold of a mobile phone that uses any of the other input mechanisms. This is a collapse of choices into one modality, that of touchscreens. We often view markets as providing a rich diversity of consumer options, but seldom note areas of the markets which have collapsed into one or a small number of options through monopolization and majoritarian utilitarianism. This example is to illustrate that ‘model collapse’ is not just ‘another issue’ with AI models, but a fundamental issue with AI becoming aligned with markets and capitalism. It is only if we understand those issues that way – aided by papers such as the present one in Nature – that we may be able to re-imagine AI as a force for inclusive social good, rather than a force of parochialism and exclusion.”

AI models collapse when trained on recursively generated data’ by Shumailov et al. was published in Nature at 16:00 Irish time on Wednesday 24^th July. 

DOI: 10.1038/s41586-024-07566-y

Declared interests

Dr Deepak Padmanabhan: none