Imitative AI Cannot Be a Dessert Topping and a Floor Wax

The headlines are now both obscure and ancient. Well done, me.

A German court recently ruled that Google was liable for the false statements returned by its AI overview feature. What interests me is not so much the specifics of the case — it is a German court, after all, and the case has yet to go through appeals — but the defense that Google did and did not use in the trial.

Google tried two main defenses neither of which worked. They argued that AI overview was merely a part of search and that people could follow the links, and therefore the fault, if any, lies in the source material, just like a normal search. The judge did not buy that argument. The output of the AI overview, according to the judge, was a distinct product, created not by pointing people to others’ data but by Google itself. This certainly feels like common sense. The Google AI system produces results that are distinct from the normal search results. Even if they search to find the information, the summary comes from Google itself. The other defense was that people should know that AI results are not reliable, but the judge found that Google did nothing, or at least nothing legally sufficient, to warn people that the AI overview results can be crap. One thing that you will notice is what is missing — the idea that the AI results are not generated by Google.

Imitative AI is created, to simplify, by training a system to predict what comes next, either in a series of words or in pictures. Again, I know this is overly simplified, but for our purposes it is sufficient. In the United States, the law relieves providers of liability for things that others post on their internet platforms, even those things that are surfaced in, for example, a search. Since the material in AI overviews is generated from training data in a way that is arguably similar to the way an algorithm surfaces search answers, it could be argued that they should have the same protection from liability. Perhaps they did not make that argument because the laws in Germany do not support it — though the discussion of the difference between search results and the overview information suggests some close cousin, at least, to the concept. But they may also have avoided it because it would cripple their business.

In order for these systems to be trained, they require an enormous amount of data. To get where we are with them now required essentially close to the entire internet and all the current published books. They require so much data that new model training is slowing in part because they have no new data to train the models on. Did they pay for this data? Of course not. Their legal argument for using all of other people’s works is that imitative AI training is transformative — it takes others’ works and turns them into something new. Put aside, for a moment, the fact that you can and always have been able to easily get these systems to regurgitate complete books and pictures, and realize the hole this places them into.

If training really is transformative, then that means that the material produced by imitative AI systems is the responsibility of the people who own and program the systems. That means firms like xAI, OpenAI, Anthropic, and Google are responsible for each lie, for all the child sexual abuse material, every dangerous piece of medical advice, and every conversation that encourages suicide that these systems produce. If they want to avoid that kind of liability, they can claim that the outputs are merely representations of the training material. But that, then, opens them to the argument that they stole their training material. Which in turn makes them liable to likely millions of people with damages approaching, in purely economic terms, a metric fuck-ton of money. Either way, they are staring at potentially company-destroying liabilities. They need, legally, for imitative AI to be both transformative and merely the regurgitation of others’ words — a floor wax and a dessert topping.

Now, it is modern capitalism where the law is most often on the side of capital. It is conceivable that they may get their way, that imitative AI could be declared both a floor wax and a dessert topping. But that has not happened in courts yet. And without it, these firms could be, no, should be, in a lot of trouble.