The way we judge AI models may be about to change.
According to a post by Andrej Karpathy highlighted on Techmeme, large language models are moving from generating artifacts to creating hyper-custom worlds on demand — but they still lack the ability to natively perceive and audit what they create.
Karpathy frames it as the end of an era in how people kick the tires on these systems. "We're starting to leave the territory where you'd test an LLM by e.g. 'create an svg of pelican on a bicycle,'" he writes, referencing the kind of small, self-contained party trick that has become a familiar informal benchmark.
The distinction he draws is between an artifact and a world. An artifact is a single output — an image, a snippet, a file — that a person can glance at and judge. A world built on demand is something larger and more open-ended, assembled to a user's specifications rather than pulled off a shelf.
The catch is the second half of Karpathy's point. These systems can produce the thing, but they cannot natively see it. A model writing code that renders a picture, or generating an environment meant to be explored, has no built-in way to look at the result and check whether it actually came out right. Auditing is left to the human on the other end.
That gap matters because it sets a ceiling on trust. The bigger and more customized the output, the harder it is for any person to eyeball it for errors — and the more a model's inability to inspect its own work becomes the bottleneck rather than its ability to produce it.