What if ChatGPT is Actually a Tour Guide From Another World?
ChatGPT seems to be a tour guide leading us through high dimensional space.
”It may look on the surface that we are just learning the statistical correlations in text. But it turns out to just learn the statistical correlations in the text, to compress them well, what the neural net learns is some representation of the process that produced the text. This text is actually a projection of the world.” — Ilya Sutskever (Fireside Chat with Ilya Sutskever and Jensen Huang March 2023)
I went back and listened to an old interview with Ilya Sutskever, co-founder of OpenAI and Jensen Huang the co-founder and chairman of Nvidia. Listening to Ilya often leaves me feeling like a thickheaded numbskull. But this time, something clicked, and it made me want to pour a stiff drink.
What if text really is a projection of a world?!
And what if that world, to use a simple analogy, is like Minecraft, the famous game created by Markus “Notch” Persson. In Minecraft, players explore an immense expanse of a procedurally generated terrain made out of cubes. “Procedurally generated” is key. It means a computer created the terrain — not people.
Procedurally generated Minecraft terrain looks like this:
Imagine, then, replacing pictures of nature with pictures of words, like this:
Then the terrain might look something like this:
Notice the person in the lower right corner? That’s our guide. That’s ChatGPT.
“Here we are in a place called ‘One,’” our guide begins. “We’re surrounded by things semantically related to the word ‘One.’ Let’s explore. Over here is ‘Step,’ and just around the corner, ‘Forward’ — that’s One Step Forward for us.” Then the guide gestures to another path. “Following this route, we find ‘Piece’ and then ‘Missing.’ And look there, ‘Last’ and just beyond, ‘Dance.’ Anyone up for One Last Dance?” our guide jokes, inviting us further into this world of words.
Now, you might be asking yourself, ‘What’s the big deal?’ Well, here’s the make-your-head-explode insight that seems to be at the core of Ilya’s thinking:
A trained language model is also a procedurally generated world.
This has big implications. Here are a few that come to mind:
- Training a large language model is like building a virtual world, which intuitively makes sense, as both language models and video games run on the same chips and rely on the same mathematical principles (matrix multiplication).
- ChatGPT doesn’t operate a large language model; it explores it. Like the method of loci, but instead of words being merely mapped to a world, words form the world itself.
- Submitting a prompt to ChatGPT is like parachuting into a specific location in the expansive world of GPT and exploring it.
- Some parts of the world are busier than others.
- Only a tiny fraction of the GPT world has been explored.
It gets even crazier. If GPT really is another world — and it certainly seems so — then it encompasses not only everything that has already been said but also many things that will be said. That conversation you’re going to have with someone you haven’t met yet? It’s already out there, somewhere in GPT world. The wedding speech for a future couple that don’t even know they’re going to be engaged? It’s already out there in GPT world too.
And there’s more. Just like in Minecraft, the GPT world can be modified. We can add structures to bring in domain expertise and provide additional context. What OpenAI has created is not just a fantastic language model; it’s the most detailed virtual world ever, and digital tourism is booming.
