"What Is ChatGPT Doing... and Why Does It Work?" by Stephen Wolfram

What Is ChatGPT Doing… and Why Does It Work? by Stephen Wolfram, published 2023.

I feel horribly behind on AI in general and LLMs in particular.

And by that I mean mostly on the technical side, because I just never took the time to try to understand how LLMs actually work. While I’m not terribly interested in using LLMs right now, I would like to know how they work. So it’s been irking me for years now that I have no clue about what’s going on inside them.

So it was high time to do something about that. Being a big fan of Stephen Wolfram, I figured this little book from early 2023 would be a good place to start.

At 102 pages, it was a quick read.

And did I learn anything?

Yes.

I learnt that a neural network — the mathematical construction behind large language models — in the most basic sense simply computes a function.

These neural networks were invented back in the 1940s already, as an idealized model of how brains work. But it wasn’t until recently that computer hardware caught up and they could actually do useful things in practice.

Under the hood, a neural network consists of “neurons” organized in layers, with the neurons connected to each other with individual weights per connection. By adjusting these weights, the network can be customized to approximate a desired function. Doing these adjustments is called training the network.

As it turns out, neural networks can be used to model human language.

And that’s what ChatGPT does.

Backed by a large neural network trained on billions and billions of words of human written text, ChatGPT simply tries to predict the next word of its input — it tries to make up a convincing continuation of whatever you prompt it with based on what it’s “seen” in its training. That’s it.

It’s an interesting example of a simple system with surprisingly complex behavior.