I think I can help clear a couple things up. It's important to understand that these models *are NOT reasoning*; they're just doing math. When they are able to accomplish theory-of-mind-type tasks it's because the text they were trained on was written by humans with minds. E.g., GPT-2 didn't "figure out" that John giving a drink to John doesn't make sense; it has no idea what makes sense and what does not. Humans do, however, and they write accordingly. So what it's doing is mathematically determining, having been trained on text written by humans, that a human is very unlikely to arrange words in that way. It's far more likely that the other noun in the compound subject would be the recipient of the drink. Likewise with the mislabeled popcorn bag. It's able to do these things not because it "knows" anything and certainly not because it reasoned about the question but simply because of statistical probabilities. This is why the models improve so much with scale: because more examples, i.e. more data points, make the statistics more accurate. The exact process it goes through to get here is opaque to us but the _principle_ is clear and relatively simple, the more so because of your excellent explanation above. E.g., with the TiKZ unicorn, the model isn't "understanding" what a unicorn is but rather there are text descriptions of unicorns in the training data, along with descriptions of shapes and colors and how to draw things and how the drawing tools in TiKZ work, etc. and it has seen enough examples to bring those things together.
Emily Bender is exactly right when she calls these models "stochastic parrots." No amount of increasing complexity can ever turn a nonrational, purely deterministic, mathematical process into a rational understanding of truth. Thus, "hallucinations." These models are something like a cultural mirror: if, when we gaze into them, what we see looks human, it's because we are human. It is decidedly NOT because the mirror has spontaneously become human.
> I think I can help clear a couple things up. It's important to understand that these models *are NOT reasoning*; they're just doing math.
An important thing to understand is that nothing in the above post is actually factual or proven correct; it's all opinion, but whenever anyone with these opinions says them, they word it as if it's a fact.
There's no reason to believe it's true because:
1. transformer models are capable of learning to reason, insofar as it's math, since they can learn any mathematical function.
2. LLMs aren't merely trained on "predicting the next word"; that's an oversimplification, where the real answer involves things like regularization, sampling methods, and RLHF.
3. stating something "is just a stochastic parrot" is near-meaningless, as we don't know what kind of thing a stochastic parrot is or how it could work.
> with the TiKZ unicorn, the model isn't "understanding" what a unicorn is but rather there are text descriptions of unicorns in the training data, along with descriptions of shapes and colors and how to draw things and how the drawing tools in TiKZ work, etc. and it has seen enough examples to bring those things together.
An example of the shortcomings of the "just" explanation here is, you just said it didn't understand the task and then described it understanding the task. How did it "bring these things together"?
(Especially since a unicorn isn't a real thing; a unicorn is nothing more than the idea described in text descriptions of unicorns. Besides, GPT-4 was trained on images as well as text.)
You seem to have taken my comment personally for some reason. You appear to have some misunderstandings as well. Briefly:
1. Reasoning is not math, is not done with math, and no computer can ever do it since computers “just” do math.
2. I never said this. You made it up.
3. Perhaps if you read Emily Bender’s paper the term “stochastic parrot” would not be meaningless to you.
4. The phase “bring those things together” in no way implies any understanding. It did it with statistics; i.e. math. Which is not at all the same as reasoning.
Physics is the study of the fundamental math that makes up the world. The brain is physical, and therefore fundamentally mathematical. Reasoning is entirely a function of the physical brain. Therefore, reasoning is math, and is done with math, and computers can do it since computers do math.
I read the thread below, but your fundamental thesis is off. Phenomena are modelled mathematically. Math is a tool. All models include assumptions and acknowledge, "unmodeled dynamics". Classical models use differential equations. Feynman challenges us with his path integrals and phase cancellation notions to reveal all paths are possible for an object but our classical equation is the actual one, only statistically.
This is not to rail against Machine Learning, and I'm not implying you don't know the above simple concepts, but intelligence is a function of a complex biological system. We might approximate it, or give ourselves myriad efficiency gainers to for example form sentences but the, "intelligence" is beyond the computer or the math.
What we might need to acknowledge is we are all psychotic to an extent greater than we are, "intelligent". This massive brain is so whacked we don't really know what is going on, but we trip around under it for a few decades.
The world is made of energy, not math; math is a tool we use to model the world. Also, reasoning does not take place in the brain; rather, the brain merely supplies the facts upon which our reason operates. As JBS Haldane once pointed out, "If my mental processes are determined wholly by the motions of atoms in my brain, I have no reason for supposing that my beliefs are true. They may be sound chemically, but that does not make them sound logically. And hence I have no reason for supposing my brain to be composed of atoms."
We don't really know what the world is made of. That's a metaphysical question beyond the reach of current science. It's also irrelevant to the question of whether reasoning is math, because we do know that whatever the world is made of behaves according to mathematical rules.
Adopt a Pragmatic definition of truth, and Haldane's comment is exposed as meaningless nonsense.
One might disagree with Haldane but to call his observation nonsense is to announce that one has entirely failed to understand the subject. Whole books and entire careers have been based on this question you're calling nonsense.
And, in point of fact, it's pretty well established that the world is made of matter and energy and that the two are interchangable. See Einstein. Regardless, it's certainly NOT made of math as you originally said.
Good note that is technically true - assuming humans aren't intuitively simulating the same math (an assumption I believe). Yet, if the Turing Test taking ai.bot simulates every aspect of intelligence, so that there is no empirical difference between the ai.bot and 19 other human students, or digital workers, does it really matter on a project team of 20. Or maybe it's just 1 human.
It is a great explainer indeed and I like your comment. It leaves me thinking,”If it can take months to determine how it thinks and what will it predict” then how is human intelligence calibrated, Who will do what based on the human grades of intelligence.”
Thank you so much Tim and Sean for writing this article - it’s the best overview I’ve read of how LLMs work and touches on some fascinating research areas.
It’s obvious to me now that we’re way past the point where the research into how these models work can keep up with the pace of development of the models themselves. This has me both excited and scared at the same time - a feeling I’m becoming very familiar with at the moment 🤓.
For me this means a couple of things:
- We definitely need to find ways to accelerate the research and I think we’re getting to the point where we need research AI models to help us understand AI models. I think we’ll see a big increase in developing specialised AI models to help us with all aspects of LLMs from research through to alignment.
- Philosophy and the Social Sciences are and will continue to become much more important in how we think, discuss and evaluate the future of LLMs and generative AI. This is almost the ‘top-down’ approach to understanding LLMs and when paired with more technical research probably gives us the best handle on how we should focus our energies around LLMs in the future.
Thank you again - really looking forward to your next article!
Excellent and incisive explanatory piece. Great use of visual analogues to explain the power of computer science, as well as social science to elucidate consequences of these models.
Great review! I was wondering if and how LLMs deal with punctuation? I would assume they could be assigned their own vector, but their meaning is far less concrete and operates primarily to *modify* the meaning of words and sentences. For example, there is a WORLD of difference between:
"Rachael Ray finds inspiration in cooking her family and her dog"
and
"Rachael Ray finds inspiration in cooking, her family, and her dog"
Any brief explanation of how punctuation and other language modifiers (accent marks for non-English language perhaps) works would be very welcome!
This is one of the aspects of LLMs that we simplified to keep the explainer manageable. The basic unit of analysis in an LLM isn't words but rather tokens. The most common words are their own tokens, but longer and less common words are often made up of multiple tokens.
So I'm pretty sure ChatGPT treats "," as its own token and represents it with a vector. As you say, punctuation largely affects sentences by changing the meaning of nearby words, so I assume that LLMs figure this out by having nearby words "notice" the comma via the attention mechanism. But I don't know of any specific research on this question.
As Tim wrote, we didn't write about tokenization, but technically, most LLMs represent input as a series of "tokens", as opposed to words. Before the LLM is trained, there's a *tokenizer* that is trained, which basically amounts to learning all the "tokens" of a language. One way of thinking about this is that all the alphabetic characters of a language ("a", "b", "c", etc.) and also punctuation (".", ",", etc.) start out as their own tokens. Then, gradually, the tokenizer learns which characters co-occur with which other characters, i.e., which it then "merges" into a newer token representation. So words like "cat" typically would be their own token. The number of tokens this process ultimately results in is constrained by the researchers, e.g., they decide they want only 50K tokens overall in the model's vocabulary. (There are also really interesting questions about to what extent the "tokens" a model learns overlap with basic morphemes, i.e., units of meaning, in a language.)
Returning to your specific example, the two sentences would indeed have different token-level representations.
Simplifying a bit***, the first would be something like:
As Tim noted, those additional "," tokens will change the model's representations of adjacent tokens, as well as predictions of upcoming words. To know exactly how that would work in this specific case——and to see how successfully GPT differentiates those meanings——we'd need to actually look at different examples involving commas and ambiguity.
***Technically, some of these words would likely be broken up into tokens of their own. E.g., perhaps "cooking" is broken up into "cook" and "ing". But we'd have to run them through the tokenizer to be sure.
I’m far from an expert, but I believe punctuation marks have their own tokens. As footnote 1 says, the “words” LLMs operate on are “tokens”, some of which are whole words and some are fragments — so “hasn't” may be broken into “has” and “n't”. And IIRC there are tokens for word separators like commas, periods, semicolons..l
Punctuation and accent marks are entirely different sorts of things. The latter are mostly just composite glyphs representing different phonemes (e.g. 'o' vs 'ö') which could just as well have been represented by completely different glyphs like most phonemes are (e.g. 'k' vs 'g').
The tokens are mostly an optimization to cut training time and make the context window bigger. LLMs work okay on untokenized text; even if they only have English tokens they can translate to Chinese pretty well.
Hey Tim, fantastic article on understanding large language models! Your use of visual analogies and insights into social science really make it accessible. Looking forward to more insightful breakdowns in the future! Keep up the great work!
Great explainer, but you only spent one sentence on the down side: you don't know how it works. This logic is being used to weed out resumes, set insurance rates, etc. It is drastically impacting people's lives. It is being hailed as impartial, but actually is the opposite. It is replacing human decision-makers with third-rate reasoning skills. Written word is only a small part of human communications. Building an entire system using only that as input is like only eating the crust and thinking you ate a piece of apple pie.
Thank you for the gentle primer. For the first time, I’ve read something that gave me an explanation of the mechanisms behind it. Do you have any concerns about biased algorithms that can be introduced to (or maybe are already in) ChatGPT?
Thanks for an awesome writeup. I have two questions:
1) Is the model "seeded" with certain rules in the Translators that are provided by humans? Like ... did *we* tell it what the difference is between a noun and a verb, or is that something that it learns from the training data itself?
2) How do the conclusions drawn from the Translators wind up back in the main vector space? Does the model need to know that "Warsaw" was correct to make an update to the "Poland" vector?
(1) No the weights of the transformers are initialized entirely at random. Everything it knows about natural language it learns "from scratch" by detecting patterns of words in its training set. (Yes this is crazy! Machine learning experts found it mind-blowing when it started happening ~5 years ago.)
(2) After the final layer outputs its word vectors, the model uses a fairly simple neural network that takes the final word vector as an input and produces a prediction for the next word. In the Poland/Warsaw example, the encoding that leads to a prediction of "Warsaw" would be encoded in the vector for ":"—the final "word" of the sequence. Nobody knows exactly how the model "encodes" the word Warsaw into this final vector, we just know that if we apply the decode network to the final vector in the input you get a next-word prediction that is usually correct.
I forgot to mention something else: Each time you submit text to ChatGPT, *all* of the previous text in the conversation is submitted to the model as well.
As explained in the article, these models work as next-word predictors and the next word is predicted based upon all the previous words in your conversation.
But the conversations you have are distinct from each other.
One thing to understand is that these models can only "reason" about a (relatively) small amount of text. In the case of GPT-3 around 4k words. So, if your conversation with GPT-3 extends beyond this number of words...including the answers the model has given you...the oldest words begin getting cut off.
What may happen instead is that OpenAI will use your conversations or some modified version as part of the corpus for future training runs or as some other sort of input into future GPT versions. You can imagine some sort of RLHF-like process that uses conversations to fine-tune output.
There's ways you can use user feedback without letting their text get into the actual model.
For instance, if you train two models on your clean data, you can give them both user questions and then keep the model whose output "looks more like" previously upvoted answers. (This is, obviously, a lot of work.)
Tim, great article! I’m wondering if we can translate your blog into Chinese and post it on AI community in China. We will highlight your name and keep the original link on the top of the translated version. Thank you.
Given every step is observable there is no inherent mystery as to what these algorithms are doing. It's just impossible to map a process with states evolving wildly. It's so complicated it's enough to make we want to stick to stuff I can easily measure. God bless you guys!
It's such a whacky, evolving state transition matrix (of a sort) that, given a bunch of preceding words outputs another. I don't know much about it, but that's a safe place to start with, "AI": your common sense.
Given this is all about, "language" there is a grave error committed on this entire topic: use of the word, "intelligence". We are putting, "artificial" in front of it, but have we defined organic, "intelligence"?
What was my impulse to write this comment? How or why did I divert from what I (should) be doing to share these words? Why do I assert an opinion on AI when I am not a algorithm guy in the space? Is this, "intelligent" for me to do? Do I do it to share my, "intelligence" or to prompt response?
This is all and everywhere, "machine learning". Combining words is not, "intelligence". Consider for yourself how very limited language is in conveying sentiment. You have a sentiment to convey, but language is so very limited. This is one reason whey some people don't say much.
A LLM could have helped me tab-through this comment to type it more quickly than I did, and with fewer errors I'm prone to commit. It otherwise has no idea from whence, "intelligence" emanates.
Really excellent article, you made a densely complex topic understandable, thank you. But so many more questions! How the model weights probable outcomes based on learning from training data raises the question of quality of training data and inherent biases within that data. Fascinating.
Great explainer! Very interesting stuff.
I think I can help clear a couple things up. It's important to understand that these models *are NOT reasoning*; they're just doing math. When they are able to accomplish theory-of-mind-type tasks it's because the text they were trained on was written by humans with minds. E.g., GPT-2 didn't "figure out" that John giving a drink to John doesn't make sense; it has no idea what makes sense and what does not. Humans do, however, and they write accordingly. So what it's doing is mathematically determining, having been trained on text written by humans, that a human is very unlikely to arrange words in that way. It's far more likely that the other noun in the compound subject would be the recipient of the drink. Likewise with the mislabeled popcorn bag. It's able to do these things not because it "knows" anything and certainly not because it reasoned about the question but simply because of statistical probabilities. This is why the models improve so much with scale: because more examples, i.e. more data points, make the statistics more accurate. The exact process it goes through to get here is opaque to us but the _principle_ is clear and relatively simple, the more so because of your excellent explanation above. E.g., with the TiKZ unicorn, the model isn't "understanding" what a unicorn is but rather there are text descriptions of unicorns in the training data, along with descriptions of shapes and colors and how to draw things and how the drawing tools in TiKZ work, etc. and it has seen enough examples to bring those things together.
Emily Bender is exactly right when she calls these models "stochastic parrots." No amount of increasing complexity can ever turn a nonrational, purely deterministic, mathematical process into a rational understanding of truth. Thus, "hallucinations." These models are something like a cultural mirror: if, when we gaze into them, what we see looks human, it's because we are human. It is decidedly NOT because the mirror has spontaneously become human.
> I think I can help clear a couple things up. It's important to understand that these models *are NOT reasoning*; they're just doing math.
An important thing to understand is that nothing in the above post is actually factual or proven correct; it's all opinion, but whenever anyone with these opinions says them, they word it as if it's a fact.
There's no reason to believe it's true because:
1. transformer models are capable of learning to reason, insofar as it's math, since they can learn any mathematical function.
2. LLMs aren't merely trained on "predicting the next word"; that's an oversimplification, where the real answer involves things like regularization, sampling methods, and RLHF.
3. stating something "is just a stochastic parrot" is near-meaningless, as we don't know what kind of thing a stochastic parrot is or how it could work.
> with the TiKZ unicorn, the model isn't "understanding" what a unicorn is but rather there are text descriptions of unicorns in the training data, along with descriptions of shapes and colors and how to draw things and how the drawing tools in TiKZ work, etc. and it has seen enough examples to bring those things together.
An example of the shortcomings of the "just" explanation here is, you just said it didn't understand the task and then described it understanding the task. How did it "bring these things together"?
(Especially since a unicorn isn't a real thing; a unicorn is nothing more than the idea described in text descriptions of unicorns. Besides, GPT-4 was trained on images as well as text.)
You seem to have taken my comment personally for some reason. You appear to have some misunderstandings as well. Briefly:
1. Reasoning is not math, is not done with math, and no computer can ever do it since computers “just” do math.
2. I never said this. You made it up.
3. Perhaps if you read Emily Bender’s paper the term “stochastic parrot” would not be meaningless to you.
4. The phase “bring those things together” in no way implies any understanding. It did it with statistics; i.e. math. Which is not at all the same as reasoning.
Physics is the study of the fundamental math that makes up the world. The brain is physical, and therefore fundamentally mathematical. Reasoning is entirely a function of the physical brain. Therefore, reasoning is math, and is done with math, and computers can do it since computers do math.
I read the thread below, but your fundamental thesis is off. Phenomena are modelled mathematically. Math is a tool. All models include assumptions and acknowledge, "unmodeled dynamics". Classical models use differential equations. Feynman challenges us with his path integrals and phase cancellation notions to reveal all paths are possible for an object but our classical equation is the actual one, only statistically.
This is not to rail against Machine Learning, and I'm not implying you don't know the above simple concepts, but intelligence is a function of a complex biological system. We might approximate it, or give ourselves myriad efficiency gainers to for example form sentences but the, "intelligence" is beyond the computer or the math.
What we might need to acknowledge is we are all psychotic to an extent greater than we are, "intelligent". This massive brain is so whacked we don't really know what is going on, but we trip around under it for a few decades.
The world is made of energy, not math; math is a tool we use to model the world. Also, reasoning does not take place in the brain; rather, the brain merely supplies the facts upon which our reason operates. As JBS Haldane once pointed out, "If my mental processes are determined wholly by the motions of atoms in my brain, I have no reason for supposing that my beliefs are true. They may be sound chemically, but that does not make them sound logically. And hence I have no reason for supposing my brain to be composed of atoms."
We don't really know what the world is made of. That's a metaphysical question beyond the reach of current science. It's also irrelevant to the question of whether reasoning is math, because we do know that whatever the world is made of behaves according to mathematical rules.
Adopt a Pragmatic definition of truth, and Haldane's comment is exposed as meaningless nonsense.
One might disagree with Haldane but to call his observation nonsense is to announce that one has entirely failed to understand the subject. Whole books and entire careers have been based on this question you're calling nonsense.
And, in point of fact, it's pretty well established that the world is made of matter and energy and that the two are interchangable. See Einstein. Regardless, it's certainly NOT made of math as you originally said.
can't agree more.
Good note that is technically true - assuming humans aren't intuitively simulating the same math (an assumption I believe). Yet, if the Turing Test taking ai.bot simulates every aspect of intelligence, so that there is no empirical difference between the ai.bot and 19 other human students, or digital workers, does it really matter on a project team of 20. Or maybe it's just 1 human.
I agree. There are definitely situations where the distinction between human reasoning and language modeling is not important.
It is a great explainer indeed and I like your comment. It leaves me thinking,”If it can take months to determine how it thinks and what will it predict” then how is human intelligence calibrated, Who will do what based on the human grades of intelligence.”
Wow, I'm stealing that mirror example to explain to some of my alpha friends :-)
Very good explainer. I also appreciated the last section that goes into a bit of philosophy and theories about how people learn.
Thank you! Please continue with this type of analysis. I'd be curious to see a similar breakdown of what is back propagation and how does it work.
Thank you so much Tim and Sean for writing this article - it’s the best overview I’ve read of how LLMs work and touches on some fascinating research areas.
It’s obvious to me now that we’re way past the point where the research into how these models work can keep up with the pace of development of the models themselves. This has me both excited and scared at the same time - a feeling I’m becoming very familiar with at the moment 🤓.
For me this means a couple of things:
- We definitely need to find ways to accelerate the research and I think we’re getting to the point where we need research AI models to help us understand AI models. I think we’ll see a big increase in developing specialised AI models to help us with all aspects of LLMs from research through to alignment.
- Philosophy and the Social Sciences are and will continue to become much more important in how we think, discuss and evaluate the future of LLMs and generative AI. This is almost the ‘top-down’ approach to understanding LLMs and when paired with more technical research probably gives us the best handle on how we should focus our energies around LLMs in the future.
Thank you again - really looking forward to your next article!
Excellent and incisive explanatory piece. Great use of visual analogues to explain the power of computer science, as well as social science to elucidate consequences of these models.
Great review! I was wondering if and how LLMs deal with punctuation? I would assume they could be assigned their own vector, but their meaning is far less concrete and operates primarily to *modify* the meaning of words and sentences. For example, there is a WORLD of difference between:
"Rachael Ray finds inspiration in cooking her family and her dog"
and
"Rachael Ray finds inspiration in cooking, her family, and her dog"
Any brief explanation of how punctuation and other language modifiers (accent marks for non-English language perhaps) works would be very welcome!
This is one of the aspects of LLMs that we simplified to keep the explainer manageable. The basic unit of analysis in an LLM isn't words but rather tokens. The most common words are their own tokens, but longer and less common words are often made up of multiple tokens.
So I'm pretty sure ChatGPT treats "," as its own token and represents it with a vector. As you say, punctuation largely affects sentences by changing the meaning of nearby words, so I assume that LLMs figure this out by having nearby words "notice" the comma via the attention mechanism. But I don't know of any specific research on this question.
Great question!
As Tim wrote, we didn't write about tokenization, but technically, most LLMs represent input as a series of "tokens", as opposed to words. Before the LLM is trained, there's a *tokenizer* that is trained, which basically amounts to learning all the "tokens" of a language. One way of thinking about this is that all the alphabetic characters of a language ("a", "b", "c", etc.) and also punctuation (".", ",", etc.) start out as their own tokens. Then, gradually, the tokenizer learns which characters co-occur with which other characters, i.e., which it then "merges" into a newer token representation. So words like "cat" typically would be their own token. The number of tokens this process ultimately results in is constrained by the researchers, e.g., they decide they want only 50K tokens overall in the model's vocabulary. (There are also really interesting questions about to what extent the "tokens" a model learns overlap with basic morphemes, i.e., units of meaning, in a language.)
Returning to your specific example, the two sentences would indeed have different token-level representations.
Simplifying a bit***, the first would be something like:
1) ['Rachael', 'Ray', 'finds', 'inspiration', 'in', 'cooking', 'her', 'family', 'and', 'her', 'dog']
Whereas the second would be something like:
2) ['Rachael', 'Ray', 'finds', 'inspiration', 'in', 'cooking', ',', 'her', 'family', ',' 'and', 'her', 'dog']
As Tim noted, those additional "," tokens will change the model's representations of adjacent tokens, as well as predictions of upcoming words. To know exactly how that would work in this specific case——and to see how successfully GPT differentiates those meanings——we'd need to actually look at different examples involving commas and ambiguity.
***Technically, some of these words would likely be broken up into tokens of their own. E.g., perhaps "cooking" is broken up into "cook" and "ing". But we'd have to run them through the tokenizer to be sure.
Thanks for the detailed answers Sean and Tim!
I’m far from an expert, but I believe punctuation marks have their own tokens. As footnote 1 says, the “words” LLMs operate on are “tokens”, some of which are whole words and some are fragments — so “hasn't” may be broken into “has” and “n't”. And IIRC there are tokens for word separators like commas, periods, semicolons..l
Punctuation and accent marks are entirely different sorts of things. The latter are mostly just composite glyphs representing different phonemes (e.g. 'o' vs 'ö') which could just as well have been represented by completely different glyphs like most phonemes are (e.g. 'k' vs 'g').
The tokens are mostly an optimization to cut training time and make the context window bigger. LLMs work okay on untokenized text; even if they only have English tokens they can translate to Chinese pretty well.
Hey Tim, fantastic article on understanding large language models! Your use of visual analogies and insights into social science really make it accessible. Looking forward to more insightful breakdowns in the future! Keep up the great work!
This explanation is wonderful, thank you!
Great explainer, but you only spent one sentence on the down side: you don't know how it works. This logic is being used to weed out resumes, set insurance rates, etc. It is drastically impacting people's lives. It is being hailed as impartial, but actually is the opposite. It is replacing human decision-makers with third-rate reasoning skills. Written word is only a small part of human communications. Building an entire system using only that as input is like only eating the crust and thinking you ate a piece of apple pie.
This may be the best Poetry 101 lesson I have encountered in years. All good poets know language vectors & how to both use them and topple them.
Thank you for the gentle primer. For the first time, I’ve read something that gave me an explanation of the mechanisms behind it. Do you have any concerns about biased algorithms that can be introduced to (or maybe are already in) ChatGPT?
Thanks for an awesome writeup. I have two questions:
1) Is the model "seeded" with certain rules in the Translators that are provided by humans? Like ... did *we* tell it what the difference is between a noun and a verb, or is that something that it learns from the training data itself?
2) How do the conclusions drawn from the Translators wind up back in the main vector space? Does the model need to know that "Warsaw" was correct to make an update to the "Poland" vector?
(1) No the weights of the transformers are initialized entirely at random. Everything it knows about natural language it learns "from scratch" by detecting patterns of words in its training set. (Yes this is crazy! Machine learning experts found it mind-blowing when it started happening ~5 years ago.)
(2) After the final layer outputs its word vectors, the model uses a fairly simple neural network that takes the final word vector as an input and produces a prediction for the next word. In the Poland/Warsaw example, the encoding that leads to a prediction of "Warsaw" would be encoded in the vector for ":"—the final "word" of the sequence. Nobody knows exactly how the model "encodes" the word Warsaw into this final vector, we just know that if we apply the decode network to the final vector in the input you get a next-word prediction that is usually correct.
Does that make sense?
I think so ... it sounds like the questions that I ask ChatGPT aren't then fed back into the model as training data then, for future questions?
I forgot to mention something else: Each time you submit text to ChatGPT, *all* of the previous text in the conversation is submitted to the model as well.
As explained in the article, these models work as next-word predictors and the next word is predicted based upon all the previous words in your conversation.
But the conversations you have are distinct from each other.
One thing to understand is that these models can only "reason" about a (relatively) small amount of text. In the case of GPT-3 around 4k words. So, if your conversation with GPT-3 extends beyond this number of words...including the answers the model has given you...the oldest words begin getting cut off.
That's correct.
What may happen instead is that OpenAI will use your conversations or some modified version as part of the corpus for future training runs or as some other sort of input into future GPT versions. You can imagine some sort of RLHF-like process that uses conversations to fine-tune output.
God, I hope not. The conversations drawn from "people who have been testing ChatGPT" is not what I'd call an unbiased data set! :)
There's ways you can use user feedback without letting their text get into the actual model.
For instance, if you train two models on your clean data, you can give them both user questions and then keep the model whose output "looks more like" previously upvoted answers. (This is, obviously, a lot of work.)
Yes, that’s correct.
Tim, great article! I’m wondering if we can translate your blog into Chinese and post it on AI community in China. We will highlight your name and keep the original link on the top of the translated version. Thank you.
Sure go ahead!
Thanks for this! Very good.
Given every step is observable there is no inherent mystery as to what these algorithms are doing. It's just impossible to map a process with states evolving wildly. It's so complicated it's enough to make we want to stick to stuff I can easily measure. God bless you guys!
It's such a whacky, evolving state transition matrix (of a sort) that, given a bunch of preceding words outputs another. I don't know much about it, but that's a safe place to start with, "AI": your common sense.
Given this is all about, "language" there is a grave error committed on this entire topic: use of the word, "intelligence". We are putting, "artificial" in front of it, but have we defined organic, "intelligence"?
What was my impulse to write this comment? How or why did I divert from what I (should) be doing to share these words? Why do I assert an opinion on AI when I am not a algorithm guy in the space? Is this, "intelligent" for me to do? Do I do it to share my, "intelligence" or to prompt response?
This is all and everywhere, "machine learning". Combining words is not, "intelligence". Consider for yourself how very limited language is in conveying sentiment. You have a sentiment to convey, but language is so very limited. This is one reason whey some people don't say much.
A LLM could have helped me tab-through this comment to type it more quickly than I did, and with fewer errors I'm prone to commit. It otherwise has no idea from whence, "intelligence" emanates.
In "The feed-forward step", is the rightmost green neuron missing an arrow? It only has two
You are correct! 300k people have read this story and you are the first to mention it.
Really excellent article, you made a densely complex topic understandable, thank you. But so many more questions! How the model weights probable outcomes based on learning from training data raises the question of quality of training data and inherent biases within that data. Fascinating.