The Chinese room argument makes no sense to me. I cant see how its different from how young children understand and learn language.
My 2 year old sometimes unmistakable start counting when playing. (Countdown for lift off) Most numbers are gibberish but often he says a real number in the midst of it. He clearly is just copying and does not understand what counting is. At some point though he will not only count correctly but he will also be able to answer math questions. At what point does he “understand” at what point would you consider that chatgpt “understands”

There was this old tv programm where some then ai experts discussed the chinese room but they used a chinese restaurant for a more realistic setting. This ended with “So if i walk into a chinese restaurant, pick sm out on the chinese menu and can answer anything the waiter may ask, in chinese. Do i know or understand chinese? I remember the parties agreeing to disagree at that point.
Yes… the chinese experiment misses the point, because the Turing test was never really about figuring out whether or not an algorithm has “conscience” (what is that even?)… but about determining if an algorithm can exhibit inteligent behavior that’s equivalent/indistinguishable from a human.
The chinese room is useless because the only thing it proves is that people don’t know what conscience is, or what are they even are trying to test.
Your underlying model is not made out of words, but out of concepts. You can have multiple words that all map to the same concept, i.e. cosmos, universe, space. Or a single word that map to different concepts.
No, they aren’t. You represent them with words. But you sure as hell aren’t responding to someone throwing you a football with words trying to figure out where it’s going.
No, a dog (while many times more intelligent than chatGPT) doesn’t understand anything.
What are your underlying models of the world built out of?
As a Bayesian, my models of the world are built on priors. That is, assumptions I’ve made based on my existing information. From that, I make an educated guess about the world with that model and see what the world does. If my guess doesn’t match reality, I update my assumptions to rebuild my model and repeat the process until it’s close enough.
This is the way the best science is done, and I fell it’s the way that humans really work. Language is just a type of model we use to communicate the world to others, each of us may have a slightly different Bayesian understanding of the language yet we can still communicate.
Studies have shown we typically use pattern matching for our choices but not statistics. One such experiment had humans view to light bulbs (I think one was red one was green). One light would turn on at a time and they were allowed or given a record of what had happened. Then they were asked to guess what would occur next for n number of steps. Same thing is done with rats. Humans are rewarded with money based on correct choices and rats with food. Here is the thing, one light (let’s say red) would light up with 70% probability and the other with 30%. But it was randomized.
The optimal solution is to always pick red. Every time. But humans pick a pattern. Rats pick red. Humans consistently do worse than rats. So while we are using a form of updating, it certainly isn’t proper bayesian updating. And just because you think we function some way doesn’t make it true. And it will forever be difficult to describe any AI as conscious, because we have really arbitrarily defined it to fit us. But we can’t truly say what it is. Not can we can why we function how we do. Or if we are all in a simulation or just a Boltzmann brain.
Honestly, something that concerns me most about AI is that it could become sentient, but we will not know if it is or just cleverly programmed so we treat it only as a tool. Because while I don’t think AI is inherently dangerous, I think becoming a slave owner of something that could be much more powerful probably is. And given their lack of chemical hormones, we will have even less of an understanding of what or how it feels.
It could still be bayesian reasoning, but a much more complex one, underlaid by a lot of preconceptions (which could have also been acquired in a bayesian way).
Even if the result is random, a highly pre-trained bayessian network that has the experience of seeing many puzzles or tests before that do follow non-random patterns might expect a non-random pattern… so those people might have learned to not expect true randomness, since most things aren’t random.
Your brain understands concepts and can self-conceptualise, LLMs cannot do either. They can sound convincingly as if they understand concepts but that’s because we fill in gaps due to how we understand language. The examples of broken or distorted sentences being understandable applies here. You and I can communicate in broken sentences because you and I understand the concepts beneath the conversation. LLMs play on that understanding but they do not understand its concepts.
LLMs are criminally simplified neural networks at minimum thousands of orders less complex than a brain. Nothing we do with current neural networks resembles intelligence.
Nothing they do is close to understanding. The fact that you can train one exclusively on the rules of a simple game and get it to eventually infer a basic rule set doesn’t imply anything like comprehension. It’s simplistic pattern matching.
No, and that definition has nothing in common with what the word means.
Autocorrect has plenty of information encoded as artifacts of how it works. ChatGPT isn’t like autocorrect. It is autocorrect, and doesn’t do anything more.
It’s fine if you think so, but then it’s a pointless argument over definitions.
You can’t have a conversation with autocomplete. It’s qualitatively different. There’s a reason we didn’t have this kind of code generation before LLM’s.
I am not sure of the relevance of the oppressed classes and with the object of duping the latter is the cravings of the oppressed classes and with the object of duping the latter
Yeah, totally. Repeating the same nonsensical sentence over and over is also how I converse. 🙄
Note that “real world truth” is something you can never accurately map with just your senses.
No model of the “real world” is accurate, and not everyone maps the “real world truth” they personally experience through their senses in the same way… or even necessarily in a way that’s really truly “correct”, since the senses are often deceiving.
A person who is blind experiences the “real world truth” by mapping it to a different set of models than someone who has additional visual information to mix into that model.
However, that doesn’t mean that the blind person can “never understand” the “real world truth” …it just means that the extent at which they experience that truth is different, since they need to rely in other senses to form their model.
Of course, the more different the senses and experiences between two intelligent beings, the harder it will be for them to communicate with each other in a way they can truly empathize. At the end of the day, when we say we “understand” someone, what we mean is that we have found enough evidence to hold the belief that some aspects of our models are similar enough. It doesn’t really mean that what we modeled is truly accurate, nor that if we didn’t understand them then our model (or theirs) is somehow invalid. Sometimes people are both technically referring to the same “real world truth”, they simply don’t understand each other and focus on different aspects/perceptions of it.
Someone (or something) not understanding an idea you hold doesn’t mean that they (or you) aren’t intelligent. It just means you both perceive/model reality in different ways.
For one thing, understanding implies that a word is linked to a mental concept. So if you say “The car is red”, you first need to mentally compare the mental concept of “red” to the car in question.
The Chinese room bypasses all of that, it can say “The car is red” without ever having seen a red object at all.
Do you maintain this line of reasoning if it only says “the car is red” when the car is in fact red. And is capable of changing the answer to correctly mentioned a different color when the item
In question is a different question.
Some ai demos show that programs like gpt-4 are already way passed this when provided with, it can not only accurate describe whats in the image but also the context.
Some examples, mind these where shown in an openAI demo for gpt4, Open ai has not yet made their version of this tech publicly available.
When i see these examples, i am not convinced that the ai truly understands everything it is saying. But it does seem to understand context, One of the theories on how it can do this (they are still a black box) is talked about in some papers that large language models may actually create an internal model of the world similar to humans and use that for logical reasoning and context.
It doesn’t matter if the answer is right. If the AI does not have an abstract understanding of “red” then it is using a different process to get to the answer than humans. And according to Searle, a Turing machine cannot have an abstract understanding of “red”, no matter how complex the question or how complex an internal model is used to determine its answers.
Going back to the Chinese Room, it is possible that the instructions carried out by the human are based on a complex model. In fact, it is possible that the human is literally calculating the output of a trained neural net by summing the weights of nodes, etc. You could even carry out these calculations yourself, if you could memorize the parameters.
Your use of “black box” gets to the heart of it. Memorizing all of the parameters of a trained NN allows you to calculate an answer, but they don’t give you any understanding what the answer means. And if they don’t tell you anything about the meaning, then they don’t tell the CPU doing that calculation anything about meaning either.
I don’t think ai will ever use a process to derive an answer the same way as a human does. Maybe thats part of the goal for the original Turing test but i don’t think the biological human ways is the only way to intelligent understanding “on par” with human intelligence.
Does a blind person have an abstract understanding of “red”?
I can imagine an intelligent alien species, unable to perceive colors like us but yet having an sense to detect to what they call “surface temperature” which allow them to recognize specific wave lengths of the ligt reflecting on surfaces, this is sort of how humans see color but maybe for the alien they hear this as sound.
They then go on and use this sensory input to make music. A song about the specific light wavelength that humans know as a deep bordeaux red color.
Do these biological Intelligent aliens not have an abstract understanding of the color red? I would say they do, its different then how we understand it for sure but both are valid. An even more supreme species might have both those understandings and combine them for an even deeper fuller sensory understanding of “red”.
I see ai similar to this, its a program contained in computer hardware. With no body of its own its depending on us to provide it with input. This is now mostly text so the ai obtains a text based understanding of the world, hence why its so decent at poetry. But when we attach more sensors like a camera then that will change.
I am not sure how to discuss “a human using instructions to calculate perfect answers, but not getting an understanding of what that answers means” wed might have to agree to disagree on that but i feel like thats all my brain has ever done. Were born in a complex place we do not comprehend, are given some instructions mostly by copying what others are doing. Then we find a personal meaning in those things, which as far as i am aware is unique for everyone. (Tbf: i am an autist, the fact that not all humans experience reality the same and that i had to find and learn my own personal understanding of the world has greatly shaped how i think about these systems)
Perhaps I should rephrase the argument as Searle did. He didn’t actually discuss “abstract understanding”, instead he made a distinction between “syntax” and “semantics”. And he claimed that computers as we know them cannot have semantics, whereas humans can (even if we don’t all have the same semantics).
Now consider a quadratic expression. If you want to solve it, you can insert the coefficients into the quadratic formula. There are other ways to solve it, but this will always give you the right answer.
If you remember your algebra class, you will recognize that the quadratic formula isn’t just some random equation to compute. You use it with intention, because the answer is semantically meaningful. It describes things like cars accelerating or apples falling.
You can teach a three year old to identify the coefficients, you can show them the symbols that make up the quadratic formula: “-”, second number, “+”, “√”, “(”, etc. And you can teach them to copy those symbols into a calculator in order. So a three year old could probably solve a quadratic expression. But they almost certainly have no idea why they are doing what they are doing. It’s just a series of symbols that they were told to copy into a calculator, their only intention was to copy them in order correctly. There are no semantics behind the equation.
For that matter, a three year old could equally well enter the symbols necessary to calculate relativistic time dilation, which is an even shorter equation. But if their parents proudly told you that their toddler can solve problems in special relativity, you might think, “Yes… but not really.”
That three year old is every computer program. Sure, an AI can enter symbols into a calculator and report the answer. If you tell them to enter a different series of symbols, they will report a different answer. You can tell the AI that one answer scores 0.1 and another scores 0.8, and to calculate a different equation that is based partly on those scores. But to the AI, those scores and equations have no semantic meaning. At some point those scores might stop increasing, and you will declare that the AI is “trained”. But at no point does the AI assign any semantic content behind those symbols or scores. It is pure syntax.
For me, I think the criteria I’d use for saying someone has a decent understanding of math is knowing that math has underlying rules and most things can be understood from those basic rules (each problem is not just an arbitrary magic trick to get an answer that was impossible figure out) and perhaps also being able to ask “novel” questions (compared that what you already know) and taking reasonable steps to answer it with the rules you do know and the tools you have (doesn’t need to be successful). I think counting could be done with any consistent set of sounds and it doesn’t matter whether yours just reading those sounds for a list or not as long as you know roughly what they correspond to in terms of time. I don’t think a lot of humans have much understanding of math, I think some computers already beat a lot of humans with respect to that.
The Chinese room argument makes no sense to me. I cant see how its different from how young children understand and learn language.
My 2 year old sometimes unmistakable start counting when playing. (Countdown for lift off) Most numbers are gibberish but often he says a real number in the midst of it. He clearly is just copying and does not understand what counting is. At some point though he will not only count correctly but he will also be able to answer math questions. At what point does he “understand” at what point would you consider that chatgpt “understands”  There was this old tv programm where some then ai experts discussed the chinese room but they used a chinese restaurant for a more realistic setting. This ended with “So if i walk into a chinese restaurant, pick sm out on the chinese menu and can answer anything the waiter may ask, in chinese. Do i know or understand chinese? I remember the parties agreeing to disagree at that point.
Yes… the chinese experiment misses the point, because the Turing test was never really about figuring out whether or not an algorithm has “conscience” (what is that even?)… but about determining if an algorithm can exhibit inteligent behavior that’s equivalent/indistinguishable from a human.
The chinese room is useless because the only thing it proves is that people don’t know what conscience is, or what are they even are trying to test.
ChatGPT will never understand. LLMs have no capacity to do so.
To understand you need underlying models of real world truth to build your word salad on top of. LLMs have none of that.
What are your underlying models of the world built out of? Because I’m human, and mine are primarily built out of words.
How do you draw a line between knowing and understanding? Does a dog understand the commands it’s been trained to obey?
Your underlying model is not made out of words, but out of concepts. You can have multiple words that all map to the same concept, i.e. cosmos, universe, space. Or a single word that map to different concepts.
No, they aren’t. You represent them with words. But you sure as hell aren’t responding to someone throwing you a football with words trying to figure out where it’s going.
No, a dog (while many times more intelligent than chatGPT) doesn’t understand anything.
As a Bayesian, my models of the world are built on priors. That is, assumptions I’ve made based on my existing information. From that, I make an educated guess about the world with that model and see what the world does. If my guess doesn’t match reality, I update my assumptions to rebuild my model and repeat the process until it’s close enough.
This is the way the best science is done, and I fell it’s the way that humans really work. Language is just a type of model we use to communicate the world to others, each of us may have a slightly different Bayesian understanding of the language yet we can still communicate.
Studies have shown we typically use pattern matching for our choices but not statistics. One such experiment had humans view to light bulbs (I think one was red one was green). One light would turn on at a time and they were allowed or given a record of what had happened. Then they were asked to guess what would occur next for n number of steps. Same thing is done with rats. Humans are rewarded with money based on correct choices and rats with food. Here is the thing, one light (let’s say red) would light up with 70% probability and the other with 30%. But it was randomized.
The optimal solution is to always pick red. Every time. But humans pick a pattern. Rats pick red. Humans consistently do worse than rats. So while we are using a form of updating, it certainly isn’t proper bayesian updating. And just because you think we function some way doesn’t make it true. And it will forever be difficult to describe any AI as conscious, because we have really arbitrarily defined it to fit us. But we can’t truly say what it is. Not can we can why we function how we do. Or if we are all in a simulation or just a Boltzmann brain.
Honestly, something that concerns me most about AI is that it could become sentient, but we will not know if it is or just cleverly programmed so we treat it only as a tool. Because while I don’t think AI is inherently dangerous, I think becoming a slave owner of something that could be much more powerful probably is. And given their lack of chemical hormones, we will have even less of an understanding of what or how it feels.
All very fair points. It’s all wildly complicated, and I agree; we don’t really understand ourselves.
It could still be bayesian reasoning, but a much more complex one, underlaid by a lot of preconceptions (which could have also been acquired in a bayesian way).
Even if the result is random, a highly pre-trained bayessian network that has the experience of seeing many puzzles or tests before that do follow non-random patterns might expect a non-random pattern… so those people might have learned to not expect true randomness, since most things aren’t random.
Your brain understands concepts and can self-conceptualise, LLMs cannot do either. They can sound convincingly as if they understand concepts but that’s because we fill in gaps due to how we understand language. The examples of broken or distorted sentences being understandable applies here. You and I can communicate in broken sentences because you and I understand the concepts beneath the conversation. LLMs play on that understanding but they do not understand its concepts.
https://thegradient.pub/othello/
LLMs are neural networks and are absolutely capable of understanding.
LLMs are criminally simplified neural networks at minimum thousands of orders less complex than a brain. Nothing we do with current neural networks resembles intelligence.
Nothing they do is close to understanding. The fact that you can train one exclusively on the rules of a simple game and get it to eventually infer a basic rule set doesn’t imply anything like comprehension. It’s simplistic pattern matching.
Does AlphaGo understand go? How about AlphaStar?
When I say LLM’s can understand things, what I mean is that there’s semantic information encoded in the network. A demonstrable fact.
You can disagree with that definition, but the point is that it’s absolutely not just autocomplete.
No, and that definition has nothing in common with what the word means.
Autocorrect has plenty of information encoded as artifacts of how it works. ChatGPT isn’t like autocorrect. It is autocorrect, and doesn’t do anything more.
It’s fine if you think so, but then it’s a pointless argument over definitions.
You can’t have a conversation with autocomplete. It’s qualitatively different. There’s a reason we didn’t have this kind of code generation before LLM’s.
Adversus solem ne loquitor.
If you just keep taking the guessed next word from autocomplete you also get a bunch of words shaped like a conversation.
Yeah, totally. Repeating the same nonsensical sentence over and over is also how I converse. 🙄
Note that “real world truth” is something you can never accurately map with just your senses.
No model of the “real world” is accurate, and not everyone maps the “real world truth” they personally experience through their senses in the same way… or even necessarily in a way that’s really truly “correct”, since the senses are often deceiving.
A person who is blind experiences the “real world truth” by mapping it to a different set of models than someone who has additional visual information to mix into that model.
However, that doesn’t mean that the blind person can “never understand” the “real world truth” …it just means that the extent at which they experience that truth is different, since they need to rely in other senses to form their model.
Of course, the more different the senses and experiences between two intelligent beings, the harder it will be for them to communicate with each other in a way they can truly empathize. At the end of the day, when we say we “understand” someone, what we mean is that we have found enough evidence to hold the belief that some aspects of our models are similar enough. It doesn’t really mean that what we modeled is truly accurate, nor that if we didn’t understand them then our model (or theirs) is somehow invalid. Sometimes people are both technically referring to the same “real world truth”, they simply don’t understand each other and focus on different aspects/perceptions of it.
Someone (or something) not understanding an idea you hold doesn’t mean that they (or you) aren’t intelligent. It just means you both perceive/model reality in different ways.
For one thing, understanding implies that a word is linked to a mental concept. So if you say “The car is red”, you first need to mentally compare the mental concept of “red” to the car in question.
The Chinese room bypasses all of that, it can say “The car is red” without ever having seen a red object at all.
Do you maintain this line of reasoning if it only says “the car is red” when the car is in fact red. And is capable of changing the answer to correctly mentioned a different color when the item In question is a different question.
Some ai demos show that programs like gpt-4 are already way passed this when provided with, it can not only accurate describe whats in the image but also the context.
Some examples, mind these where shown in an openAI demo for gpt4, Open ai has not yet made their version of this tech publicly available.
When i see these examples, i am not convinced that the ai truly understands everything it is saying. But it does seem to understand context, One of the theories on how it can do this (they are still a black box) is talked about in some papers that large language models may actually create an internal model of the world similar to humans and use that for logical reasoning and context.
It doesn’t matter if the answer is right. If the AI does not have an abstract understanding of “red” then it is using a different process to get to the answer than humans. And according to Searle, a Turing machine cannot have an abstract understanding of “red”, no matter how complex the question or how complex an internal model is used to determine its answers.
Going back to the Chinese Room, it is possible that the instructions carried out by the human are based on a complex model. In fact, it is possible that the human is literally calculating the output of a trained neural net by summing the weights of nodes, etc. You could even carry out these calculations yourself, if you could memorize the parameters.
Your use of “black box” gets to the heart of it. Memorizing all of the parameters of a trained NN allows you to calculate an answer, but they don’t give you any understanding what the answer means. And if they don’t tell you anything about the meaning, then they don’t tell the CPU doing that calculation anything about meaning either.
I don’t think ai will ever use a process to derive an answer the same way as a human does. Maybe thats part of the goal for the original Turing test but i don’t think the biological human ways is the only way to intelligent understanding “on par” with human intelligence.
Does a blind person have an abstract understanding of “red”?
I can imagine an intelligent alien species, unable to perceive colors like us but yet having an sense to detect to what they call “surface temperature” which allow them to recognize specific wave lengths of the ligt reflecting on surfaces, this is sort of how humans see color but maybe for the alien they hear this as sound. They then go on and use this sensory input to make music. A song about the specific light wavelength that humans know as a deep bordeaux red color.
Do these biological Intelligent aliens not have an abstract understanding of the color red? I would say they do, its different then how we understand it for sure but both are valid. An even more supreme species might have both those understandings and combine them for an even deeper fuller sensory understanding of “red”.
I see ai similar to this, its a program contained in computer hardware. With no body of its own its depending on us to provide it with input. This is now mostly text so the ai obtains a text based understanding of the world, hence why its so decent at poetry. But when we attach more sensors like a camera then that will change.
I am not sure how to discuss “a human using instructions to calculate perfect answers, but not getting an understanding of what that answers means” wed might have to agree to disagree on that but i feel like thats all my brain has ever done. Were born in a complex place we do not comprehend, are given some instructions mostly by copying what others are doing. Then we find a personal meaning in those things, which as far as i am aware is unique for everyone. (Tbf: i am an autist, the fact that not all humans experience reality the same and that i had to find and learn my own personal understanding of the world has greatly shaped how i think about these systems)
Perhaps I should rephrase the argument as Searle did. He didn’t actually discuss “abstract understanding”, instead he made a distinction between “syntax” and “semantics”. And he claimed that computers as we know them cannot have semantics, whereas humans can (even if we don’t all have the same semantics).
Now consider a quadratic expression. If you want to solve it, you can insert the coefficients into the quadratic formula. There are other ways to solve it, but this will always give you the right answer.
If you remember your algebra class, you will recognize that the quadratic formula isn’t just some random equation to compute. You use it with intention, because the answer is semantically meaningful. It describes things like cars accelerating or apples falling.
You can teach a three year old to identify the coefficients, you can show them the symbols that make up the quadratic formula: “-”, second number, “+”, “√”, “(”, etc. And you can teach them to copy those symbols into a calculator in order. So a three year old could probably solve a quadratic expression. But they almost certainly have no idea why they are doing what they are doing. It’s just a series of symbols that they were told to copy into a calculator, their only intention was to copy them in order correctly. There are no semantics behind the equation.
For that matter, a three year old could equally well enter the symbols necessary to calculate relativistic time dilation, which is an even shorter equation. But if their parents proudly told you that their toddler can solve problems in special relativity, you might think, “Yes… but not really.”
That three year old is every computer program. Sure, an AI can enter symbols into a calculator and report the answer. If you tell them to enter a different series of symbols, they will report a different answer. You can tell the AI that one answer scores 0.1 and another scores 0.8, and to calculate a different equation that is based partly on those scores. But to the AI, those scores and equations have no semantic meaning. At some point those scores might stop increasing, and you will declare that the AI is “trained”. But at no point does the AI assign any semantic content behind those symbols or scores. It is pure syntax.
For me, I think the criteria I’d use for saying someone has a decent understanding of math is knowing that math has underlying rules and most things can be understood from those basic rules (each problem is not just an arbitrary magic trick to get an answer that was impossible figure out) and perhaps also being able to ask “novel” questions (compared that what you already know) and taking reasonable steps to answer it with the rules you do know and the tools you have (doesn’t need to be successful). I think counting could be done with any consistent set of sounds and it doesn’t matter whether yours just reading those sounds for a list or not as long as you know roughly what they correspond to in terms of time. I don’t think a lot of humans have much understanding of math, I think some computers already beat a lot of humans with respect to that.