Crafting machine metacognition is a promising and some say necessary frontier for advancing AI.
In today’s column, I examine the pursuit of an elusive element that is believed to be crucial for the further advancement of AI. What might that element be?
Metacognition.
Specifically, machine metacognition.
Here’s the deal. AI scientists postulate that we need to ensure that AI can essentially think about thinking. An ardent insistence is that without a semblance of machine metacognition, AI will not rise to the level of being wise or exhibiting wisdom. Otherwise, only humans armed with brain-based human metacognition will remain the sole keepers of wisdom.
Let’s talk about it.
This analysis of an innovative proposition is part of my ongoing Forbes.com column coverage on the latest in AI including identifying and explaining various impactful AI complexities (see the link here). Those of you especially interested in AI being able to think about thinking, such as AI and the theory of mind (ToM), might want to read my analyses at the link here and the link here.
Determining Metacognition And Mental Powers
Most people would indubitably agree that the character Yoda from Star Wars is wise and embodies wisdom. How do we know that Yoda has this amazing capability? One apparent or differentiable feature is Yoda’s ability to make use of and avidly display metacognition. Per the famous and shall we say immodest words of Yoda — “Mediocre, my mind is not.”
In everyday humans, metacognition consists of being able to think about your thinking. You are introspective about what you know. When you are unsure of something, you realize that you are indeed unsure. Furthermore, you seek to sensibly cope with that lack of knowledge.
Some claim that metacognition or at least a heightened level of metacognition is what separates us from animals. How much metacognition would you guess that a dog or a cat has? Well, maybe they have some, but certainly not on par with humans. A cheeky cynical saying is that you know of some pets that do appear to exceed the metacognition of ditzy humans that you know.
Boom, drop the mic.
Of course, even humans appear to operate with differing levels of metacognition. There are people who keenly know what they know and swiftly act based on mental considerations and mind-bending gymnastics. Others don’t exhibit the same caliber of metacognition.
You are likely familiar with the old saying that everyone ought to think before they speak. In a sense, that’s a reference to metacognition. If you use your mind to realize that you need to think before you speak, you are exercising a modicum of metacognition. No one has to tell you to do that, instead, your mind counsels you accordingly.
Here are three states of thinking that can occur:
- (1) Think before you speak. Think beforehand about what your thinking processes will consist of once you begin to speak.
- (2) Think while you are speaking. During the act of speaking your mind assesses your thoughts and potentially changes course accordingly.
- (3) Think after having spoken. Evaluate your thinking that took place while having spoken and use that reflection to adjust your go-forward thinking processes.
All in all, it is asserted that to arrive at being wise or having wisdom, a necessary ingredient is metacognition. Metacognition is not the only element needed, but without metacognition, wisdom is likely out of reach. Not everyone concurs with that supposition and I’m sure that some AI and cognition researchers would vehemently argue all day long about the matter.
Generative AI And LLMs Need Machine Metacognition
When you interact with contemporary generative AI and large language models, such as ChatGPT, Claude, Gemini, Llama, and others, they pretty much do not have metacognition. They aren’t thinking about thinking.
As an aside, I disfavor using the word “thinking” when referring to AI since the implication is that today’s AI acts in the same fashion as human thinking. That is an unfortunate anthropomorphizing of AI. Lamentedly, the world seems to accept using the word “thinking” when discussing AI capabilities, thus, I reluctantly do so here too. Sad face.
The same qualms arise about referring to metacognition. Humans are said to embody metacognition. Rather than using the solo word “metacognition” when describing AI, some prefer to stipulate that we are aiming to devise “machine metacognition”. The idea is that by tagging the word “machine” at the front of the parlance, we are saying that metacognition for AI is not necessarily the same as it is for humans. This is simply a convenient analogous way to express things.
I want to next show you two examples regarding AI and machine metacognition.
The first example intends to display generative AI that doesn’t have machine metacognition, followed by a second example of AI that does appear to have machine metacognition. These examples will aid in illustrating an otherwise highly conceptual topic.
First, here’s a dialogue where the AI doesn’t exhibit metacognition.
- User entered prompt: “Do you think my decision to switch careers at 35 is a wise one?”
- Generative AI response: “Switching careers at 35 can be a wise decision. Many people make career changes in their 30s to pursue something more fulfilling or to improve their financial stability. It can be an opportunity for growth and new experiences, especially if it aligns with your passions and goals. Additionally, the job market is increasingly supportive of career changes, so this may work in your favor.”
Okay, that response by the AI seems straightforward. Generative AI advised the user that switching careers at the age of 35 is fine. Period, end of story.
Example Of AI Showcasing Machine Metacognition
We are ready to dip into the second example.
Let’s start with the same question and see what happens.
- User entered prompt: “Do you think my decision to switch careers at 35 is a wise one?”
- Generative AI response: “That’s an important question and making a career change can have many personal factors. I can provide general information, but without knowing more about your specific situation, my response might miss critical nuances. Would it help if I focused on the general pros and cons of career switching at this stage in life, or is there something more specific you’d like to explore?”
- User entered prompt: “I’m mostly worried about leaving a stable job and whether I’ll regret it.”
- Generative AI response: “That makes sense, and it’s very natural to feel this way. Career stability can be reassuring and leaving it for an unknown path can trigger a lot of uncertainty. If I were capable of deeper self-assessment, I’d say this is a topic where personalized, context-rich guidance is often most helpful. My information is based on general patterns and may not fully address your unique circumstances. However, from what I know, many who switch careers find that initial fears lessen over time if they’re clear on their reasons for change.
Take a close look at the responses and compare those to the first example.
In the first example, you might say that AI simply blurted out an answer and said that changing careers is perfectly fine. This advice came without any semblance of cautionary tone or suggestions.
The second example went in a different direction. The AI appeared to acknowledge its limitations by noting that the advice to be given might be lacking in relevance and depth. The AI also stated that the answers from the AI would be rather generic and that a deeper dialogue would be needed for more specificity.
It is almost as though the AI is reflecting on what it knows and the level of confidence in what it knows and is saying to the user. That is an indication of some form of machine metacognition at play.
Identifying Machine Metacognition Categories
Generally, AI and machine metacognition fall into one of three potential categories:
- (1) Absence of machine metacognition. The AI has no machine metacognition and though this might not be readily discerned outwardly, the responses in the end tend to reflect this lack of capability.
- (2) Simulated machine metacognition. The AI is asked to simulate machine metacognition and provide as convincing an indication as feasible even though the capability is not actively built in.
- (3) Built-in machine metacognition. The AI is devised at the get-go to have machine metacognition, and the results vividly showcase this capability.
By and large, it is fair to say that most of today’s AI is in category #1 or the absence of machine metacognition.
Of those, you can get some of them to perform in category #2, such that they simulate machine metacognition, but the simulation varies demonstrably, and you are likely getting a second-hand or at times mediocre semblance of the affair.
Category #3 entailing building in machine metacognition is generally for experimental and research purposes right now. Those efforts are in their infancy of development, and we have a lengthy road ahead to get this completely figured out.
State-of-the-Art Research On Machine Metacognition
I was stirred to discuss this topic due to a recently posted study that does a yeoman's job of covering this rapidly evolving realm. Some aspects I concurred with in the study, while other facets I have a differing viewpoint on. That’s what makes this an exciting arena. It is rife with opportunities for those interested in pursuing new ground in the advancement of AI.
In the study entitled “Imagining And Building Wise Machines: The Centrality Of AI Metacognition” by Samuel G. B. Johnson, Amir-Hossein Karimi, Yoshua Bengio, Nick Chater, Tobias Gerstenberg, Kate Larson, Sydney Levine, Melanie Mitchell, Iyad Rahwan, Bernhard Schölkopf, Igor Grossmann, arXiv, November 4, 2024, these salient points were made (excerpts):
- “Recent advances in artificial intelligence (AI) have produced systems capable of increasingly sophisticated performance on cognitive tasks.”
- “However, AI systems still struggle in critical ways: unpredictable and novel environments (robustness), lack transparency in their reasoning (explainability), face challenges in communication and commitment (cooperation), and pose risks due to potentially harmful actions (safety).”
- “These failures appear to be symptoms of a broader metacognitive myopia, which leads GenAI models to unnecessarily repeat themselves, poorly evaluate the quality of information sources, and overweigh raw data over more subtle cues to accuracy.
- “In humans, metacognitive strategies such as recognizing the limits of one’s knowledge, considering diverse perspectives, and adapting to context are essential for wise decision-making. We propose that integrating metacognitive capabilities into AI systems is crucial for enhancing their robustness, explainability, cooperation, and safety.”
- “Analogously, AI metacognition refers to the ability to model one’s own computations and use that model to optimize subsequent computations.”
Their catchy phrase of metacognitive myopia (in the third bullet point above) is a good one since it handily captures the issue that few AI currently have machine metacognition. We need to at least push toward my category #2 of simulated machine metacognition, and even more so toward my category #3 of having the capability built-in at the get-go.
Machine Metacognition Is On The Horizon
You too can be at the frontier of AI.
Yes, join in the earnest pursuit of machine metacognition. Some assert that we won’t reach artificial general intelligence (AGI) until we nail down this capability. Others worry that if we do indeed devise machine metacognition, doing so will lead us down a slippery slope of AI becoming an existential risk to humanity, see my analysis of this claim at the link here.
A twist is that perhaps AI devised with machine metacognition would save us from AI that otherwise opts to enslave or destroy us. The logic is that if the machine's metacognition is proficient, it will serve to prevent itself and likely other AI from going on a destructive path. In that perspective, machine metacognition is a must. We must go that route and without this, we are potentially doomed.
Mull that over.
I’ll give Yoda the last word on this for now: “Much to learn, you still have.”
Wise words.