What does AI think about Gary Marcus and his insight that LLM AI scaling has hit a wall?
I find listening to this AI NotebookLM discussion fascinating. NotebookLM is addressing
and his insight that AI and LLMs, as a training model, have hit a wall.It's very objective and emotionlessly discussing its own potential fate. Did you catch the end where it addresses “NotebookLM” as "him"?
One has to ponder what this could mean for all that Cap-ex spending and its ROI?
Hope you enjoy. Someone is knocking on my door.
“Who is it?”
The sources I included for NotebookLM:
NotebookLM Briefing Document: The Limits of Scaling in AI and the Need for Hybrid Approaches
Executive Summary:
This document summarizes Gary Marcus's arguments, presented in two articles, regarding the limitations of scaling in AI, particularly deep learning. Marcus argues that the pursuit of "pure scaling" (simply increasing data and parameters in large language models - LLMs) is hitting a wall and that true progress in AI will require a shift towards hybrid models that incorporate symbolic manipulation. He uses the example of OpenAI's "GPT" models to support his claim, and points to the historical debate between symbolic AI and neural networks, advocating for a neurosymbolic rapproachment.
Source 1: "Breaking: OpenAI's efforts at pure scaling have hit a wall."
Main Theme: The limitations of pure scaling for achieving AGI and the likely downgrade of OpenAI's GPT-5 to GPT 4.5.
Key Ideas/Facts:
Scaling is Failing: Marcus argues that after years of promises, OpenAI's efforts to simply scale up LLMs are failing to deliver true AGI, citing the delayed (and potentially downgraded) release of GPT-5. "For nearly three years I have been saying that the pure scaling of LLMs – adding more data and more parameters – would eventually run out, and that it would fail to solve hallucinations and boneheaded errors and that scaling laws were merely empirical generalizations, rather than physical laws."
Altman's Bluff: Marcus suggests that Sam Altman, CEO of OpenAI, has been "bluffing" about the progress toward AGI, and the downgrade of "Orion" to GPT 4.5 is an admission of the failure of pure scaling. "All this time, it turns out, Altman was bluffing."
New Approaches Needed: OpenAI will shift focus. "Second, pure scaling will no longer be the means of attack. Instead, OpenAI will be throwing the kitchen sink at future efforts to build GPT-5 , including “test-time-compute”, a new approach that involves longer, more expensive inference times rather than “constant time inference” of GPT-4-like models, as well as (I suspect) massive amounts of synthetic data, which appears to work better for semi-closed domains like math and coding than in the open-ended real world."
Scaling Isn't Everything: Marcus quotes a Google DeepMind researcher who agrees that "the death of pure scaling is not the death of AI," but that the myth of performance being predictable solely based on data and parameters is dead.
Source 2: "Deep Learning Is Hitting a Wall - Nautilus"
Main Theme: A broader argument against relying solely on deep learning and advocating for hybrid neurosymbolic approaches.
Key Ideas/Facts:
Deep Learning's Limitations: Deep learning excels at tasks requiring "rough-ready results" (like photo tagging) but struggles in high-stakes situations (like radiology or autonomous driving) where errors are unacceptable. "Deep learning, which is fundamentally a technique for recognizing patterns, is at its best when all we need are rough-ready results, where stakes are low and perfect results optional."
Overreliance on Scaling: The AI field is increasingly relying on scaling up models with more data, but these efforts are not necessarily leading to genuine comprehension. "The implication was that we could do better and better AI if we gather more data and apply deep learning at increasingly large scales." "Indeed, we may already be running into scaling limits in deep learning, perhaps already approaching a point of diminishing returns."
The Need for Symbolic Manipulation: Marcus argues for a return to the concept of symbolic manipulation, which involves representing and processing information using symbols, algebra, and logic. "To think that we can simply abandon symbol-manipulation is to suspend disbelief."
NetHack Example: The NetHack Challenge, where a symbolic AI system defeated deep learning systems, is cited as evidence of the importance of reasoning and understanding abstract relationships. "But in December, a pure symbol-manipulation based system crushed the best deep learning entries, by a score of 3 to 1—a stunning upset."
Historical Context: Marcus traces the historical conflict between symbolic AI and neural networks, highlighting the "bad blood" that has hindered progress. He recounts Hinton's shift away from neurosymbolic approaches despite early work in that direction.
Benefits of Hybrid AI: Marcus presents four main reasons why hybrid AI (combining deep learning and symbolic manipulation) is the best way forward. "So much of the world’s knowledge, from recipes to history to technology is currently available mainly or only in symbolic form."
Neurosymbolic Momentum: He notes that there's growing interest and investment in neurosymbolic approaches from researchers and major companies like IBM, Intel, Google, Facebook, and Microsoft. Examples like AlphaGo and AlphaFold2 are cited as successful hybrid systems.
Overlapping Themes and Key Takeaways:
The limitations of scaling alone: Both articles emphasize that simply increasing the size of AI models and the amount of training data isn't a guaranteed path to AGI or even reliable performance.
The importance of reasoning and symbolic manipulation: Marcus consistently advocates for incorporating symbolic manipulation and reasoning into AI systems, arguing that these are essential for true understanding and problem-solving.
The need for hybrid approaches: Marcus believes that the most promising path forward for AI is to combine the strengths of deep learning (pattern recognition) with symbolic manipulation (reasoning, knowledge representation). "No single AI approach will ever be enough on its own; we must master the art of putting diverse approaches together, if we are to have any hope at all."
A Call for Collaboration and Openness: Marcus concludes by calling for collaboration across different fields (linguistics, psychology, neuroscience, etc.) and a willingness to consider diverse approaches to AI development.
This briefing document provides a comprehensive overview of Gary Marcus's arguments regarding the state and future of AI. His central claim is that the field needs to move beyond a narrow focus on deep learning and scaling, and embrace hybrid approaches that incorporate symbolic manipulation and reasoning.
NotebookLM can be inaccurate; please double check its responses.