TTT #36: Managing the AI Hype Cycle

Stephen Collins • Mar 23, 2024

It seems like every day there are new headlines proclaiming the next AI breakthrough that will supposedly revolutionize everything from healthcare to education to creative industries. While there have certainly been impressive advances in AI capabilities, it’s important to view many of the breathless claims through a skeptical lens.

The Financial Incentives Driving Hype

A major factor driving the torrent of AI hype is the unprecedented amount of capital that has flooded into the field over the past few years. Venture capitalists poured a massive $50 billion into AI startups in 2023 alone. With so much money at stake, there is an enormous financial incentive for companies and researchers to make bold claims about their AI systems in order to attract investment and media attention.

Curated Demos and Overstated Capabilities

Many of the most sensational claims tend to come from companies that are seeking to raise funds or trying to drum up publicity. They may overstate an AI’s general capabilities or imply that it has human-like reasoning that it does not actually possess. Terms like “artificial general intelligence” get thrown around liberally despite no system meeting that criteria.

It’s also common for companies to carefully curate AI demo scenarios that showcase strengths while hiding weaknesses and failure cases. An AI might seem to engage in coherent dialog during a limited text interaction, only to produce nonsensical outputs when pushed slightly off script.

The Risks of Extraordinary AI Claims

Even respected academic labs and research institutions have faced criticism for making exaggerated or premature claims about their AI work. The pressure for headlines and competition can lead elite academics down the path of overblown marketing at times.

While genuine breakthroughs occur, extraordinary claims require extraordinary evidence that can withstand rigorous scrutiny. Achievements may represent meaningful progress within specific domains or benchmarks without indicating human-level general intelligence.

Recognizing the nuances and true limitations is critical when evaluating any AI claim, regardless of the prestige of its source.

Maintaining a Critical Eye

So how can you view the steady stream of AI announcements with an appropriately critical eye?

  1. Evaluate the source - startups raising funds or seeking publicity are incentivized to hype.
  2. Look at demos in context - evaluate claims in the context of the specific capabilities being shown rather than extrapolating too far.
  3. Seek third-party expert analysis - look for objective analysis from those without a financial stake.

Maintaining a healthy skepticism amidst the all of the AI hype is key for separating the “wheat from the chaff”.