As competition heats up, plowing money into foundation model startups is fraught with risk. Building foundation models from scratch requires enormous resources in the form of computing power and human talent — both of which remain scarce.
Iuri Struta, S&P Global (source)
I was listening to Bret Taylor, the founder and CEO of Sierra, former co-CEO of Salesforce, ex CTO of Facebook and current chairman of OpenAI on the 20VC podcast. He talked about where we are in the timeline of generative AI and it made me realise that we’re making an interesting shift now.
While most VC money (in terms of $ amount) went to foundational models (think OpenAI, Anthropic and open source including Mistral AI, Meta and to an extent NVIDIA), the LLMs are reaching a bit of a saturation (until the next major breakthrough) and they are ‘good enough’ to build applications on.
In 2023 foundation models raised $28.67 billion across 9 top pure-play companies. Application companies on the other hand, those that build on top of existing models instead of training their own, attracted over twice as much funding in Q1 2024 vs Q1 2023.
However, we’re starting to see that foundational models are starting to plateau. LLMs, while revolutionary, are flawed in significant ways. The issues are well known: they hallucinate, they largely lack reasoning skills, they require obscene amounts of training data and computational power, making them costly to scale. A recent research paper by Apple on the limitations of LLMs suggests that brute force is not going to solve these problems either.
Venture capital fund Accel noted this tension between the major and minor league in AI in their Euroscape 2024 report. Basically, to be able to compete in what Accel called the ‘AI Majors’ league, companies need to be able to raise at least $100 million, almost at inception. Case in point, Mira Murati, OpenAI’s co-founder and former CTO who recently left the company, is rumoured to be raising $100M for her new startup which “will train proprietary models to build AI products”. Even the AI challengers, require deep pockets to get started unless they have a real competitive edge.
As Gil Dibner of Angular Ventures put it, to succeed in this market as a startup, companies that raise inception capital (and their early stage investors like my firm, Remagine Ventures) will need one or more of these competitive advantages to win:
- A genuine technical edge (rare but not impossible)
- Deep domain expertise.
- Durable distribution advantages.
- Hyper complex products/roadmaps
- Super painful sales cycles
- Willingness to out-hustle literally everyone on a shoestring
So, in the current point in time, there’s no real reason for most companies today to train a new model unless they’re truly going for AGI. They can fine tune, calibrate and train custom models as described in the bottom part of the Menlo Ventures “Modern AI tech stack” image below, but there’s no need to add another foundational model (unless you have very deep pockets like SSI Inc.).
If not LLMs, where will the value accrue in Generative AI?
But the challenges in LLMs don’t remotely mean that AI is dead. So where should VC investors place their bets? A recent article by Vivek Wadhwa published in Fortune suggests 4 key areas:
- Neurosymbolic AI: Combining neural networks with symbolic AI to enable true understanding and reasoning. Such progress could help AI systems better handle multi-step reasoning and systematic problem analysis (a space being tackled by autonomous AI agents).
- Efficient AI models: Smaller, more scalable models that are less resource-intensive. Rather than ‘one model to rule them all’, the next wave of AI innovation may focus on making models smarter, more efficient, and problem or vertical specific. Vertical AI and fine tuning of vision models for a specific task is a step in that direction.
- Context-aware AI: Improving AI’s ability to maintain context in conversations for more meaningful interactions. Current language models often struggle to maintain consistency across longer conversations – similar to talking with someone who has short-term memory issues.
- Ethical and Explainable AI: Addressing bias, misinformation, and potential misuse to ensure responsible AI development. Embedding ethical considerations directly into AI systems can help deliver more responsible, nuanced recommendations – particularly crucial when AI assists with life-impacting decisions in fields like healthcare, legal sentencing, or educational assessment
The current pace of innovation has been relentless and investment has promptly followed. A lot of it has been driven by FOMO, as Sequoia pointed out in their “AI’s $600 billion question” But the cracks are starting to show, and we’re starting to see how applications, and the above trends are extending the promise of AI beyond LLMs.
I'm a former general partner at google ventures, head of Google for Entrepreneurs in Europe and founding head of Campus London, Google's first physical hub for startups.
I'm also the founder of Techbikers, a non-profit bringing together the startup ecosystem on cycling challenges in support of Room to Read. Since inception in 2012 we've built 11 schools and 50 libraries in the developing world.
- Breaking the mold: Pattern Breakers book review - November 18, 2024
- Weekly Firgun Newsletter – November 15 2024 - November 15, 2024
- Kindling Early B2C Growth: Getting to 1,000 Users - November 11, 2024
