The high-order bit that changed in AI:
— Andrej Karpathy (@karpathy) August 3, 2023
"I'll give you 10X bigger computer"
- 10 years ago: I'm not immediately sure what to do with it
- Now: Not only do I know exactly what to do with it but I can predict the metrics I will achieve
Algorithmic progress was necessity, now bonus.
Turning scaling into a systematic science is the biggest advance enabled by LLMs. https://t.co/BgHoDxeX7m
— Jim Fan (@DrJimFan) August 4, 2023
It will be interested to see if or when we hit scaling limits to training more powerful models and what our new bottleneck becomes. For now, there appears to be a lot of greenfield.