AI Explained in 10 Minutes: A Simple Formulaic Approach
Why CUDA is Instrumental to Nvidia’s Success
This post is inspired on this video from a CUDA Hackathon featuring Raja Koduri (ex EVP at Intel, ex SVP at AMD), who eloquently laid out what AI really means in 10 minutes. Here are the main points from his presentation which is an excellent way to get a deeper understanding of what AI means from a Semiconductor Expert.
Skeptics of GPU: A Common Occurrence
Since 1995, every five years or so, skeptics have predicted the demise of GPUs. Similar doubts were cast on CPUs during the early internet era, with predictions of specialized network accelerators and Java accelerators taking over a limited ARM x86. Fast forward to 2024, x86 ARM CPUs are still prevalent. The introduction of Google's TPU in 2016 caused Nvidia’s stock to drop temporarily, with a famous presentation by David Patterson (TPU Lead @ Google) predicting that all there would be in the future was a domain specific language. However, the anticipated domain-specific language for AI did not prevail, and Nvidia continued to lead the industry.
What Does AI Stand For?
AI doesn't stand for artificial intelligence; it stands for "Architecture Input." This concept, highlighted by Raja Koduri, emphasizes the balance between generality and performance.
Architecture Input = Generality x Performance
Today’s CPUs are highly general but lack performance in graphics and AI workloads. On the other hand, AI ASIC hardware excels in performance but lacks generality. The optimal balance is achieved by GPUs, more specifically NVDA GPU’s which has the best score across many measures.
Concept of Generality
In the history of parallel programming, there hasn’t been a successful architecture whose programming model was different than the execution model of the machine. Successful parallel programming architectures align their programming models with their execution models. CUDA exemplifies this principle, making GPUs versatile and powerful. Raja Koduri notes that a big mistake he sees in many hardware companies is the lack of integration of the GPU into the rest of the system software stack (Linux drivers, etc). This is why the concept of Generality is so important & is a big reason why Nvidia is lightyears away from other competitors like AMD.
Generality = F( Compatibility, Extensibility, Accessibility)
A) Compatibility
Initially, GPU architecture focused on graphics rendering, evolving to programmable shapes and floating-point computation. Nvidia’s CUDA introduced pointers, bridging the gap between GPU languages and the CPU world. This innovation made GPUs compatible with mainstream programming languages like C++.
B) Extensibility
Nvidia’s GPUs support the creation of new languages and libraries, enhancing their extensibility. This ensures that the technology can evolve with new innovations and requirements.
C) Accessibility
Nvidia has democratized GPU programming, making it accessible through affordable gaming GPUs, not just cloud-based solutions. This accessibility allows a wider range of developers to innovate with GPU-based applications.
Concept of Performance
Performance in AI hardware is measured by the formula:
Performance= F(Flops, Bandwidth, Capacity) / (Power x Cost)
A few notes here:
One megawatt now costs $30m-$40m to build and that needs to come down to <$10m for for AI to reach internet scale
The next generations of GPU’s will bring compute & memory together. Nvidia’s advancements from the H100 to B200 showcase continuous improvements, aiming to optimize memory capacity and compute power while reducing costs.