How the GPU Works: The Graphics Pipeline (and How It Became the Engine of Deep Learning)
We’ve talked at a high level about the graphics pipeline — geometry, transforms, rasterization, shading, compositing. That’s the conceptual model. But what does this actually look like when it runs on real hardware? This post goes one level deeper: how OpenGL ES implements that pipeline on a GPU, what happens when there’s no GPU at all, and why understanding this still matters in 2026.
The Pipeline, As the GPU Sees It
Strip away the API and OpenGL ES really boils down to two programmable stages bookending a fixed-function step:
Vertex shader — runs once per vertex, transforming it into screen space.
Rasterization — the GPU’s fixed-function hardware converts each triangle into fragments (candidate pixels).
Fragment shader — runs once per fragment, deciding its final color.
Almost everything interesting in modern rendering happens in those two shader stages. The vertex shader answers “where does this point go,” and the fragment shader answers “what color is this point.” Rasterization is the bridge between them — mechanical, fast, and (on real hardware) not something you write code for.
What a Fragment Shader Actually Does
A fragment shader isn’t just “pick a color.” At minimum it typically:
Interpolates values from the vertices — colors, normals, texture coordinates are calculated per-vertex, then smoothly interpolated across the triangle’s surface for each fragment.
Reads textures, if any are bound, sampling color data from an image.
Computes the final pixel color — applying lighting, combining texture samples, or running whatever arbitrary math the shader author wrote.
Writes color, depth, and any other output values to the framebuffer.
This is also where blending and depth testing live conceptually: alpha blending lets a fragment combine with whatever’s already in the buffer, and z-buffering ensures that when multiple objects overlap, the one closest to the camera wins (other resolution strategies exist too, but nearest-to-eye is the default and most common).
How This Maps to Real Silicon
It’s easy to think of the pipeline as an abstract flowchart, but mobile GPUs implement it as actual dedicated hardware blocks. This part is corroborated by public sources rather than any single internal document: NVIDIA’s own publicly released Tegra 4 GPU architecture whitepaper describes dedicated “VLIW pixel fragment shader pipelines” and separate “texture filtering units” as distinct hardware blocks. Independently, the open-source grate-driver project — which reverse-engineered and publicly documented the Tegra fragment shader instruction set — describes the fragment pipeline as being split across a handful of separate units, including:
PSEQ (program sequencer) — fetches and feeds instructions/data into the pipeline
MFU (multi-function unit) — handles interpolation and special functions
ALU — the arithmetic unit performing the actual per-fragment math
The point isn’t to memorize these specific names — they vary across GPU vendors and even across generations of the same vendor’s chips — but to notice the pattern, which both public sources agree on: there’s a dedicated piece of hardware for sequencing instructions, a separate one for interpolation, and a separate one for arithmetic. This division of labor into small, specialized, parallel units — rather than one general-purpose core doing everything serially — is the architectural DNA that GPUs carried directly into deep learning hardware. The dedicated ALU doing per-fragment math in 2012-era mobile graphics is a direct conceptual ancestor of the matrix and tensor units doing per-neuron math in 2026-era AI accelerators.
Sources: NVIDIA, “Tegra 4 Family GPU Architecture” whitepaper (publicly published); grate-driver project, “Fragment Shader ISA” documentation (open-source, public wiki).
When There’s No GPU: Software Rendering
Not every device has dedicated graphics hardware, and not every situation wants it — testing, virtualization, and certain embedded/CI environments often render graphics entirely on the CPU. This is “software rendering,” and several real implementations exist:
Mesa (commonly used on Linux), with its Gallium llvmpipe backend — an open-source project hosted publicly on freedesktop.org/GitHub
Android’s libagl — part of the public Android Open Source Project
Adobe’s SwiftShader — open-sourced and publicly available on GitHub
These projects reimplement the entire GPU pipeline as ordinary CPU code. It’s slower than dedicated hardware, but it proves something important: the pipeline is fundamentally just a sequence of well-defined operations, and any sufficiently capable processor can execute them — just without the parallel hardware acceleration.
A simplified version of what a software rasterizer’s main loop looks like:
foreach tri in triangles {
// run the vertex program on each vertex
v1 = process_vertex(tri.vertex1);
v2 = process_vertex(tri.vertex2);
v3 = process_vertex(tri.vertex3);
// assemble the vertices into a triangle
assembledtriangle = setup_tri(v1, v2, v3);
// rasterize the assembled triangle into fragments
fragments = rasterize(assembledtriangle);
// run the fragment program on each fragment
foreach frag in fragments {
outbuffer[frag.position] = process_fragment(frag);
}
}
This is the entire pipeline in roughly ten lines: transform vertices, assemble a triangle, rasterize it into fragments, then shade each fragment. Everything else — lighting models, texturing, blending — is detail layered on top of this loop.
How Software Rendering Actually Executes a Shader
Here’s the part that’s genuinely clever: GLSL shaders are written assuming a GPU will compile and run them as native instructions. A software renderer has no GPU to hand that off to, so it builds a small virtual machine instead. The shader gets parsed into an instruction stream, and the “VM” steps through it, maintaining state such as:
Fragment varyings (the interpolated per-fragment values)
Vertex attributes
Uniforms (constants passed in from the application)
Registers and condition codes
Sampler and texture info
Then it executes each shader instruction against that state, one opcode at a time. As a concrete example, here’s the kind of register-machine pattern used in Mesa’s actual open-source software path to implement the LERP (linear interpolation) opcode — the same blending math LERP(a, b, c) = a*b + (1-a)*c that shows up constantly in shading and texturing:
case OPCODE_LRP:
{
GLfloat a[4], b[4], c[4], result[4];
fetch_vector4(&inst->SrcReg[0], machine, a);
fetch_vector4(&inst->SrcReg[1], machine, b);
fetch_vector4(&inst->SrcReg[2], machine, c);
result[0] = a[0] * b[0] + (1.0F - a[0]) * c[0];
result[1] = a[1] * b[1] + (1.0F - a[1]) * c[1];
result[2] = a[2] * b[2] + (1.0F - a[2]) * c[2];
result[3] = a[3] * b[3] + (1.0F - a[3]) * c[3];
store_vector4(inst, machine, result);
}
Each of the four components ([0..3]) corresponds to a color channel — red, green, blue, and alpha — computed independently with the exact same formula. That’s not a coincidence; it’s the same “same operation, repeated across independent lanes” pattern that defines GPU work in general, just running serially on a CPU instead of across thousands of parallel cores.
How This Same Hardware Got Adapted for Deep Learning
Everything described above — vertex shaders, fragment shaders, the PSEQ/MFU/ALU-style block split — was designed to do one thing extremely well: run the same small program across thousands of independent data points at once (one vertex, one fragment, one pixel at a time). A neural network layer needs exactly that same capability, just with different data: multiply an input vector by a weight matrix, across millions of neurons, independently and in parallel.
That overlap is why GPUs ended up training nearly every modern AI model:
The ALU block that crunches per-fragment lighting math is architecturally the same kind of unit that, scaled up and specialized, became the matrix-multiply (”tensor core”) units in modern AI accelerators.
The virtual machine that steps through shader opcodes in software rendering is a small, readable preview of what a deep learning compiler does when it lowers a model into GPU kernels — both are “interpret a small instruction stream, applied independently across many lanes of data.”
The interpolation and texture-sampling hardware built for shading is no longer the bottleneck in AI workloads, but the surrounding philosophy — dedicate fixed hardware to the operation you’ll do millions of times — is exactly why today’s chips ship with dedicated matrix units instead of relying on general-purpose cores.
In short: the graphics pipeline didn’t get replaced by AI hardware — it got repurposed. The chips changed their internal proportions (more matrix units, less texture/raster hardware), but the core idea — many identical operations, executed in parallel across dedicated silicon — is a straight line from rendering triangles to training transformers.
Why This Still Matters
It’s tempting to think shader internals and software rasterizers are a relic of the early mobile-GPU era. But the underlying lesson generalizes cleanly to anything you build today:
Specialized hardware blocks for specialized math (interpolation, sampling, arithmetic, sequencing) is the exact same design philosophy behind today’s AI accelerators, which split work across tensor cores, memory-bandwidth-optimized units, and instruction schedulers.
Compiling a small language down to a virtual machine that executes per-element is precisely what happens when a deep learning framework compiles a model graph down to GPU kernels that execute the same operation across every element of a tensor.
Software fallbacks exist in both worlds — just as Mesa or SwiftShader let you run graphics without a GPU, CPU-based inference exists for running smaller AI models without an accelerator, trading speed for availability.
The shader pipeline, in other words, isn’t just a relic worth knowing for graphics nostalgia. It’s a small, fully-readable example of the exact computational pattern — many independent, identical operations, executed either by dedicated parallel hardware or emulated step-by-step in software — that now underlies the entire AI compute stack.