Sizing hardware for running models locally
Technique
To run an AI model on your own machine, the whole model must fit in the graphics chip's memory. Splitting a model across two consumer graphics cards doesn't really work, because the link between them is far too slow to share memory.
Theo's current stance
He says the amount of memory, not chip speed, decides what you can run.
Try it
- Before buying hardware, check the model's size against the memory of a single machine.
- Don't plan on linking two consumer graphics cards to fit a bigger model.
- Look at machines with large unified memory (shared by processor and graphics), like the Mac Studio.
First seen
7:25
You can't just plug the chips in together. If you have a model that is 40 gigs, hell, if you have a model that's 33 gigs and it doesn't fit on your 5090, you now have to split the data across the two GPUs, which means ideally you
Every mention
7:25
You can't just plug the chips in together. If you have a model that is 40 gigs, hell, if you have a model that's 33 gigs and it doesn't fit on your 5090, you now have to split the data across the two GPUs, which means ideally you
Theo Daily is unofficial and not affiliated with Theo or t3.gg. Written by Claude Opus 5.5 from each video's transcript.