Local Models and GPU computation

I’m a huge fan of the Qwen model series and PaddlePaddle’s selection of models. They punch above their weight and relatively cheap to host on our machines at work. The push for better open source models will usher in an era to have local, private “intelligence” become more of the norm and hopefully we will become less dependent on frontier labs. But this might be true if you have a couple of A100s or a couple of Mac Studios, and definitely not if you have consumer-grade hardware. The current state of the art open models require on the order of at least 50-100 GBs which is well out of the range of most consumers. I don’t think model improvements will substantially change this. It remains to be seen, but I seriously doubt that a 7B-parameter LLM will ever have the performance of a 1-trillion parameter one. Hence, I think that the main bottleneck boils down to the consumer hardware getting better and cheaper and less so whether we can compress models further.

This is a bit awkward since this implies that a person, who wishes to run the next Pareto-frontier model with performance akin to Opus, will still need insane hardware requirements at the end of the day. What is the alternative? Is the future one where every family buys a $20k home cluster to run their designated lobotomized butler and treat the expense as nothing more than a monthly bill? Definitely not - the future local LLM stack won’t be reliable enough to warrant paying a fortune to use run a private coding agent, nor is the chat/CLI interface the correct one for most tasks. For example, I can see a future where robots are the correct interface for completing tasks but achieving parity with human performance will be a slow, fruitful grind. Will it be local though? I seriously doubt it too without a fundamental shift in architecture. The margin of error is surprisingly low, and the current stack consists of autoregressive vision and policy transformers, with diffusion, and reinforcement learning which I personally don’t find too compelling. Don’t get me wrong. The last couple of years have shown that scaling and good engineering can go a loooong way to get phenomenal coding models. It’s just that if that is the method of choice, expect for all general-purpose robots to forever relegate their “intelligence” to the cloud.

In the meantime, the best we can do is strive for better hardware utilization. Everything from local LLMs to general on-device computation should be done as efficiently as possible. Let’s not be so wasteful and obtain the largest, baddest GPU specs if running the program on an M1 Mac would suffice. In my old job, I was interested in moving some of our JAX inference code, which usually required one to install Python and as of recently Nvidia chips, to online pedagogical resources that anyone can use¹. Using Javascript on its own was clearly not enough, so WebGPU was needed.² I haven’t tried, but I expect Opus to be even better at writing GPU kernels than me. Yes, that means using LLMs to complete the loop and implement what would otherwise be labor-intensive optimizations…