Biomimicry and Efficient Compute
Honing in on biologically-inspired inductive biases is important for improving data efficiency. I believe this will be especially important in robotics as it is running a frontier model on a GPU cluster runs contrary with the effectivness of most organisms’ brains. Here we explore two inductive biases
- Visual foveation as a downsampler
- Learned visual token chunking
Visual Foveation
Foveation is a trait found in humans and birds that allows the eyes to focus intently at a specific point. The region around the focus is sampled at a high resolution whereas the periphery, the region outside of the center, is undersampled and “blurry”. There are quite a few works that add foveation to vision models.
- FoVi
- NeVa
- FoaVis
The last work happens to summarize the state of the field as of October 2026 in Table D.1.
Learned Visual Token Chunking
Look at your monitor and ask does the solid-color background contribute anything to the understanding of the scene. The answer is probably very little (although context matters), and and it would be the dream to compress the background into one token. SOTA VL(A)s use vision models that uniformly expend compute on the whole image which goes against this intuition.
H-Nets are appealing because they train a model to learn how to chunk text rather than using a prebuilt tokenizer. H-Nets are hierarchical afterall so chunked bytes can be further chunked as higher level constructs. In their evals, they showed that it scales well for OOD programming languages.
The Cartesia blog highlighted a couple of image outputs, so they appeared to have got it working on 2D data though they didn’t release any details on the updated architecture.
My plan is to tinker around with image and possibly video data with H-Net as the inspiration.
A couple of points to think about:
Chunking in text is natural with boundaries but it isn’t as obvious with 2D data. I thought for a while on how to influence the model to learn semantic groups. Should the groupings be contiguous pixels like a segmentation mask? Should it be a set of class-labeled pixels? Should it be like a quadtree? And once we group the pixels into this patches, how then should we feed them into the base model? A RoPE extension isn’t obvious so this leads us to learned embeddings.
I really liked the lightweight SSM for chunking text. While I don’t think this is appropriate for images, I think it can work with video. If I stare at a scene, there are just some elements that don’t move as often so chunking these regions temporally ought to save on tokens. Again, how do we do this and how are the processed chunks fed into the base model?
Prior Attempts TODO: expand on