← Worklog

Honing in on biologically-inspired inductive biases is important for improving data efficiency. I believe this will be especially important in robotics as it is running a frontier model on a GPU cluster runs contrary with the effectivness of most organisms’ brains. Here we explore two inductive biases

Visual Foveation

Foveation is a trait found in humans and birds that allows the eyes to focus intently at a specific point. The region around the focus is sampled at a high resolution whereas the periphery, the region outside of the center, is undersampled and “blurry”. There are quite a few works that add foveation to vision models.

The last work happens to summarize the state of the field as of October 2026 in Table D.1.

Learned Visual Token Chunking

Look at your monitor and ask does the solid-color background contribute anything to the understanding of the scene. The answer is probably very little (although context matters), and and it would be the dream to compress the background into one token. SOTA VL(A)s use vision models that uniformly expend compute on the whole image which goes against this intuition.

H-Nets are appealing because they train a model to learn how to chunk text rather than using a prebuilt tokenizer. H-Nets are hierarchical afterall so chunked bytes can be further chunked as higher level constructs. In their evals, they showed that it scales well for OOD programming languages.

The Cartesia blog highlighted a couple of image outputs, so they appeared to have got it working on 2D data though they didn’t release any details on the updated architecture.

My plan is to tinker around with image and possibly video data with H-Net as the inspiration.

A couple of points to think about:

Prior Attempts TODO: expand on

Entries