← Biomimicry and Efficient Compute

Nov 15

Follow up from yesterday’s thought. It’s been challenging finding a way to reason about chunking spatially, so let’s instead do the simpler but inferior. Let’s treat each row of the image as a 1D sequence and use the router to create boundary cuts. This creates a brick pattern where individual rows are partitioned into segments. Each segment intuitively captures regions that are self-consistent and predictable.

On Miniworld, qualitative figure show that it does learn to chunk walls and not the floor tiles. https://claude.ai/artifact/LQ6grPVXgDR6QBrvwKWLPu.

Thoughts

  • Right now I am using Mamba with bidirection scans to reduce any causality bias, but it might make sense to just use self-attention here. At the image sizes we typically deal with (max 5k), self-attention is substantially faster and consumes a smaller memory footprint. Since it is also bidirection, we don’t have to deal with causality bias.
  • Cleary there is no information passing between rows (technically every 3rd row), and yet we can see there are a lot of adjacent segments that ought to be merged together.
  • How can we merge adjacent segments? What if we just self-attend all of the segments and decide how similar each segment is to each other. One can view this as a second H-Net chunker that combines vertically.