<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Biomimicry and Efficient Compute on Ian Limarta</title><link>https://ian.limarta.org/worklog/biomimicry-and-adaptive-compute/</link><description>Recent content in Biomimicry and Efficient Compute on Ian Limarta</description><generator>Hugo</generator><language>en-us</language><lastBuildDate>Sun, 15 Nov 2026 15:35:55 -0600</lastBuildDate><atom:link href="https://ian.limarta.org/worklog/biomimicry-and-adaptive-compute/index.xml" rel="self" type="application/rss+xml"/><item><title>Nov 15</title><link>https://ian.limarta.org/worklog/biomimicry-and-adaptive-compute/nov_14/</link><pubDate>Sun, 15 Nov 2026 15:35:55 -0600</pubDate><guid>https://ian.limarta.org/worklog/biomimicry-and-adaptive-compute/nov_14/</guid><description>&lt;p>Follow up from yesterday&amp;rsquo;s thought. It&amp;rsquo;s been challenging finding a way to reason about chunking spatially, so let&amp;rsquo;s instead do the simpler but inferior. Let&amp;rsquo;s treat each row of the image as a 1D sequence and use the router to create
boundary cuts. This creates a brick pattern where individual rows are partitioned into segments. Each segment intuitively captures regions that are self-consistent and predictable.&lt;/p>
&lt;p>On Miniworld, qualitative figure show that it does learn to chunk walls and not the floor tiles. &lt;a href="https://claude.ai/artifact/LQ6grPVXgDR6QBrvwKWLPu">https://claude.ai/artifact/LQ6grPVXgDR6QBrvwKWLPu&lt;/a>.&lt;/p></description></item><item><title>Nov 13</title><link>https://ian.limarta.org/worklog/biomimicry-and-adaptive-compute/nov_13/</link><pubDate>Fri, 13 Nov 2026 12:12:28 -0600</pubDate><guid>https://ian.limarta.org/worklog/biomimicry-and-adaptive-compute/nov_13/</guid><description>&lt;p>First tests with hnets on images show promising patching, but there are still a couple of issues&lt;/p>
&lt;h2 id="encoder-performance">Encoder performance&lt;/h2>
&lt;p>There is also an interesting trade-off with the encoder. For small feature tensors (e.g. 64x64), it is faster to use self-attention compared to Mamba-style scans. Since self-attention
is more general, is it just better to use self-attention? To get a sense of the performance profile of self-attention and Mamba2D, I asked Opus to do a few benchmarks:&lt;/p></description></item><item><title>Nov 12</title><link>https://ian.limarta.org/worklog/biomimicry-and-adaptive-compute/nov_12/</link><pubDate>Thu, 12 Nov 2026 23:56:49 -0600</pubDate><guid>https://ian.limarta.org/worklog/biomimicry-and-adaptive-compute/nov_12/</guid><description>&lt;p>I&amp;rsquo;m putting foveation on the back-burner because I don&amp;rsquo;t think it is the right step yet for more efficient compute. The dynamic chunking
ideas from H-Net are more appealing. And once we work out how to chunk appropriately, I think we can bring foveation back. The chunker will benefit because the periphery can be more aggressively grouped together.&lt;/p>
&lt;h2 id="anchors-with-dit">Anchors with DiT&lt;/h2>
&lt;p>The architecture is&lt;/p>
&lt;ul>
&lt;li>Encoder: conv 3x3, self-attention, conv 3x3, self-attention. Outputs 256-dim &amp;lsquo;fine-grained&amp;rsquo; tokens.&lt;/li>
&lt;li>Anchors: select an ordering of the patches (e.g. row-major or Hilbert) and use a router to select anchor patches one at a time. To select an anchor patch,
inspect all previously discovered anchors, select the anchor that has the closest dot product in query-key space (64-dim), and include the anchor only if it exceeds a learned threshold &amp;rsquo;theta&amp;rsquo;. Keys and queries and&lt;/li>
&lt;li>Chunks: Each anchor produces a chunk by doing a weighted average of all the patches together. Positional information is encoded by pooling the learned position tokens.&lt;/li>
&lt;li>Base model: JiT. Bottleneck embedding 768-&amp;gt;128-&amp;gt;768. 12 blocks and 12 heads. Logit-normal time distribution.&lt;/li>
&lt;li>Upsampling: Same as H-Net&lt;/li>
&lt;li>No smoothing or EMA but the rest of the non-anchor patches are interpolated from the anchors.&lt;/li>
&lt;li>Apply a confidence weight similar to H-Net except it is optional.&lt;/li>
&lt;/ul>
&lt;p>Comments:&lt;/p></description></item><item><title>Nov 8</title><link>https://ian.limarta.org/worklog/biomimicry-and-adaptive-compute/nov_8/</link><pubDate>Sun, 08 Nov 2026 17:35:03 -0600</pubDate><guid>https://ian.limarta.org/worklog/biomimicry-and-adaptive-compute/nov_8/</guid><description>&lt;p>NeVA-style foveation is fully-differentiable and uses Gaussian blurring to simulate the periphery. I want to first investigate how well
foveation scales compared to a vanilla model&lt;/p>
&lt;p>So I&amp;rsquo;m trying to reconile some observations/known phenomena&lt;/p>
&lt;ul>
&lt;li>A vanilla encoder can perform the task well on sharp images and possibly better than foveation. Foavis found similar behavior, and the intuition is that foveating
reduces the amount of information in the scene. Throwing information only reduces classification accuracy.&lt;/li>
&lt;li>At the same time, the last two vanilla training runs (one on sharp and one with oracle foveation) shows that blurring near the focus of interest dramatically improves
training. The number of steps needed to get near 100% accuracy between foveation and sharp images for the plain vision encoder is night and day. There is a trade-off here.
Foveating reduces the amount of information, so this is great when the model only has to pay attention to the region of interest. However if the focus is not in the
region of interest, you are almost at a disadvantage because the lack of information forces you to search the space. Moreover, there is a question of whether foveation
caused overfitting to just ignore all blurred rings and focus on teh Landolt-C.&lt;/li>
&lt;li>Humans use foveation remarkably well and this helps dramatically reduce compute on the periphery. Our foveation mechanism only blurs and reduces information content but
not pixels per-se.&lt;/li>
&lt;li>The current vision encoders are designed for classification of orientation, but the Landolt-C may not lie&lt;/li>
&lt;li>How can we reconcile attention with the focus?&lt;/li>
&lt;li>We have remarkably good ViTs that were trained on a ton of high-resolution imagery. Is it possible to extend these ViTs to add a periphery?&lt;/li>
&lt;li>Blurry objects in the periphery help us detect far away. People can recognize what a blurry object is without seeing the fully resolution. How do we learn a relationship between
blurry objects and their sharper counterparts?&lt;/li>
&lt;li>Should we finetune SigLIP on rendered foveation masks? The only issue is how do we decide foveation paths? Can we use SigLIP&amp;rsquo;s attention heatmaps to influence how foveation
moves?&lt;/li>
&lt;/ul></description></item></channel></rss>