← Biomimicry and Efficient Compute

Nov 8

NeVA-style foveation is fully-differentiable and uses Gaussian blurring to simulate the periphery. I want to first investigate how well foveation scales compared to a vanilla model

So I’m trying to reconile some observations/known phenomena

  • A vanilla encoder can perform the task well on sharp images and possibly better than foveation. Foavis found similar behavior, and the intuition is that foveating reduces the amount of information in the scene. Throwing information only reduces classification accuracy.
  • At the same time, the last two vanilla training runs (one on sharp and one with oracle foveation) shows that blurring near the focus of interest dramatically improves training. The number of steps needed to get near 100% accuracy between foveation and sharp images for the plain vision encoder is night and day. There is a trade-off here. Foveating reduces the amount of information, so this is great when the model only has to pay attention to the region of interest. However if the focus is not in the region of interest, you are almost at a disadvantage because the lack of information forces you to search the space. Moreover, there is a question of whether foveation caused overfitting to just ignore all blurred rings and focus on teh Landolt-C.
  • Humans use foveation remarkably well and this helps dramatically reduce compute on the periphery. Our foveation mechanism only blurs and reduces information content but not pixels per-se.
  • The current vision encoders are designed for classification of orientation, but the Landolt-C may not lie
  • How can we reconcile attention with the focus?
  • We have remarkably good ViTs that were trained on a ton of high-resolution imagery. Is it possible to extend these ViTs to add a periphery?
  • Blurry objects in the periphery help us detect far away. People can recognize what a blurry object is without seeing the fully resolution. How do we learn a relationship between blurry objects and their sharper counterparts?
  • Should we finetune SigLIP on rendered foveation masks? The only issue is how do we decide foveation paths? Can we use SigLIP’s attention heatmaps to influence how foveation moves?