Nov 12
I’m putting foveation on the back-burner because I don’t think it is the right step yet for more efficient compute. The dynamic chunking ideas from H-Net are more appealing. And once we work out how to chunk appropriately, I think we can bring foveation back. The chunker will benefit because the periphery can be more aggressively grouped together.
Anchors with DiT
The architecture is
- Encoder: conv 3x3, self-attention, conv 3x3, self-attention. Outputs 256-dim ‘fine-grained’ tokens.
- Anchors: select an ordering of the patches (e.g. row-major or Hilbert) and use a router to select anchor patches one at a time. To select an anchor patch, inspect all previously discovered anchors, select the anchor that has the closest dot product in query-key space (64-dim), and include the anchor only if it exceeds a learned threshold ’theta’. Keys and queries and
- Chunks: Each anchor produces a chunk by doing a weighted average of all the patches together. Positional information is encoded by pooling the learned position tokens.
- Base model: JiT. Bottleneck embedding 768->128->768. 12 blocks and 12 heads. Logit-normal time distribution.
- Upsampling: Same as H-Net
- No smoothing or EMA but the rest of the non-anchor patches are interpolated from the anchors.
- Apply a confidence weight similar to H-Net except it is optional.
Comments:
- The ordering is unfortunate but it’s hard to think of a way to process patches without selecting an order. This probably biases the model
- The anchor sweep is quite expensive though surprisingly not as expensive as the base model. If there are many sweeps, I expect it will be hard for the model to learn.
- The model size ends up being ~20M parameters.
I ran the model on CIFAR-10 and Imagenet, but it seems that I practically will only be able to train on datasets with ~100k samples or less.
Hardware: RTX Pro 6000 Dataset: CIFAR-10, 32x32, 2x2 patches Batch size: 256 Steps: 120,000 Compression: 1/6
| Run | Size | loss | FID cfg 1.0 | FID cfg 1.5 | wandb id |
|---|---|---|---|---|---|
| JiT baseline | 21.6M | 0.134 | 32.08 | 23.96 | f05mkpfx |
| HNet with no confidence | 24.3M | 0.129 | 34.53 | 26.31 | ra9r82f5 |
| HNet with confidence | 24.3M | 0.133 | 34.66 | 26.52 | 376j92p9 |
RIP, but visually the results are basically the same.
Plain JiT uses 256 tokens sicne use a 2x2 grid size. In contrast, the H-Net was able to compress to 1/6 the number of tokens (~40). I am not so interested in JiT itself, but these next two tables are interesting:
Chunking by sampling step
| Heun step | 0 | 10 | 20 | 30 | 40 | 45 | 49 |
|---|---|---|---|---|---|---|---|
| No confidence gate | 40 | 40 | 35 | 39 | 63 | 89 | 89 |
| Confidence gate | 62 | 40 | 27 | 37 | 86 | 123 | 113 |
Chunking behavior by quartile during training
| Run | Steps | t < 0.25 | 0.25–0.5 | 0.5–0.75 | t > 0.75 |
|---|---|---|---|---|---|
| No confidence gate | 0–12k | 44.8 | 40.5 | 41.3 | 44.5 |
| No confidence gate | 30k–60k | 47.5 | 38.9 | 42.2 | 59.5 |
| No confidence gate | 60k–90k | 48.1 | 39.3 | 41.1 | 65.2 |
| No confidence gate | 99.6k–120k | 45.0 | 39.7 | 44.9 | 74.8 |
| Confidence gate | 0–12k | 35.3 | 41.7 | 60.5 | 85.1 |
| Confidence gate | 30k–60k | 46.6 | 38.9 | 45.7 | 58.0 |
| Confidence gate | 60k–90k | 54.1 | 33.7 | 41.8 | 64.3 |
| Confidence gate | 99.6k–120k | 55.6 | 31.8 | 43.5 | 93.0 |
There is sort of a parabolic behavior with how chunks are distributed. Early during the training run there roughly 40-50 chunks which indicates the noise forces the model to just chunk at random. In the middle phase, the model is able to chunk more aggressively because only low frequency features start to appear in the image. As we $t\rightarrow 1$, the high frequency details emerge which then forces the model to give up and chunk less efficiently.
Thought
- Rather than using JiT, can we try a different base?
- It’s sort of unfortunate that clean images end up being the ones with more chunks (i.e. the last low). The compression factor for t > 0.75 for both is usually than 3.