Nov 13
First tests with hnets on images show promising patching, but there are still a couple of issues
Encoder performance
There is also an interesting trade-off with the encoder. For small feature tensors (e.g. 64x64), it is faster to use self-attention compared to Mamba-style scans. Since self-attention is more general, is it just better to use self-attention? To get a sense of the performance profile of self-attention and Mamba2D, I asked Opus to do a few benchmarks:
Hardware: RTX Pro 6000
Attention: Pytorch’s compiled FlashAttention.
Mamba2D: See https://arxiv.org/pdf/2412.16146. The authors published a public implementation of the CUDA kernel. Opus wrote a reimplementation, so I wouldn’t be surprised it ripped off the original. As far as I can tell, Opus’s implementation uses standard optimizations found in the original Mamba implementations.
Data: 512x512 images using patch sizes of 1,2,4,8, or 16.
Forward+Backward time per step (ms)
| Patch (grid) | Batch | Attention | Mamba2D, 1 dir | Mamba2D, 2 dirs | Mamba2D, 4 dirs |
|---|---|---|---|---|---|
| 1 (512²) | 1 | 1,002 | 62.8 (16×) | 103 | 140 (7.1×) |
| 1 (512²) | 2 | 2,037 | 128 (15.9×) | 171 | 327 (6.2×) |
| 2 (256²) | 1 | 64.7 | 14.5 (4.5×) | 22.9 | 31.6 (2.0×) |
| 2 (256²) | 2 | 131.9 | 28.3 (4.7×) | 37.4 | 71.3 (1.8×) |
| 4 (128²) | 1 | 5.4 | 5.8 | 6.5 | 7.5 |
| 4 (128²) | 2 | 8.5 | 6.6 | 8.8 | 16.0 |
| 8 (64²) | 1 | 3.5 | 4.4 | 4.8 | 4.9 |
| 8 (64²) | 2 | 3.8 | 4.3 | 4.7 | 6.1 |
| 16 (32²) | 1 | 3.8 | 3.9 | 2.9† | 4.1 |
| 16 (32²) | 2 | 3.6 | 4.2 | 4.7 | 4.7 |
Forward+backward memory held (MB)
| Patch (grid) | Batch | Attention | Mamba2D, 1 dir | Mamba2D, 4 dirs |
|---|---|---|---|---|
| 1 (512²) | 1 | 3,359 | 3,711 | 4,454 |
| 1 (512²) | 2 | 6,714 | 7,418 | 8,906 |
| 2 (256²) | 1 | 842 | 928 | 1,107 |
| 2 (256²) | 2 | 1,681 | 1,853 | 2,213 |
| 4 (128²) | 1 | 213 | 252 | 339 |
| 4 (128²) | 2 | 423 | 496 | 676 |
| 8 (64²) | 1 | 55 | 65 | 85 |
| 8 (64²) | 2 | 107 | 126 | 166 |
| 16 (32²) | 1 | 15 | 19 | 31 |
| 16 (32²) | 2 | 28 | 36 | 57 |
Forward time (ms)
| Patch (grid) | Batch | Attention | Mamba2D, 1 dir | Mamba2D, 2 dirs | Mamba2D, 4 dirs |
|---|---|---|---|---|---|
| 1 (512²) | 1 | 283 | 18.4 (15.4×) | 27.3 | 32.0 (8.9×) |
| 1 (512²) | 2 | 575 | 33.4 (17.2×) | 41.0 | 65.2 (8.8×) |
| 2 (256²) | 1 | 18.8 | 3.8 (5.0×) | 5.6 | 6.9 (2.7×) |
| 2 (256²) | 2 | 36.8 | 7.4 (5.0×) | 8.2 | 15.1 (2.4×) |
| 4 (128²) | 1 | 1.42 | 1.24 | 1.51 | 1.80 |
| 4 (128²) | 2 | 2.39 | 1.68 (1.4×) | 1.96 | 3.33 |
| 8 (64²) | 1 | 0.64 | 0.74 | 0.76 | 0.87 |
| 8 (64²) | 2 | 0.69 | 0.76 | 0.81 | 1.12 |
| 16 (32²) | 1 | 0.61 | 0.69 | 0.67 | 0.67 |
| 16 (32²) | 2 | 0.62 | 0.67 | 0.68 | 0.71 |
Forward peak memory (MB)
| Patch (grid) | Batch | Attention | Mamba2D, 1 dir | Mamba2D, 4 dirs |
|---|---|---|---|---|
| 1 (512²) | 1 | 705 | 705 | 1,104 |
| 1 (512²) | 2 | 1,407 | 1,408 | 2,208 |
| 2 (256²) | 2 | 353 | 353 | 560 |
| 4 (128²) | 2 | 88 | 89 | 144 |
| 8 (64²) | 2 | 22 | 22 | 38 |
Forward+backward time (ms)
| Shape | Attention | Mamba-1 , 1 dir | Mamba-2 , 1 dir | Mamba-3 , 1 dir | Mamba-1 , 4 dirs | Mamba-2 , 4 dirs | Mamba-3 , 4 dirs |
|---|---|---|---|---|---|---|---|
| 64 × 16² | 3.13 | 3.42 | 3.62 | 6.39 | 6.46 | 7.41 | 13.7 |
| 4 × 64² | 4.62 | 6.00 | 5.54 | 8.46 | 7.77 | ||
| 1 × 128² | 7.07 | 6.70 | 5.50 | 6.85 | 9.75 | 8.45 | 13.6 |
| 1 × 256² | 75.9 | 17.5 | 13.0 | 20.6 | 39.9 | 27.3 | 58.4 |
| 1 × 512² | 1,136 | 71.7 | 57.0 | 89.8 | 170 | 128 | 254 |
I don’t have a good intuition for why Mamba uses more memory than flash attention in all cases. It’s likely the implementation though I will need time to understand what the heck it is doing.
Thoughts
- Video requires some sort of self-consistent patches between frames. Ideally two frames that don’t differ too much are chunked in a similar manner minus some diffs. We can try scoring D(chunker(I), chunker(I’)) and ensure this is small.
- How does chunking do with different types of tasks (e.g. supervised labels). Does the model learn to chunk just as good as self-supervised prediction?
Can we do something like this – mamba scan –> |- AAAAAAAAAAAAA self atten - | BBBBBBBBBBBBB |- CCCCCCCCCCCCC
to produce [p1,p2,p3]. But let’s simplify the picture futher. Suppose we have an grid of patches H*W big and ran parallel Mamba scans along the rows of patches. If we treat this similar to how H-Net works, we will get a sequence of probabilities $(p_ij)_j$ for each row i. This gives us a set of boundaries/non-boundaries $(b_ij)_j$$ per row. This naturally produces segments that are “contiguous” pieces of information. We can do a similar scan along the columns to get probabilities (q_ij)_i.
Note that this gives us the opportunity to make the same fix on Lazy H-Nets as specified in the Cartesia blog.
Let’s say we are between two boundaries b_i1 and b_i8. Grab a random token within this segment - this will be an achor. You end up with a varying number of anchors per row. Some anchors will be close to each other and so we can view this as redundant, but the question is how can we reduce the number of anchors if they are redundant (e.g. two anchors one on top of the other).
The ideal thing is to do irregular self-attention from one anchor on one row to all of the anchors above and below it.
Thought: wait what if we just pass these tokens? What happens???