Conv2d Layers Decomposer

Decompose Conv2d layers via Tucker decomposition

Overview

The Conv_Decomposer class reduces model size and FLOPs by factorizing Conv2d layers into three smaller convolutions using Tucker decomposition. This is the Conv2d counterpart of FC_Decomposer (which uses SVD for Linear layers).

How it works: A Conv2d weight [C_out, C_in, H, W] is decomposed into: 1. Conv2d(C_in, R_in, 1) — pointwise input channel compression 2. Conv2d(R_in, R_out, (H, W)) — spatial convolution at reduced rank 3. Conv2d(R_out, C_out, 1) — pointwise output channel expansion

When to Use

Scenario	Recommendation
Large 3x3 or larger convolutions	Highly recommended — significant FLOP savings
1x1 pointwise convolutions	Skipped automatically (already minimal)
Depthwise / grouped convolutions	Skipped (Tucker assumes standard convolution)
First layer (C_in=3)	Works but limited benefit
Post-training compression	Fine-tune after decomposition for best accuracy

source

Conv_Decomposer

def Conv_Decomposer():

Decompose Conv2d layers to reduce parameters and FLOPs

source

Conv_Decomposer.decompose

def decompose(
    model:torch.nn.modules.module.Module, percent_removed:float=0.5, # Fraction of rank to remove [0, 1)
    method:str='tucker', # 'tucker', 'svd', 'spatial', or 'cp'
    energy_threshold:float | None=None, # Auto rank via energy retention (0-1)
    layers:list[str] | None=None, # Layer names to decompose (None = all eligible)
    exclude:list[str] | None=None, # Layer names to skip
    n_iter:int=10, # Max HOOI iterations (tucker only)
    tol:float=0.0001, # HOOI convergence tolerance (tucker only)
)->torch.nn.modules.module.Module:

Decompose eligible Conv2d layers using the specified method.

Usage Example

from fasterai.misc.conv_decomposer import Conv_Decomposer
from torchvision.models import resnet18

model = resnet18(pretrained=True)
decomposer = Conv_Decomposer()
compressed = decomposer.decompose(model, percent_removed=0.5)

# Check parameter reduction
orig = sum(p.numel() for p in model.parameters())
comp = sum(p.numel() for p in compressed.parameters())
print(f"Compression: {orig/comp:.2f}x")

Note: Tucker decomposition uses an iterative algorithm (HOOI), so even at percent_removed=0.0 there will be small reconstruction error. Fine-tuning after decomposition is recommended.

Future Work

LayerNorm_Folder: Fold LayerNorm into adjacent Linear layers for transformer inference (analogous to BN_Folder)
NuclearNormCallback: Add nuclear norm regularization during training to pre-condition weights for better SVD/Tucker decomposition (Low-Rank Prehab, arxiv 2512.01980)
Latency-aware rank selection: Use fasterlatency to predict actual speedup at each rank, selecting ranks to hit a target latency budget rather than parameter budget (FLAR-SVD, CVPRW 2025)