Check parameter reduction

Note: Tucker decomposition uses an iterative algorithm (HOOI), so even at percent_removed=0.0 there will be small reconstruction error. Fine-tuning after decomposition is recommended.

—description: Decompose Conv2d layers via Tucker decompositionoutput-file: conv_decomposer.htmltitle: Conv2d Layers Decomposerskip_showdoc: true—

Overview

The Conv_Decomposer class reduces model size and FLOPs by factorizing Conv2d layers into three smaller convolutions using Tucker decomposition. This is the Conv2d counterpart of FC_Decomposer (which uses SVD for Linear layers).

How it works: A Conv2d weight [C_out, C_in, H, W] is decomposed into: 1. Conv2d(C_in, R_in, 1) — pointwise input channel compression 2. Conv2d(R_in, R_out, (H, W)) — spatial convolution at reduced rank 3. Conv2d(R_out, C_out, 1) — pointwise output channel expansion

When to Use

Scenario Recommendation
Large 3x3 or larger convolutions Highly recommended — significant FLOP savings
1x1 pointwise convolutions Skipped automatically (already minimal)
Depthwise / grouped convolutions Skipped (Tucker assumes standard convolution)
First layer (C_in=3) Works but limited benefit
Post-training compression Fine-tune after decomposition for best accuracy

source

Conv_Decomposer


def Conv_Decomposer(
    
):

Decompose Conv2d layers to reduce parameters and FLOPs

source

Conv_Decomposer.decompose


def decompose(
    model:Module, # The model to decompose
    percent_removed:float=0.5, # Fraction of rank to remove [0, 1)
    method:str='tucker', # 'tucker', 'svd', 'spatial', or 'cp'
    energy_threshold:float | None=None, # Auto rank via energy retention (0-1)
    layers:list[str] | None=None, # Layer names to decompose (None = all eligible)
    exclude:list[str] | None=None, # Layer names to skip
    n_iter:int=10, # Max HOOI iterations (tucker only)
    tol:float=0.0001, # HOOI convergence tolerance (tucker only)
)->Module:

Decompose eligible Conv2d layers using the specified method.

Future Work

  • LayerNorm_Folder: Fold LayerNorm into adjacent Linear layers for transformer inference (analogous to BN_Folder)
  • NuclearNormCallback: Add nuclear norm regularization during training to pre-condition weights for better SVD/Tucker decomposition (Low-Rank Prehab, arxiv 2512.01980)
  • Latency-aware rank selection: Use fasterlatency to predict actual speedup at each rank, selecting ranks to hit a target latency budget rather than parameter budget (FLAR-SVD, CVPRW 2025)

See Also

  • FC Decomposer - SVD decomposition for Linear layers
  • BN Folding - Fold BatchNorm into preceding Conv/Linear layers
  • Pruner - Structured pruning that removes entire filters