Conv2d Layer Decomposition

Factorize Conv2d layers with SVD, spatial, Tucker or CP decomposition

Conv_Decomposer replaces a Conv2d by a stack of smaller convolutions. Four factorizations are available. This page shows what each one does to the shapes and to the parameter count of a pretrained ResNet-18 — it measures no latency and no accuracy.

import torch
from torchvision.models import resnet18, ResNet18_Weights
from fasterai.misc.all import Conv_Decomposer

1. One layer, four ways

layers= restricts the rewrite to the convolutions named in it, so the four calls below differ only by method.

model = resnet18(weights=ResNet18_Weights.IMAGENET1K_V1)
decomposer = Conv_Decomposer()

print('original:', model.layer1[0].conv1)

for method in ['svd', 'spatial', 'tucker', 'cp']:
    one = decomposer.decompose(model, 0.5, method=method, layers=['layer1.0.conv1'])
    print(f'\n{method}:')
    print(one.layer1[0].conv1)

original: Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)

svd:
Sequential(
  (0): Conv2d(64, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  (1): Conv2d(32, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
)

spatial:
Sequential(
  (0): Conv2d(64, 64, kernel_size=(3, 1), stride=(1, 1), padding=(1, 0), bias=False)
  (1): Conv2d(64, 64, kernel_size=(1, 3), stride=(1, 1), padding=(0, 1), groups=64, bias=False)
)

tucker:
Sequential(
  (0): Conv2d(64, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)
  (1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  (2): Conv2d(32, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
)

cp:
Sequential(
  (0): Conv2d(64, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)
  (1): Conv2d(32, 32, kernel_size=(3, 1), stride=(1, 1), padding=(1, 0), groups=32, bias=False)
  (2): Conv2d(32, 32, kernel_size=(1, 3), stride=(1, 1), padding=(0, 1), groups=32, bias=False)
  (3): Conv2d(32, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
)

The same Conv2d(64, 64, 3x3) comes back as two, two, three or four convolutions. svd and tucker cut the channel dimensions, spatial splits the 3x3 kernel into a 3x1 and a 1x3, and cp does both.

2. The whole model

Called without layers=, the decomposer rewrites every eligible convolution: groups == 1 and a kernel larger than 1×1, which leaves the 1×1 downsample convolutions of ResNet out.

base = sum(p.numel() for p in model.parameters())

print(f'{"method":<10}{"parameters":>12}{"ratio":>9}')
print(f'{"original":<10}{base:>12,}{"1.00x":>9}')

for method in ['svd', 'spatial', 'tucker', 'cp']:
    dec = decomposer.decompose(model, 0.5, method=method)
    p = sum(q.numel() for q in dec.parameters())
    print(f'{method:<10}{p:>12,}{base / p:>8.2f}x')

method      parameters    ratio
original    11,689,512    1.00x
svd          6,410,811    1.82x
spatial      4,373,352    2.67x
tucker       4,708,235    2.48x
cp           1,882,489    6.21x

One ratio, four parameter counts, from 1.82x to 6.21x. The number of layers a method produces says nothing about the ratio it reaches: svd and spatial both produce two layers and land at 6,410,811 and 4,373,352 parameters. What these ratios cost in accuracy or in latency is not measured here.

3. Let the energy choose the rank

energy_threshold replaces percent_removed: each layer keeps the smallest rank retaining that fraction of the squared singular values, so the ranks differ from layer to layer.

by_energy = decomposer.decompose(model, energy_threshold=0.9)

print(by_energy.layer1[0].conv1)
print(f'\n{sum(p.numel() for p in by_energy.parameters()):,} parameters')

Sequential(
  (0): Conv2d(64, 25, kernel_size=(1, 1), stride=(1, 1), bias=False)
  (1): Conv2d(25, 33, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  (2): Conv2d(33, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
)

6,719,925 parameters

The ranks now differ per mode, 25 input and 33 output channels for this layer instead of a fixed 32, and the whole model lands at 6,719,925 parameters — above the 4,708,235 of the fixed-0.5 Tucker run.

4. Selecting layers

layers= keeps only the named convolutions, exclude= skips them.

kept = decomposer.decompose(model, 0.5, method='tucker', exclude=['conv1'])

print('conv1, excluded:              ', kept.conv1)
print('layer2.0.downsample.0, 1x1:   ', kept.layer2[0].downsample[0])
print('\nlayer2.0.conv1, decomposed:')
print(kept.layer2[0].conv1)
print(f'\n{sum(p.numel() for p in kept.parameters()):,} parameters')

conv1, excluded:               Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
layer2.0.downsample.0, 1x1:    Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)

layer2.0.conv1, decomposed:
Sequential(
  (0): Conv2d(64, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)
  (1): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
  (2): Conv2d(64, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
)

4,714,024 parameters

conv1 is untouched because it is excluded. layer2.0.downsample.0 is untouched because it is a 1x1 convolution, which the decomposer skips.

Summary

Call	What it gives you
`Conv_Decomposer().decompose(model, 0.5)`	a copy of the model with every eligible `Conv2d` factorized, `method='tucker'` by default
`method='svd' \\| 'spatial' \\| 'tucker' \\| 'cp'`	2, 2, 3 or 4 convolutions in place of one, factorizing channels, kernel, or both
`energy_threshold=0.9`	one rank per layer instead of a fixed ratio
`layers=[...]`, `exclude=[...]`	restrict or skip layers by name

Eligible layers are the Conv2d with groups == 1 and a kernel larger than 1x1. Every figure above is a shape or a parameter count on an ImageNet-pretrained ResNet-18: no accuracy and no latency were measured on this page.

1. One layer, four ways

2. The whole model

3. Let the energy choose the rank

4. Selecting layers

Summary

See Also