Regularize Callback

Perform Group Regularization in fastai Callback system 
from fasterai.core.criteria import *
from fasterai.core.schedule import *
from fasterai.regularize.all import *
from fastai.vision.all import *

RegularizeCallback adds a penalty on groups of weights to the training loss, so that whole groups are pushed towards zero.

Get your data

path = untar_data(URLs.PETS)
files = get_image_files(path/"images")

def label_func(f): return f[0].isupper()

dls = ImageDataLoaders.from_name_func(path, files, label_func, item_tfms=Resize(64))

The task is binary (cat/dog). The two accuracies below are read against this split’s size and majority-class rate.

def accuracy_report(learn, z=1.96):
    "Validation accuracy as k/n, with its Wilson 95% interval"
    n = len(learn.dls.valid_ds); k = round(learn.validate()[1]*n); p, d = k/n, 1 + z**2/n
    c, h = (p + z**2/(2*n))/d, z*(p*(1-p)/n + z**2/(4*n**2))**0.5/d
    print(f'{k}/{n} = {p:.2%}, Wilson 95% [{c-h:.2%}, {c+h:.2%}]')
    return k

labels = [label_func(f.name) for f in dls.valid_ds.items]
print(f'validation set: n={len(labels)}, majority class {max(sum(labels), len(labels)-sum(labels))/len(labels):.2%}')
validation set: n=1478, majority class 68.06%

Train a model without regularization as a baseline

learn = vision_learner(dls, resnet18, metrics=accuracy)
learn.unfreeze()

learn.fit_one_cycle(5)
epoch train_loss valid_loss accuracy time
0 0.662255 0.531558 0.839648 00:04
1 0.366927 0.258699 0.893099 00:04
2 0.235529 0.319857 0.880920 00:04
3 0.128605 0.195322 0.924222 00:04
4 0.079388 0.182196 0.931664 00:04 
k_base = accuracy_report(learn)
baseline_model = learn.model
1377/1478 = 93.17%, Wilson 95% [91.76%, 94.34%]

Create the RegularizeCallback: criteria scores the weights, granularity says how they are grouped, weight scales the penalty and schedule ramps it over the training. With verbose=True, the callback prints the penalty weight it reached at each epoch.

reg_cb = RegularizeCallback(squared_final, 'filter', 1e-3, schedule=one_cycle, verbose=True)
learn = vision_learner(dls, resnet18, metrics=accuracy)
learn.unfreeze()
learn.fit_one_cycle(5, cbs=reg_cb)
epoch train_loss valid_loss accuracy time
0 0.800137 0.824260 0.847091 00:04
1 1.381801 2.572556 0.777402 00:05
2 3.521760 4.494071 0.893099 00:05
3 4.274886 4.434783 0.915426 00:05
4 4.260731 4.380584 0.922869 00:05 
Current regularization weight: 0.000039
Current regularization weight: 0.000401
Current regularization weight: 0.000917
Current regularization weight: 0.000995
Current regularization weight: 0.001000

The reported losses include the penalty term, which is why they sit well above the baseline’s. verbose=True prints the penalty weight the schedule reached at each epoch: 0.000039, then 0.000401, 0.000917, 0.000995 and finally the requested 0.001000.

k_reg = accuracy_report(learn)
1364/1478 = 92.29%, Wilson 95% [90.81%, 93.54%]

The penalty acts on the weights themselves. Here are the per-filter L2 norms of the last convolution, for the baseline and for the regularized model:

def filter_norms(model, name='0.7.1.conv2'):
    w = dict(model.named_modules())[name].weight.detach()
    return w.flatten(1).norm(dim=1)

for tag, m in [('baseline   ', baseline_model), ('regularized', learn.model)]:
    n_ = filter_norms(m)
    print(f'{tag}: min {n_.min():.4f}  median {n_.median():.4f}  max {n_.max():.4f}')
baseline   : min 0.8409  median 0.9306  max 1.1626
regularized: min 0.7467  median 0.8220  max 1.0142

In this single run, the minimum, median and maximum per-filter norm all moved down for the regularized model. Nothing has been removed from the network — the filters are smaller, not absent.

Both accuracies are printed above, each from a single run, and their Wilson 95% intervals overlap: this page shows what the callback does to the weights, not that it helps or hurts accuracy.

Summary

Argument What it gives you
criteria The score applied to the weights before grouping (squared_final, large_final, …)
granularity How the weights are grouped (weight, kernel, channel, filter, …)
weight The scale of the penalty added to the loss
schedule How that scale evolves over the training
verbose Prints the penalty weight reached at each epoch

See Also