Regularize Callback

Perform Group Regularization in fastai Callback system 
from fasterai.core.criteria import *
from fasterai.core.schedule import *
from fasterai.regularize.all import *
from fastai.vision.all import *

RegularizeCallback adds a penalty on the weights to the training loss. With norm=1 (the default) it is an elementwise L1, the same whatever the granularity; with norm=2 it sums the L2 norm of each group of the granularity (group lasso), so each group is penalized as a unit, within its layer.

In a network where each convolution is followed by BatchNorm, the BatchNorm rescales each filter’s output, so the loss does not depend on a filter’s scale and penalizing the convolution weights alone does not select filters; to prepare structured pruning, penalize the BatchNorm scale or use a group penalty that includes it.

Get your data

path = untar_data(URLs.PETS)
files = get_image_files(path/"images")

def label_func(f): return f[0].isupper()

dls = ImageDataLoaders.from_name_func(path, files, label_func, item_tfms=Resize(64))

The task is binary (cat/dog). The two accuracies below are read against this split’s size and majority-class rate.

def accuracy_report(learn, z=1.96):
    "Validation accuracy as k/n, with its Wilson 95% interval"
    n = len(learn.dls.valid_ds); k = round(learn.validate()[1]*n); p, d = k/n, 1 + z**2/n
    c, h = (p + z**2/(2*n))/d, z*(p*(1-p)/n + z**2/(4*n**2))**0.5/d
    print(f'{k}/{n} = {p:.2%}, Wilson 95% [{c-h:.2%}, {c+h:.2%}]')
    return k

labels = [label_func(f.name) for f in dls.valid_ds.items]
print(f'validation set: n={len(labels)}, majority class {max(sum(labels), len(labels)-sum(labels))/len(labels):.2%}')
validation set: n=1478, majority class 68.27%

Train a model without regularization as a baseline

learn = vision_learner(dls, resnet18, metrics=accuracy)
learn.unfreeze()

learn.fit_one_cycle(5)
epoch train_loss valid_loss accuracy time
0 0.663662 0.516258 0.846414 00:37
1 0.413637 0.417955 0.881597 00:33
2 0.254523 0.266877 0.899865 00:57
3 0.131538 0.197689 0.924222 00:30
4 0.072530 0.207578 0.924222 00:30 
k_base = accuracy_report(learn)
1366/1478 = 92.42%, Wilson 95% [90.96%, 93.66%]

Create the RegularizeCallback: criteria scores the weights, granularity says how they are grouped, weight scales the penalty, norm=2 penalizes each group as a unit and schedule ramps the weight over the training. With verbose=True, the callback prints the penalty weight it reached at each epoch.

reg_cb = RegularizeCallback(large_final, 'filter', 1e-3, norm=2, schedule=one_cycle, verbose=True)
learn = vision_learner(dls, resnet18, metrics=accuracy)
learn.unfreeze()
learn.fit_one_cycle(5, cbs=reg_cb)
epoch train_loss valid_loss accuracy time
0 0.777113 0.638687 0.826116 00:32
1 1.343649 2.177101 0.895129 00:33
2 3.455848 4.423439 0.916103 00:33
3 4.408343 4.669721 0.936401 00:33
4 4.542003 4.661158 0.941137 00:32 
Current regularization weight: 0.000039
Current regularization weight: 0.000401
Current regularization weight: 0.000917
Current regularization weight: 0.000995
Current regularization weight: 0.001000

The reported losses include the penalty term. verbose=True prints the penalty weight the schedule reached at each epoch, ending at the requested 1e-3.

k_reg = accuracy_report(learn)
1391/1478 = 94.11%, Wilson 95% [92.80%, 95.20%]

Both accuracies come from a single run each: this page shows how to use the callback, not that it helps or hurts accuracy.

Summary

Argument What it gives you
criteria The score applied to the weights before grouping (large_final, squared_final, …)
granularity How the weights are grouped (weight, kernel, channel, filter, …)
weight The scale of the penalty added to the loss
norm 1: elementwise L1; 2: each group penalized as a unit (group lasso)
schedule How that scale evolves over the training
verbose Prints the penalty weight reached at each epoch

See Also