Schedules

When should I prune my network ?

Schedule

 Schedule (sched_func:Callable, start_pct:float=0.0, end_pct:float=1.0,
           start_sparsity:float=0.0)

Base class to create schedules

	Type	Default	Details
sched_func	Callable		Function that computes sparsity at given training percentage
start_pct	float	0.0	Percentage of training to start pruning
end_pct	float	1.0	Percentage of training to end pruning (default: 1.0)
start_sparsity	float	0.0	Initial sparsity level

The Schedule class allows you to create any schedule function according to 4 parameters:
- the sched_func: the function according to which the sparsity will evolve (e.g. linear, cos, …)
- the start_pct: the percentage of training steps at which the sparsification process will start
- the end_pct: the percentage of training steps at which the sparsification process will end
- the start_sparsity: the percentage of sparsity at which the model starts

One-Shot

The easiest schedule is the one-shot pruning, i.e. prune the network once. This can be done by simply returning the desired sparsity value. The moment when you want to prune will be controlled by the start_epoch argument in the SparsifyCallback.

source

sched_oneshot

 sched_oneshot (start:float, end:float, pos:float)

One-shot pruning: jump directly to target sparsity

	Type	Details
start	float	Starting sparsity level
end	float	Target sparsity level
pos	float	Current position in schedule (0-1)
Returns	float

one_shot.plot(50)

Iterative

Instead of pruning the network to desired sparsity in one step, you can do it iteratively. In fasterai, you can change the amount of iterations

source

sched_iterative

 sched_iterative (start:float, end:float, pos:float, n_steps:int=3)

Perform iterative pruning in discrete steps

	Type	Default	Details
start	float		Starting sparsity level
end	float		Target sparsity level
pos	float		Current position in schedule (0-1)
n_steps	int	3	Number of pruning steps
Returns	float

iterative.plot(50)

To modify the default n_steps, you can use the partial function.

iterative = Schedule(partial(sched_iterative, n_steps=5), start_pct=0.2)

iterative.plot(50)

Automated Gradual Pruning

Some researchers have come up with more sophisticated schedules, such as the Automated Gradual Pruning.

source

sched_agp

 sched_agp (start:float, end:float, pos:float)

Automated gradual pruning schedule with cubic decay

	Type	Details
start	float	Starting sparsity level
end	float	Target sparsity level
pos	float	Current position in schedule (0-1)
Returns	float

agp.plot(50)

One-Cycle Pruning

source

sched_onecycle

 sched_onecycle (start:float, end:float, pos:float, α:float=14, β:float=6)

One-cycle schedule based on logistic function

	Type	Default	Details
start	float		Starting sparsity level
end	float		Target sparsity level
pos	float		Current position in schedule (0-1)
α	float	14	Steepness parameter
β	float	6	Offset parameter
Returns	float

one_cycle.plot(50)

On top of that, all of the schedules available in fastai by default are also available: - sched_cos - sched_linear

fig, ax = plt.subplots(1, 1, figsize=(8,5), dpi=100)
fig.patch.set_alpha(0.)
ax.patch.set_alpha(0.)

prune = np.linspace(0, 1, 1000)
sps = [cos([50], p) for p in prune]
plt.plot(prune, sps, c='#89d6c9', linewidth=2)
plt.xlabel('training iterations (Normalized)')
plt.ylabel('sparsity')

        
ax.spines['bottom'].set_color('#808080')
ax.spines['top'].set_color('#808080') 
ax.spines['right'].set_color('#808080')
ax.spines['left'].set_color('#808080')
ax.tick_params(axis='x', colors='#808080')
ax.tick_params(axis='y', colors='#808080')
ax.yaxis.label.set_color('#808080')
ax.xaxis.label.set_color('#808080')

fig, ax = plt.subplots(1, 1, figsize=(8,5), dpi=100)
fig.patch.set_alpha(0.)
ax.patch.set_alpha(0.)

prune = np.linspace(0, 1, 1000)
sps = [lin([50], p) for p in prune]
plt.plot(prune, sps, c='#89d6c9', linewidth=2)
plt.xlabel('training iterations (Normalized)')
plt.ylabel('sparsity')

        
ax.spines['bottom'].set_color('#808080')
ax.spines['top'].set_color('#808080') 
ax.spines['right'].set_color('#808080')
ax.spines['left'].set_color('#808080')
ax.tick_params(axis='x', colors='#808080')
ax.tick_params(axis='y', colors='#808080')
ax.yaxis.label.set_color('#808080')
ax.xaxis.label.set_color('#808080')

Dense-Sparse-Dense

You can also create even more interesting behaviours such as the DSD method, where you prune the model in the first place, then re-grow it to its initial amount of parameter.

source

sched_dsd

 sched_dsd (start:float, end:float, pos:float)

Dense-Sparse-Dense schedule: increase then decrease sparsity

	Type	Details
start	float	Starting sparsity level
end	float	Target sparsity level
pos	float	Current position in schedule (0-1)
Returns	float

dsd.plot(50)