How to add a L2 regularization term in my loss function

Question

I’m going to compare the difference between with and without regularization, so I want to custom two loss functions.

My loss function with L2 norm:

enter image description here

###NET 
class CNN(nn.Module):
def __init__(self):
 super(CNN,self).__init__()
 self.layer1 = nn.Sequential(
 nn.Conv2d(3, 16, kernel_size = 5, padding=2),
 nn.ReLU(),
 nn.MaxPool2d(2))
 self.layer2 = nn.Sequential(
 nn.Conv2d(16, 32, kernel_size = 5, padding=2),
 nn.ReLU(),
 nn.MaxPool2d(2))
 self.layer3 = nn.Sequential(
 nn.Conv2d(32, 32, kernel_size = 5, padding=2),
 nn.ReLU(),
 nn.MaxPool2d(4))
 self.fc = nn.Linear(32*32*32,11)
def forward(self, x):
 out = self.layer1(x)
 out = self.layer2(out)
 out = self.layer3(out)
 out = out.view(out.size(0), -1)
 out = self.fc(out)
 return out
net = CNN()
###OPTIMIZER
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr = LR, momentum = MOMENTUM)

1.How can I add a L2 norm in my loss function?

2.If I want to write the loss function by myself (without using optim.SGD) and do the grad-decent by autograd, how can I do?

Thanks for your help!

• Kefeng91: You don't need to write two different loss functions if you want to try with and without regularization. You just need to write the one with regularization, and set the damping parameter alpha to zero when you want to try without regularization. Please edit and write the loss function with regularization so we can guide you.
• Weimin Chan: @Kefeng91 criterion = nn.CrossEntropyLoss() optimizer = optim.SGD(net.parameters(), lr = LR, weight_decay=0.01) -------------- I got an advice that I can add a parameter 'weight_decay'. If I set it to zero, it means the loss function without regularization, and if I set to to arbitrary value I'll get what I need. Right?
• Kefeng91: What I meant is that SO is not a "please write the code for me" forum. You should first try to write the forward function yourself and then come back to us with more details about what you tried.
• Weimin Chan: @Kefeng91 Sure, there's some misunderstanding.I wrote the net by myself but I don't know how to make a custom loss function. This is just a tiny question/part in my code.And I'm sorry for that I didn't notice that this is almost the main structure of all network.
• Kefeng91: Given the equation of the entropy, if you set alpha to zero, you will have no regularization. If you set alpha to anything else, you will have regularization.

Answer 1

You can explicitly compute the norm of the weights yourself, and add it to the loss.

reg = 0
for param in CNN.parameters():
 reg += 0.5 * (param ** 2).sum() # you can replace it with abs().sum() to get L1 regularization
loss = criterion(CNN(x), y) + reg_lambda * reg # make the regularization part of the loss
loss.backward() # continue as usuall

See this thread for more info.