MR_Lab_6_Pytorch_tutorial_Part3_Neural networks Background
ipynb283 kBPython · 54 cellsIn Lab 6: Neural networks background
NN Background
Neural networks (NNs) are a collection of nested functions that are executed on some input data. These functions are defined by parameters (consisting of weights and biases), which in PyTorch are stored in tensors.
Training a NN happens in two steps:
Forward Propagation: In forward prop, the NN makes its best guess about the correct output. It runs the input data through each of its functions to make this guess.
Backward Propagation: In backprop, the NN adjusts its parameters proportionate to the error in its guess. It does this by traversing backwards from the output, collecting the derivatives of the error with respect to the parameters of the functions (gradients), and optimizing the parameters using gradient descent.
For a more detailed walkthrough of backprop, check out this [video from 3Blue1Brown] (https://www.youtube.com/watch?v=tIeHLnjs5U8).

Using PyTorch
Let's take a look at a single training step.
For this example, we load a pretrained resnet18 model from torchvision.
We create a random data tensor to represent a single image with 3 channels, and height & width of 64,
and its corresponding label initialized to some random values.
torchvision Key Components:
- Datasets: Provides standard computer vision loaders like ImageNet, CIFAR-10, and MNIST.
- Models: Includes definitions for classic and modern architectures (like ResNet, VGG, and MobileNet) with pre-trained weights.
- Transforms: Offers common preprocessing and data augmentation operations for images, videos, and bounding boxes.
import torch, torchvision
from torch import nnfrom torchvision.models import resnet18, ResNet18_Weights
Download the model and save the model into path to be used later without downloading
The code snippet model = resnet18(weights=ResNet18_Weights.DEFAULT) initializes a ResNet-18 model pre-loaded with the highest-performing weights available in the PyTorch library.
By using ResNet18_Weights.DEFAULT, PyTorch automatically downloads and applies weights trained on the ImageNet-1K dataset (1000 class).
This configures your network for transfer learning or fine-tuning (seen later), saving you massive amounts of computational time and data requirements compared to training from scratch.
model = resnet18(weights=ResNet18_Weights.DEFAULT)
print(model)
PATH = r'D:\Teaching\UL\AI\Lab\resnetModel\resnet18.pth'
#save model parameters in dictionary
torch.save(model.state_dict(), PATH) #state_dict is simply a Python dictionary object that maps each layer to its parameter tensor (such as weights and biases).ResNet(
(conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
(layer1): Sequential(
(0): BasicBlock(
(conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(1): BasicBlock(
(conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(layer2): Sequential(
(0): BasicBlock(
(conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(downsample): Sequential(
(0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
(1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(layer3): Sequential(
(0): BasicBlock(
(conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(downsample): Sequential(
(0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
(1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(layer4): Sequential(
(0): BasicBlock(
(conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(downsample): Sequential(
(0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
(fc): Linear(in_features=512, out_features=1000, bias=True)
)
Notice the ''out_features=1000''
Inspecting Model Weights or Tensors
As the .pt file contains a model's state dictionary (weights and biases), you can loop through the dictionary keys and inspect individual tensor shapesIf your .pt file contains a model's state dictionary (weights and biases), you can loop through the dictionary keys and inspect individual tensor shapes`.
import torch
state_dict = torch.load(PATH, map_location="cpu")
for key, value in state_dict.items():
print(f"{key}: {value.shape}")conv1.weight: torch.Size([64, 3, 7, 7]) bn1.weight: torch.Size([64]) bn1.bias: torch.Size([64]) bn1.running_mean: torch.Size([64]) bn1.running_var: torch.Size([64]) bn1.num_batches_tracked: torch.Size([]) layer1.0.conv1.weight: torch.Size([64, 64, 3, 3]) layer1.0.bn1.weight: torch.Size([64]) layer1.0.bn1.bias: torch.Size([64]) layer1.0.bn1.running_mean: torch.Size([64]) layer1.0.bn1.running_var: torch.Size([64]) layer1.0.bn1.num_batches_tracked: torch.Size([]) layer1.0.conv2.weight: torch.Size([64, 64, 3, 3]) layer1.0.bn2.weight: torch.Size([64]) layer1.0.bn2.bias: torch.Size([64]) layer1.0.bn2.running_mean: torch.Size([64]) layer1.0.bn2.running_var: torch.Size([64]) layer1.0.bn2.num_batches_tracked: torch.Size([]) layer1.1.conv1.weight: torch.Size([64, 64, 3, 3]) layer1.1.bn1.weight: torch.Size([64]) layer1.1.bn1.bias: torch.Size([64]) layer1.1.bn1.running_mean: torch.Size([64]) layer1.1.bn1.running_var: torch.Size([64]) layer1.1.bn1.num_batches_tracked: torch.Size([]) layer1.1.conv2.weight: torch.Size([64, 64, 3, 3]) layer1.1.bn2.weight: torch.Size([64]) layer1.1.bn2.bias: torch.Size([64]) layer1.1.bn2.running_mean: torch.Size([64]) layer1.1.bn2.running_var: torch.Size([64]) layer1.1.bn2.num_batches_tracked: torch.Size([]) layer2.0.conv1.weight: torch.Size([128, 64, 3, 3]) layer2.0.bn1.weight: torch.Size([128]) layer2.0.bn1.bias: torch.Size([128]) layer2.0.bn1.running_mean: torch.Size([128]) layer2.0.bn1.running_var: torch.Size([128]) layer2.0.bn1.num_batches_tracked: torch.Size([]) layer2.0.conv2.weight: torch.Size([128, 128, 3, 3]) layer2.0.bn2.weight: torch.Size([128]) layer2.0.bn2.bias: torch.Size([128]) layer2.0.bn2.running_mean: torch.Size([128]) layer2.0.bn2.running_var: torch.Size([128]) layer2.0.bn2.num_batches_tracked: torch.Size([]) layer2.0.downsample.0.weight: torch.Size([128, 64, 1, 1]) layer2.0.downsample.1.weight: torch.Size([128]) layer2.0.downsample.1.bias: torch.Size([128]) layer2.0.downsample.1.running_mean: torch.Size([128]) layer2.0.downsample.1.running_var: torch.Size([128]) layer2.0.downsample.1.num_batches_tracked: torch.Size([]) layer2.1.conv1.weight: torch.Size([128, 128, 3, 3]) layer2.1.bn1.weight: torch.Size([128]) layer2.1.bn1.bias: torch.Size([128]) layer2.1.bn1.running_mean: torch.Size([128]) layer2.1.bn1.running_var: torch.Size([128]) layer2.1.bn1.num_batches_tracked: torch.Size([]) layer2.1.conv2.weight: torch.Size([128, 128, 3, 3]) layer2.1.bn2.weight: torch.Size([128]) layer2.1.bn2.bias: torch.Size([128]) layer2.1.bn2.running_mean: torch.Size([128]) layer2.1.bn2.running_var: torch.Size([128]) layer2.1.bn2.num_batches_tracked: torch.Size([]) layer3.0.conv1.weight: torch.Size([256, 128, 3, 3]) layer3.0.bn1.weight: torch.Size([256]) layer3.0.bn1.bias: torch.Size([256]) layer3.0.bn1.running_mean: torch.Size([256]) layer3.0.bn1.running_var: torch.Size([256]) layer3.0.bn1.num_batches_tracked: torch.Size([]) layer3.0.conv2.weight: torch.Size([256, 256, 3, 3]) layer3.0.bn2.weight: torch.Size([256]) layer3.0.bn2.bias: torch.Size([256]) layer3.0.bn2.running_mean: torch.Size([256]) layer3.0.bn2.running_var: torch.Size([256]) layer3.0.bn2.num_batches_tracked: torch.Size([]) layer3.0.downsample.0.weight: torch.Size([256, 128, 1, 1]) layer3.0.downsample.1.weight: torch.Size([256]) layer3.0.downsample.1.bias: torch.Size([256]) layer3.0.downsample.1.running_mean: torch.Size([256]) layer3.0.downsample.1.running_var: torch.Size([256]) layer3.0.downsample.1.num_batches_tracked: torch.Size([]) layer3.1.conv1.weight: torch.Size([256, 256, 3, 3]) layer3.1.bn1.weight: torch.Size([256]) layer3.1.bn1.bias: torch.Size([256]) layer3.1.bn1.running_mean: torch.Size([256]) layer3.1.bn1.running_var: torch.Size([256]) layer3.1.bn1.num_batches_tracked: torch.Size([]) layer3.1.conv2.weight: torch.Size([256, 256, 3, 3]) layer3.1.bn2.weight: torch.Size([256]) layer3.1.bn2.bias: torch.Size([256]) layer3.1.bn2.running_mean: torch.Size([256]) layer3.1.bn2.running_var: torch.Size([256]) layer3.1.bn2.num_batches_tracked: torch.Size([]) layer4.0.conv1.weight: torch.Size([512, 256, 3, 3]) layer4.0.bn1.weight: torch.Size([512]) layer4.0.bn1.bias: torch.Size([512]) layer4.0.bn1.running_mean: torch.Size([512]) layer4.0.bn1.running_var: torch.Size([512]) layer4.0.bn1.num_batches_tracked: torch.Size([]) layer4.0.conv2.weight: torch.Size([512, 512, 3, 3]) layer4.0.bn2.weight: torch.Size([512]) layer4.0.bn2.bias: torch.Size([512]) layer4.0.bn2.running_mean: torch.Size([512]) layer4.0.bn2.running_var: torch.Size([512]) layer4.0.bn2.num_batches_tracked: torch.Size([]) layer4.0.downsample.0.weight: torch.Size([512, 256, 1, 1]) layer4.0.downsample.1.weight: torch.Size([512]) layer4.0.downsample.1.bias: torch.Size([512]) layer4.0.downsample.1.running_mean: torch.Size([512]) layer4.0.downsample.1.running_var: torch.Size([512]) layer4.0.downsample.1.num_batches_tracked: torch.Size([]) layer4.1.conv1.weight: torch.Size([512, 512, 3, 3]) layer4.1.bn1.weight: torch.Size([512]) layer4.1.bn1.bias: torch.Size([512]) layer4.1.bn1.running_mean: torch.Size([512]) layer4.1.bn1.running_var: torch.Size([512]) layer4.1.bn1.num_batches_tracked: torch.Size([]) layer4.1.conv2.weight: torch.Size([512, 512, 3, 3]) layer4.1.bn2.weight: torch.Size([512]) layer4.1.bn2.bias: torch.Size([512]) layer4.1.bn2.running_mean: torch.Size([512]) layer4.1.bn2.running_var: torch.Size([512]) layer4.1.bn2.num_batches_tracked: torch.Size([]) fc.weight: torch.Size([1000, 512]) fc.bias: torch.Size([1000])
Load the model from path
The code snippet model = resnet18(weights=None) initializes a ResNet-18 convolutional neural network architecture with randomly initialized weights. (https://docs.pytorch.org/vision/0.24/models/generated/torchvision.models.resnet18.html)
In modern versions of PyTorch Torchvision (v0.13+), passing weights=None explicitely tells the framework to build the model structure from scratch rather than downloading or loading any pre-trained feature weights (like those trained on ImageNet).
This setup is typically used when you want to train the entire network from scratch on a completely custom dataset.(https://docs.pytorch.org/vision/0.24/models/generated/torchvision.models.resnet18.html)
model = resnet18(weights=None) #initializes a ResNet-18 convolutional neural network architecture with randomly initialized weights.
print(model)
# Access the first convolutional layer's weights
print("model.conv1.weight: \n:" , model.conv1.weight)
# Access the final fully connected layer's weights and biases
print("model.fc.weight: \n:" ,model.fc.weight)
print("model.fc.bias: \n:" , model.fc.bias)ResNet(
(conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
(layer1): Sequential(
(0): BasicBlock(
(conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(1): BasicBlock(
(conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(layer2): Sequential(
(0): BasicBlock(
(conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(downsample): Sequential(
(0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
(1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(layer3): Sequential(
(0): BasicBlock(
(conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(downsample): Sequential(
(0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
(1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(layer4): Sequential(
(0): BasicBlock(
(conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(downsample): Sequential(
(0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
(fc): Linear(in_features=512, out_features=1000, bias=True)
)
model.conv1.weight:
: Parameter containing:
tensor([[[[-1.3086e-02, -3.1112e-02, -3.6389e-02, ..., -5.4175e-02,
5.4395e-02, -2.5005e-02],
[ 3.8673e-02, 1.3881e-02, -1.7389e-02, ..., -6.0029e-02,
1.8984e-02, -2.6445e-02],
[-1.4626e-02, 5.0921e-03, 1.9642e-02, ..., 1.2843e-02,
2.4105e-02, -3.7248e-02],
...,
[-9.4977e-03, -1.4338e-02, -5.4294e-02, ..., -3.8254e-02,
4.4293e-02, -2.0731e-03],
[-1.5389e-02, 3.2904e-02, 1.2909e-02, ..., 3.6685e-02,
1.4250e-02, 2.9495e-02],
[-4.9284e-02, 2.9430e-02, 7.5745e-03, ..., -3.1274e-04,
4.1180e-02, 5.2326e-02]],
[[ 1.7679e-02, -2.7999e-03, 1.7758e-02, ..., -9.4958e-04,
7.4765e-03, -7.6752e-03],
[-2.3257e-03, 3.3647e-04, 1.6102e-02, ..., -1.8317e-03,
4.0178e-02, 1.5281e-02],
[-1.2810e-02, 3.8825e-03, 3.6682e-02, ..., 1.3285e-02,
-4.0815e-02, 2.3961e-02],
...,
[-4.2712e-03, 1.8843e-02, 4.9674e-02, ..., 2.8423e-02,
2.9430e-02, -1.7722e-02],
[-3.1959e-02, 2.4426e-02, 3.1414e-04, ..., -2.3043e-02,
4.6490e-04, 1.2840e-02],
[-3.6689e-02, 2.9443e-02, -1.7633e-03, ..., 3.0678e-02,
1.9297e-02, 1.1338e-04]],
[[-1.2344e-02, -1.7498e-02, -1.3338e-02, ..., -7.0555e-03,
2.2704e-02, 1.4261e-02],
[ 9.0282e-03, 9.2836e-04, -3.0686e-02, ..., -3.8379e-02,
-1.9619e-04, -2.3578e-02],
[ 3.4334e-02, 1.6076e-02, -1.5786e-04, ..., 7.9590e-03,
1.8516e-02, 2.9628e-03],
...,
[ 2.0154e-02, 3.6974e-02, 8.3311e-03, ..., 2.4418e-03,
-4.4408e-02, -1.0242e-03],
[-1.5331e-02, -1.9499e-02, 5.9082e-03, ..., 8.8046e-03,
-6.9272e-03, 3.1608e-02],
[-2.2746e-02, 6.4436e-03, 1.7021e-02, ..., -1.1822e-02,
9.9754e-03, -3.1941e-02]]],
[[[ 3.3431e-02, 1.4508e-02, 1.5699e-02, ..., -1.7253e-02,
-3.1298e-02, -2.6068e-02],
[ 1.5179e-02, 1.6135e-02, -2.2674e-03, ..., -3.0985e-02,
-1.0993e-02, -1.8818e-02],
[-1.7652e-02, -3.3454e-02, -3.7808e-03, ..., 3.9122e-02,
2.2577e-02, -2.4700e-02],
...,
[-3.5682e-03, -2.6182e-02, -1.4607e-02, ..., -3.7405e-02,
-5.4776e-03, -5.1396e-03],
[-3.7800e-02, 4.6840e-03, -1.6311e-02, ..., -4.7859e-02,
1.4744e-02, -6.6287e-04],
[ 4.1918e-03, -6.1411e-04, -4.3261e-02, ..., 9.7292e-03,
1.8748e-02, -1.0631e-02]],
[[ 5.1505e-03, 3.2858e-02, 4.2537e-02, ..., 9.7442e-04,
3.5499e-02, 2.9583e-02],
[ 2.0632e-02, 1.4485e-02, 8.5249e-03, ..., -3.0861e-02,
1.2778e-02, -1.5692e-02],
[ 3.9443e-03, -2.8744e-02, 1.3396e-03, ..., 2.2632e-02,
-9.9038e-03, -4.2819e-02],
...,
[ 3.9354e-02, 1.5974e-02, 3.5986e-03, ..., 2.7315e-03,
3.6516e-03, -4.7933e-02],
[ 1.4287e-03, -4.5966e-03, -6.9759e-03, ..., -3.0708e-02,
-4.2372e-02, 1.1567e-02],
[ 3.7950e-02, 1.1240e-03, 4.6108e-02, ..., -2.0777e-02,
-3.0915e-02, -1.8039e-02]],
[[ 1.5250e-02, -4.9164e-02, -3.4645e-02, ..., 3.6571e-02,
6.3953e-03, -5.7051e-03],
[ 2.0486e-02, -1.1681e-02, -2.4832e-02, ..., 9.5862e-03,
3.5506e-02, -5.1502e-03],
[-1.5379e-02, 1.3257e-02, -1.8376e-02, ..., -1.4773e-02,
6.6286e-03, -7.5662e-03],
...,
[-3.1635e-02, 1.1548e-02, 1.3147e-02, ..., -1.8167e-02,
-1.9000e-02, -8.1491e-03],
[ 1.5846e-03, 3.7215e-02, 1.2012e-02, ..., 1.7006e-02,
-2.8085e-02, -1.1328e-02],
[-3.1765e-02, 2.7048e-03, -3.3641e-02, ..., -4.3699e-02,
3.0042e-02, 5.0639e-03]]],
[[[-9.7802e-03, -8.7122e-03, -3.6396e-02, ..., 4.1968e-03,
-1.1881e-03, 6.7793e-03],
[-2.7434e-03, 4.7240e-02, 1.4657e-03, ..., -3.6633e-03,
3.1844e-02, -2.8830e-02],
[ 8.4211e-03, -2.2644e-02, -9.1989e-03, ..., -1.2080e-02,
1.6848e-02, -2.2363e-02],
...,
[ 3.3904e-03, -6.5938e-03, -4.7103e-02, ..., 1.3903e-02,
-1.0441e-02, 2.0235e-02],
[ 2.6037e-02, -1.0579e-03, -3.4679e-02, ..., 4.1253e-02,
2.9493e-02, -1.6104e-02],
[ 2.5633e-03, -1.6440e-02, 6.2533e-03, ..., -1.7470e-02,
1.3753e-02, -2.4870e-03]],
[[-1.8049e-02, -6.5977e-02, -3.8066e-02, ..., 4.1479e-03,
8.4795e-03, -9.5176e-03],
[-2.0926e-02, -3.8062e-02, -1.9736e-02, ..., 1.3850e-02,
1.8331e-02, -2.0075e-02],
[-2.3712e-02, -3.5960e-02, -2.6790e-03, ..., 1.8628e-02,
-1.2573e-02, -5.1130e-04],
...,
[-1.9927e-02, 1.9317e-02, 2.9754e-02, ..., -3.1934e-02,
-3.7766e-03, 2.8451e-02],
[ 1.3600e-03, -4.1692e-02, 4.6905e-02, ..., 2.9793e-02,
-2.0200e-03, -4.8869e-02],
[-4.2642e-03, 4.6504e-02, 3.7192e-02, ..., -1.3980e-03,
6.8348e-03, -1.5888e-02]],
[[ 3.3332e-03, 2.7567e-02, -4.7204e-02, ..., -2.6919e-02,
8.7295e-03, 1.6939e-02],
[-1.3619e-02, 1.0928e-02, -1.1339e-02, ..., 7.6049e-03,
2.5139e-02, -9.0158e-03],
[-4.7401e-03, 7.3096e-03, -2.7770e-02, ..., 1.8326e-02,
-1.5990e-02, 2.6291e-02],
...,
[ 4.5445e-02, -9.4544e-03, 2.4782e-02, ..., -2.9071e-02,
-3.2365e-02, 8.8268e-03],
[ 4.0765e-02, 2.2570e-02, -1.9113e-02, ..., -1.8645e-02,
8.1035e-03, -1.0949e-02],
[-2.1200e-02, 1.7700e-02, -4.4319e-02, ..., 5.2274e-03,
2.6197e-02, 1.6100e-02]]],
...,
[[[ 3.3714e-02, 5.5556e-03, 2.0587e-02, ..., 2.5200e-02,
3.8924e-03, -4.0617e-02],
[ 4.1854e-02, 2.8174e-02, -6.2013e-03, ..., 2.5031e-03,
-3.4022e-03, -3.4871e-02],
[-1.6821e-02, -3.6620e-02, -2.3534e-02, ..., -1.9622e-02,
8.2826e-03, 3.7131e-03],
...,
[ 2.0634e-02, 2.2744e-02, 4.5861e-03, ..., 2.2088e-02,
2.2627e-03, -1.6724e-02],
[-3.1468e-02, -3.2112e-02, -1.2265e-02, ..., -1.7597e-02,
-2.8272e-02, -1.5313e-02],
[ 1.3026e-02, 2.0893e-02, 3.9027e-02, ..., 3.4923e-02,
9.5520e-03, -1.8129e-02]],
[[ 8.8904e-03, 3.8515e-02, 4.4425e-02, ..., -1.3852e-02,
4.8040e-03, 5.1235e-02],
[ 3.3025e-03, 2.5527e-02, -9.4300e-03, ..., 1.4510e-02,
-3.0307e-02, -3.5022e-02],
[-3.4314e-02, 1.3796e-02, -9.5422e-03, ..., -6.8030e-03,
1.0676e-02, 3.8653e-02],
...,
[ 2.5927e-02, -6.6149e-05, 1.0275e-02, ..., 3.4913e-02,
1.4676e-02, 3.8507e-02],
[-2.4562e-02, -4.3279e-02, -1.0328e-02, ..., 4.7027e-03,
-3.7987e-03, 1.2375e-02],
[ 2.4334e-02, 1.5566e-02, -2.9046e-02, ..., -5.4959e-02,
2.5540e-02, 2.3308e-02]],
[[-2.9602e-02, 3.7904e-02, -1.9280e-03, ..., 9.4172e-03,
-8.8574e-04, 3.4184e-02],
[-1.7332e-02, 2.1015e-02, 1.2678e-02, ..., -3.1239e-02,
-2.9880e-02, 3.0622e-02],
[ 1.9555e-02, 1.6769e-02, -6.0038e-02, ..., -1.5786e-02,
-8.0458e-03, -2.1345e-02],
...,
[-1.4373e-02, 1.9178e-02, 1.5497e-03, ..., 5.3318e-03,
-7.8935e-02, -7.5253e-03],
[ 4.6438e-02, 3.9494e-02, -2.2136e-03, ..., 1.1705e-02,
5.7578e-03, -9.7146e-03],
[-1.8809e-02, 1.0231e-02, 6.0930e-02, ..., -4.3130e-02,
2.5284e-02, 1.3115e-02]]],
[[[ 1.8473e-02, 1.7480e-02, -1.1784e-02, ..., 3.5469e-02,
1.7634e-02, -2.5749e-02],
[ 1.8930e-02, 3.1237e-02, 1.7283e-02, ..., -1.6515e-03,
-1.8021e-02, -2.1672e-02],
[-3.6499e-02, -1.3839e-03, -7.3642e-03, ..., 5.1207e-02,
-3.3824e-02, -2.6073e-02],
...,
[-2.4507e-02, 1.0007e-02, 4.2950e-02, ..., -8.0031e-03,
-1.4661e-02, -2.9270e-02],
[-1.3482e-02, 2.3737e-03, -2.6394e-02, ..., -1.9343e-02,
2.8561e-02, 3.1980e-02],
[ 2.1348e-02, -1.4224e-02, -1.2017e-02, ..., 4.0921e-03,
1.1764e-02, -1.1095e-03]],
[[-1.3138e-02, 3.3096e-02, -9.2037e-03, ..., 4.1546e-02,
2.5493e-02, 2.6701e-02],
[ 6.4069e-03, -5.4193e-03, -1.0854e-02, ..., -3.1360e-02,
-9.0770e-03, -2.9249e-02],
[-2.7682e-02, -5.4734e-03, 2.0376e-02, ..., -7.6030e-02,
-1.8631e-02, 1.6634e-02],
...,
[ 8.2631e-03, -2.4483e-02, 9.4689e-04, ..., -1.9582e-02,
-1.1446e-02, 5.0442e-03],
[-5.7068e-02, -4.0914e-03, -6.1443e-03, ..., -2.9335e-02,
-1.2029e-02, 1.0121e-02],
[-1.5769e-02, -8.2811e-03, -6.3039e-03, ..., -1.3446e-02,
1.1988e-02, -3.2937e-02]],
[[-1.6036e-02, 1.4213e-02, 4.2961e-02, ..., -8.8144e-03,
1.7176e-02, 4.1237e-02],
[ 1.9985e-02, 3.7185e-02, 2.9738e-02, ..., 2.2794e-02,
2.8896e-02, -3.9143e-02],
[ 3.7034e-03, -4.1056e-02, -5.3449e-02, ..., -2.2375e-02,
4.3491e-03, 8.5502e-03],
...,
[ 1.6544e-02, -1.0767e-02, -9.2308e-03, ..., -1.0372e-02,
-1.7222e-03, -1.9121e-02],
[-2.9051e-02, 5.9398e-02, 2.5404e-02, ..., 2.5333e-02,
1.2202e-02, -1.2848e-02],
[-3.1142e-02, -8.7013e-03, -2.6656e-02, ..., 2.8492e-02,
-1.6816e-02, -1.6935e-02]]],
[[[ 2.3377e-02, 1.6590e-02, 2.1225e-02, ..., 1.9169e-02,
1.3304e-02, -2.0508e-02],
[-4.4765e-03, -1.4903e-02, -1.9340e-02, ..., 2.5715e-02,
1.3914e-02, 2.5560e-02],
[-1.0967e-02, 2.7451e-02, 2.2388e-02, ..., 5.3966e-03,
2.0108e-02, 1.7914e-02],
...,
[-4.5706e-02, -2.0524e-02, 1.0587e-02, ..., 2.0916e-02,
2.9460e-03, -2.7240e-02],
[-2.9380e-03, 6.5324e-03, -5.4407e-03, ..., -3.2431e-02,
-1.4979e-02, 5.0215e-03],
[-3.3599e-02, -1.2113e-02, -4.7314e-02, ..., 7.1167e-03,
3.9054e-03, -1.8187e-02]],
[[-7.1775e-02, -2.3840e-03, 5.4217e-03, ..., 3.5302e-02,
1.1647e-02, -7.6884e-03],
[-2.4945e-02, -9.5715e-03, 3.9911e-03, ..., 1.0853e-03,
3.1263e-02, 1.9822e-02],
[ 1.9756e-02, 1.1643e-02, 2.3243e-02, ..., -4.1315e-02,
1.8440e-03, 5.3246e-03],
...,
[-3.3249e-03, -3.0070e-02, 3.7844e-02, ..., 1.4901e-02,
-4.1918e-02, -9.8526e-03],
[ 3.2752e-02, 3.5931e-04, 1.8155e-02, ..., -1.9029e-02,
2.6716e-02, 5.6834e-03],
[-1.1175e-02, -1.2387e-02, 3.6924e-03, ..., -1.1679e-03,
-1.4800e-02, -2.4505e-02]],
[[ 4.7387e-02, 5.6009e-02, -4.1617e-02, ..., 1.9190e-03,
-7.0090e-03, 1.6482e-02],
[ 1.1483e-02, -2.3147e-03, -1.0276e-02, ..., -3.1490e-04,
1.7676e-02, 2.6198e-03],
[-1.6541e-02, -1.5416e-02, -2.7207e-03, ..., -1.1480e-02,
1.3632e-02, -1.2555e-02],
...,
[ 1.8812e-02, 1.4795e-02, -1.2419e-03, ..., 3.8935e-02,
-9.3657e-03, -3.0454e-02],
[-2.8176e-03, 3.4499e-02, -1.9260e-02, ..., 1.8121e-02,
2.6551e-02, 1.9333e-03],
[-5.5164e-03, 7.3086e-03, -2.4567e-02, ..., 2.6827e-02,
-3.1277e-03, 6.8055e-02]]]], requires_grad=True)
model.fc.weight:
: Parameter containing:
tensor([[-0.0317, 0.0102, 0.0418, ..., 0.0017, -0.0421, -0.0211],
[-0.0054, -0.0069, 0.0378, ..., 0.0120, 0.0188, 0.0349],
[ 0.0145, -0.0107, -0.0368, ..., 0.0419, -0.0330, 0.0325],
...,
[ 0.0222, 0.0330, 0.0063, ..., -0.0017, -0.0049, -0.0205],
[ 0.0370, -0.0050, -0.0362, ..., 0.0046, 0.0302, 0.0294],
[ 0.0261, -0.0161, -0.0196, ..., 0.0142, -0.0029, -0.0329]],
requires_grad=True)
model.fc.bias:
: Parameter containing:
tensor([-2.4117e-02, -3.7933e-02, 9.7963e-04, -1.0820e-02, 2.6935e-03,
-4.3112e-02, -4.0845e-02, -2.7943e-02, -1.4312e-02, -4.9923e-03,
-1.7133e-02, -1.3386e-02, 3.3876e-02, -7.5625e-04, -3.4157e-02,
3.4147e-02, 2.4332e-02, 2.9141e-02, 3.9899e-02, 4.3666e-02,
-2.5052e-02, 3.2035e-02, -3.5455e-03, -3.3262e-02, 3.8754e-02,
2.8115e-03, -3.1822e-02, -9.6254e-03, -2.0181e-02, 1.2686e-02,
3.3901e-02, 8.1084e-03, -2.8161e-02, 1.9934e-02, -2.0706e-02,
2.7123e-02, -3.6538e-02, -1.2426e-02, 3.0698e-02, 2.3470e-03,
1.3638e-02, 3.5717e-02, -3.8454e-02, 3.0099e-02, -1.5004e-02,
-3.9532e-02, 1.8980e-02, 4.3724e-02, 8.7017e-03, -1.7741e-02,
4.0132e-02, 2.5724e-02, -1.7622e-02, 1.9998e-02, -9.2812e-03,
3.6942e-02, -3.5272e-02, 3.5106e-02, 5.0608e-05, -1.5971e-03,
-2.0890e-02, -2.7758e-02, 3.4425e-02, 3.5607e-02, 2.6190e-02,
3.6996e-02, 3.5285e-02, 2.0461e-02, -3.0377e-03, 3.2752e-02,
2.5453e-02, 3.2056e-02, -1.5961e-02, -3.2892e-02, -2.1479e-03,
3.0227e-02, 2.4508e-02, 3.4686e-02, -3.2594e-02, -5.8296e-03,
4.2219e-02, 2.9144e-03, 3.6139e-02, 3.5438e-02, 1.2660e-02,
-3.8065e-02, -3.0929e-02, 3.0298e-02, -2.3649e-02, 3.0673e-02,
1.1556e-02, 2.8964e-02, -8.2362e-03, -2.6992e-02, -3.7070e-02,
-3.8203e-02, -3.6611e-02, 5.0848e-03, 3.9825e-02, -1.6918e-02,
3.6449e-02, 3.4152e-02, 3.6988e-02, -1.5581e-02, -3.4491e-02,
-2.4126e-03, 4.0758e-02, 2.5801e-02, -2.2247e-02, 4.6956e-04,
-4.2760e-02, 3.0353e-02, -2.0677e-02, -3.9425e-02, 4.2200e-02,
-3.9686e-02, -1.9109e-02, -3.6948e-02, 3.0631e-02, -4.2770e-02,
1.6430e-02, 9.1258e-03, 1.8666e-02, 2.1389e-02, 8.0537e-03,
9.3363e-03, 7.1609e-03, -1.4121e-02, -1.2703e-02, 1.9672e-02,
2.3520e-02, -6.8812e-03, -2.2805e-02, -2.5721e-02, -1.7154e-03,
2.8376e-02, -1.4995e-02, -3.9956e-02, -4.1373e-02, -9.0296e-04,
-4.0154e-02, 4.5745e-03, -7.6859e-03, -2.8322e-02, 2.1376e-03,
4.7361e-03, 4.1377e-02, -6.5688e-03, 3.1163e-02, -2.7303e-03,
-1.6573e-02, -4.3335e-02, -3.4337e-02, 3.0882e-02, 1.0500e-02,
-8.9397e-03, 9.2963e-03, 3.8369e-02, -3.8378e-02, -3.5385e-02,
6.1201e-03, 1.0476e-02, -6.6138e-03, 2.8645e-02, -3.1419e-03,
2.6717e-02, -9.1592e-03, 1.1920e-03, -2.0479e-02, -3.9804e-02,
-4.9046e-04, -1.7230e-02, -1.0690e-02, 1.8671e-02, 4.0673e-02,
-3.3376e-02, -2.8610e-02, 1.4348e-02, 1.1179e-02, -1.7704e-02,
4.9393e-03, 3.8353e-03, 8.8076e-03, -3.0779e-02, -4.0104e-02,
-1.6398e-02, 2.8576e-02, 2.5871e-02, 1.7129e-02, -8.8134e-03,
-1.8121e-02, -2.9167e-02, -1.4626e-02, -4.3377e-02, -1.5319e-02,
-3.5164e-03, -4.3643e-02, -3.1311e-02, 3.5664e-02, 2.9657e-02,
-1.8056e-02, 4.2979e-02, 2.0364e-02, 6.4372e-03, 2.0411e-02,
3.9069e-02, 2.6211e-02, 1.5878e-02, -2.0572e-02, -3.5655e-02,
3.7120e-02, 3.4514e-02, 1.9353e-02, -2.5075e-02, 1.0266e-02,
2.1584e-02, -5.9859e-03, 1.3424e-02, -3.0735e-02, -4.3302e-02,
-2.6999e-04, 1.2025e-02, -6.9452e-03, -2.6935e-02, -1.2855e-03,
-2.2445e-02, -2.0281e-02, 2.4479e-03, -3.3410e-02, -3.9525e-02,
3.5385e-02, 1.2765e-04, 2.6417e-02, -3.2118e-02, -1.7006e-02,
-4.2093e-02, -6.0259e-03, -7.0972e-03, 1.6043e-02, -9.3101e-03,
-1.5537e-02, -1.6476e-03, 3.9265e-03, -1.8971e-02, 1.4587e-02,
1.2138e-02, -3.6850e-02, 2.0978e-02, 1.5084e-02, -2.6001e-02,
-2.4886e-02, -9.7027e-03, -3.5997e-02, 2.9726e-03, 3.9254e-02,
8.5588e-03, 1.3460e-02, 1.5860e-02, 6.5122e-05, 2.4562e-02,
2.7875e-02, -1.0083e-02, -3.0848e-02, -3.7830e-02, 3.5487e-02,
2.7684e-02, 2.7864e-02, 8.2255e-03, 4.1666e-02, 4.9674e-03,
-2.1683e-02, -3.3825e-02, -2.8923e-02, -1.0127e-02, -8.8842e-03,
4.2261e-02, 4.3752e-02, 2.5294e-02, -3.6889e-02, -1.5303e-02,
1.1560e-02, -9.9936e-03, -1.1775e-02, 1.8346e-02, 2.7728e-02,
-1.0631e-02, 7.5223e-03, -3.0359e-02, -3.5956e-02, 3.7859e-02,
4.3767e-02, 2.5341e-02, 4.1024e-03, 4.3200e-02, -4.3022e-02,
6.4712e-03, -1.9079e-02, 3.9142e-02, 2.4755e-03, -4.4076e-03,
3.1958e-02, 3.5230e-02, -2.9868e-02, -1.5399e-02, 1.4958e-03,
7.7926e-03, -6.0195e-03, -3.7798e-02, 3.3347e-02, 1.7290e-02,
-3.0063e-02, 1.1058e-02, -2.9781e-02, 1.8921e-02, 2.4075e-02,
-3.6636e-02, 5.4383e-03, 3.3325e-03, -4.0438e-02, 5.6742e-03,
6.4007e-03, -1.2074e-02, 2.4905e-02, -2.5758e-02, -1.6563e-03,
3.0467e-02, -3.4637e-02, -2.2088e-03, -2.5514e-02, 1.4704e-02,
1.5550e-02, 1.6906e-02, 3.7830e-02, 4.0259e-02, 8.8870e-03,
3.5739e-02, -2.2575e-02, 4.0584e-02, -8.7720e-03, -7.5378e-03,
1.6779e-02, 1.4646e-02, -1.6356e-02, 1.3372e-02, 2.8916e-02,
1.9030e-02, -1.0482e-02, 4.7930e-03, -5.4075e-03, -3.2016e-02,
-2.9992e-02, -6.9195e-03, -3.0331e-02, 1.8318e-02, -3.0350e-02,
-2.0404e-03, 1.5233e-02, -3.5137e-02, 1.1771e-02, -3.3765e-02,
-3.1399e-02, 2.2244e-02, -1.6434e-02, -2.2659e-02, -1.0327e-02,
-4.4141e-02, -2.7299e-04, -4.1546e-02, 3.4935e-02, -2.5936e-02,
3.5849e-03, -3.5562e-02, -1.0521e-03, 3.2509e-03, 4.3745e-02,
-2.0231e-02, 3.1748e-02, 2.7664e-02, 3.3826e-02, -4.1992e-03,
9.3350e-03, -3.9497e-02, 1.9879e-02, -2.5770e-03, -3.0591e-02,
1.0491e-02, -4.2451e-02, -2.0855e-02, -1.1940e-02, -3.2995e-02,
1.8019e-02, 4.8992e-03, -7.4720e-03, 4.7281e-03, 3.4419e-02,
2.2742e-02, 1.0580e-02, 7.3134e-03, 2.0333e-02, 1.9470e-02,
2.3032e-02, 2.1075e-02, -5.7767e-03, 2.9818e-02, 5.1262e-03,
-8.6886e-03, 6.5642e-03, 1.7946e-02, -9.3283e-03, -8.9844e-03,
1.2400e-02, -4.0898e-02, 1.5983e-02, 9.4346e-03, 1.9510e-02,
-7.3141e-03, 2.4978e-02, 2.3099e-02, -2.6190e-02, 2.1889e-02,
4.4012e-02, -3.5401e-02, 3.5011e-02, 4.3788e-02, 1.6963e-02,
-2.8752e-02, -5.5295e-03, 1.0810e-02, -2.0480e-02, 1.8403e-02,
-2.8802e-02, 2.6105e-02, -3.7679e-02, -5.4581e-03, 2.1271e-02,
1.9543e-02, -8.8569e-03, -4.3007e-02, 1.6702e-02, 4.1331e-02,
8.8604e-03, -1.6685e-02, 4.3630e-02, 2.5360e-02, 1.5615e-03,
-2.1782e-02, -2.7161e-02, 2.3562e-02, 1.9055e-02, -1.9156e-02,
3.9857e-02, -1.5368e-02, 5.1632e-03, 7.9231e-03, -2.2148e-02,
-4.3949e-02, -1.6331e-02, 4.0527e-02, 1.3421e-02, -1.5041e-02,
9.1700e-03, 4.0154e-02, -4.0652e-02, 4.1407e-02, 1.8387e-02,
-1.4700e-02, 3.8101e-02, 2.5207e-02, -2.5500e-02, -3.8884e-02,
1.7042e-02, -2.7773e-02, -4.8815e-03, 3.6693e-02, 2.4061e-02,
4.0061e-02, 1.5907e-02, 2.6083e-02, -5.1275e-03, -3.8483e-02,
-3.3031e-02, -1.0499e-02, -2.9237e-02, 2.6979e-02, 3.0750e-02,
-2.6906e-02, 3.0771e-02, 1.9476e-02, 2.3910e-02, -1.7738e-02,
3.7423e-02, -1.7996e-02, -5.4705e-04, 2.0078e-02, 2.8025e-02,
3.8187e-02, -3.6417e-02, 8.6021e-03, -1.8486e-02, 2.9207e-02,
2.6874e-02, -3.4264e-02, -2.2227e-02, -3.4211e-02, -3.6697e-02,
-3.5154e-02, -2.0070e-02, 1.9241e-02, -2.5349e-02, 3.8762e-02,
1.0143e-02, 4.1868e-02, -4.3937e-02, 3.4043e-02, 1.4809e-02,
-3.7942e-02, -9.3666e-04, 2.6658e-02, -1.9103e-04, -3.5837e-02,
-1.0050e-03, -2.8763e-02, -1.3865e-02, -2.7477e-02, 3.6949e-02,
2.8948e-02, -4.2287e-02, -1.1895e-02, -1.3435e-02, 1.5027e-04,
6.6853e-03, -1.2845e-03, 1.9734e-02, 2.3675e-02, -4.0662e-02,
-7.3640e-03, 2.0968e-02, -2.5376e-02, -2.5132e-02, -2.3242e-02,
-7.5041e-03, 1.6724e-02, 1.3873e-02, 2.4314e-02, -1.1329e-02,
-3.7322e-02, 2.0102e-02, 3.5288e-02, 2.7534e-02, -3.3670e-02,
2.2089e-03, 1.9606e-02, -1.7628e-02, 2.8141e-02, 6.5642e-03,
2.1986e-02, -4.3573e-02, -3.7474e-02, 2.9580e-02, -1.8226e-02,
2.9937e-02, -1.6855e-02, 1.6723e-02, 4.0913e-02, 2.4750e-03,
-9.8366e-03, -3.7451e-02, -3.0095e-02, -4.9803e-03, -2.1592e-02,
-1.5901e-02, -2.6885e-02, -2.9261e-02, 3.0110e-02, 2.2930e-02,
4.0616e-02, -3.7937e-02, 1.2429e-02, 2.2823e-02, -3.9039e-02,
2.1306e-03, 2.4029e-02, -3.2754e-02, 4.2982e-02, -2.4327e-02,
4.3583e-02, -2.8492e-03, -2.0034e-02, 1.2236e-02, -2.4791e-02,
6.2349e-03, -3.9219e-02, -1.6985e-02, 1.7883e-02, 2.0381e-02,
-1.6358e-02, 1.2848e-02, 5.8549e-03, 1.3015e-02, 3.4301e-02,
4.3225e-02, -7.8592e-03, -1.6479e-02, 9.9257e-03, 1.3167e-02,
1.2724e-03, 4.2138e-02, -3.9785e-02, 3.9593e-02, -2.7750e-02,
-3.7402e-02, 3.2865e-02, 9.2096e-03, 1.4094e-02, 4.0992e-02,
-1.7410e-02, -3.6079e-02, 4.2521e-02, 7.5902e-03, -7.7598e-03,
3.0627e-03, -2.0672e-02, -6.3373e-03, 4.2614e-02, -2.2152e-02,
-2.5578e-02, 6.4597e-03, -4.4014e-02, 5.0339e-03, 7.5887e-03,
1.2718e-02, -1.7058e-02, -2.2254e-02, 2.5091e-02, -2.1790e-02,
-5.4421e-03, 2.2981e-02, 2.0453e-02, -2.4298e-03, -1.5837e-02,
-4.2211e-02, -3.0118e-02, 2.4715e-02, 2.4730e-02, -4.2527e-02,
-1.2538e-02, 3.2136e-03, -1.6710e-03, 2.3268e-02, 9.1679e-03,
3.1571e-02, -1.5815e-02, -1.1096e-02, 9.8921e-03, 7.9392e-04,
3.2971e-02, -6.6990e-03, 1.3557e-02, -3.5064e-02, 3.5403e-02,
4.2664e-02, -1.5800e-02, 4.3398e-02, 4.0640e-02, -3.3751e-02,
1.3580e-02, -3.8144e-02, -1.8090e-02, 9.3731e-04, 2.0840e-02,
-1.2009e-02, 3.4912e-02, -1.7679e-02, -3.9539e-02, 3.4680e-03,
3.9684e-02, -3.3325e-02, 2.6713e-03, -3.8206e-02, -3.9972e-02,
-3.1282e-03, -3.8188e-02, -2.8257e-02, 3.2987e-02, -3.8457e-02,
3.2170e-02, 2.4518e-02, 2.9780e-02, 1.2534e-02, -2.0685e-02,
-1.8907e-02, 2.2013e-02, -1.6981e-02, 4.1264e-02, -2.2175e-02,
1.0159e-02, -3.2666e-02, 5.2799e-03, -1.0847e-02, 2.8523e-02,
-3.9825e-02, -3.2843e-02, 3.6515e-02, -2.5685e-02, 2.2891e-02,
-2.4747e-02, 5.3063e-03, 4.3668e-02, -3.4650e-02, 3.6823e-02,
-2.8244e-02, 4.2452e-02, 1.3816e-02, -1.5793e-02, -7.1594e-04,
-4.2329e-02, 4.0547e-03, 3.7834e-02, 2.1341e-02, -1.3693e-02,
-3.0579e-02, 7.9177e-03, -3.5001e-02, 3.6562e-03, -2.6743e-02,
3.7220e-02, -2.7752e-02, -1.7973e-02, -4.1521e-03, 2.5362e-02,
-1.7720e-02, 2.5441e-02, -2.9623e-03, -1.1528e-03, 8.1567e-03,
-1.4192e-02, -8.7125e-03, 3.3447e-02, -3.4173e-02, -2.4116e-02,
1.4293e-03, -7.6724e-03, -1.9860e-02, -9.7157e-03, -3.2511e-02,
1.1183e-02, 4.2441e-02, -3.2266e-02, 9.8930e-04, 3.0115e-02,
4.1097e-02, -3.0117e-02, 1.6868e-03, 6.0134e-03, -5.1189e-03,
3.2756e-02, 9.7878e-03, -1.6064e-02, -1.9183e-02, -4.0395e-02,
-1.4749e-02, -3.4433e-02, 3.3717e-02, 3.5716e-02, -3.5766e-02,
3.5245e-03, 4.3115e-02, 3.1519e-02, -3.2467e-02, -1.0816e-02,
-6.2082e-03, -1.1707e-03, 4.0959e-02, -1.3239e-02, 5.0986e-03,
1.4094e-02, 8.3844e-03, 2.7789e-02, 4.2431e-02, 3.3608e-02,
2.8772e-02, 4.0176e-02, 6.3305e-03, -7.1043e-04, 1.3736e-02,
9.9628e-03, 2.4579e-02, -1.7120e-02, 2.8982e-02, 1.6079e-02,
-2.7343e-02, 5.3231e-03, 2.2070e-03, -1.0944e-02, -1.1517e-02,
4.6747e-03, 5.4891e-03, 2.0638e-02, 1.0116e-02, 3.7790e-03,
-4.4093e-02, 3.0005e-02, 2.8393e-02, -1.8299e-02, 8.7084e-03,
2.1353e-02, -3.4671e-02, 2.4020e-02, 3.2436e-03, 1.5599e-02,
-1.1943e-02, -2.5489e-02, -1.4920e-02, 4.7223e-03, -3.5522e-02,
-3.2982e-02, -8.6737e-03, 2.7788e-03, -4.1496e-02, 2.4575e-02,
-7.5421e-03, -1.3912e-02, -1.9188e-02, -2.0763e-02, 3.5815e-02,
7.2995e-04, -4.0451e-03, -2.8788e-03, 1.0944e-02, 2.5841e-02,
-1.1748e-02, 1.3788e-02, -1.9165e-02, 3.1892e-02, -1.9015e-02,
2.8376e-02, 1.5461e-03, -3.8323e-02, -1.4483e-02, -3.0304e-02,
1.4016e-03, 1.6440e-02, 3.6433e-02, 2.3168e-02, 6.6936e-03,
-2.2053e-02, -4.2852e-02, 1.9193e-02, 1.5811e-02, -2.1865e-02,
-4.8377e-03, -2.0948e-02, 8.0655e-03, 7.0774e-03, -2.2731e-02,
4.3879e-02, 2.0405e-02, -2.5863e-02, 1.0083e-02, 3.8987e-02,
9.2835e-03, -3.6676e-02, 6.4604e-03, 2.6685e-02, -1.7186e-02,
-2.5990e-02, 2.1052e-02, 4.2546e-02, -1.4339e-03, -4.2522e-02,
1.8104e-02, -1.9512e-02, 1.9702e-02, -1.6975e-02, 1.6440e-02,
-7.9617e-03, 1.3070e-02, 4.1806e-02, 3.2236e-02, 1.6571e-02,
9.1078e-03, 3.6698e-02, 1.0057e-02, -1.4100e-02, 2.4984e-02,
-1.5487e-02, 3.0544e-02, 1.0943e-02, -4.2081e-02, 2.1963e-02,
4.4177e-02, -1.6660e-02, -1.2521e-03, 2.9780e-02, -3.1117e-02,
2.4003e-03, -7.6231e-03, -4.3868e-02, 1.5720e-03, 2.2478e-02,
-8.9548e-03, 1.5770e-03, -4.2172e-02, -2.1309e-02, 3.5271e-02,
9.2696e-03, -3.6379e-02, -1.7342e-02, 5.1165e-03, -3.8103e-02,
-2.7791e-02, -3.8910e-02, -4.6293e-03, -1.0099e-03, -2.9652e-02,
4.0334e-02, 5.8412e-03, -3.1735e-02, 3.9167e-02, -4.3619e-02,
2.2345e-02, 2.3266e-02, -1.0621e-03, -3.5152e-02, 1.9975e-02,
-3.6231e-03, -1.9427e-02, -5.8426e-03, -1.5828e-02, -5.2194e-03,
3.4676e-02, -3.4097e-02, -1.8852e-02, -9.0134e-03, -7.2298e-03,
-2.3476e-02, -1.4194e-02, -8.1652e-03, 2.9315e-02, 1.0901e-02,
-2.9688e-02, -1.3316e-02, 9.5899e-03, -4.6760e-03, -8.2487e-03,
-3.4587e-02, -2.7070e-03, 2.8827e-02, -1.9739e-02, 2.8208e-02,
-9.0963e-03, 3.4196e-02, 3.3862e-02, 2.2745e-02, 1.0312e-02,
-3.7694e-02, -4.1191e-02, -2.2713e-02, -2.2979e-02, -3.8086e-02,
2.8223e-02, -7.8154e-03, 1.4612e-02, -7.1470e-03, 4.0933e-02,
-4.1522e-02, 3.5925e-02, 2.9106e-02, -3.5312e-02, -1.0882e-02,
-1.7263e-02, 1.3144e-03, 4.8318e-03, -3.4144e-02, 2.9715e-02,
1.2130e-02, 1.9185e-02, -3.6789e-02, 1.6646e-02, -2.9841e-02,
-1.4231e-02, 2.1365e-02, 1.3357e-02, -2.0553e-02, -8.5419e-03,
-2.4267e-02, 1.2274e-02, -4.3875e-02, -3.8952e-02, 5.2353e-03,
-1.4806e-04, -2.3930e-02, -5.3433e-03, -2.7817e-02, 1.5436e-02,
-2.6638e-02, -3.4437e-02, -2.3703e-02, 2.1001e-02, 2.1898e-02],
requires_grad=True)
PATH = r'D:\Teaching\UL\AI\Lab\resnetModel\resnet18.pth'
#LOAD model parameters in dictionary
model.load_state_dict(torch.load(PATH))
print(model)
# Access the first convolutional layer's weights
print("model.conv1.weight: \n:" , model.conv1.weight)
# Access the final fully connected layer's weights and biases
print("model.fc.weight: \n:" , model.fc.weight)
print("model.fc.bias: \n:" , model.fc.bias)ResNet(
(conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
(layer1): Sequential(
(0): BasicBlock(
(conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(1): BasicBlock(
(conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(layer2): Sequential(
(0): BasicBlock(
(conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(downsample): Sequential(
(0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
(1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(layer3): Sequential(
(0): BasicBlock(
(conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(downsample): Sequential(
(0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
(1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(layer4): Sequential(
(0): BasicBlock(
(conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(downsample): Sequential(
(0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
(fc): Linear(in_features=512, out_features=1000, bias=True)
)
model.conv1.weight:
: Parameter containing:
tensor([[[[-1.0419e-02, -6.1356e-03, -1.8098e-03, ..., 5.6615e-02,
1.7083e-02, -1.2694e-02],
[ 1.1083e-02, 9.5276e-03, -1.0993e-01, ..., -2.7124e-01,
-1.2907e-01, 3.7424e-03],
[-6.9434e-03, 5.9089e-02, 2.9548e-01, ..., 5.1972e-01,
2.5632e-01, 6.3573e-02],
...,
[-2.7535e-02, 1.6045e-02, 7.2595e-02, ..., -3.3285e-01,
-4.2058e-01, -2.5781e-01],
[ 3.0613e-02, 4.0960e-02, 6.2850e-02, ..., 4.1384e-01,
3.9359e-01, 1.6606e-01],
[-1.3736e-02, -3.6746e-03, -2.4084e-02, ..., -1.5070e-01,
-8.2230e-02, -5.7828e-03]],
[[-1.1397e-02, -2.6619e-02, -3.4641e-02, ..., 3.2521e-02,
6.6221e-04, -2.5743e-02],
[ 4.5687e-02, 3.3603e-02, -1.0453e-01, ..., -3.1253e-01,
-1.6051e-01, -1.2826e-03],
[-8.3730e-04, 9.8420e-02, 4.0210e-01, ..., 7.0789e-01,
3.6887e-01, 1.2455e-01],
...,
[-5.5926e-02, -5.2239e-03, 2.7081e-02, ..., -4.6178e-01,
-5.7080e-01, -3.6552e-01],
[ 3.2860e-02, 5.5574e-02, 9.9670e-02, ..., 5.4636e-01,
4.8276e-01, 1.9867e-01],
[ 5.3051e-03, 6.6938e-03, -1.7254e-02, ..., -1.4822e-01,
-7.7248e-02, 7.2183e-04]],
[[-2.0315e-03, -9.1617e-03, 2.1209e-02, ..., 8.9177e-02,
3.3655e-02, -2.0102e-02],
[ 1.5398e-02, -1.8648e-02, -1.2591e-01, ..., -2.5342e-01,
-1.2980e-01, -2.7975e-02],
[ 9.8454e-03, 4.9047e-02, 2.1699e-01, ..., 3.4872e-01,
1.0433e-01, 1.8413e-02],
...,
[-2.8356e-02, 1.8404e-02, 9.8647e-02, ..., -1.1740e-01,
-2.5760e-01, -1.5451e-01],
[ 2.0766e-02, -2.6286e-03, -3.7825e-02, ..., 2.4141e-01,
2.4345e-01, 1.1796e-01],
[ 7.4684e-04, 7.7677e-04, -1.0050e-02, ..., -1.4865e-01,
-1.1754e-01, -3.8350e-02]]],
[[[-4.4154e-03, -4.0645e-03, 3.1589e-03, ..., -3.7026e-02,
-2.5158e-02, -4.7945e-02],
[ 5.1310e-02, 5.3402e-02, 8.0436e-02, ..., 1.4480e-01,
1.4287e-01, 1.2312e-01],
[-7.3337e-03, 2.1755e-03, 3.7580e-02, ..., 6.1517e-02,
8.0324e-02, 1.1715e-01],
...,
[-2.6754e-02, -1.2297e-01, -1.3653e-01, ..., -1.4068e-01,
-1.1155e-01, -4.9556e-02],
[ 2.3524e-02, -1.7288e-02, -1.1122e-02, ..., -1.8826e-02,
-2.3320e-02, -2.9474e-02],
[ 2.8689e-02, 2.1659e-02, 4.7888e-02, ..., 2.5498e-02,
3.5346e-02, 1.1280e-02]],
[[ 4.6919e-04, 1.2153e-02, 4.2035e-02, ..., 4.6403e-02,
4.0423e-02, -1.4439e-02],
[ 4.3463e-02, 6.8779e-02, 1.3268e-01, ..., 2.8606e-01,
2.6905e-01, 2.0935e-01],
[-5.7621e-02, -2.2642e-02, 3.0547e-02, ..., 1.3763e-01,
1.6538e-01, 1.7946e-01],
...,
[-1.0816e-01, -2.5227e-01, -2.9742e-01, ..., -2.8503e-01,
-2.1493e-01, -1.0320e-01],
[ 4.0709e-02, -3.2771e-02, -6.3450e-02, ..., -9.2360e-02,
-6.9876e-02, -4.9841e-02],
[ 8.2942e-02, 8.7580e-02, 1.0111e-01, ..., 5.2714e-02,
6.0968e-02, 4.1198e-02]],
[[-1.6391e-02, -1.3870e-02, 5.2810e-03, ..., 4.3698e-02,
2.2707e-02, -4.5983e-02],
[ 3.3202e-02, 4.2014e-02, 9.3500e-02, ..., 2.6162e-01,
2.2970e-01, 1.6694e-01],
[-4.5987e-02, -1.6365e-02, 2.6811e-02, ..., 1.4951e-01,
1.3216e-01, 1.3579e-01],
...,
[-7.2129e-02, -1.8902e-01, -2.3389e-01, ..., -1.9038e-01,
-1.5609e-01, -7.5974e-02],
[ 5.1161e-02, -2.5815e-02, -6.9357e-02, ..., -5.8999e-02,
-6.1550e-02, -4.4555e-02],
[ 1.1174e-01, 7.8979e-02, 6.5849e-02, ..., 3.1617e-02,
2.5221e-02, 7.4257e-03]]],
[[[-7.0826e-08, -6.4306e-08, -7.3806e-08, ..., -9.8000e-08,
-1.0905e-07, -8.3421e-08],
[-6.1125e-09, 2.0613e-09, -8.0922e-09, ..., -4.9840e-08,
-4.3836e-08, -3.0538e-09],
[ 7.1953e-08, 7.5616e-08, 5.9282e-08, ..., -9.7509e-09,
-1.0951e-09, 4.2442e-08],
...,
[ 9.5889e-08, 1.0039e-07, 7.9817e-08, ..., -1.7491e-08,
-4.7666e-08, -1.3265e-08],
[ 1.2904e-07, 1.4762e-07, 1.7477e-07, ..., 1.3233e-07,
1.0628e-07, 9.3316e-08],
[ 1.2558e-07, 1.3644e-07, 1.8431e-07, ..., 2.1399e-07,
1.7710e-07, 1.7166e-07]],
[[-1.2690e-07, -9.6139e-08, -1.0372e-07, ..., -1.1808e-07,
-1.3309e-07, -1.0820e-07],
[-5.7412e-08, -2.5055e-08, -3.0115e-08, ..., -7.2922e-08,
-6.7022e-08, -2.2574e-08],
[ 2.1813e-08, 4.8608e-08, 3.1222e-08, ..., -1.8694e-08,
-7.9591e-09, 3.9750e-08],
...,
[ 5.6013e-08, 7.5526e-08, 4.4496e-08, ..., -4.4128e-08,
-5.9930e-08, -1.8247e-08],
[ 7.7614e-08, 9.8348e-08, 1.0455e-07, ..., 6.3272e-08,
4.1781e-08, 4.5901e-08],
[ 5.9834e-08, 7.1006e-08, 9.0437e-08, ..., 1.1654e-07,
8.7550e-08, 9.8837e-08]],
[[-4.3810e-08, 1.3270e-08, 7.8275e-09, ..., -5.8804e-09,
-2.6217e-08, -1.5649e-08],
[ 4.1700e-08, 1.0778e-07, 1.0946e-07, ..., 7.6403e-08,
7.1450e-08, 9.7615e-08],
[ 1.0436e-07, 1.6586e-07, 1.5933e-07, ..., 1.3517e-07,
1.3487e-07, 1.6449e-07],
...,
[ 9.8763e-08, 1.5072e-07, 1.2547e-07, ..., 6.8316e-08,
6.8382e-08, 1.1367e-07],
[ 9.1435e-08, 1.3576e-07, 1.3793e-07, ..., 1.1678e-07,
1.1723e-07, 1.4394e-07],
[ 6.2183e-08, 8.8184e-08, 1.0456e-07, ..., 1.3941e-07,
1.3333e-07, 1.5844e-07]]],
...,
[[[-6.1896e-02, -3.0206e-02, 1.9225e-02, ..., 4.3665e-02,
-2.2114e-02, -4.2214e-02],
[-3.8061e-02, 6.0774e-03, 4.5797e-02, ..., 9.6029e-02,
5.9254e-02, 2.9958e-02],
[-2.9672e-02, 2.7766e-03, 2.0457e-02, ..., 5.9828e-02,
4.1422e-02, 2.3134e-02],
...,
[ 1.1916e-02, 4.5701e-02, 4.4892e-02, ..., 4.7419e-02,
2.2274e-02, -5.4993e-03],
[-3.2468e-02, -1.2210e-02, 2.2023e-02, ..., 5.8061e-02,
-7.5033e-03, -5.9736e-02],
[-4.3314e-02, -2.8162e-02, -5.9126e-03, ..., 8.8460e-02,
8.4406e-03, -5.0019e-02]],
[[-6.1292e-02, -1.4004e-02, 1.7229e-02, ..., 1.8349e-02,
-3.2708e-02, -4.1060e-02],
[-3.1506e-02, 2.4460e-02, 4.5516e-02, ..., 6.6806e-02,
4.6687e-02, 3.3248e-02],
[-3.2216e-02, 2.0718e-02, 2.3343e-02, ..., 3.5265e-02,
3.6478e-02, 3.1291e-02],
...,
[ 1.7739e-02, 6.1040e-02, 4.8247e-02, ..., 3.7785e-02,
2.8894e-02, 1.3984e-02],
[-1.0890e-02, 2.2079e-02, 4.2737e-02, ..., 6.0247e-02,
1.6197e-02, -1.2493e-02],
[-2.2284e-02, 1.3220e-02, 3.0897e-02, ..., 1.0403e-01,
4.0119e-02, -5.3310e-03]],
[[-8.5322e-02, -4.2603e-02, 6.8145e-03, ..., 3.0751e-02,
-3.4818e-02, -4.9945e-02],
[-2.9215e-02, 1.8165e-02, 5.1092e-02, ..., 9.0200e-02,
5.3438e-02, 4.0169e-02],
[-3.9932e-02, -1.1100e-03, 9.6176e-03, ..., 2.4114e-02,
2.6298e-02, 2.5489e-02],
...,
[-3.1890e-03, 3.0454e-02, 1.6316e-02, ..., 5.5054e-03,
-6.2689e-03, -8.4638e-03],
[-2.2995e-02, -2.8211e-03, 2.3203e-02, ..., 3.5888e-02,
-1.4296e-02, -3.2419e-02],
[-9.8894e-03, 7.0542e-03, 1.0659e-02, ..., 7.0495e-02,
1.2996e-02, -8.3417e-03]]],
[[[-7.8699e-03, 1.9911e-02, 3.4208e-02, ..., 2.8694e-02,
1.2820e-02, 1.8142e-02],
[ 8.7942e-03, -3.2875e-02, -3.5713e-02, ..., 7.2533e-02,
4.5889e-02, 5.2383e-02],
[-3.6122e-02, -1.1878e-01, -1.3767e-01, ..., 3.3811e-02,
3.7806e-02, 2.6944e-02],
...,
[ 1.7322e-02, 3.9589e-03, -8.2269e-03, ..., 2.7543e-03,
1.8313e-02, 1.6057e-02],
[-9.5007e-04, 1.6428e-02, 1.7156e-02, ..., 3.3672e-03,
2.2857e-02, 6.5783e-04],
[ 6.1727e-03, 2.7145e-02, 1.4340e-02, ..., 7.5867e-03,
1.8770e-02, 1.5624e-02]],
[[-1.3423e-02, -5.0696e-04, 8.0959e-03, ..., -6.0963e-03,
9.2341e-03, 1.5751e-02],
[-1.8343e-02, -6.7982e-02, -7.0685e-02, ..., 2.9855e-02,
2.6264e-02, 2.3773e-02],
[-5.4359e-02, -1.4663e-01, -1.6211e-01, ..., 1.1781e-02,
3.2477e-02, 1.1980e-02],
...,
[ 8.3686e-04, -1.7564e-02, -1.9535e-02, ..., -4.1382e-03,
2.4658e-02, 1.2893e-02],
[-6.3183e-04, 1.1788e-02, 2.4810e-02, ..., 6.1105e-03,
3.9210e-02, 9.6696e-03],
[-7.1831e-03, 6.6918e-03, 5.2723e-03, ..., -7.6077e-03,
2.7253e-02, 1.7735e-02]],
[[-2.3753e-04, -4.9343e-03, 2.2991e-03, ..., -4.7958e-02,
-2.6154e-02, -2.3525e-02],
[-3.3053e-04, -5.1502e-02, -5.9977e-02, ..., -1.7369e-02,
-2.3337e-02, -3.7312e-02],
[-2.2674e-02, -9.9412e-02, -1.1176e-01, ..., -1.1725e-02,
-8.3744e-03, -4.0615e-02],
...,
[ 1.1437e-02, -8.0313e-03, -1.4955e-03, ..., -3.4133e-02,
-8.7267e-03, -2.3526e-02],
[ 2.9522e-03, 6.7770e-04, 1.9933e-02, ..., -2.2002e-02,
1.4814e-02, -1.4487e-02],
[-1.9085e-02, -2.9430e-02, -2.3284e-02, ..., -4.8587e-02,
-1.3049e-02, -2.4368e-02]]],
[[[-3.6296e-02, 7.1996e-03, 1.9100e-02, ..., 1.9602e-02,
1.4870e-02, -1.7298e-02],
[-1.1061e-02, 8.5665e-02, 1.2667e-01, ..., 1.3744e-02,
-5.5036e-05, -3.0162e-02],
[ 1.1322e-01, 1.8634e-01, 5.0658e-02, ..., -1.7333e-01,
-7.2041e-02, -6.2474e-02],
...,
[-5.3062e-02, -2.5781e-01, -2.6747e-01, ..., 2.6781e-01,
1.4344e-01, 5.5145e-02],
[-2.1009e-02, -2.9969e-02, 1.0245e-01, ..., 2.0843e-01,
-4.1518e-03, -3.8118e-02],
[-2.2155e-02, 1.2380e-02, 8.4302e-02, ..., -4.4992e-02,
-1.4687e-01, -9.0890e-02]],
[[-5.3969e-03, 3.2799e-02, 1.5486e-02, ..., -7.7451e-03,
3.0229e-03, 1.1216e-03],
[ 6.1723e-02, 1.4899e-01, 1.4645e-01, ..., -2.8897e-02,
-2.0227e-02, -9.1878e-03],
[ 1.6146e-01, 2.0886e-01, -2.5589e-02, ..., -2.7278e-01,
-1.0735e-01, -6.2971e-02],
...,
[-1.3723e-01, -4.0863e-01, -3.8551e-01, ..., 4.0846e-01,
2.6202e-01, 1.3491e-01],
[-5.9388e-02, -6.1187e-02, 1.4197e-01, ..., 3.5780e-01,
9.0893e-02, -1.7392e-03],
[ 7.8613e-03, 5.8403e-02, 1.5339e-01, ..., 4.7045e-02,
-1.0095e-01, -9.7920e-02]],
[[-5.6799e-03, 1.3425e-02, -2.6461e-02, ..., 4.4881e-03,
2.0666e-03, 1.3902e-02],
[ 6.5943e-03, 4.5181e-02, 6.0260e-02, ..., 1.4368e-02,
-5.0725e-03, 4.0505e-03],
[ 5.5257e-02, 1.2397e-01, 4.3193e-02, ..., -1.4486e-01,
-7.4489e-02, -5.7533e-02],
...,
[-3.1513e-02, -1.6334e-01, -1.5795e-01, ..., 2.2904e-01,
1.2017e-01, 7.1998e-02],
[-1.0456e-02, -1.1248e-03, 8.4582e-02, ..., 1.5748e-01,
2.2142e-02, -1.0083e-02],
[-4.8639e-03, -5.0065e-03, 3.6341e-02, ..., -2.4361e-02,
-7.1195e-02, -6.6788e-02]]]], requires_grad=True)
model.fc.weight:
: Parameter containing:
tensor([[-0.0185, -0.0705, -0.0518, ..., -0.0390, 0.1735, -0.0410],
[-0.0818, -0.0944, 0.0174, ..., 0.2028, -0.0248, 0.0372],
[-0.0332, -0.0566, -0.0242, ..., -0.0344, -0.0227, 0.0197],
...,
[-0.0103, 0.0033, -0.0359, ..., -0.0279, -0.0115, 0.0128],
[-0.0359, -0.0353, -0.0296, ..., -0.0330, -0.0110, -0.0513],
[ 0.0021, -0.0248, -0.0829, ..., 0.0417, -0.0500, 0.0663]],
requires_grad=True)
model.fc.bias:
: Parameter containing:
tensor([-2.6341e-03, 3.0005e-03, 6.5581e-04, -2.6909e-02, 6.3637e-03,
1.3260e-02, -1.1178e-02, 2.0639e-02, -3.6373e-03, -1.2325e-02,
-1.2629e-02, -7.2057e-03, -1.9321e-02, -2.4960e-02, -1.1885e-02,
-8.3259e-03, -9.5745e-03, -1.6658e-02, 9.1804e-03, -1.5354e-02,
7.1358e-03, 3.0737e-02, 1.3239e-02, -7.7528e-03, 4.7448e-03,
1.1175e-02, 1.5949e-02, -1.6712e-02, -1.0130e-03, -3.7167e-03,
6.5269e-03, -1.2041e-02, 9.0427e-03, -8.3279e-04, 8.8647e-03,
-2.6307e-02, -1.4588e-02, 2.9433e-03, 2.9718e-03, -1.9125e-02,
-4.7922e-03, 1.3828e-02, 9.8802e-03, -1.8417e-02, 1.9734e-02,
1.6941e-03, 1.2420e-02, -5.5842e-03, -1.0612e-02, 3.9847e-04,
4.2733e-03, -1.3298e-02, 2.0661e-02, 1.6963e-02, 2.7952e-03,
7.4214e-04, 1.3168e-02, 3.2213e-03, 1.0458e-02, 1.6511e-02,
9.1717e-04, 3.9388e-03, -5.6534e-03, 1.9372e-02, 7.5238e-03,
1.3437e-02, -1.3185e-02, -1.0026e-02, 7.1920e-03, -2.3166e-03,
-1.8895e-02, 1.2519e-02, 1.9583e-03, 7.3836e-03, -9.6664e-03,
2.0189e-02, 7.6652e-03, 1.8529e-02, 1.5710e-02, 1.8582e-02,
-6.9314e-03, 1.7090e-02, 9.1268e-03, -3.8876e-02, -2.4116e-02,
-6.8715e-03, -1.1648e-02, 7.8817e-03, 1.8046e-03, 2.8480e-02,
-1.9379e-02, -1.6295e-02, 1.0468e-02, -1.3027e-02, -8.4211e-03,
-2.9210e-02, -2.4856e-03, -8.7141e-03, -1.6397e-02, -9.3054e-03,
-1.5931e-02, -2.6346e-02, -1.1091e-03, 2.2589e-02, 2.1387e-03,
-2.3212e-02, -1.4085e-02, -5.6224e-03, -2.0090e-02, -3.0284e-02,
-4.9574e-02, 2.3283e-02, 1.4954e-02, -7.7501e-03, -3.9482e-03,
-3.7629e-02, -2.4220e-02, -1.0194e-02, -7.7038e-03, -4.1312e-03,
-2.9553e-03, -6.2174e-03, -1.2076e-02, -7.0168e-03, -3.8948e-03,
-1.6953e-02, -2.4585e-02, 5.5353e-03, -8.3370e-03, -7.0759e-03,
-2.4023e-02, -6.3686e-03, 7.3420e-04, 5.2883e-03, -2.2181e-02,
-2.6972e-02, -1.7990e-02, -1.6393e-02, 2.1485e-03, -1.6122e-02,
-1.6112e-02, 6.5931e-03, -2.0045e-02, 6.4149e-03, -1.2601e-02,
-7.6238e-03, 1.1411e-02, -4.5084e-02, -9.2018e-03, -1.5563e-02,
-1.3590e-02, -1.4374e-03, -1.9466e-02, 2.0737e-02, -1.0476e-02,
6.3229e-03, 8.3229e-03, -1.0791e-02, -1.8903e-02, 5.8624e-03,
-2.0189e-03, 3.2436e-02, 4.0581e-02, -4.0820e-05, 1.0886e-02,
-1.6544e-02, -5.3365e-04, -2.2903e-02, 4.6295e-03, -4.8402e-03,
1.0187e-02, 1.7954e-02, 4.8211e-03, 6.1831e-03, 1.4419e-02,
-1.2094e-02, -8.7460e-03, 1.9488e-03, 1.4685e-02, 1.2464e-02,
7.0523e-03, -4.1783e-03, 1.2048e-02, -2.0199e-02, 9.9144e-03,
1.3978e-02, -1.0321e-03, 5.7394e-03, 1.4019e-03, 6.0113e-04,
-5.5790e-04, 2.4424e-02, 2.3076e-02, -1.4610e-02, 1.1185e-02,
3.4608e-02, 1.6944e-02, 4.3295e-03, -2.5606e-02, 1.2279e-02,
-2.5810e-02, 8.5365e-03, 2.0437e-02, 2.2557e-02, 2.2966e-02,
8.8420e-03, -1.3894e-02, 3.8719e-03, -9.3046e-03, 2.3220e-02,
-1.4949e-02, 6.9258e-03, 5.0070e-03, -1.7302e-02, 1.0364e-03,
-1.0223e-02, -9.6949e-03, 3.4534e-02, 6.1337e-03, 1.1582e-02,
-2.0529e-02, -2.1956e-02, 5.3109e-03, 3.4101e-02, -5.8079e-03,
2.9406e-02, -8.7954e-03, -5.2505e-03, -1.9088e-02, 3.0350e-02,
1.8445e-02, -2.1225e-02, 1.8432e-02, 1.3832e-02, 1.7848e-02,
-4.4762e-03, 3.5858e-02, 2.1762e-02, 1.0880e-02, 4.0255e-02,
-2.0049e-03, -3.0348e-03, 9.3293e-03, -1.6304e-02, 9.6253e-04,
1.8673e-02, -1.6567e-02, 1.4964e-02, -3.7206e-03, -7.6734e-03,
-7.9254e-06, 3.9732e-03, -9.5979e-03, -1.6833e-02, 5.8524e-05,
-6.4126e-03, 8.2977e-03, 4.8207e-03, -1.1467e-03, 4.8869e-03,
1.7349e-02, 3.9222e-03, -7.8080e-03, 1.6051e-02, 9.8802e-03,
-1.0144e-02, 2.0912e-02, -6.3203e-03, -2.3139e-02, 1.1646e-03,
2.2468e-02, -6.6953e-03, 1.8311e-02, 1.4623e-02, -1.1654e-02,
-1.4306e-02, 1.2974e-02, -9.6865e-03, -6.2351e-03, 1.3180e-02,
6.7543e-03, 4.6418e-02, -2.7962e-02, -1.5111e-02, 2.8716e-02,
9.1991e-03, -5.3710e-03, -6.0361e-03, -7.2140e-03, -9.2421e-03,
1.8536e-03, -3.1078e-03, -8.4004e-03, -1.6766e-02, 4.0936e-03,
6.2426e-03, -1.2470e-03, -1.2919e-02, 3.5819e-03, 1.1006e-02,
-1.3282e-02, 2.6395e-03, 8.9953e-03, 6.5421e-03, -1.2031e-02,
1.7149e-02, 1.7949e-02, -1.0581e-02, -2.6962e-02, -1.3564e-02,
-9.7173e-03, -2.1176e-03, 3.5370e-02, 1.8392e-02, 2.6676e-02,
-1.0594e-03, -3.3949e-03, -4.8838e-03, 1.3427e-02, -1.3948e-02,
-1.9559e-02, -2.3295e-02, -3.7834e-02, -1.4637e-02, -2.1323e-02,
-3.0952e-02, -3.0822e-02, 1.9438e-03, 2.8637e-03, -2.1198e-02,
1.0448e-02, -1.1316e-02, -4.2609e-03, 2.2647e-02, -1.2867e-02,
-1.1018e-02, 1.2336e-02, -2.0057e-02, -2.1837e-02, -6.8067e-03,
-1.0488e-02, -2.6298e-02, -9.9579e-03, -1.2966e-02, -2.4832e-03,
-7.0940e-03, 1.7997e-02, -6.4257e-03, 7.9069e-03, -1.2287e-02,
9.8176e-03, 2.6674e-03, -2.1524e-02, 2.7511e-03, -9.2075e-03,
-1.7541e-02, -1.7103e-03, -1.4588e-02, 4.4247e-03, 3.4405e-02,
1.2725e-02, 3.0885e-02, 1.3090e-03, -7.0084e-05, -2.3165e-03,
-3.7989e-03, -1.1148e-02, 1.7210e-02, -6.7575e-03, -1.3694e-04,
-8.9166e-03, -1.6281e-02, -4.4920e-03, 1.1332e-02, -1.5909e-03,
-8.8193e-03, -4.9399e-03, 4.5732e-03, -1.0949e-02, 1.2890e-02,
-6.6586e-03, -2.5605e-03, 2.7965e-03, 1.1225e-02, -2.2055e-02,
-3.9271e-03, -6.6467e-03, -1.8840e-02, -2.1687e-02, -7.4066e-04,
-2.7281e-02, 5.0448e-03, -2.0709e-02, -3.4103e-02, -2.2374e-02,
-1.6656e-02, -2.7916e-02, -9.8977e-03, 5.5252e-03, 1.6013e-02,
-1.4895e-02, 3.5091e-03, 9.0003e-03, -8.3982e-03, -3.7479e-02,
2.0727e-02, -5.8799e-03, 9.1768e-03, -2.0297e-02, -7.3148e-03,
-1.6966e-03, -1.4029e-03, 3.2229e-03, 2.9212e-02, 1.2487e-02,
-2.0100e-02, 2.1170e-02, -2.5300e-02, 3.1815e-02, -1.0645e-03,
-1.0449e-02, -2.3419e-02, 1.4564e-02, 2.1245e-02, 1.6530e-02,
-3.2436e-03, -2.0437e-02, -3.6982e-02, -8.7213e-03, 5.4575e-03,
1.1048e-03, 2.2012e-03, 2.9512e-03, -5.9939e-05, 5.5785e-04,
-3.6906e-03, 5.3763e-03, -2.4765e-02, 9.2729e-03, 9.6081e-03,
9.1647e-03, 9.0880e-03, 7.4842e-03, -1.1946e-02, 2.1395e-02,
2.7922e-02, 1.4692e-02, -2.4958e-03, 2.8887e-02, 1.3422e-02,
1.7173e-03, 2.5018e-03, -2.1253e-02, -8.2424e-04, 4.2183e-03,
8.5981e-03, 1.8735e-02, 8.5622e-03, -8.8255e-03, 1.7462e-02,
-1.3693e-02, 2.1955e-03, 1.0772e-02, 2.8693e-03, 3.1032e-02,
8.5460e-03, -1.4198e-02, -2.2472e-03, 1.8740e-02, -1.2905e-02,
4.0370e-02, -7.7538e-04, 1.8671e-03, 7.2793e-03, -2.6508e-02,
-1.7609e-02, -2.4142e-02, 2.9577e-03, -1.5917e-02, 1.6273e-03,
1.1132e-02, 1.4574e-02, -8.1919e-03, -7.6581e-03, -1.8452e-02,
-9.0419e-03, 4.0883e-03, 4.4482e-02, -2.3664e-02, -5.2547e-03,
-1.9529e-02, 3.2860e-03, 5.4667e-03, -4.9558e-03, 7.6805e-03,
-3.3026e-03, -2.6248e-03, -1.1094e-02, 2.3922e-02, 1.8079e-02,
-1.8135e-02, 5.2204e-03, -1.3559e-02, 1.9448e-02, 1.0981e-02,
2.6869e-02, -6.6801e-03, -8.9389e-04, -3.4924e-03, -1.9667e-02,
-1.8511e-02, -7.6262e-04, -1.6382e-02, -1.5862e-02, -1.3717e-02,
1.7528e-02, -1.1419e-03, -6.3346e-03, -1.1118e-02, 1.3159e-02,
-2.3464e-02, 2.7993e-04, -3.6273e-04, 2.3797e-02, -2.7353e-03,
-2.2223e-02, 1.3415e-02, 1.0443e-02, -2.3512e-02, 1.6832e-02,
4.3699e-03, -1.3243e-02, -2.8605e-03, 5.4212e-03, 1.9924e-03,
-6.8664e-04, -3.9092e-04, 1.7806e-02, 1.8391e-02, 2.8473e-02,
-3.3835e-02, -1.0778e-02, -1.2371e-02, -1.9110e-03, -1.6381e-03,
1.7288e-02, -3.9813e-03, -1.5167e-02, -1.0781e-02, 5.3808e-03,
-3.3947e-04, 3.3885e-04, -1.0162e-02, -4.0266e-03, -3.4751e-03,
4.2359e-03, -1.4677e-03, 1.3207e-02, 7.5580e-03, 1.9397e-04,
3.0048e-03, 8.6283e-03, -1.1193e-02, 3.8466e-02, -2.6220e-02,
-2.0251e-02, -6.3872e-03, 2.1906e-02, -7.3400e-03, 5.2753e-03,
-1.1709e-02, 8.4009e-03, 2.8530e-03, -4.7220e-03, 2.3118e-02,
-7.6039e-03, 2.8136e-03, -1.1701e-02, -4.4118e-03, 1.1846e-02,
-1.7632e-03, -1.2260e-02, -2.1210e-03, 1.2072e-02, 6.7523e-03,
-1.9128e-04, -2.5105e-02, 1.2693e-02, 1.6062e-02, 8.1264e-03,
1.3857e-03, 3.0087e-03, -1.4111e-02, 1.9784e-02, -9.2301e-04,
-1.8428e-02, 7.8059e-03, 1.5319e-02, -1.2768e-02, -9.0166e-03,
1.8031e-02, 2.4853e-02, 1.7788e-02, 8.8640e-03, -9.4422e-03,
-1.3652e-03, 1.2932e-02, 9.0133e-03, 1.6655e-02, -5.4321e-03,
2.7480e-02, -3.1781e-02, -1.3331e-02, 5.5792e-03, -1.3278e-02,
-1.9219e-02, -1.3307e-02, 4.2390e-03, 3.0246e-02, -8.1990e-03,
8.3008e-03, 1.8993e-02, 1.0643e-02, 3.1324e-02, 1.9283e-02,
3.3642e-03, 1.9669e-02, 2.2673e-03, -1.9630e-02, 2.0147e-02,
-1.1433e-02, -7.6073e-03, 1.5071e-02, -3.0395e-03, -9.3430e-03,
-4.1657e-03, 2.2972e-03, -5.0985e-03, -1.4499e-02, -2.7673e-02,
-3.8721e-02, 5.4249e-03, 1.3504e-02, -1.2811e-03, 3.7465e-02,
1.5154e-03, 2.4035e-02, -2.0557e-02, 9.8406e-03, 1.0352e-02,
3.8597e-02, -1.1905e-02, -2.1718e-02, 8.3778e-03, 1.4691e-02,
2.2631e-02, -3.7629e-03, 1.5570e-02, -9.3990e-03, 5.3536e-03,
1.9584e-02, -1.1156e-02, 1.5190e-02, 5.4622e-03, 2.2995e-02,
2.9260e-02, -1.5236e-03, 6.6009e-03, -3.1939e-02, -1.0486e-02,
-4.4617e-03, 3.1853e-02, 1.3736e-02, 1.3561e-02, 7.0907e-03,
-1.6753e-02, -2.5470e-02, 1.9752e-02, 2.6715e-02, -4.6859e-03,
1.7682e-02, 3.2496e-02, 1.4553e-02, 2.6101e-02, 1.1341e-02,
-2.2271e-03, 3.5237e-02, -1.1892e-02, -1.8683e-02, -5.5245e-03,
-7.0732e-03, -5.2670e-03, 7.5946e-03, -1.8465e-02, -1.6897e-02,
1.0127e-02, 1.3006e-02, -1.8251e-03, 6.6651e-04, -1.1207e-02,
1.3563e-02, -1.8153e-02, -2.6487e-02, 6.0652e-03, 3.9711e-02,
-1.4285e-02, 1.8001e-02, -1.4039e-02, -1.8762e-02, -1.1778e-02,
-1.6449e-02, 9.0423e-03, -7.2730e-03, 1.7517e-02, -7.3016e-04,
1.0212e-02, 2.3785e-02, 8.1286e-03, 8.0260e-03, 1.1922e-02,
6.2416e-03, -2.4625e-02, 2.9461e-02, -1.4183e-02, -1.8672e-02,
-1.4057e-02, 7.9872e-03, -2.1081e-02, -2.7560e-02, -3.5690e-03,
1.5993e-03, 9.4720e-03, 1.8272e-02, -2.3742e-03, 1.1843e-03,
-5.7722e-04, 8.7818e-03, 2.7804e-03, 6.7973e-04, 1.5877e-02,
-7.0359e-03, 2.5487e-03, -1.7925e-02, 8.4912e-03, 4.3375e-03,
2.4508e-02, 3.6686e-03, 1.0252e-02, -1.3396e-02, 4.5706e-04,
1.0313e-02, 1.5229e-02, 3.9907e-02, -8.0809e-03, 1.3760e-02,
-6.5863e-03, 6.6066e-03, -3.1480e-02, 2.4665e-02, 3.4374e-03,
2.0973e-02, 1.9384e-02, -2.0880e-02, 7.1465e-03, 1.0406e-02,
2.2273e-05, -1.9182e-02, 6.3135e-03, -1.6891e-04, 8.8664e-03,
-4.7666e-03, -1.4493e-02, 3.2176e-03, 7.3346e-03, 2.0694e-02,
-4.9972e-04, 1.8820e-02, 3.9147e-02, -2.7095e-02, -1.8293e-02,
-1.9868e-02, -9.4048e-03, 4.1552e-03, 5.3837e-03, -4.6663e-03,
-1.3019e-02, -2.4452e-02, -8.9231e-03, -1.4603e-02, 2.5529e-03,
-3.0766e-02, 1.1169e-02, -6.8113e-03, -7.5967e-03, -9.3191e-03,
1.4919e-03, -2.3428e-03, 4.4398e-04, -1.0810e-02, 8.8498e-03,
-2.1022e-02, -8.0380e-03, -1.0818e-02, -6.4815e-03, -2.0681e-03,
2.2326e-02, -1.9234e-02, 4.0844e-03, 7.7233e-04, 1.7226e-03,
-1.7454e-02, -1.3190e-02, -7.4112e-03, -1.7550e-03, 1.2926e-03,
-6.7029e-03, -7.0588e-03, 6.2745e-03, -1.8068e-02, -9.4855e-03,
-2.0856e-02, 8.9604e-03, 2.1294e-02, 1.7025e-02, 2.1015e-02,
8.8233e-03, -9.8277e-03, -2.2293e-02, 2.4295e-02, -1.1174e-02,
-7.5753e-03, 6.1182e-03, -2.0653e-02, -1.6264e-02, 2.6457e-02,
-1.4782e-02, 1.8654e-02, 2.5488e-02, 2.4106e-02, 4.7888e-03,
2.3329e-02, 3.5806e-04, 2.5154e-02, 1.7094e-02, 1.7803e-02,
2.4687e-02, 9.0085e-03, 2.3610e-03, 2.6088e-02, -1.4110e-02,
-5.4212e-04, 8.9498e-04, 2.1150e-02, 4.8484e-03, -3.0503e-02,
-7.5025e-03, -3.3718e-02, -2.8913e-02, 1.5691e-02, 6.2047e-03,
-1.0853e-02, 1.9524e-02, -1.6188e-02, 8.9890e-03, 9.1894e-03,
-2.8592e-03, -1.0911e-02, 1.0848e-02, 4.8784e-02, -1.9687e-03,
2.6843e-02, -4.8715e-03, 1.3489e-02, -1.4523e-02, -2.7585e-02,
6.1228e-03, 4.8171e-03, 2.1566e-03, -3.7561e-02, 3.0775e-02,
1.9977e-02, 1.8480e-02, 3.0368e-03, 9.3825e-03, 4.5243e-04,
6.1650e-02, -8.6416e-03, -2.6913e-02, 6.3527e-03, 7.7985e-03,
1.3180e-02, -1.6666e-03, 2.0865e-02, 9.9480e-03, 8.8136e-03,
1.4841e-02, 3.3211e-03, 3.6342e-03, 2.8740e-02, -2.2120e-02,
-7.1567e-03, 1.0352e-02, 1.6433e-02, 1.1683e-02, -5.8058e-03,
-6.9297e-04, 2.6578e-02, 8.7967e-03, -3.1689e-02, 1.8949e-02,
-8.5859e-03, 3.4228e-02, -1.5237e-02, -5.9709e-03, 1.1069e-03,
-1.8394e-02, -1.9246e-02, -3.6361e-02, 3.9839e-03, 4.1237e-02,
1.3816e-02, -7.3304e-03, 3.8832e-03, 2.4367e-03, -2.1625e-02,
-1.4523e-02, -1.6281e-04, 6.2566e-04, -1.6798e-02, 2.3083e-02,
9.7114e-03, -8.2207e-03, 1.1595e-03, -2.0983e-02, -6.6540e-03,
-1.4097e-02, 3.4067e-03, -7.7575e-03, -1.4738e-02, -2.1343e-02,
5.4123e-03, 3.9747e-03, -4.6185e-03, -1.5462e-02, -7.6229e-03,
1.2211e-02, -4.8453e-03, -8.8757e-03, -1.0275e-02, 7.3482e-03,
-6.0349e-03, 2.3658e-03, 2.1053e-02, -8.5688e-03, -1.1630e-02,
-2.7332e-02, -2.0648e-02, 4.4952e-03, -1.8649e-02, -1.1564e-02,
4.5905e-04, -6.1831e-03, -2.4435e-02, -7.1187e-03, -1.4394e-02,
-2.3544e-03, 2.1556e-02, 2.2924e-02, -1.3725e-02, 7.7785e-03,
-8.5513e-03, 2.4221e-02, 3.8192e-03, 7.0947e-04, 1.6114e-02,
2.5932e-02, 1.8108e-02, 2.9306e-02, 1.6773e-03, -3.0166e-03,
3.2015e-02, -1.4034e-02, 2.7365e-02, -1.8858e-02, 2.5832e-03,
1.3498e-02, -1.3502e-02, -1.4940e-02, -1.0904e-02, 1.8642e-02,
4.2593e-03, -1.6742e-02, -1.2638e-02, -4.5468e-02, -5.0823e-03,
-2.5093e-02, 6.7847e-03, -1.7868e-02, -7.8250e-04, -6.3448e-03],
requires_grad=True)
Change number of features if needed
n_features = 10
model.fc = nn.Linear(model.fc.in_features, n_features)
print("model.fc: \n" , model.fc )
print("\n")
print("model: \n" , model)
# Access the first convolutional layer's weights
print("model.conv1.weight: \n:" , model.conv1.weight)
# Access the final fully connected layer's weights and biases
print("model.fc.weight: \n:" , model.fc.weight)
print("model.fc.bias: \n:" , model.fc.bias)model.fc:
Linear(in_features=512, out_features=10, bias=True)
model:
ResNet(
(conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
(layer1): Sequential(
(0): BasicBlock(
(conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(1): BasicBlock(
(conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(layer2): Sequential(
(0): BasicBlock(
(conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(downsample): Sequential(
(0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
(1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(layer3): Sequential(
(0): BasicBlock(
(conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(downsample): Sequential(
(0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
(1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(layer4): Sequential(
(0): BasicBlock(
(conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(downsample): Sequential(
(0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
(fc): Linear(in_features=512, out_features=10, bias=True)
)
model.conv1.weight:
: Parameter containing:
tensor([[[[-1.0419e-02, -6.1356e-03, -1.8098e-03, ..., 5.6615e-02,
1.7083e-02, -1.2694e-02],
[ 1.1083e-02, 9.5276e-03, -1.0993e-01, ..., -2.7124e-01,
-1.2907e-01, 3.7424e-03],
[-6.9434e-03, 5.9089e-02, 2.9548e-01, ..., 5.1972e-01,
2.5632e-01, 6.3573e-02],
...,
[-2.7535e-02, 1.6045e-02, 7.2595e-02, ..., -3.3285e-01,
-4.2058e-01, -2.5781e-01],
[ 3.0613e-02, 4.0960e-02, 6.2850e-02, ..., 4.1384e-01,
3.9359e-01, 1.6606e-01],
[-1.3736e-02, -3.6746e-03, -2.4084e-02, ..., -1.5070e-01,
-8.2230e-02, -5.7828e-03]],
[[-1.1397e-02, -2.6619e-02, -3.4641e-02, ..., 3.2521e-02,
6.6221e-04, -2.5743e-02],
[ 4.5687e-02, 3.3603e-02, -1.0453e-01, ..., -3.1253e-01,
-1.6051e-01, -1.2826e-03],
[-8.3730e-04, 9.8420e-02, 4.0210e-01, ..., 7.0789e-01,
3.6887e-01, 1.2455e-01],
...,
[-5.5926e-02, -5.2239e-03, 2.7081e-02, ..., -4.6178e-01,
-5.7080e-01, -3.6552e-01],
[ 3.2860e-02, 5.5574e-02, 9.9670e-02, ..., 5.4636e-01,
4.8276e-01, 1.9867e-01],
[ 5.3051e-03, 6.6938e-03, -1.7254e-02, ..., -1.4822e-01,
-7.7248e-02, 7.2183e-04]],
[[-2.0315e-03, -9.1617e-03, 2.1209e-02, ..., 8.9177e-02,
3.3655e-02, -2.0102e-02],
[ 1.5398e-02, -1.8648e-02, -1.2591e-01, ..., -2.5342e-01,
-1.2980e-01, -2.7975e-02],
[ 9.8454e-03, 4.9047e-02, 2.1699e-01, ..., 3.4872e-01,
1.0433e-01, 1.8413e-02],
...,
[-2.8356e-02, 1.8404e-02, 9.8647e-02, ..., -1.1740e-01,
-2.5760e-01, -1.5451e-01],
[ 2.0766e-02, -2.6286e-03, -3.7825e-02, ..., 2.4141e-01,
2.4345e-01, 1.1796e-01],
[ 7.4684e-04, 7.7677e-04, -1.0050e-02, ..., -1.4865e-01,
-1.1754e-01, -3.8350e-02]]],
[[[-4.4154e-03, -4.0645e-03, 3.1589e-03, ..., -3.7026e-02,
-2.5158e-02, -4.7945e-02],
[ 5.1310e-02, 5.3402e-02, 8.0436e-02, ..., 1.4480e-01,
1.4287e-01, 1.2312e-01],
[-7.3337e-03, 2.1755e-03, 3.7580e-02, ..., 6.1517e-02,
8.0324e-02, 1.1715e-01],
...,
[-2.6754e-02, -1.2297e-01, -1.3653e-01, ..., -1.4068e-01,
-1.1155e-01, -4.9556e-02],
[ 2.3524e-02, -1.7288e-02, -1.1122e-02, ..., -1.8826e-02,
-2.3320e-02, -2.9474e-02],
[ 2.8689e-02, 2.1659e-02, 4.7888e-02, ..., 2.5498e-02,
3.5346e-02, 1.1280e-02]],
[[ 4.6919e-04, 1.2153e-02, 4.2035e-02, ..., 4.6403e-02,
4.0423e-02, -1.4439e-02],
[ 4.3463e-02, 6.8779e-02, 1.3268e-01, ..., 2.8606e-01,
2.6905e-01, 2.0935e-01],
[-5.7621e-02, -2.2642e-02, 3.0547e-02, ..., 1.3763e-01,
1.6538e-01, 1.7946e-01],
...,
[-1.0816e-01, -2.5227e-01, -2.9742e-01, ..., -2.8503e-01,
-2.1493e-01, -1.0320e-01],
[ 4.0709e-02, -3.2771e-02, -6.3450e-02, ..., -9.2360e-02,
-6.9876e-02, -4.9841e-02],
[ 8.2942e-02, 8.7580e-02, 1.0111e-01, ..., 5.2714e-02,
6.0968e-02, 4.1198e-02]],
[[-1.6391e-02, -1.3870e-02, 5.2810e-03, ..., 4.3698e-02,
2.2707e-02, -4.5983e-02],
[ 3.3202e-02, 4.2014e-02, 9.3500e-02, ..., 2.6162e-01,
2.2970e-01, 1.6694e-01],
[-4.5987e-02, -1.6365e-02, 2.6811e-02, ..., 1.4951e-01,
1.3216e-01, 1.3579e-01],
...,
[-7.2129e-02, -1.8902e-01, -2.3389e-01, ..., -1.9038e-01,
-1.5609e-01, -7.5974e-02],
[ 5.1161e-02, -2.5815e-02, -6.9357e-02, ..., -5.8999e-02,
-6.1550e-02, -4.4555e-02],
[ 1.1174e-01, 7.8979e-02, 6.5849e-02, ..., 3.1617e-02,
2.5221e-02, 7.4257e-03]]],
[[[-7.0826e-08, -6.4306e-08, -7.3806e-08, ..., -9.8000e-08,
-1.0905e-07, -8.3421e-08],
[-6.1125e-09, 2.0613e-09, -8.0922e-09, ..., -4.9840e-08,
-4.3836e-08, -3.0538e-09],
[ 7.1953e-08, 7.5616e-08, 5.9282e-08, ..., -9.7509e-09,
-1.0951e-09, 4.2442e-08],
...,
[ 9.5889e-08, 1.0039e-07, 7.9817e-08, ..., -1.7491e-08,
-4.7666e-08, -1.3265e-08],
[ 1.2904e-07, 1.4762e-07, 1.7477e-07, ..., 1.3233e-07,
1.0628e-07, 9.3316e-08],
[ 1.2558e-07, 1.3644e-07, 1.8431e-07, ..., 2.1399e-07,
1.7710e-07, 1.7166e-07]],
[[-1.2690e-07, -9.6139e-08, -1.0372e-07, ..., -1.1808e-07,
-1.3309e-07, -1.0820e-07],
[-5.7412e-08, -2.5055e-08, -3.0115e-08, ..., -7.2922e-08,
-6.7022e-08, -2.2574e-08],
[ 2.1813e-08, 4.8608e-08, 3.1222e-08, ..., -1.8694e-08,
-7.9591e-09, 3.9750e-08],
...,
[ 5.6013e-08, 7.5526e-08, 4.4496e-08, ..., -4.4128e-08,
-5.9930e-08, -1.8247e-08],
[ 7.7614e-08, 9.8348e-08, 1.0455e-07, ..., 6.3272e-08,
4.1781e-08, 4.5901e-08],
[ 5.9834e-08, 7.1006e-08, 9.0437e-08, ..., 1.1654e-07,
8.7550e-08, 9.8837e-08]],
[[-4.3810e-08, 1.3270e-08, 7.8275e-09, ..., -5.8804e-09,
-2.6217e-08, -1.5649e-08],
[ 4.1700e-08, 1.0778e-07, 1.0946e-07, ..., 7.6403e-08,
7.1450e-08, 9.7615e-08],
[ 1.0436e-07, 1.6586e-07, 1.5933e-07, ..., 1.3517e-07,
1.3487e-07, 1.6449e-07],
...,
[ 9.8763e-08, 1.5072e-07, 1.2547e-07, ..., 6.8316e-08,
6.8382e-08, 1.1367e-07],
[ 9.1435e-08, 1.3576e-07, 1.3793e-07, ..., 1.1678e-07,
1.1723e-07, 1.4394e-07],
[ 6.2183e-08, 8.8184e-08, 1.0456e-07, ..., 1.3941e-07,
1.3333e-07, 1.5844e-07]]],
...,
[[[-6.1896e-02, -3.0206e-02, 1.9225e-02, ..., 4.3665e-02,
-2.2114e-02, -4.2214e-02],
[-3.8061e-02, 6.0774e-03, 4.5797e-02, ..., 9.6029e-02,
5.9254e-02, 2.9958e-02],
[-2.9672e-02, 2.7766e-03, 2.0457e-02, ..., 5.9828e-02,
4.1422e-02, 2.3134e-02],
...,
[ 1.1916e-02, 4.5701e-02, 4.4892e-02, ..., 4.7419e-02,
2.2274e-02, -5.4993e-03],
[-3.2468e-02, -1.2210e-02, 2.2023e-02, ..., 5.8061e-02,
-7.5033e-03, -5.9736e-02],
[-4.3314e-02, -2.8162e-02, -5.9126e-03, ..., 8.8460e-02,
8.4406e-03, -5.0019e-02]],
[[-6.1292e-02, -1.4004e-02, 1.7229e-02, ..., 1.8349e-02,
-3.2708e-02, -4.1060e-02],
[-3.1506e-02, 2.4460e-02, 4.5516e-02, ..., 6.6806e-02,
4.6687e-02, 3.3248e-02],
[-3.2216e-02, 2.0718e-02, 2.3343e-02, ..., 3.5265e-02,
3.6478e-02, 3.1291e-02],
...,
[ 1.7739e-02, 6.1040e-02, 4.8247e-02, ..., 3.7785e-02,
2.8894e-02, 1.3984e-02],
[-1.0890e-02, 2.2079e-02, 4.2737e-02, ..., 6.0247e-02,
1.6197e-02, -1.2493e-02],
[-2.2284e-02, 1.3220e-02, 3.0897e-02, ..., 1.0403e-01,
4.0119e-02, -5.3310e-03]],
[[-8.5322e-02, -4.2603e-02, 6.8145e-03, ..., 3.0751e-02,
-3.4818e-02, -4.9945e-02],
[-2.9215e-02, 1.8165e-02, 5.1092e-02, ..., 9.0200e-02,
5.3438e-02, 4.0169e-02],
[-3.9932e-02, -1.1100e-03, 9.6176e-03, ..., 2.4114e-02,
2.6298e-02, 2.5489e-02],
...,
[-3.1890e-03, 3.0454e-02, 1.6316e-02, ..., 5.5054e-03,
-6.2689e-03, -8.4638e-03],
[-2.2995e-02, -2.8211e-03, 2.3203e-02, ..., 3.5888e-02,
-1.4296e-02, -3.2419e-02],
[-9.8894e-03, 7.0542e-03, 1.0659e-02, ..., 7.0495e-02,
1.2996e-02, -8.3417e-03]]],
[[[-7.8699e-03, 1.9911e-02, 3.4208e-02, ..., 2.8694e-02,
1.2820e-02, 1.8142e-02],
[ 8.7942e-03, -3.2875e-02, -3.5713e-02, ..., 7.2533e-02,
4.5889e-02, 5.2383e-02],
[-3.6122e-02, -1.1878e-01, -1.3767e-01, ..., 3.3811e-02,
3.7806e-02, 2.6944e-02],
...,
[ 1.7322e-02, 3.9589e-03, -8.2269e-03, ..., 2.7543e-03,
1.8313e-02, 1.6057e-02],
[-9.5007e-04, 1.6428e-02, 1.7156e-02, ..., 3.3672e-03,
2.2857e-02, 6.5783e-04],
[ 6.1727e-03, 2.7145e-02, 1.4340e-02, ..., 7.5867e-03,
1.8770e-02, 1.5624e-02]],
[[-1.3423e-02, -5.0696e-04, 8.0959e-03, ..., -6.0963e-03,
9.2341e-03, 1.5751e-02],
[-1.8343e-02, -6.7982e-02, -7.0685e-02, ..., 2.9855e-02,
2.6264e-02, 2.3773e-02],
[-5.4359e-02, -1.4663e-01, -1.6211e-01, ..., 1.1781e-02,
3.2477e-02, 1.1980e-02],
...,
[ 8.3686e-04, -1.7564e-02, -1.9535e-02, ..., -4.1382e-03,
2.4658e-02, 1.2893e-02],
[-6.3183e-04, 1.1788e-02, 2.4810e-02, ..., 6.1105e-03,
3.9210e-02, 9.6696e-03],
[-7.1831e-03, 6.6918e-03, 5.2723e-03, ..., -7.6077e-03,
2.7253e-02, 1.7735e-02]],
[[-2.3753e-04, -4.9343e-03, 2.2991e-03, ..., -4.7958e-02,
-2.6154e-02, -2.3525e-02],
[-3.3053e-04, -5.1502e-02, -5.9977e-02, ..., -1.7369e-02,
-2.3337e-02, -3.7312e-02],
[-2.2674e-02, -9.9412e-02, -1.1176e-01, ..., -1.1725e-02,
-8.3744e-03, -4.0615e-02],
...,
[ 1.1437e-02, -8.0313e-03, -1.4955e-03, ..., -3.4133e-02,
-8.7267e-03, -2.3526e-02],
[ 2.9522e-03, 6.7770e-04, 1.9933e-02, ..., -2.2002e-02,
1.4814e-02, -1.4487e-02],
[-1.9085e-02, -2.9430e-02, -2.3284e-02, ..., -4.8587e-02,
-1.3049e-02, -2.4368e-02]]],
[[[-3.6296e-02, 7.1996e-03, 1.9100e-02, ..., 1.9602e-02,
1.4870e-02, -1.7298e-02],
[-1.1061e-02, 8.5665e-02, 1.2667e-01, ..., 1.3744e-02,
-5.5036e-05, -3.0162e-02],
[ 1.1322e-01, 1.8634e-01, 5.0658e-02, ..., -1.7333e-01,
-7.2041e-02, -6.2474e-02],
...,
[-5.3062e-02, -2.5781e-01, -2.6747e-01, ..., 2.6781e-01,
1.4344e-01, 5.5145e-02],
[-2.1009e-02, -2.9969e-02, 1.0245e-01, ..., 2.0843e-01,
-4.1518e-03, -3.8118e-02],
[-2.2155e-02, 1.2380e-02, 8.4302e-02, ..., -4.4992e-02,
-1.4687e-01, -9.0890e-02]],
[[-5.3969e-03, 3.2799e-02, 1.5486e-02, ..., -7.7451e-03,
3.0229e-03, 1.1216e-03],
[ 6.1723e-02, 1.4899e-01, 1.4645e-01, ..., -2.8897e-02,
-2.0227e-02, -9.1878e-03],
[ 1.6146e-01, 2.0886e-01, -2.5589e-02, ..., -2.7278e-01,
-1.0735e-01, -6.2971e-02],
...,
[-1.3723e-01, -4.0863e-01, -3.8551e-01, ..., 4.0846e-01,
2.6202e-01, 1.3491e-01],
[-5.9388e-02, -6.1187e-02, 1.4197e-01, ..., 3.5780e-01,
9.0893e-02, -1.7392e-03],
[ 7.8613e-03, 5.8403e-02, 1.5339e-01, ..., 4.7045e-02,
-1.0095e-01, -9.7920e-02]],
[[-5.6799e-03, 1.3425e-02, -2.6461e-02, ..., 4.4881e-03,
2.0666e-03, 1.3902e-02],
[ 6.5943e-03, 4.5181e-02, 6.0260e-02, ..., 1.4368e-02,
-5.0725e-03, 4.0505e-03],
[ 5.5257e-02, 1.2397e-01, 4.3193e-02, ..., -1.4486e-01,
-7.4489e-02, -5.7533e-02],
...,
[-3.1513e-02, -1.6334e-01, -1.5795e-01, ..., 2.2904e-01,
1.2017e-01, 7.1998e-02],
[-1.0456e-02, -1.1248e-03, 8.4582e-02, ..., 1.5748e-01,
2.2142e-02, -1.0083e-02],
[-4.8639e-03, -5.0065e-03, 3.6341e-02, ..., -2.4361e-02,
-7.1195e-02, -6.6788e-02]]]], requires_grad=True)
model.fc.weight:
: Parameter containing:
tensor([[-0.0166, -0.0325, -0.0027, ..., 0.0415, 0.0074, 0.0252],
[ 0.0046, -0.0348, -0.0101, ..., -0.0329, -0.0399, -0.0235],
[ 0.0127, 0.0429, -0.0368, ..., 0.0294, 0.0339, 0.0413],
...,
[ 0.0107, 0.0406, -0.0318, ..., 0.0140, 0.0297, -0.0383],
[ 0.0313, 0.0032, 0.0164, ..., -0.0195, 0.0222, 0.0037],
[ 0.0385, -0.0376, 0.0073, ..., 0.0049, -0.0066, 0.0169]],
requires_grad=True)
model.fc.bias:
: Parameter containing:
tensor([-0.0133, -0.0391, -0.0252, -0.0067, -0.0403, -0.0097, 0.0135, -0.0066,
0.0415, 0.0095], requires_grad=True)
create input data
The code torch.rand(1, 3, 64, 64) creates a PyTorch tensor filled with random floating-point numbers sampled from a uniform distribution between 0 and 1 (specifically, on the interval ([0, 1))). (https://docs.pytorch.org/docs/stable/generated/torch.rand.html)
In machine learning and computer vision applications, this specific syntax is most commonly used to simulate a batch of image data:
- 1: The Batch Size. It generates a batch containing exactly 1 image.
- 3: The number of Channels. This represents the 3 color channels (Red, Green, Blue) of a standard RGB image.
- 64: The Height of the image in pixels.
- 64: The Width of the image in pixels.
inputData = torch.rand(1, 3, 64, 64) # imitating a 3-channel (RGB) image of resolution 64x64
print(inputData.size())
#x= torch.count_nonzero(data, dim=-2)
#print(x)
print(inputData)torch.Size([1, 3, 64, 64])
tensor([[[[9.5841e-01, 6.0746e-01, 2.8166e-01, ..., 9.3681e-01,
6.1917e-01, 2.4135e-01],
[1.6661e-01, 6.9673e-01, 6.4277e-01, ..., 1.2155e-01,
2.6584e-02, 1.4099e-01],
[5.5063e-01, 5.4008e-01, 3.9730e-02, ..., 4.8003e-01,
6.5066e-01, 7.1473e-02],
...,
[6.9182e-01, 5.7999e-01, 1.4790e-01, ..., 8.3422e-02,
6.3316e-01, 8.0672e-01],
[2.4326e-01, 9.4158e-01, 6.8585e-01, ..., 7.0906e-01,
7.0271e-01, 3.0782e-01],
[3.3045e-01, 7.6311e-01, 2.2229e-01, ..., 6.8798e-01,
5.2555e-01, 8.8321e-01]],
[[6.7163e-01, 7.9909e-01, 5.3082e-01, ..., 6.0547e-01,
8.3251e-01, 7.6736e-01],
[9.3180e-01, 7.0346e-02, 3.9334e-01, ..., 3.4731e-01,
3.2179e-01, 1.4288e-02],
[8.0640e-01, 7.2000e-01, 8.1161e-01, ..., 5.3906e-01,
7.1333e-01, 6.7825e-01],
...,
[2.6193e-01, 6.2501e-02, 1.8817e-01, ..., 3.4501e-02,
9.5582e-01, 4.5406e-01],
[2.2799e-01, 3.9165e-01, 1.4729e-02, ..., 2.2498e-01,
4.5451e-01, 4.1192e-01],
[9.8269e-01, 7.4705e-01, 8.5231e-01, ..., 7.1067e-01,
2.8151e-01, 8.7332e-01]],
[[4.8994e-01, 8.7905e-01, 5.8813e-01, ..., 7.6099e-01,
1.5596e-01, 3.4634e-01],
[4.3577e-01, 7.1509e-01, 5.1868e-01, ..., 1.3920e-01,
1.3287e-01, 8.8131e-01],
[2.7214e-03, 6.3753e-01, 6.1534e-01, ..., 8.2451e-01,
2.7781e-01, 1.8892e-01],
...,
[9.3951e-01, 7.3669e-02, 4.5974e-01, ..., 8.0792e-01,
1.8557e-01, 4.8001e-01],
[9.7458e-02, 9.1575e-01, 4.9707e-01, ..., 7.7584e-01,
7.4184e-01, 2.2200e-01],
[6.3333e-01, 8.4113e-01, 8.1350e-01, ..., 9.8211e-04,
6.7820e-01, 4.8692e-01]]]])
visualize the data: functions to show an image
import torch
import matplotlib.pyplot as plt
import numpy as np
import torchvision
# 1. Define the original function (without unnormalization for raw random data)
def Display(img):
print("img shape:", img.shape) # Helpful debugging check
npimg = img.numpy()
# Transpose from (Channels, Height, Width) to (Height, Width, Channels)
plt.imshow(np.transpose(npimg, (1, 2, 0)))
plt.show()
# 3. Create a grid (make_grid converts a 4D batch into a single 3D image grid)
image = torchvision.utils.make_grid(inputData)
# 4. Display the grid image
Display(image)img shape: torch.Size([3, 64, 64])
![Output of In [9]](/courses/media/436a1cecc0bc.png)
create labels
- random in this demo
- the size of the labels should be compatible with features (and therefore with predictions)
# Generates a matrix of shape (1, n_features) filled with random floating-point values between 0 and 1.
labels = torch.rand(1, n_features) #Because these are random floats, the probability of any value being exactly 0 is near impossible
print("labels:",labels)
print(labels.size())
# count how many elements along that row are not equal to 0. Since torch.rand yields non-zero decimals (e.g., 0.234, 0.891), all elements are non-zero.
#x= torch.count_nonzero(labels, dim=1) #
#print(x)labels: tensor([[0.3839, 0.1817, 0.1820, 0.0781, 0.6000, 0.2717, 0.2384, 0.4251, 0.4012,
0.1795]])
torch.Size([1, 10])
Next, we run the input data through the model through each of its layers to make a prediction. This is the forward pass.
prediction = model(inputData) # forward pass
print("prediction: \n", prediction)
#x= torch.count_nonzero(prediction, dim=+1)
print(prediction.size())prediction:
tensor([[-0.8056, -1.1892, -0.9612, 0.8434, 0.0793, -0.3005, -0.6426, -0.5022,
-0.5883, 0.9827]], grad_fn=<AddmmBackward0>)
torch.Size([1, 10])
We use the model's prediction and the corresponding label to calculate the error (loss).
loss = (prediction - labels).sum()
print("loss; \n" , loss)loss; tensor(-6.0259, grad_fn=<SumBackward0>)
The next step is to backpropagate this error through the network.
Backward propagation is kicked off when we call .backward() on the error tensor.
Autograd then calculates and stores the gradients for each model parameter in the parameter's .grad attribute.
loss.backward() # backward passNext, we load an optimizer, in this case stochastic gradient descent SGD with a learning rate of 0.01 and momentum of 0.9. We register all the parameters of the model in the optimizer.
optim = torch.optim.SGD(model.parameters(), lr=1e-2, momentum=0.9)Finally, we call .step() to initiate gradient descent. The optimizer adjusts each parameter by its gradient stored in .grad.
optim.step() #gradient descent
# Access the first convolutional layer's weights
print("model.conv1.weight: \n:" , model.conv1.weight)
# Access the final fully connected layer's weights and biases
print("model.fc.weight: \n:" , model.fc.weight)
print("model.fc.bias: \n:" , model.fc.bias)model.conv1.weight:
: Parameter containing:
tensor([[[[-1.0419e-02, -6.1356e-03, -1.8098e-03, ..., 5.6615e-02,
1.7083e-02, -1.2694e-02],
[ 1.1083e-02, 9.5276e-03, -1.0993e-01, ..., -2.7124e-01,
-1.2907e-01, 3.7424e-03],
[-6.9434e-03, 5.9089e-02, 2.9548e-01, ..., 5.1972e-01,
2.5632e-01, 6.3573e-02],
...,
[-2.7535e-02, 1.6045e-02, 7.2595e-02, ..., -3.3285e-01,
-4.2058e-01, -2.5781e-01],
[ 3.0613e-02, 4.0960e-02, 6.2850e-02, ..., 4.1384e-01,
3.9359e-01, 1.6606e-01],
[-1.3736e-02, -3.6746e-03, -2.4084e-02, ..., -1.5070e-01,
-8.2230e-02, -5.7828e-03]],
[[-1.1397e-02, -2.6619e-02, -3.4641e-02, ..., 3.2521e-02,
6.6221e-04, -2.5743e-02],
[ 4.5687e-02, 3.3603e-02, -1.0453e-01, ..., -3.1253e-01,
-1.6051e-01, -1.2826e-03],
[-8.3730e-04, 9.8420e-02, 4.0210e-01, ..., 7.0789e-01,
3.6887e-01, 1.2455e-01],
...,
[-5.5926e-02, -5.2239e-03, 2.7081e-02, ..., -4.6178e-01,
-5.7080e-01, -3.6552e-01],
[ 3.2860e-02, 5.5574e-02, 9.9670e-02, ..., 5.4636e-01,
4.8276e-01, 1.9867e-01],
[ 5.3051e-03, 6.6938e-03, -1.7254e-02, ..., -1.4822e-01,
-7.7248e-02, 7.2183e-04]],
[[-2.0315e-03, -9.1617e-03, 2.1209e-02, ..., 8.9177e-02,
3.3655e-02, -2.0102e-02],
[ 1.5398e-02, -1.8648e-02, -1.2591e-01, ..., -2.5342e-01,
-1.2980e-01, -2.7975e-02],
[ 9.8454e-03, 4.9047e-02, 2.1699e-01, ..., 3.4872e-01,
1.0433e-01, 1.8413e-02],
...,
[-2.8356e-02, 1.8404e-02, 9.8647e-02, ..., -1.1740e-01,
-2.5760e-01, -1.5451e-01],
[ 2.0766e-02, -2.6286e-03, -3.7825e-02, ..., 2.4141e-01,
2.4345e-01, 1.1796e-01],
[ 7.4684e-04, 7.7677e-04, -1.0050e-02, ..., -1.4865e-01,
-1.1754e-01, -3.8350e-02]]],
[[[-4.4154e-03, -4.0645e-03, 3.1589e-03, ..., -3.7026e-02,
-2.5158e-02, -4.7945e-02],
[ 5.1310e-02, 5.3402e-02, 8.0436e-02, ..., 1.4480e-01,
1.4287e-01, 1.2312e-01],
[-7.3337e-03, 2.1755e-03, 3.7580e-02, ..., 6.1517e-02,
8.0324e-02, 1.1715e-01],
...,
[-2.6754e-02, -1.2297e-01, -1.3653e-01, ..., -1.4068e-01,
-1.1155e-01, -4.9556e-02],
[ 2.3524e-02, -1.7288e-02, -1.1122e-02, ..., -1.8826e-02,
-2.3320e-02, -2.9474e-02],
[ 2.8689e-02, 2.1659e-02, 4.7888e-02, ..., 2.5498e-02,
3.5346e-02, 1.1280e-02]],
[[ 4.6919e-04, 1.2153e-02, 4.2035e-02, ..., 4.6403e-02,
4.0423e-02, -1.4439e-02],
[ 4.3463e-02, 6.8779e-02, 1.3268e-01, ..., 2.8606e-01,
2.6905e-01, 2.0935e-01],
[-5.7621e-02, -2.2642e-02, 3.0547e-02, ..., 1.3763e-01,
1.6538e-01, 1.7946e-01],
...,
[-1.0816e-01, -2.5227e-01, -2.9742e-01, ..., -2.8503e-01,
-2.1493e-01, -1.0320e-01],
[ 4.0709e-02, -3.2771e-02, -6.3450e-02, ..., -9.2360e-02,
-6.9876e-02, -4.9841e-02],
[ 8.2942e-02, 8.7580e-02, 1.0111e-01, ..., 5.2714e-02,
6.0968e-02, 4.1198e-02]],
[[-1.6391e-02, -1.3870e-02, 5.2810e-03, ..., 4.3698e-02,
2.2707e-02, -4.5983e-02],
[ 3.3202e-02, 4.2014e-02, 9.3500e-02, ..., 2.6162e-01,
2.2970e-01, 1.6694e-01],
[-4.5987e-02, -1.6365e-02, 2.6811e-02, ..., 1.4951e-01,
1.3216e-01, 1.3579e-01],
...,
[-7.2129e-02, -1.8902e-01, -2.3389e-01, ..., -1.9038e-01,
-1.5609e-01, -7.5974e-02],
[ 5.1161e-02, -2.5815e-02, -6.9357e-02, ..., -5.8999e-02,
-6.1550e-02, -4.4555e-02],
[ 1.1174e-01, 7.8979e-02, 6.5849e-02, ..., 3.1617e-02,
2.5221e-02, 7.4257e-03]]],
[[[-7.0826e-08, -6.4306e-08, -7.3806e-08, ..., -9.8000e-08,
-1.0905e-07, -8.3421e-08],
[-6.1125e-09, 2.0613e-09, -8.0922e-09, ..., -4.9840e-08,
-4.3836e-08, -3.0538e-09],
[ 7.1953e-08, 7.5616e-08, 5.9282e-08, ..., -9.7509e-09,
-1.0951e-09, 4.2442e-08],
...,
[ 9.5889e-08, 1.0039e-07, 7.9817e-08, ..., -1.7491e-08,
-4.7666e-08, -1.3265e-08],
[ 1.2904e-07, 1.4762e-07, 1.7477e-07, ..., 1.3233e-07,
1.0628e-07, 9.3316e-08],
[ 1.2558e-07, 1.3644e-07, 1.8431e-07, ..., 2.1399e-07,
1.7710e-07, 1.7166e-07]],
[[-1.2690e-07, -9.6139e-08, -1.0372e-07, ..., -1.1808e-07,
-1.3309e-07, -1.0820e-07],
[-5.7412e-08, -2.5055e-08, -3.0115e-08, ..., -7.2922e-08,
-6.7022e-08, -2.2574e-08],
[ 2.1813e-08, 4.8608e-08, 3.1222e-08, ..., -1.8694e-08,
-7.9591e-09, 3.9750e-08],
...,
[ 5.6013e-08, 7.5526e-08, 4.4496e-08, ..., -4.4128e-08,
-5.9930e-08, -1.8247e-08],
[ 7.7614e-08, 9.8348e-08, 1.0455e-07, ..., 6.3272e-08,
4.1781e-08, 4.5901e-08],
[ 5.9834e-08, 7.1006e-08, 9.0437e-08, ..., 1.1654e-07,
8.7550e-08, 9.8837e-08]],
[[-4.3810e-08, 1.3270e-08, 7.8275e-09, ..., -5.8804e-09,
-2.6217e-08, -1.5649e-08],
[ 4.1700e-08, 1.0778e-07, 1.0946e-07, ..., 7.6403e-08,
7.1450e-08, 9.7615e-08],
[ 1.0436e-07, 1.6586e-07, 1.5933e-07, ..., 1.3517e-07,
1.3487e-07, 1.6449e-07],
...,
[ 9.8763e-08, 1.5072e-07, 1.2547e-07, ..., 6.8316e-08,
6.8382e-08, 1.1367e-07],
[ 9.1435e-08, 1.3576e-07, 1.3793e-07, ..., 1.1678e-07,
1.1723e-07, 1.4394e-07],
[ 6.2183e-08, 8.8184e-08, 1.0456e-07, ..., 1.3941e-07,
1.3333e-07, 1.5844e-07]]],
...,
[[[-6.1896e-02, -3.0206e-02, 1.9225e-02, ..., 4.3665e-02,
-2.2114e-02, -4.2214e-02],
[-3.8061e-02, 6.0774e-03, 4.5797e-02, ..., 9.6029e-02,
5.9254e-02, 2.9958e-02],
[-2.9672e-02, 2.7766e-03, 2.0457e-02, ..., 5.9828e-02,
4.1422e-02, 2.3134e-02],
...,
[ 1.1916e-02, 4.5701e-02, 4.4892e-02, ..., 4.7419e-02,
2.2274e-02, -5.4993e-03],
[-3.2468e-02, -1.2210e-02, 2.2023e-02, ..., 5.8061e-02,
-7.5033e-03, -5.9736e-02],
[-4.3314e-02, -2.8162e-02, -5.9126e-03, ..., 8.8460e-02,
8.4406e-03, -5.0019e-02]],
[[-6.1292e-02, -1.4004e-02, 1.7229e-02, ..., 1.8349e-02,
-3.2708e-02, -4.1060e-02],
[-3.1506e-02, 2.4460e-02, 4.5516e-02, ..., 6.6806e-02,
4.6687e-02, 3.3248e-02],
[-3.2216e-02, 2.0718e-02, 2.3343e-02, ..., 3.5265e-02,
3.6478e-02, 3.1291e-02],
...,
[ 1.7739e-02, 6.1040e-02, 4.8247e-02, ..., 3.7785e-02,
2.8894e-02, 1.3984e-02],
[-1.0890e-02, 2.2079e-02, 4.2737e-02, ..., 6.0247e-02,
1.6197e-02, -1.2493e-02],
[-2.2284e-02, 1.3220e-02, 3.0897e-02, ..., 1.0403e-01,
4.0119e-02, -5.3310e-03]],
[[-8.5322e-02, -4.2603e-02, 6.8145e-03, ..., 3.0751e-02,
-3.4818e-02, -4.9945e-02],
[-2.9215e-02, 1.8165e-02, 5.1092e-02, ..., 9.0200e-02,
5.3438e-02, 4.0169e-02],
[-3.9932e-02, -1.1100e-03, 9.6176e-03, ..., 2.4114e-02,
2.6298e-02, 2.5489e-02],
...,
[-3.1890e-03, 3.0454e-02, 1.6316e-02, ..., 5.5054e-03,
-6.2689e-03, -8.4638e-03],
[-2.2995e-02, -2.8211e-03, 2.3203e-02, ..., 3.5888e-02,
-1.4296e-02, -3.2419e-02],
[-9.8894e-03, 7.0542e-03, 1.0659e-02, ..., 7.0495e-02,
1.2996e-02, -8.3417e-03]]],
[[[-7.8699e-03, 1.9911e-02, 3.4208e-02, ..., 2.8694e-02,
1.2820e-02, 1.8142e-02],
[ 8.7942e-03, -3.2875e-02, -3.5713e-02, ..., 7.2533e-02,
4.5889e-02, 5.2383e-02],
[-3.6122e-02, -1.1878e-01, -1.3767e-01, ..., 3.3811e-02,
3.7806e-02, 2.6944e-02],
...,
[ 1.7322e-02, 3.9589e-03, -8.2269e-03, ..., 2.7543e-03,
1.8313e-02, 1.6057e-02],
[-9.5007e-04, 1.6428e-02, 1.7156e-02, ..., 3.3672e-03,
2.2857e-02, 6.5783e-04],
[ 6.1727e-03, 2.7145e-02, 1.4340e-02, ..., 7.5867e-03,
1.8770e-02, 1.5624e-02]],
[[-1.3423e-02, -5.0696e-04, 8.0959e-03, ..., -6.0963e-03,
9.2341e-03, 1.5751e-02],
[-1.8343e-02, -6.7982e-02, -7.0685e-02, ..., 2.9855e-02,
2.6264e-02, 2.3773e-02],
[-5.4359e-02, -1.4663e-01, -1.6211e-01, ..., 1.1781e-02,
3.2477e-02, 1.1980e-02],
...,
[ 8.3686e-04, -1.7564e-02, -1.9535e-02, ..., -4.1382e-03,
2.4658e-02, 1.2893e-02],
[-6.3183e-04, 1.1788e-02, 2.4810e-02, ..., 6.1105e-03,
3.9210e-02, 9.6696e-03],
[-7.1831e-03, 6.6918e-03, 5.2723e-03, ..., -7.6077e-03,
2.7253e-02, 1.7735e-02]],
[[-2.3753e-04, -4.9343e-03, 2.2991e-03, ..., -4.7958e-02,
-2.6154e-02, -2.3525e-02],
[-3.3053e-04, -5.1502e-02, -5.9977e-02, ..., -1.7369e-02,
-2.3337e-02, -3.7312e-02],
[-2.2674e-02, -9.9412e-02, -1.1176e-01, ..., -1.1725e-02,
-8.3744e-03, -4.0615e-02],
...,
[ 1.1437e-02, -8.0313e-03, -1.4955e-03, ..., -3.4133e-02,
-8.7267e-03, -2.3526e-02],
[ 2.9522e-03, 6.7770e-04, 1.9933e-02, ..., -2.2002e-02,
1.4814e-02, -1.4487e-02],
[-1.9085e-02, -2.9430e-02, -2.3284e-02, ..., -4.8587e-02,
-1.3049e-02, -2.4368e-02]]],
[[[-3.6296e-02, 7.1996e-03, 1.9100e-02, ..., 1.9602e-02,
1.4870e-02, -1.7298e-02],
[-1.1061e-02, 8.5665e-02, 1.2667e-01, ..., 1.3744e-02,
-5.5036e-05, -3.0162e-02],
[ 1.1322e-01, 1.8634e-01, 5.0658e-02, ..., -1.7333e-01,
-7.2041e-02, -6.2474e-02],
...,
[-5.3062e-02, -2.5781e-01, -2.6747e-01, ..., 2.6781e-01,
1.4344e-01, 5.5145e-02],
[-2.1009e-02, -2.9969e-02, 1.0245e-01, ..., 2.0843e-01,
-4.1518e-03, -3.8118e-02],
[-2.2155e-02, 1.2380e-02, 8.4302e-02, ..., -4.4992e-02,
-1.4687e-01, -9.0890e-02]],
[[-5.3969e-03, 3.2799e-02, 1.5486e-02, ..., -7.7451e-03,
3.0229e-03, 1.1216e-03],
[ 6.1723e-02, 1.4899e-01, 1.4645e-01, ..., -2.8897e-02,
-2.0227e-02, -9.1878e-03],
[ 1.6146e-01, 2.0886e-01, -2.5589e-02, ..., -2.7278e-01,
-1.0735e-01, -6.2971e-02],
...,
[-1.3723e-01, -4.0863e-01, -3.8551e-01, ..., 4.0846e-01,
2.6202e-01, 1.3491e-01],
[-5.9388e-02, -6.1187e-02, 1.4197e-01, ..., 3.5780e-01,
9.0893e-02, -1.7392e-03],
[ 7.8613e-03, 5.8403e-02, 1.5339e-01, ..., 4.7045e-02,
-1.0095e-01, -9.7920e-02]],
[[-5.6799e-03, 1.3425e-02, -2.6461e-02, ..., 4.4881e-03,
2.0666e-03, 1.3902e-02],
[ 6.5943e-03, 4.5181e-02, 6.0260e-02, ..., 1.4368e-02,
-5.0725e-03, 4.0505e-03],
[ 5.5257e-02, 1.2397e-01, 4.3193e-02, ..., -1.4486e-01,
-7.4489e-02, -5.7533e-02],
...,
[-3.1513e-02, -1.6334e-01, -1.5795e-01, ..., 2.2904e-01,
1.2017e-01, 7.1998e-02],
[-1.0456e-02, -1.1248e-03, 8.4582e-02, ..., 1.5748e-01,
2.2142e-02, -1.0083e-02],
[-4.8639e-03, -5.0065e-03, 3.6341e-02, ..., -2.4361e-02,
-7.1195e-02, -6.6788e-02]]]])
model.fc.weight:
: Parameter containing:
tensor([[ 0.0239, 0.0007, 0.0160, ..., -0.0038, -0.0065, -0.0493],
[-0.0181, -0.0027, -0.0182, ..., 0.0116, -0.0535, -0.0410],
[ 0.0030, -0.0332, -0.0371, ..., 0.0209, -0.0114, -0.0002],
...,
[ 0.0300, -0.0078, -0.0450, ..., 0.0222, -0.0317, -0.0387],
[-0.0316, 0.0289, 0.0025, ..., 0.0335, -0.0127, -0.0493],
[-0.0281, 0.0035, -0.0534, ..., -0.0429, -0.0104, -0.0041]],
requires_grad=True)
model.fc.bias:
: Parameter containing:
tensor([-0.0447, 0.0140, 0.0217, -0.0294, -0.0181, -0.0119, 0.0264, 0.0185,
-0.0064, 0.0130], requires_grad=True)
Differentiation in Autograd
Let's take a look at how autograd collects gradients. We create two tensors a and b with
requires_grad=True. This signals to autograd that every operation on them should be tracked.
import torch
a = torch.tensor([2., 3.], requires_grad=True)
b = torch.tensor([6., 4.], requires_grad=True)
print(a)
print(b)tensor([2., 3.], requires_grad=True) tensor([6., 4.], requires_grad=True)
We create another tensor Q from a and b.
\begin{align}Q = 3a^3 - b^2\end{align}
Q = 3*a**3 - b**2Let's assume a and b to be parameters of an NN, and Q
to be the error. In NN training, we want gradients of the error
w.r.t. parameters, i.e.
\begin{align}\frac{\partial Q}{\partial a} = 9a^2\end{align}
\begin{align}\frac{\partial Q}{\partial b} = -2b\end{align}
When we call .backward() on Q, autograd calculates these gradients
and stores them in the respective tensors' .grad attribute.
We need to explicitly pass a gradient argument in Q.backward() because it is a vector.
gradient is a tensor of the same shape as Q, and it represents the
gradient of Q w.r.t. itself, i.e.
\begin{align}\frac{dQ}{dQ} = 1\end{align}
Mathematically, the relationship between and forms a Jacobian matrix :
\frac{\partial Q_1}{\partial a_1} & \frac{\partial Q_1}{\partial a_2} \\[8pt] \frac{\partial Q_2}{\partial a_1} & \frac{\partial Q_2}{\partial a_2} \end{bmatrix}$$external_grad = torch.tensor([1., 1.])
Q.backward(gradient=external_grad)Gradients are now deposited in a.grad and b.grad
check if collected gradients are correct
print("a.grad:", a.grad)
print("b.grad:", b.grad)
print(9*a**2 == a.grad)
print(-2*b == b.grad)a.grad: tensor([36., 81.]) b.grad: tensor([-12., -8.]) tensor([True, True]) tensor([True, True])
Computational Graph
Conceptually, autograd keeps a record of data (tensors) & all executed operations (along with the resulting new tensors) in a directed acyclic graph (DAG) consisting of [Function] (https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function) objects. In this DAG, leaves are the input tensors, roots are the output tensors. By tracing this graph from roots to leaves, you can automatically compute the gradients using the chain rule.
In a forward pass, autograd does two things simultaneously:
- run the requested operation to compute a resulting tensor, and
- maintain the operation’s gradient function in the DAG.
The backward pass kicks off when .backward() is called on the DAG
root. autograd then:
- computes the gradients from each
.grad_fn, - accumulates them in the respective tensor’s
.gradattribute, and - using the chain rule, propagates all the way to the leaf tensors.**
** NOTE: DAGs are dynamic in PyTorch**
An important thing to note is that the graph is recreated from scratch; after each
.backward() call, autograd starts populating a new graph. This is
exactly what allows you to use control flow statements in your model;
you can change the shape, size and operations at every iteration if
needed.
Exclusion from the DAG
torch.autograd tracks operations on all tensors which have their
requires_grad flag set to True. For tensors that don’t require
gradients, setting this attribute to False excludes it from the
gradient computation DAG.
The output tensor of an operation will require gradients even if only a
single input tensor has requires_grad=True.
x = torch.rand(5, 5)
y = torch.rand(5, 5)
z = torch.rand((5, 5), requires_grad=True)
a = x + y
print(f"Does `a` require gradients? : {a.requires_grad}")
b = x + z
print(f"Does `b` require gradients?: {b.requires_grad}")Does `a` require gradients? : False Does `b` require gradients?: True
Freezing Layers and transfer learning
In a NN, parameters that don't compute gradients are usually called frozen parameters. It is useful to "freeze" part of your model if you know in advance that you won't need the gradients of those parameters (this offers some performance benefits by reducing autograd computations).
Another common usecase where exclusion from the DAG is important is for transfer learning (see finetuning a pretrained network)
In transfer learning, we freeze most of the model and typically only modify the classifier layers (last fully connected layers) to make predictions on new labels. Let's walk through a small example to demonstrate this. As before, we load a pretrained resnet18 model, and freeze all the parameters.
from torch import nn, optim
#model = torchvision.models.resnet18(pretrained=True)
model = resnet18(weights=None)
PATH = r'D:\Teaching\UL\AI\Lab\resnetModel\resnet18.pth'
#LOAD model parameters in dictionary
model.load_state_dict(torch.load(PATH))
#print(model)
print("model.conv1:\n", model.conv1 )
print("model.conv1.weight:\n", model.conv1.weight)
print("+++++++++++++++++++++++++++++++++++++")
print("model.fc:\n", model.fc )
print("model.fc.weight:\n", model.fc.weight)
# Freeze all the parameters in the network
# Iterate through every parameter (weights and biases) in the model
for param in model.parameters():
# Disable gradient tracking for the parameter to freeze it,
# reducing memory usage and preventing parameter updates during backward()
param.requires_grad = False
print("++++++++++++++++ After requires_grad = False +++++++++++++++++++++")
print("model.conv1:\n", model.conv1 )
print("model.conv1.weight:\n", model.conv1.weight)
print("+++++++++++++++++++++++++++++++++++++")
print("model.fc:\n", model.fc )
print("model.fc.weight:\n", model.fc.weight)model.conv1:
Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
model.conv1.weight:
Parameter containing:
tensor([[[[-1.0419e-02, -6.1356e-03, -1.8098e-03, ..., 5.6615e-02,
1.7083e-02, -1.2694e-02],
[ 1.1083e-02, 9.5276e-03, -1.0993e-01, ..., -2.7124e-01,
-1.2907e-01, 3.7424e-03],
[-6.9434e-03, 5.9089e-02, 2.9548e-01, ..., 5.1972e-01,
2.5632e-01, 6.3573e-02],
...,
[-2.7535e-02, 1.6045e-02, 7.2595e-02, ..., -3.3285e-01,
-4.2058e-01, -2.5781e-01],
[ 3.0613e-02, 4.0960e-02, 6.2850e-02, ..., 4.1384e-01,
3.9359e-01, 1.6606e-01],
[-1.3736e-02, -3.6746e-03, -2.4084e-02, ..., -1.5070e-01,
-8.2230e-02, -5.7828e-03]],
[[-1.1397e-02, -2.6619e-02, -3.4641e-02, ..., 3.2521e-02,
6.6221e-04, -2.5743e-02],
[ 4.5687e-02, 3.3603e-02, -1.0453e-01, ..., -3.1253e-01,
-1.6051e-01, -1.2826e-03],
[-8.3730e-04, 9.8420e-02, 4.0210e-01, ..., 7.0789e-01,
3.6887e-01, 1.2455e-01],
...,
[-5.5926e-02, -5.2239e-03, 2.7081e-02, ..., -4.6178e-01,
-5.7080e-01, -3.6552e-01],
[ 3.2860e-02, 5.5574e-02, 9.9670e-02, ..., 5.4636e-01,
4.8276e-01, 1.9867e-01],
[ 5.3051e-03, 6.6938e-03, -1.7254e-02, ..., -1.4822e-01,
-7.7248e-02, 7.2183e-04]],
[[-2.0315e-03, -9.1617e-03, 2.1209e-02, ..., 8.9177e-02,
3.3655e-02, -2.0102e-02],
[ 1.5398e-02, -1.8648e-02, -1.2591e-01, ..., -2.5342e-01,
-1.2980e-01, -2.7975e-02],
[ 9.8454e-03, 4.9047e-02, 2.1699e-01, ..., 3.4872e-01,
1.0433e-01, 1.8413e-02],
...,
[-2.8356e-02, 1.8404e-02, 9.8647e-02, ..., -1.1740e-01,
-2.5760e-01, -1.5451e-01],
[ 2.0766e-02, -2.6286e-03, -3.7825e-02, ..., 2.4141e-01,
2.4345e-01, 1.1796e-01],
[ 7.4684e-04, 7.7677e-04, -1.0050e-02, ..., -1.4865e-01,
-1.1754e-01, -3.8350e-02]]],
[[[-4.4154e-03, -4.0645e-03, 3.1589e-03, ..., -3.7026e-02,
-2.5158e-02, -4.7945e-02],
[ 5.1310e-02, 5.3402e-02, 8.0436e-02, ..., 1.4480e-01,
1.4287e-01, 1.2312e-01],
[-7.3337e-03, 2.1755e-03, 3.7580e-02, ..., 6.1517e-02,
8.0324e-02, 1.1715e-01],
...,
[-2.6754e-02, -1.2297e-01, -1.3653e-01, ..., -1.4068e-01,
-1.1155e-01, -4.9556e-02],
[ 2.3524e-02, -1.7288e-02, -1.1122e-02, ..., -1.8826e-02,
-2.3320e-02, -2.9474e-02],
[ 2.8689e-02, 2.1659e-02, 4.7888e-02, ..., 2.5498e-02,
3.5346e-02, 1.1280e-02]],
[[ 4.6919e-04, 1.2153e-02, 4.2035e-02, ..., 4.6403e-02,
4.0423e-02, -1.4439e-02],
[ 4.3463e-02, 6.8779e-02, 1.3268e-01, ..., 2.8606e-01,
2.6905e-01, 2.0935e-01],
[-5.7621e-02, -2.2642e-02, 3.0547e-02, ..., 1.3763e-01,
1.6538e-01, 1.7946e-01],
...,
[-1.0816e-01, -2.5227e-01, -2.9742e-01, ..., -2.8503e-01,
-2.1493e-01, -1.0320e-01],
[ 4.0709e-02, -3.2771e-02, -6.3450e-02, ..., -9.2360e-02,
-6.9876e-02, -4.9841e-02],
[ 8.2942e-02, 8.7580e-02, 1.0111e-01, ..., 5.2714e-02,
6.0968e-02, 4.1198e-02]],
[[-1.6391e-02, -1.3870e-02, 5.2810e-03, ..., 4.3698e-02,
2.2707e-02, -4.5983e-02],
[ 3.3202e-02, 4.2014e-02, 9.3500e-02, ..., 2.6162e-01,
2.2970e-01, 1.6694e-01],
[-4.5987e-02, -1.6365e-02, 2.6811e-02, ..., 1.4951e-01,
1.3216e-01, 1.3579e-01],
...,
[-7.2129e-02, -1.8902e-01, -2.3389e-01, ..., -1.9038e-01,
-1.5609e-01, -7.5974e-02],
[ 5.1161e-02, -2.5815e-02, -6.9357e-02, ..., -5.8999e-02,
-6.1550e-02, -4.4555e-02],
[ 1.1174e-01, 7.8979e-02, 6.5849e-02, ..., 3.1617e-02,
2.5221e-02, 7.4257e-03]]],
[[[-7.0826e-08, -6.4306e-08, -7.3806e-08, ..., -9.8000e-08,
-1.0905e-07, -8.3421e-08],
[-6.1125e-09, 2.0613e-09, -8.0922e-09, ..., -4.9840e-08,
-4.3836e-08, -3.0538e-09],
[ 7.1953e-08, 7.5616e-08, 5.9282e-08, ..., -9.7509e-09,
-1.0951e-09, 4.2442e-08],
...,
[ 9.5889e-08, 1.0039e-07, 7.9817e-08, ..., -1.7491e-08,
-4.7666e-08, -1.3265e-08],
[ 1.2904e-07, 1.4762e-07, 1.7477e-07, ..., 1.3233e-07,
1.0628e-07, 9.3316e-08],
[ 1.2558e-07, 1.3644e-07, 1.8431e-07, ..., 2.1399e-07,
1.7710e-07, 1.7166e-07]],
[[-1.2690e-07, -9.6139e-08, -1.0372e-07, ..., -1.1808e-07,
-1.3309e-07, -1.0820e-07],
[-5.7412e-08, -2.5055e-08, -3.0115e-08, ..., -7.2922e-08,
-6.7022e-08, -2.2574e-08],
[ 2.1813e-08, 4.8608e-08, 3.1222e-08, ..., -1.8694e-08,
-7.9591e-09, 3.9750e-08],
...,
[ 5.6013e-08, 7.5526e-08, 4.4496e-08, ..., -4.4128e-08,
-5.9930e-08, -1.8247e-08],
[ 7.7614e-08, 9.8348e-08, 1.0455e-07, ..., 6.3272e-08,
4.1781e-08, 4.5901e-08],
[ 5.9834e-08, 7.1006e-08, 9.0437e-08, ..., 1.1654e-07,
8.7550e-08, 9.8837e-08]],
[[-4.3810e-08, 1.3270e-08, 7.8275e-09, ..., -5.8804e-09,
-2.6217e-08, -1.5649e-08],
[ 4.1700e-08, 1.0778e-07, 1.0946e-07, ..., 7.6403e-08,
7.1450e-08, 9.7615e-08],
[ 1.0436e-07, 1.6586e-07, 1.5933e-07, ..., 1.3517e-07,
1.3487e-07, 1.6449e-07],
...,
[ 9.8763e-08, 1.5072e-07, 1.2547e-07, ..., 6.8316e-08,
6.8382e-08, 1.1367e-07],
[ 9.1435e-08, 1.3576e-07, 1.3793e-07, ..., 1.1678e-07,
1.1723e-07, 1.4394e-07],
[ 6.2183e-08, 8.8184e-08, 1.0456e-07, ..., 1.3941e-07,
1.3333e-07, 1.5844e-07]]],
...,
[[[-6.1896e-02, -3.0206e-02, 1.9225e-02, ..., 4.3665e-02,
-2.2114e-02, -4.2214e-02],
[-3.8061e-02, 6.0774e-03, 4.5797e-02, ..., 9.6029e-02,
5.9254e-02, 2.9958e-02],
[-2.9672e-02, 2.7766e-03, 2.0457e-02, ..., 5.9828e-02,
4.1422e-02, 2.3134e-02],
...,
[ 1.1916e-02, 4.5701e-02, 4.4892e-02, ..., 4.7419e-02,
2.2274e-02, -5.4993e-03],
[-3.2468e-02, -1.2210e-02, 2.2023e-02, ..., 5.8061e-02,
-7.5033e-03, -5.9736e-02],
[-4.3314e-02, -2.8162e-02, -5.9126e-03, ..., 8.8460e-02,
8.4406e-03, -5.0019e-02]],
[[-6.1292e-02, -1.4004e-02, 1.7229e-02, ..., 1.8349e-02,
-3.2708e-02, -4.1060e-02],
[-3.1506e-02, 2.4460e-02, 4.5516e-02, ..., 6.6806e-02,
4.6687e-02, 3.3248e-02],
[-3.2216e-02, 2.0718e-02, 2.3343e-02, ..., 3.5265e-02,
3.6478e-02, 3.1291e-02],
...,
[ 1.7739e-02, 6.1040e-02, 4.8247e-02, ..., 3.7785e-02,
2.8894e-02, 1.3984e-02],
[-1.0890e-02, 2.2079e-02, 4.2737e-02, ..., 6.0247e-02,
1.6197e-02, -1.2493e-02],
[-2.2284e-02, 1.3220e-02, 3.0897e-02, ..., 1.0403e-01,
4.0119e-02, -5.3310e-03]],
[[-8.5322e-02, -4.2603e-02, 6.8145e-03, ..., 3.0751e-02,
-3.4818e-02, -4.9945e-02],
[-2.9215e-02, 1.8165e-02, 5.1092e-02, ..., 9.0200e-02,
5.3438e-02, 4.0169e-02],
[-3.9932e-02, -1.1100e-03, 9.6176e-03, ..., 2.4114e-02,
2.6298e-02, 2.5489e-02],
...,
[-3.1890e-03, 3.0454e-02, 1.6316e-02, ..., 5.5054e-03,
-6.2689e-03, -8.4638e-03],
[-2.2995e-02, -2.8211e-03, 2.3203e-02, ..., 3.5888e-02,
-1.4296e-02, -3.2419e-02],
[-9.8894e-03, 7.0542e-03, 1.0659e-02, ..., 7.0495e-02,
1.2996e-02, -8.3417e-03]]],
[[[-7.8699e-03, 1.9911e-02, 3.4208e-02, ..., 2.8694e-02,
1.2820e-02, 1.8142e-02],
[ 8.7942e-03, -3.2875e-02, -3.5713e-02, ..., 7.2533e-02,
4.5889e-02, 5.2383e-02],
[-3.6122e-02, -1.1878e-01, -1.3767e-01, ..., 3.3811e-02,
3.7806e-02, 2.6944e-02],
...,
[ 1.7322e-02, 3.9589e-03, -8.2269e-03, ..., 2.7543e-03,
1.8313e-02, 1.6057e-02],
[-9.5007e-04, 1.6428e-02, 1.7156e-02, ..., 3.3672e-03,
2.2857e-02, 6.5783e-04],
[ 6.1727e-03, 2.7145e-02, 1.4340e-02, ..., 7.5867e-03,
1.8770e-02, 1.5624e-02]],
[[-1.3423e-02, -5.0696e-04, 8.0959e-03, ..., -6.0963e-03,
9.2341e-03, 1.5751e-02],
[-1.8343e-02, -6.7982e-02, -7.0685e-02, ..., 2.9855e-02,
2.6264e-02, 2.3773e-02],
[-5.4359e-02, -1.4663e-01, -1.6211e-01, ..., 1.1781e-02,
3.2477e-02, 1.1980e-02],
...,
[ 8.3686e-04, -1.7564e-02, -1.9535e-02, ..., -4.1382e-03,
2.4658e-02, 1.2893e-02],
[-6.3183e-04, 1.1788e-02, 2.4810e-02, ..., 6.1105e-03,
3.9210e-02, 9.6696e-03],
[-7.1831e-03, 6.6918e-03, 5.2723e-03, ..., -7.6077e-03,
2.7253e-02, 1.7735e-02]],
[[-2.3753e-04, -4.9343e-03, 2.2991e-03, ..., -4.7958e-02,
-2.6154e-02, -2.3525e-02],
[-3.3053e-04, -5.1502e-02, -5.9977e-02, ..., -1.7369e-02,
-2.3337e-02, -3.7312e-02],
[-2.2674e-02, -9.9412e-02, -1.1176e-01, ..., -1.1725e-02,
-8.3744e-03, -4.0615e-02],
...,
[ 1.1437e-02, -8.0313e-03, -1.4955e-03, ..., -3.4133e-02,
-8.7267e-03, -2.3526e-02],
[ 2.9522e-03, 6.7770e-04, 1.9933e-02, ..., -2.2002e-02,
1.4814e-02, -1.4487e-02],
[-1.9085e-02, -2.9430e-02, -2.3284e-02, ..., -4.8587e-02,
-1.3049e-02, -2.4368e-02]]],
[[[-3.6296e-02, 7.1996e-03, 1.9100e-02, ..., 1.9602e-02,
1.4870e-02, -1.7298e-02],
[-1.1061e-02, 8.5665e-02, 1.2667e-01, ..., 1.3744e-02,
-5.5036e-05, -3.0162e-02],
[ 1.1322e-01, 1.8634e-01, 5.0658e-02, ..., -1.7333e-01,
-7.2041e-02, -6.2474e-02],
...,
[-5.3062e-02, -2.5781e-01, -2.6747e-01, ..., 2.6781e-01,
1.4344e-01, 5.5145e-02],
[-2.1009e-02, -2.9969e-02, 1.0245e-01, ..., 2.0843e-01,
-4.1518e-03, -3.8118e-02],
[-2.2155e-02, 1.2380e-02, 8.4302e-02, ..., -4.4992e-02,
-1.4687e-01, -9.0890e-02]],
[[-5.3969e-03, 3.2799e-02, 1.5486e-02, ..., -7.7451e-03,
3.0229e-03, 1.1216e-03],
[ 6.1723e-02, 1.4899e-01, 1.4645e-01, ..., -2.8897e-02,
-2.0227e-02, -9.1878e-03],
[ 1.6146e-01, 2.0886e-01, -2.5589e-02, ..., -2.7278e-01,
-1.0735e-01, -6.2971e-02],
...,
[-1.3723e-01, -4.0863e-01, -3.8551e-01, ..., 4.0846e-01,
2.6202e-01, 1.3491e-01],
[-5.9388e-02, -6.1187e-02, 1.4197e-01, ..., 3.5780e-01,
9.0893e-02, -1.7392e-03],
[ 7.8613e-03, 5.8403e-02, 1.5339e-01, ..., 4.7045e-02,
-1.0095e-01, -9.7920e-02]],
[[-5.6799e-03, 1.3425e-02, -2.6461e-02, ..., 4.4881e-03,
2.0666e-03, 1.3902e-02],
[ 6.5943e-03, 4.5181e-02, 6.0260e-02, ..., 1.4368e-02,
-5.0725e-03, 4.0505e-03],
[ 5.5257e-02, 1.2397e-01, 4.3193e-02, ..., -1.4486e-01,
-7.4489e-02, -5.7533e-02],
...,
[-3.1513e-02, -1.6334e-01, -1.5795e-01, ..., 2.2904e-01,
1.2017e-01, 7.1998e-02],
[-1.0456e-02, -1.1248e-03, 8.4582e-02, ..., 1.5748e-01,
2.2142e-02, -1.0083e-02],
[-4.8639e-03, -5.0065e-03, 3.6341e-02, ..., -2.4361e-02,
-7.1195e-02, -6.6788e-02]]]], requires_grad=True)
+++++++++++++++++++++++++++++++++++++
model.fc:
Linear(in_features=512, out_features=1000, bias=True)
model.fc.weight:
Parameter containing:
tensor([[-0.0185, -0.0705, -0.0518, ..., -0.0390, 0.1735, -0.0410],
[-0.0818, -0.0944, 0.0174, ..., 0.2028, -0.0248, 0.0372],
[-0.0332, -0.0566, -0.0242, ..., -0.0344, -0.0227, 0.0197],
...,
[-0.0103, 0.0033, -0.0359, ..., -0.0279, -0.0115, 0.0128],
[-0.0359, -0.0353, -0.0296, ..., -0.0330, -0.0110, -0.0513],
[ 0.0021, -0.0248, -0.0829, ..., 0.0417, -0.0500, 0.0663]],
requires_grad=True)
++++++++++++++++ After requires_grad = False +++++++++++++++++++++
model.conv1:
Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
model.conv1.weight:
Parameter containing:
tensor([[[[-1.0419e-02, -6.1356e-03, -1.8098e-03, ..., 5.6615e-02,
1.7083e-02, -1.2694e-02],
[ 1.1083e-02, 9.5276e-03, -1.0993e-01, ..., -2.7124e-01,
-1.2907e-01, 3.7424e-03],
[-6.9434e-03, 5.9089e-02, 2.9548e-01, ..., 5.1972e-01,
2.5632e-01, 6.3573e-02],
...,
[-2.7535e-02, 1.6045e-02, 7.2595e-02, ..., -3.3285e-01,
-4.2058e-01, -2.5781e-01],
[ 3.0613e-02, 4.0960e-02, 6.2850e-02, ..., 4.1384e-01,
3.9359e-01, 1.6606e-01],
[-1.3736e-02, -3.6746e-03, -2.4084e-02, ..., -1.5070e-01,
-8.2230e-02, -5.7828e-03]],
[[-1.1397e-02, -2.6619e-02, -3.4641e-02, ..., 3.2521e-02,
6.6221e-04, -2.5743e-02],
[ 4.5687e-02, 3.3603e-02, -1.0453e-01, ..., -3.1253e-01,
-1.6051e-01, -1.2826e-03],
[-8.3730e-04, 9.8420e-02, 4.0210e-01, ..., 7.0789e-01,
3.6887e-01, 1.2455e-01],
...,
[-5.5926e-02, -5.2239e-03, 2.7081e-02, ..., -4.6178e-01,
-5.7080e-01, -3.6552e-01],
[ 3.2860e-02, 5.5574e-02, 9.9670e-02, ..., 5.4636e-01,
4.8276e-01, 1.9867e-01],
[ 5.3051e-03, 6.6938e-03, -1.7254e-02, ..., -1.4822e-01,
-7.7248e-02, 7.2183e-04]],
[[-2.0315e-03, -9.1617e-03, 2.1209e-02, ..., 8.9177e-02,
3.3655e-02, -2.0102e-02],
[ 1.5398e-02, -1.8648e-02, -1.2591e-01, ..., -2.5342e-01,
-1.2980e-01, -2.7975e-02],
[ 9.8454e-03, 4.9047e-02, 2.1699e-01, ..., 3.4872e-01,
1.0433e-01, 1.8413e-02],
...,
[-2.8356e-02, 1.8404e-02, 9.8647e-02, ..., -1.1740e-01,
-2.5760e-01, -1.5451e-01],
[ 2.0766e-02, -2.6286e-03, -3.7825e-02, ..., 2.4141e-01,
2.4345e-01, 1.1796e-01],
[ 7.4684e-04, 7.7677e-04, -1.0050e-02, ..., -1.4865e-01,
-1.1754e-01, -3.8350e-02]]],
[[[-4.4154e-03, -4.0645e-03, 3.1589e-03, ..., -3.7026e-02,
-2.5158e-02, -4.7945e-02],
[ 5.1310e-02, 5.3402e-02, 8.0436e-02, ..., 1.4480e-01,
1.4287e-01, 1.2312e-01],
[-7.3337e-03, 2.1755e-03, 3.7580e-02, ..., 6.1517e-02,
8.0324e-02, 1.1715e-01],
...,
[-2.6754e-02, -1.2297e-01, -1.3653e-01, ..., -1.4068e-01,
-1.1155e-01, -4.9556e-02],
[ 2.3524e-02, -1.7288e-02, -1.1122e-02, ..., -1.8826e-02,
-2.3320e-02, -2.9474e-02],
[ 2.8689e-02, 2.1659e-02, 4.7888e-02, ..., 2.5498e-02,
3.5346e-02, 1.1280e-02]],
[[ 4.6919e-04, 1.2153e-02, 4.2035e-02, ..., 4.6403e-02,
4.0423e-02, -1.4439e-02],
[ 4.3463e-02, 6.8779e-02, 1.3268e-01, ..., 2.8606e-01,
2.6905e-01, 2.0935e-01],
[-5.7621e-02, -2.2642e-02, 3.0547e-02, ..., 1.3763e-01,
1.6538e-01, 1.7946e-01],
...,
[-1.0816e-01, -2.5227e-01, -2.9742e-01, ..., -2.8503e-01,
-2.1493e-01, -1.0320e-01],
[ 4.0709e-02, -3.2771e-02, -6.3450e-02, ..., -9.2360e-02,
-6.9876e-02, -4.9841e-02],
[ 8.2942e-02, 8.7580e-02, 1.0111e-01, ..., 5.2714e-02,
6.0968e-02, 4.1198e-02]],
[[-1.6391e-02, -1.3870e-02, 5.2810e-03, ..., 4.3698e-02,
2.2707e-02, -4.5983e-02],
[ 3.3202e-02, 4.2014e-02, 9.3500e-02, ..., 2.6162e-01,
2.2970e-01, 1.6694e-01],
[-4.5987e-02, -1.6365e-02, 2.6811e-02, ..., 1.4951e-01,
1.3216e-01, 1.3579e-01],
...,
[-7.2129e-02, -1.8902e-01, -2.3389e-01, ..., -1.9038e-01,
-1.5609e-01, -7.5974e-02],
[ 5.1161e-02, -2.5815e-02, -6.9357e-02, ..., -5.8999e-02,
-6.1550e-02, -4.4555e-02],
[ 1.1174e-01, 7.8979e-02, 6.5849e-02, ..., 3.1617e-02,
2.5221e-02, 7.4257e-03]]],
[[[-7.0826e-08, -6.4306e-08, -7.3806e-08, ..., -9.8000e-08,
-1.0905e-07, -8.3421e-08],
[-6.1125e-09, 2.0613e-09, -8.0922e-09, ..., -4.9840e-08,
-4.3836e-08, -3.0538e-09],
[ 7.1953e-08, 7.5616e-08, 5.9282e-08, ..., -9.7509e-09,
-1.0951e-09, 4.2442e-08],
...,
[ 9.5889e-08, 1.0039e-07, 7.9817e-08, ..., -1.7491e-08,
-4.7666e-08, -1.3265e-08],
[ 1.2904e-07, 1.4762e-07, 1.7477e-07, ..., 1.3233e-07,
1.0628e-07, 9.3316e-08],
[ 1.2558e-07, 1.3644e-07, 1.8431e-07, ..., 2.1399e-07,
1.7710e-07, 1.7166e-07]],
[[-1.2690e-07, -9.6139e-08, -1.0372e-07, ..., -1.1808e-07,
-1.3309e-07, -1.0820e-07],
[-5.7412e-08, -2.5055e-08, -3.0115e-08, ..., -7.2922e-08,
-6.7022e-08, -2.2574e-08],
[ 2.1813e-08, 4.8608e-08, 3.1222e-08, ..., -1.8694e-08,
-7.9591e-09, 3.9750e-08],
...,
[ 5.6013e-08, 7.5526e-08, 4.4496e-08, ..., -4.4128e-08,
-5.9930e-08, -1.8247e-08],
[ 7.7614e-08, 9.8348e-08, 1.0455e-07, ..., 6.3272e-08,
4.1781e-08, 4.5901e-08],
[ 5.9834e-08, 7.1006e-08, 9.0437e-08, ..., 1.1654e-07,
8.7550e-08, 9.8837e-08]],
[[-4.3810e-08, 1.3270e-08, 7.8275e-09, ..., -5.8804e-09,
-2.6217e-08, -1.5649e-08],
[ 4.1700e-08, 1.0778e-07, 1.0946e-07, ..., 7.6403e-08,
7.1450e-08, 9.7615e-08],
[ 1.0436e-07, 1.6586e-07, 1.5933e-07, ..., 1.3517e-07,
1.3487e-07, 1.6449e-07],
...,
[ 9.8763e-08, 1.5072e-07, 1.2547e-07, ..., 6.8316e-08,
6.8382e-08, 1.1367e-07],
[ 9.1435e-08, 1.3576e-07, 1.3793e-07, ..., 1.1678e-07,
1.1723e-07, 1.4394e-07],
[ 6.2183e-08, 8.8184e-08, 1.0456e-07, ..., 1.3941e-07,
1.3333e-07, 1.5844e-07]]],
...,
[[[-6.1896e-02, -3.0206e-02, 1.9225e-02, ..., 4.3665e-02,
-2.2114e-02, -4.2214e-02],
[-3.8061e-02, 6.0774e-03, 4.5797e-02, ..., 9.6029e-02,
5.9254e-02, 2.9958e-02],
[-2.9672e-02, 2.7766e-03, 2.0457e-02, ..., 5.9828e-02,
4.1422e-02, 2.3134e-02],
...,
[ 1.1916e-02, 4.5701e-02, 4.4892e-02, ..., 4.7419e-02,
2.2274e-02, -5.4993e-03],
[-3.2468e-02, -1.2210e-02, 2.2023e-02, ..., 5.8061e-02,
-7.5033e-03, -5.9736e-02],
[-4.3314e-02, -2.8162e-02, -5.9126e-03, ..., 8.8460e-02,
8.4406e-03, -5.0019e-02]],
[[-6.1292e-02, -1.4004e-02, 1.7229e-02, ..., 1.8349e-02,
-3.2708e-02, -4.1060e-02],
[-3.1506e-02, 2.4460e-02, 4.5516e-02, ..., 6.6806e-02,
4.6687e-02, 3.3248e-02],
[-3.2216e-02, 2.0718e-02, 2.3343e-02, ..., 3.5265e-02,
3.6478e-02, 3.1291e-02],
...,
[ 1.7739e-02, 6.1040e-02, 4.8247e-02, ..., 3.7785e-02,
2.8894e-02, 1.3984e-02],
[-1.0890e-02, 2.2079e-02, 4.2737e-02, ..., 6.0247e-02,
1.6197e-02, -1.2493e-02],
[-2.2284e-02, 1.3220e-02, 3.0897e-02, ..., 1.0403e-01,
4.0119e-02, -5.3310e-03]],
[[-8.5322e-02, -4.2603e-02, 6.8145e-03, ..., 3.0751e-02,
-3.4818e-02, -4.9945e-02],
[-2.9215e-02, 1.8165e-02, 5.1092e-02, ..., 9.0200e-02,
5.3438e-02, 4.0169e-02],
[-3.9932e-02, -1.1100e-03, 9.6176e-03, ..., 2.4114e-02,
2.6298e-02, 2.5489e-02],
...,
[-3.1890e-03, 3.0454e-02, 1.6316e-02, ..., 5.5054e-03,
-6.2689e-03, -8.4638e-03],
[-2.2995e-02, -2.8211e-03, 2.3203e-02, ..., 3.5888e-02,
-1.4296e-02, -3.2419e-02],
[-9.8894e-03, 7.0542e-03, 1.0659e-02, ..., 7.0495e-02,
1.2996e-02, -8.3417e-03]]],
[[[-7.8699e-03, 1.9911e-02, 3.4208e-02, ..., 2.8694e-02,
1.2820e-02, 1.8142e-02],
[ 8.7942e-03, -3.2875e-02, -3.5713e-02, ..., 7.2533e-02,
4.5889e-02, 5.2383e-02],
[-3.6122e-02, -1.1878e-01, -1.3767e-01, ..., 3.3811e-02,
3.7806e-02, 2.6944e-02],
...,
[ 1.7322e-02, 3.9589e-03, -8.2269e-03, ..., 2.7543e-03,
1.8313e-02, 1.6057e-02],
[-9.5007e-04, 1.6428e-02, 1.7156e-02, ..., 3.3672e-03,
2.2857e-02, 6.5783e-04],
[ 6.1727e-03, 2.7145e-02, 1.4340e-02, ..., 7.5867e-03,
1.8770e-02, 1.5624e-02]],
[[-1.3423e-02, -5.0696e-04, 8.0959e-03, ..., -6.0963e-03,
9.2341e-03, 1.5751e-02],
[-1.8343e-02, -6.7982e-02, -7.0685e-02, ..., 2.9855e-02,
2.6264e-02, 2.3773e-02],
[-5.4359e-02, -1.4663e-01, -1.6211e-01, ..., 1.1781e-02,
3.2477e-02, 1.1980e-02],
...,
[ 8.3686e-04, -1.7564e-02, -1.9535e-02, ..., -4.1382e-03,
2.4658e-02, 1.2893e-02],
[-6.3183e-04, 1.1788e-02, 2.4810e-02, ..., 6.1105e-03,
3.9210e-02, 9.6696e-03],
[-7.1831e-03, 6.6918e-03, 5.2723e-03, ..., -7.6077e-03,
2.7253e-02, 1.7735e-02]],
[[-2.3753e-04, -4.9343e-03, 2.2991e-03, ..., -4.7958e-02,
-2.6154e-02, -2.3525e-02],
[-3.3053e-04, -5.1502e-02, -5.9977e-02, ..., -1.7369e-02,
-2.3337e-02, -3.7312e-02],
[-2.2674e-02, -9.9412e-02, -1.1176e-01, ..., -1.1725e-02,
-8.3744e-03, -4.0615e-02],
...,
[ 1.1437e-02, -8.0313e-03, -1.4955e-03, ..., -3.4133e-02,
-8.7267e-03, -2.3526e-02],
[ 2.9522e-03, 6.7770e-04, 1.9933e-02, ..., -2.2002e-02,
1.4814e-02, -1.4487e-02],
[-1.9085e-02, -2.9430e-02, -2.3284e-02, ..., -4.8587e-02,
-1.3049e-02, -2.4368e-02]]],
[[[-3.6296e-02, 7.1996e-03, 1.9100e-02, ..., 1.9602e-02,
1.4870e-02, -1.7298e-02],
[-1.1061e-02, 8.5665e-02, 1.2667e-01, ..., 1.3744e-02,
-5.5036e-05, -3.0162e-02],
[ 1.1322e-01, 1.8634e-01, 5.0658e-02, ..., -1.7333e-01,
-7.2041e-02, -6.2474e-02],
...,
[-5.3062e-02, -2.5781e-01, -2.6747e-01, ..., 2.6781e-01,
1.4344e-01, 5.5145e-02],
[-2.1009e-02, -2.9969e-02, 1.0245e-01, ..., 2.0843e-01,
-4.1518e-03, -3.8118e-02],
[-2.2155e-02, 1.2380e-02, 8.4302e-02, ..., -4.4992e-02,
-1.4687e-01, -9.0890e-02]],
[[-5.3969e-03, 3.2799e-02, 1.5486e-02, ..., -7.7451e-03,
3.0229e-03, 1.1216e-03],
[ 6.1723e-02, 1.4899e-01, 1.4645e-01, ..., -2.8897e-02,
-2.0227e-02, -9.1878e-03],
[ 1.6146e-01, 2.0886e-01, -2.5589e-02, ..., -2.7278e-01,
-1.0735e-01, -6.2971e-02],
...,
[-1.3723e-01, -4.0863e-01, -3.8551e-01, ..., 4.0846e-01,
2.6202e-01, 1.3491e-01],
[-5.9388e-02, -6.1187e-02, 1.4197e-01, ..., 3.5780e-01,
9.0893e-02, -1.7392e-03],
[ 7.8613e-03, 5.8403e-02, 1.5339e-01, ..., 4.7045e-02,
-1.0095e-01, -9.7920e-02]],
[[-5.6799e-03, 1.3425e-02, -2.6461e-02, ..., 4.4881e-03,
2.0666e-03, 1.3902e-02],
[ 6.5943e-03, 4.5181e-02, 6.0260e-02, ..., 1.4368e-02,
-5.0725e-03, 4.0505e-03],
[ 5.5257e-02, 1.2397e-01, 4.3193e-02, ..., -1.4486e-01,
-7.4489e-02, -5.7533e-02],
...,
[-3.1513e-02, -1.6334e-01, -1.5795e-01, ..., 2.2904e-01,
1.2017e-01, 7.1998e-02],
[-1.0456e-02, -1.1248e-03, 8.4582e-02, ..., 1.5748e-01,
2.2142e-02, -1.0083e-02],
[-4.8639e-03, -5.0065e-03, 3.6341e-02, ..., -2.4361e-02,
-7.1195e-02, -6.6788e-02]]]])
+++++++++++++++++++++++++++++++++++++
model.fc:
Linear(in_features=512, out_features=1000, bias=True)
model.fc.weight:
Parameter containing:
tensor([[-0.0185, -0.0705, -0.0518, ..., -0.0390, 0.1735, -0.0410],
[-0.0818, -0.0944, 0.0174, ..., 0.2028, -0.0248, 0.0372],
[-0.0332, -0.0566, -0.0242, ..., -0.0344, -0.0227, 0.0197],
...,
[-0.0103, 0.0033, -0.0359, ..., -0.0279, -0.0115, 0.0128],
[-0.0359, -0.0353, -0.0296, ..., -0.0330, -0.0110, -0.0513],
[ 0.0021, -0.0248, -0.0829, ..., 0.0417, -0.0500, 0.0663]])
Notice that the requires_grad=True disapears after setting param.requires_grad = False
Let's say we want to finetune the model on a new dataset with 10 labels.
In resnet, the classifier is the last linear layer model.fc.
We can simply replace it with a new linear layer (unfrozen by default)
that acts as our classifier.
model.fc = nn.Linear(512, 10)Now all parameters in the model, except the parameters of model.fc, are frozen.
The only parameters that compute gradients are the weights and bias of model.fc.
This means also that during traininig, the only parameters that are computing gradients (and hence updated in gradient descent)
are the weights and bias of the classifier (model.fc).
NOTE:
The same exclusionary functionality is available as a context manager in
[torch.no_grad()] (https://pytorch.org/docs/stable/generated/torch.no_grad.html), that we will be using for Transfer Learning
print("model.conv1:\n", model.conv1 )
print("model.conv1.weight:\n", model.conv1.weight)
print("+++++++++++++++++++++++++++++++++++++")
print("model.fc:\n", model.fc )
print("model.fc.weight:\n", model.fc.weight)model.conv1:
Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
model.conv1.weight:
Parameter containing:
tensor([[[[-1.0419e-02, -6.1356e-03, -1.8098e-03, ..., 5.6615e-02,
1.7083e-02, -1.2694e-02],
[ 1.1083e-02, 9.5276e-03, -1.0993e-01, ..., -2.7124e-01,
-1.2907e-01, 3.7424e-03],
[-6.9434e-03, 5.9089e-02, 2.9548e-01, ..., 5.1972e-01,
2.5632e-01, 6.3573e-02],
...,
[-2.7535e-02, 1.6045e-02, 7.2595e-02, ..., -3.3285e-01,
-4.2058e-01, -2.5781e-01],
[ 3.0613e-02, 4.0960e-02, 6.2850e-02, ..., 4.1384e-01,
3.9359e-01, 1.6606e-01],
[-1.3736e-02, -3.6746e-03, -2.4084e-02, ..., -1.5070e-01,
-8.2230e-02, -5.7828e-03]],
[[-1.1397e-02, -2.6619e-02, -3.4641e-02, ..., 3.2521e-02,
6.6221e-04, -2.5743e-02],
[ 4.5687e-02, 3.3603e-02, -1.0453e-01, ..., -3.1253e-01,
-1.6051e-01, -1.2826e-03],
[-8.3730e-04, 9.8420e-02, 4.0210e-01, ..., 7.0789e-01,
3.6887e-01, 1.2455e-01],
...,
[-5.5926e-02, -5.2239e-03, 2.7081e-02, ..., -4.6178e-01,
-5.7080e-01, -3.6552e-01],
[ 3.2860e-02, 5.5574e-02, 9.9670e-02, ..., 5.4636e-01,
4.8276e-01, 1.9867e-01],
[ 5.3051e-03, 6.6938e-03, -1.7254e-02, ..., -1.4822e-01,
-7.7248e-02, 7.2183e-04]],
[[-2.0315e-03, -9.1617e-03, 2.1209e-02, ..., 8.9177e-02,
3.3655e-02, -2.0102e-02],
[ 1.5398e-02, -1.8648e-02, -1.2591e-01, ..., -2.5342e-01,
-1.2980e-01, -2.7975e-02],
[ 9.8454e-03, 4.9047e-02, 2.1699e-01, ..., 3.4872e-01,
1.0433e-01, 1.8413e-02],
...,
[-2.8356e-02, 1.8404e-02, 9.8647e-02, ..., -1.1740e-01,
-2.5760e-01, -1.5451e-01],
[ 2.0766e-02, -2.6286e-03, -3.7825e-02, ..., 2.4141e-01,
2.4345e-01, 1.1796e-01],
[ 7.4684e-04, 7.7677e-04, -1.0050e-02, ..., -1.4865e-01,
-1.1754e-01, -3.8350e-02]]],
[[[-4.4154e-03, -4.0645e-03, 3.1589e-03, ..., -3.7026e-02,
-2.5158e-02, -4.7945e-02],
[ 5.1310e-02, 5.3402e-02, 8.0436e-02, ..., 1.4480e-01,
1.4287e-01, 1.2312e-01],
[-7.3337e-03, 2.1755e-03, 3.7580e-02, ..., 6.1517e-02,
8.0324e-02, 1.1715e-01],
...,
[-2.6754e-02, -1.2297e-01, -1.3653e-01, ..., -1.4068e-01,
-1.1155e-01, -4.9556e-02],
[ 2.3524e-02, -1.7288e-02, -1.1122e-02, ..., -1.8826e-02,
-2.3320e-02, -2.9474e-02],
[ 2.8689e-02, 2.1659e-02, 4.7888e-02, ..., 2.5498e-02,
3.5346e-02, 1.1280e-02]],
[[ 4.6919e-04, 1.2153e-02, 4.2035e-02, ..., 4.6403e-02,
4.0423e-02, -1.4439e-02],
[ 4.3463e-02, 6.8779e-02, 1.3268e-01, ..., 2.8606e-01,
2.6905e-01, 2.0935e-01],
[-5.7621e-02, -2.2642e-02, 3.0547e-02, ..., 1.3763e-01,
1.6538e-01, 1.7946e-01],
...,
[-1.0816e-01, -2.5227e-01, -2.9742e-01, ..., -2.8503e-01,
-2.1493e-01, -1.0320e-01],
[ 4.0709e-02, -3.2771e-02, -6.3450e-02, ..., -9.2360e-02,
-6.9876e-02, -4.9841e-02],
[ 8.2942e-02, 8.7580e-02, 1.0111e-01, ..., 5.2714e-02,
6.0968e-02, 4.1198e-02]],
[[-1.6391e-02, -1.3870e-02, 5.2810e-03, ..., 4.3698e-02,
2.2707e-02, -4.5983e-02],
[ 3.3202e-02, 4.2014e-02, 9.3500e-02, ..., 2.6162e-01,
2.2970e-01, 1.6694e-01],
[-4.5987e-02, -1.6365e-02, 2.6811e-02, ..., 1.4951e-01,
1.3216e-01, 1.3579e-01],
...,
[-7.2129e-02, -1.8902e-01, -2.3389e-01, ..., -1.9038e-01,
-1.5609e-01, -7.5974e-02],
[ 5.1161e-02, -2.5815e-02, -6.9357e-02, ..., -5.8999e-02,
-6.1550e-02, -4.4555e-02],
[ 1.1174e-01, 7.8979e-02, 6.5849e-02, ..., 3.1617e-02,
2.5221e-02, 7.4257e-03]]],
[[[-7.0826e-08, -6.4306e-08, -7.3806e-08, ..., -9.8000e-08,
-1.0905e-07, -8.3421e-08],
[-6.1125e-09, 2.0613e-09, -8.0922e-09, ..., -4.9840e-08,
-4.3836e-08, -3.0538e-09],
[ 7.1953e-08, 7.5616e-08, 5.9282e-08, ..., -9.7509e-09,
-1.0951e-09, 4.2442e-08],
...,
[ 9.5889e-08, 1.0039e-07, 7.9817e-08, ..., -1.7491e-08,
-4.7666e-08, -1.3265e-08],
[ 1.2904e-07, 1.4762e-07, 1.7477e-07, ..., 1.3233e-07,
1.0628e-07, 9.3316e-08],
[ 1.2558e-07, 1.3644e-07, 1.8431e-07, ..., 2.1399e-07,
1.7710e-07, 1.7166e-07]],
[[-1.2690e-07, -9.6139e-08, -1.0372e-07, ..., -1.1808e-07,
-1.3309e-07, -1.0820e-07],
[-5.7412e-08, -2.5055e-08, -3.0115e-08, ..., -7.2922e-08,
-6.7022e-08, -2.2574e-08],
[ 2.1813e-08, 4.8608e-08, 3.1222e-08, ..., -1.8694e-08,
-7.9591e-09, 3.9750e-08],
...,
[ 5.6013e-08, 7.5526e-08, 4.4496e-08, ..., -4.4128e-08,
-5.9930e-08, -1.8247e-08],
[ 7.7614e-08, 9.8348e-08, 1.0455e-07, ..., 6.3272e-08,
4.1781e-08, 4.5901e-08],
[ 5.9834e-08, 7.1006e-08, 9.0437e-08, ..., 1.1654e-07,
8.7550e-08, 9.8837e-08]],
[[-4.3810e-08, 1.3270e-08, 7.8275e-09, ..., -5.8804e-09,
-2.6217e-08, -1.5649e-08],
[ 4.1700e-08, 1.0778e-07, 1.0946e-07, ..., 7.6403e-08,
7.1450e-08, 9.7615e-08],
[ 1.0436e-07, 1.6586e-07, 1.5933e-07, ..., 1.3517e-07,
1.3487e-07, 1.6449e-07],
...,
[ 9.8763e-08, 1.5072e-07, 1.2547e-07, ..., 6.8316e-08,
6.8382e-08, 1.1367e-07],
[ 9.1435e-08, 1.3576e-07, 1.3793e-07, ..., 1.1678e-07,
1.1723e-07, 1.4394e-07],
[ 6.2183e-08, 8.8184e-08, 1.0456e-07, ..., 1.3941e-07,
1.3333e-07, 1.5844e-07]]],
...,
[[[-6.1896e-02, -3.0206e-02, 1.9225e-02, ..., 4.3665e-02,
-2.2114e-02, -4.2214e-02],
[-3.8061e-02, 6.0774e-03, 4.5797e-02, ..., 9.6029e-02,
5.9254e-02, 2.9958e-02],
[-2.9672e-02, 2.7766e-03, 2.0457e-02, ..., 5.9828e-02,
4.1422e-02, 2.3134e-02],
...,
[ 1.1916e-02, 4.5701e-02, 4.4892e-02, ..., 4.7419e-02,
2.2274e-02, -5.4993e-03],
[-3.2468e-02, -1.2210e-02, 2.2023e-02, ..., 5.8061e-02,
-7.5033e-03, -5.9736e-02],
[-4.3314e-02, -2.8162e-02, -5.9126e-03, ..., 8.8460e-02,
8.4406e-03, -5.0019e-02]],
[[-6.1292e-02, -1.4004e-02, 1.7229e-02, ..., 1.8349e-02,
-3.2708e-02, -4.1060e-02],
[-3.1506e-02, 2.4460e-02, 4.5516e-02, ..., 6.6806e-02,
4.6687e-02, 3.3248e-02],
[-3.2216e-02, 2.0718e-02, 2.3343e-02, ..., 3.5265e-02,
3.6478e-02, 3.1291e-02],
...,
[ 1.7739e-02, 6.1040e-02, 4.8247e-02, ..., 3.7785e-02,
2.8894e-02, 1.3984e-02],
[-1.0890e-02, 2.2079e-02, 4.2737e-02, ..., 6.0247e-02,
1.6197e-02, -1.2493e-02],
[-2.2284e-02, 1.3220e-02, 3.0897e-02, ..., 1.0403e-01,
4.0119e-02, -5.3310e-03]],
[[-8.5322e-02, -4.2603e-02, 6.8145e-03, ..., 3.0751e-02,
-3.4818e-02, -4.9945e-02],
[-2.9215e-02, 1.8165e-02, 5.1092e-02, ..., 9.0200e-02,
5.3438e-02, 4.0169e-02],
[-3.9932e-02, -1.1100e-03, 9.6176e-03, ..., 2.4114e-02,
2.6298e-02, 2.5489e-02],
...,
[-3.1890e-03, 3.0454e-02, 1.6316e-02, ..., 5.5054e-03,
-6.2689e-03, -8.4638e-03],
[-2.2995e-02, -2.8211e-03, 2.3203e-02, ..., 3.5888e-02,
-1.4296e-02, -3.2419e-02],
[-9.8894e-03, 7.0542e-03, 1.0659e-02, ..., 7.0495e-02,
1.2996e-02, -8.3417e-03]]],
[[[-7.8699e-03, 1.9911e-02, 3.4208e-02, ..., 2.8694e-02,
1.2820e-02, 1.8142e-02],
[ 8.7942e-03, -3.2875e-02, -3.5713e-02, ..., 7.2533e-02,
4.5889e-02, 5.2383e-02],
[-3.6122e-02, -1.1878e-01, -1.3767e-01, ..., 3.3811e-02,
3.7806e-02, 2.6944e-02],
...,
[ 1.7322e-02, 3.9589e-03, -8.2269e-03, ..., 2.7543e-03,
1.8313e-02, 1.6057e-02],
[-9.5007e-04, 1.6428e-02, 1.7156e-02, ..., 3.3672e-03,
2.2857e-02, 6.5783e-04],
[ 6.1727e-03, 2.7145e-02, 1.4340e-02, ..., 7.5867e-03,
1.8770e-02, 1.5624e-02]],
[[-1.3423e-02, -5.0696e-04, 8.0959e-03, ..., -6.0963e-03,
9.2341e-03, 1.5751e-02],
[-1.8343e-02, -6.7982e-02, -7.0685e-02, ..., 2.9855e-02,
2.6264e-02, 2.3773e-02],
[-5.4359e-02, -1.4663e-01, -1.6211e-01, ..., 1.1781e-02,
3.2477e-02, 1.1980e-02],
...,
[ 8.3686e-04, -1.7564e-02, -1.9535e-02, ..., -4.1382e-03,
2.4658e-02, 1.2893e-02],
[-6.3183e-04, 1.1788e-02, 2.4810e-02, ..., 6.1105e-03,
3.9210e-02, 9.6696e-03],
[-7.1831e-03, 6.6918e-03, 5.2723e-03, ..., -7.6077e-03,
2.7253e-02, 1.7735e-02]],
[[-2.3753e-04, -4.9343e-03, 2.2991e-03, ..., -4.7958e-02,
-2.6154e-02, -2.3525e-02],
[-3.3053e-04, -5.1502e-02, -5.9977e-02, ..., -1.7369e-02,
-2.3337e-02, -3.7312e-02],
[-2.2674e-02, -9.9412e-02, -1.1176e-01, ..., -1.1725e-02,
-8.3744e-03, -4.0615e-02],
...,
[ 1.1437e-02, -8.0313e-03, -1.4955e-03, ..., -3.4133e-02,
-8.7267e-03, -2.3526e-02],
[ 2.9522e-03, 6.7770e-04, 1.9933e-02, ..., -2.2002e-02,
1.4814e-02, -1.4487e-02],
[-1.9085e-02, -2.9430e-02, -2.3284e-02, ..., -4.8587e-02,
-1.3049e-02, -2.4368e-02]]],
[[[-3.6296e-02, 7.1996e-03, 1.9100e-02, ..., 1.9602e-02,
1.4870e-02, -1.7298e-02],
[-1.1061e-02, 8.5665e-02, 1.2667e-01, ..., 1.3744e-02,
-5.5036e-05, -3.0162e-02],
[ 1.1322e-01, 1.8634e-01, 5.0658e-02, ..., -1.7333e-01,
-7.2041e-02, -6.2474e-02],
...,
[-5.3062e-02, -2.5781e-01, -2.6747e-01, ..., 2.6781e-01,
1.4344e-01, 5.5145e-02],
[-2.1009e-02, -2.9969e-02, 1.0245e-01, ..., 2.0843e-01,
-4.1518e-03, -3.8118e-02],
[-2.2155e-02, 1.2380e-02, 8.4302e-02, ..., -4.4992e-02,
-1.4687e-01, -9.0890e-02]],
[[-5.3969e-03, 3.2799e-02, 1.5486e-02, ..., -7.7451e-03,
3.0229e-03, 1.1216e-03],
[ 6.1723e-02, 1.4899e-01, 1.4645e-01, ..., -2.8897e-02,
-2.0227e-02, -9.1878e-03],
[ 1.6146e-01, 2.0886e-01, -2.5589e-02, ..., -2.7278e-01,
-1.0735e-01, -6.2971e-02],
...,
[-1.3723e-01, -4.0863e-01, -3.8551e-01, ..., 4.0846e-01,
2.6202e-01, 1.3491e-01],
[-5.9388e-02, -6.1187e-02, 1.4197e-01, ..., 3.5780e-01,
9.0893e-02, -1.7392e-03],
[ 7.8613e-03, 5.8403e-02, 1.5339e-01, ..., 4.7045e-02,
-1.0095e-01, -9.7920e-02]],
[[-5.6799e-03, 1.3425e-02, -2.6461e-02, ..., 4.4881e-03,
2.0666e-03, 1.3902e-02],
[ 6.5943e-03, 4.5181e-02, 6.0260e-02, ..., 1.4368e-02,
-5.0725e-03, 4.0505e-03],
[ 5.5257e-02, 1.2397e-01, 4.3193e-02, ..., -1.4486e-01,
-7.4489e-02, -5.7533e-02],
...,
[-3.1513e-02, -1.6334e-01, -1.5795e-01, ..., 2.2904e-01,
1.2017e-01, 7.1998e-02],
[-1.0456e-02, -1.1248e-03, 8.4582e-02, ..., 1.5748e-01,
2.2142e-02, -1.0083e-02],
[-4.8639e-03, -5.0065e-03, 3.6341e-02, ..., -2.4361e-02,
-7.1195e-02, -6.6788e-02]]]])
+++++++++++++++++++++++++++++++++++++
model.fc:
Linear(in_features=512, out_features=10, bias=True)
model.fc.weight:
Parameter containing:
tensor([[ 0.0313, 0.0096, 0.0258, ..., 0.0060, 0.0032, -0.0439],
[-0.0108, 0.0062, -0.0084, ..., 0.0214, -0.0438, -0.0356],
[ 0.0103, -0.0243, -0.0273, ..., 0.0307, -0.0017, 0.0052],
...,
[ 0.0374, 0.0011, -0.0352, ..., 0.0320, -0.0220, -0.0333],
[-0.0243, 0.0378, 0.0123, ..., 0.0434, -0.0030, -0.0438],
[-0.0207, 0.0123, -0.0436, ..., -0.0331, -0.0007, 0.0013]],
requires_grad=True)