Tensors, gradients & finite differences

PyTorch tensors carry shape, dtype and device. Matrix multiplication x @ w maps an (N,D) feature matrix and (D,) weights to N predictions. A scalar loss turns these predictions into an objective: mean squared error is mean((prediction-y)^2). A gradient tells how this scalar changes for a small change in each weight.

A scalar objective supplies the starting point for a backward pass through the recorded operations.

import torch

w = torch.tensor([2.0], dtype=torch.float64, requires_grad=True)
loss = (w * w).sum()
loss.backward()
print(w.grad.tolist())  # derivative 2*w

Operations on tensors with requires_grad=True build a graph. backward() follows it from a scalar loss and accumulates gradients into leaf .grad fields. detach() breaks the graph; clone() copies storage. To differentiate a caller's supplied weights independently, detach().clone().requires_grad_(True) gives a fresh leaf.

import torch

x = torch.tensor([2.0], requires_grad=True)
y = x.detach().clone().requires_grad_(True)
(3 * y).sum().backward()
print(x.grad is None, y.grad.tolist())

Mean MSE has gradient 2*x.T@(x@w-y)/N. Summed error differs by N; a silent sum makes effective learning rates depend on batch size. Compare autograd against this analytic expression and central finite differences: [L(w+eps e_i) - L(w-eps e_i)]/(2 eps). Use float64 and a moderate eps for this numerical check.

A numerical check samples nearby objective values; it checks the derivative of this objective, not whether the objective answers the research question.

The build returns a scalar loss value and a detached gradient tensor. It must leave inputs and any existing caller gradient untouched. Test a non-square matrix, N>1, an exact fit and one finite-difference check. A finite-difference match is evidence about the implemented computation, not proof that the loss represents the intended research question.