Cameras and rays from real datasets

A radiance field is only as good as its cameras. NeRF and 3D Gaussian splatting both start from a set of photos with known poses, and the first thing every codebase does is turn those poses into one ray per pixel (NeRF) or a world-to-camera matrix plus intrinsics (3DGS, gsplat). The maths is the pinhole camera from gfx-cameras-projection; the difficulty is that real datasets disagree about almost every convention — which way the camera looks, which way y points, whether a pose maps world to camera or camera to world, where a quaternion keeps its scalar, and where in a pixel the ray goes. A sign flipped anywhere still produces something that looks like a set of rays. This lesson goes through the two formats you will meet in every job — NeRF-synthetic transforms.json and COLMAP — and builds a vectorised get_rays that is right on all of those points.

1. Two conventions

• OpenCV (computer vision): x right, y down, the camera looks down +z. Matches image coordinates, where rows grow downwards. Used by COLMAP, OpenCV, the 3DGS reference code and gsplat.
• OpenGL (graphics): x right, y up, the camera looks down −z. Used by Blender (and so by the NeRF-synthetic scenes rendered in it), bmild/nerf, nerf-pytorch and nerfstudio.

The two differ by a 180° rotation about the camera's x axis, so converting a camera-to-world matrix is "negate its second and third columns": c2w_gl = c2w_cv @ diag(1, −1, −1, 1). The same multiplication converts back. That's the whole of 3DGS's readCamerasFromTransforms: c2w[:3, 1:3] *= -1 # change from OpenGL/Blender camera axes (Y up, Z back) to COLMAP (Y down, Z forward).

Independently of the axes, a pose matrix goes one of two ways: camera-to-world (c2w: its columns are the camera's axes in world coordinates and its last column is the camera centre) or world-to-camera (w2c = c2w⁻¹, which is what you multiply points by to project them). Ray generation wants c2w; projection and rasterisation want w2c. 3DGS adds a third twist — it stores w2c transposed for its row-vector CUDA code (py-numpy-geometry, §3).

2. NeRF-synthetic: transforms.json

The Blender scenes (lego, chair, ship, …) come with one JSON file per split:

{
  "camera_angle_x": 0.6911112070083618,          # full horizontal field of view, radians
  "frames": [
    {"file_path": "./train/r_0",
     "rotation": 0.012566370614359171,           # unused
     "transform_matrix": [[...4 floats...], ...]} # 4x4 camera-to-world, OpenGL axes
  ]
}

There is no K: the images are square with square pixels and the principal point in the middle, so everything follows from the field of view. In the right triangle from the optical centre to the middle and the edge of the image, the half-width W/2 is opposite the half-angle:

tan(fov / 2) = (W / 2) / f ⇒ f = 0.5·W / tan(0.5·fov), fov = 2·atan(0.5·W / f)

which is bmild/nerf's focal = .5 * W / np.tan(.5 * camera_angle_x). Halving the resolution (--half_res) halves f along with W; the field of view doesn't change. nerfstudio's transforms.json variant writes fl_x, fl_y, cx, cy, w, h explicitly instead.

import json
import numpy as np

meta = json.loads("""{
  "camera_angle_x": 0.6911112070083618,
  "frames": [{"file_path": "./train/r_0",
              "transform_matrix": [[-0.9999, 0.0042, -0.0134, -0.0538],
                                   [-0.0140, -0.2997, 0.9539, 3.8455],
                                   [0.0, 0.9540, 0.2997, 1.2081],
                                   [0.0, 0.0, 0.0, 1.0]]}]
}""")
W = H = 800
f = 0.5 * W / np.tan(0.5 * meta["camera_angle_x"])
print(f"focal = {f:.3f} px   (fov {np.degrees(meta['camera_angle_x']):.2f}°)")
print(f"half-res 400 px: focal = {0.5 * 400 / np.tan(0.5 * meta['camera_angle_x']):.3f} px")

c2w = np.array(meta["frames"][0]["transform_matrix"], dtype=np.float32)
print("dtype from JSON:", np.array(meta["frames"][0]["transform_matrix"]).dtype, "-> stored as", c2w.dtype)
print("camera centre:", c2w[:3, 3])
# OpenGL camera: looks down its -z axis, so the viewing direction is -(third column).
view = -c2w[:3, 2]
print("looks along:", view.round(3), "  towards the origin?", bool(view @ -c2w[:3, 3] > 0))

Reading a c2w is mechanical: column 0 is the camera's right vector, column 1 its up vector (OpenGL) and column 2 its backward vector, all in world coordinates; column 3 is where the camera is. The Blender cameras sit on a sphere of radius ~4 around the object and look at the origin — a quick sanity check for any loader.

3. COLMAP: world-to-camera, scalar-first

COLMAP (structure-from-motion) is how real captures get their poses — the 3DGS and Mip-NeRF 360 datasets ship its sparse/0/ output. cameras.txt holds the intrinsics per camera model (PINHOLE fx fy cx cy, SIMPLE_RADIAL f cx cy k, OPENCV fx fy cx cy k1 k2 p1 p2), and images.txt one pose per image:

Three things to get right, each a classic bug:

1. w first. COLMAP writes QW QX QY QZ (its read_write_model.py has qvec2rotmat(qvec) with qvec[0] the scalar), as does the 3DGS PLY. SciPy's Rotation.from_quat, ROS, glTF and Unity use (x, y, z, w). A misread quaternion is still a unit quaternion — every orthonormality check passes — so only a known value catches it.
2. World-to-camera. x_cam = R X + t. The inverse is c2w = [Rᵀ | −Rᵀ t], so the camera centre is −Rᵀ t. Treating tvec as the camera position is the most common COLMAP mistake.
3. OpenCV axes. To feed COLMAP poses to an OpenGL-convention NeRF codebase (nerfstudio's colmap_utils, instant-ngp's colmap2nerf.py), invert and flip.

import numpy as np

def qvec_to_rotmat(q):
    q = np.asarray(q, dtype=np.float64)
    w, x, y, z = q / np.linalg.norm(q)
    return np.array([
        [1 - 2 * (y * y + z * z), 2 * (x * y - w * z), 2 * (x * z + w * y)],
        [2 * (x * y + w * z), 1 - 2 * (x * x + z * z), 2 * (y * z - w * x)],
        [2 * (x * z - w * y), 2 * (y * z + w * x), 1 - 2 * (x * x + y * y)]])

# One line of COLMAP's images.txt: IMAGE_ID QW QX QY QZ TX TY TZ CAMERA_ID NAME
line = "1 0.9238795 0 0.3826834 0 0 0 4 1 frame_0001.png"
fields = line.split()
qvec = np.array(fields[1:5], dtype=np.float64)
tvec = np.array(fields[5:8], dtype=np.float64)

R = qvec_to_rotmat(qvec)                 # world-to-camera rotation (45° about y)
centre = -R.T @ tvec                     # the camera centre in world coordinates
print("tvec          :", tvec.round(4))
print("camera centre :", centre.round(4), "  (not tvec!)")
c2w = np.eye(4)
c2w[:3, :3] = R.T
c2w[:3, 3] = centre
print("viewing dir (+z_cam in world):", c2w[:3, 2].round(4))
print("world origin in camera coords:", (R @ np.zeros(3) + tvec).round(4), "-> straight ahead, 4 units")

# SciPy, ROS, glTF, Unity store (x, y, z, w). Reading COLMAP's w-first numbers that way:
wrong = qvec_to_rotmat(np.roll(qvec, -1))
print("x-first misread is still a rotation:", np.allclose(wrong @ wrong.T, np.eye(3)),
      "| angle vs truth:", round(float(np.degrees(np.arccos((np.trace(wrong.T @ R) - 1) / 2))), 1), "deg")

The batched version of qvec_to_rotmat is the same nine expressions on (..., 4) input — w, x, y, z = np.moveaxis(q, -1, 0) unpacks the last axis — stacked into (..., 3, 3); it is 3DGS's build_rotation in NumPy.

4. Pixel centres

Image coordinates are continuous: pixel (row r, column c) is the square [c, c + 1) × [r, r + 1), so its centre is (c + 0.5, r + 0.5), and the principal point of a centred camera is (W/2, H/2) — a pixel corner when W is even. A ray through the centre of each pixel is the unbiased single-sample choice, and it is what COLMAP's and OpenCV's K assume.

The original NeRF code (get_rays in bmild/nerf and nerf-pytorch, and kwea123/nerf_pl's get_ray_directions) uses the integer coordinates — rays through pixel corners. On the synthetic scenes it trains fine, because the renderer and the evaluation share the offset, but the learned scene is shifted by half a pixel against the ground-truth images and against any other renderer of the same K — a 3DGS model, a mesh, a different NeRF. nerfstudio (pixel_offset=0.5), Mip-NeRF and gsplat use centres. When you mix codebases, this is the first thing to check.

5. A vectorised get_rays

The recipe, for all H × W pixels at once:

1. a pixel grid with np.meshgrid(..., indexing="xy"), shape (H, W);
2. camera-space directions by inverting K: ((u − cx)/fx, (v − cy)/fy, 1) in OpenCV, (x, −y, −1) in OpenGL;
3. rotate them into the world with c2w[:3, :3], one einsum that also broadcasts a batch of cameras;
4. every origin is the camera centre c2w[:3, 3].

meshgrid's indexing matters: 'xy' (NumPy's default) returns (len(ys), len(xs)) = (H, W) — image layout — while 'ij' returns (W, H). torch.meshgrid historically defaulted to 'ij', which is why nerf-pytorch transposes both grids with .t(). Pass indexing= explicitly in both libraries, and test with H ≠ W.

import numpy as np

H, W = 3, 4
u, v = np.meshgrid(np.arange(W), np.arange(H), indexing="xy")
print("indexing='xy':", u.shape, "  u row 0:", u[0], "  v column 0:", v[:, 0])
a, b = np.meshgrid(np.arange(W), np.arange(H), indexing="ij")
print("indexing='ij':", a.shape, "  (W, H): transposed for an image")

def get_rays(H, W, K, c2w, convention="opencv", offset=0.5):
    u, v = np.meshgrid(np.arange(W) + offset, np.arange(H) + offset, indexing="xy")  # (H, W)
    x = (u - K[0, 2]) / K[0, 0]
    y = (v - K[1, 2]) / K[1, 1]
    one = np.ones_like(x)
    d = np.stack([x, y, one], -1) if convention == "opencv" else np.stack([x, -y, -one], -1)
    rays_d = np.einsum("...ij,hwj->...hwi", c2w[..., :3, :3], d)          # R @ d per pixel
    rays_o = np.broadcast_to(c2w[..., None, None, :3, 3], rays_d.shape).copy()
    return rays_o, rays_d

K = np.array([[2.0, 0, 2], [0, 2.0, 1.5], [0, 0, 1]])     # W = 4, H = 3: principal point at the centre
o, d = get_rays(H, W, K, np.eye(4))
print("rays:", o.shape, d.shape)
print("corner pixel (0, 0):", d[0, 0], "  mirror pixel (2, 3):", d[2, 3])
o0, d0 = get_rays(H, W, K, np.eye(4), offset=0.0)
print("without +0.5      :", d0[0, 0], "  ", d0[2, 3], " <- not symmetric")
print("mean direction, centres vs corners:", d.reshape(-1, 3).mean(0), d0.reshape(-1, 3).mean(0))

flip = np.diag([1.0, -1, -1, 1])                          # OpenCV <-> OpenGL camera axes
c2w_cv = np.eye(4)
c2w_cv[:3, 3] = [0, 0, -5]
_, d_cv = get_rays(H, W, K, c2w_cv, "opencv")
_, d_gl = get_rays(H, W, K, c2w_cv @ flip, "opengl")
print("same camera, both conventions agree:", np.allclose(d_cv, d_gl))

C = 5                                                      # a batch of cameras
batch = np.tile(np.eye(4), (C, 1, 1))
batch[:, 0, 3] = np.arange(C)
ob, db = get_rays(H, W, K, batch)
print("batched:", ob.shape, "| origin of camera 3:", ob[3, 0, 0])

// The C++ version (gfx-cameras-projection) does the same per pixel:
for (int r = 0; r < H; ++r)
    for (int c = 0; c < W; ++c) {
        vec3 d_cam{(c + 0.5f - cx) / fx, -(r + 0.5f - cy) / fy, -1.0f};   // OpenGL camera
        rays[r * W + c] = {cam_pos, R * d_cam};
    }

Design choices worth stating in the spec of any get_rays:

• Direction length. NeRF leaves d unnormalised with camera-axis component 1, so the sample parameter t is depth and near/far are depth planes (2 and 6 for the Blender scenes). nerf_pl and nerfstudio normalise d (then t is distance along the ray) and keep the norms for depth maps. Either works; mixing them makes depth maps wrong by 1/cos θ towards the edges.
• Origins. rays_o is the same three numbers for every pixel. np.broadcast_to (PyTorch: expand) shares that memory with stride 0 — free, but read-only in NumPy and silently aliased in PyTorch; copy before anything writes into it (jitter, NDC conversion in place).
• Rows, not columns. Directions are rows of an (H, W, 3) array, so the rotation is d @ R.T or einsum("ij,hwj->hwi", R, d). nerf-pytorch writes torch.sum(dirs[..., None, :] * c2w[:3, :3], -1), the same thing spelled with broadcasting.
• Distortion. Real COLMAP cameras (SIMPLE_RADIAL, OPENCV) have lens distortion; undistort the images first (colmap image_undistorter, as the 3DGS pipeline does) so a pure pinhole K applies.

The test that catches every one of these bugs at once is the round trip: cast rays, take a point at some depth on each, project it back with the world-to-camera pose and K, and require the pixel centres you started from.

import numpy as np

def rot_y(a):
    c, s = np.cos(a), np.sin(a)
    return np.array([[c, 0, s], [0, 1, 0], [-s, 0, c]])

H, W = 4, 6
K = np.array([[50.0, 0, 3.1], [0, 55.0, 1.9], [0, 0, 1]])
R_w2c, t = rot_y(0.4), np.array([0.2, -0.1, 3.0])          # a COLMAP-style pose
c2w = np.eye(4)
c2w[:3, :3], c2w[:3, 3] = R_w2c.T, -R_w2c.T @ t

u, v = np.meshgrid(np.arange(W) + 0.5, np.arange(H) + 0.5, indexing="xy")
d_cam = np.stack([(u - K[0, 2]) / K[0, 0], (v - K[1, 2]) / K[1, 1], np.ones_like(u)], -1)
rays_d = np.einsum("ij,hwj->hwi", c2w[:3, :3], d_cam)
rays_o = np.broadcast_to(c2w[:3, 3], rays_d.shape)

X = rays_o + 2.0 * rays_d                                  # a point at depth 2 on every ray
Xc = X @ R_w2c.T + t                                       # back into the camera...
uv = (Xc @ K.T)[..., :2] / Xc[..., 2:]                     # ...and through K
print("depths:", np.unique(Xc[..., 2].round(9)))
print("max pixel error:", np.abs(uv - np.stack([u, v], -1)).max())
print("pixel (1, 2) ->", uv[1, 2])

6. Normalising the scene

COLMAP's world has arbitrary units, orientation and origin (often the first camera, or somewhere far from the object). Radiance-field methods want the scene in a known box: NeRF's near/far, instant-NGP's hash-grid bounds, Mip-NeRF 360's contraction of everything beyond radius 1. The standard fix is a similarity transform applied to all poses: subtract the mean camera centre, then scale so the farthest camera sits at radius 1 (nerfstudio's auto_scale_poses, NeRF++'s unit sphere). Rotations don't change; a world point maps to (X − center)·scale, and the same map must be applied to the sparse point cloud 3DGS initialises from. Measure the radius after centring — measuring from the old origin squeezes the scene:

import numpy as np

rng = np.random.default_rng(0)
# A COLMAP-like capture: 12 cameras on a noisy ring, far from the origin, in arbitrary units.
ang = np.linspace(0, 2 * np.pi, 12, endpoint=False)
centres = np.stack([40 * np.cos(ang), 40 * np.sin(ang), 5 + rng.normal(size=12)], -1) + [300, -120, 60]
c2w = np.tile(np.eye(4), (12, 1, 1))
c2w[:, :3, 3] = centres

centre = c2w[:, :3, 3].mean(0)
scale = 1.0 / np.linalg.norm(c2w[:, :3, 3] - centre, axis=-1).max()
out = c2w.copy()
out[:, :3, 3] = (c2w[:, :3, 3] - centre) * scale
print("before: |t| from", np.linalg.norm(centres, axis=1).min().round(1), "to", np.linalg.norm(centres, axis=1).max().round(1))
print("centre", centre.round(2), " scale", round(float(scale), 5))
print("after : max |t| =", np.linalg.norm(out[:, :3, 3], axis=1).max().round(6),
      " mean t =", np.abs(out[:, :3, 3].mean(0)).max().round(12))
wrong = 1.0 / np.linalg.norm(c2w[:, :3, 3], axis=-1).max()   # measured from the OLD origin
print("scale measured before centring:", round(float(wrong), 5), "-> cameras squeezed to radius",
      (np.linalg.norm(c2w[:, :3, 3] - centre, axis=-1).max() * wrong).round(3))

Fuller pipelines also re-orient the scene so the average camera "up" is +z (nerfstudio's auto_orient_and_center_poses), and LLFF forward-facing scenes use NDC rays instead — both build on exactly these pieces.

In an interview

Expect "given a transforms.json, how do you get the focal length?" (half the width over the tangent of half the angle), "COLMAP gives you qvec and tvec — where is the camera?" (−Rᵀ t, and the quaternion is w-first), "how do you convert an OpenCV pose to OpenGL?" (flip the y and z columns of c2w), and "write get_rays without loops", where the details that show experience are the half-pixel offset, meshgrid indexing, unnormalised directions with depth as t, and testing by round-tripping pixels. For a 3DGS role, add: gsplat and the 3DGS rasteriser take world-to-camera matrices in OpenCV axes, so Blender data is flipped and inverted at load time.