How to Concatenate Tensors in PyTorch: torch.cat, stack, and shapes

Concatenation joins tensors along an axis you already have—batch dimension, channels, time, and so on. PyTorch’s main API is torch.cat; torch.stack is different because it adds a dimension.

Run the snippets locally or in a notebook with PyTorch installed (for example after you set up a project or venv as in Create a Python package). For NumPy-style matrix joins without tensors, see combine two column matrices in Python; for ML-oriented NumPy patterns, see introduction to Python for machine learning.

Tested on: Python 3.13.3; kernel 6.14.0-37-generic.

All runnable code paths in this article were executed successfully with PyTorch 2.12.1+cpu in a local virtual environment (CPU wheels from the PyTorch package index). The device-mismatch example stays commented because it requires CUDA hardware to demonstrate meaningfully.

Quick answer: concatenate tensors in PyTorch

Pass a sequence of tensors and a dim (axis) to torch.cat. Sizes must match on every dimension except dim, which grows to the sum of the input sizes on that axis. If you need a new batch-like axis instead of extending an existing one, use torch.stack, not cat.

import torch

a = torch.tensor([[1, 2], [3, 4]])
b = torch.tensor([[5, 6], [7, 8]])
out = torch.cat((a, b), dim=0)
print(out.shape)  # torch.Size([4, 2])

torch.cat: what concatenation does, dimensions, aliases, and shapes

Meaning: concatenation builds one tensor by appending slices along a single dimension. Values are not combined elementwise; whole rows, columns, or higher-dimensional slabs are placed in order. Inputs should share dtype, and they must already live on the same device unless you move them first (for example with .to(device)).

torch.cat(tensors, dim=0, out=None) takes a tuple or list of tensors whose shapes agree on every axis except the one you concatenate.

Concatenate along dim=0 and dim=1

dim=0 walks the first axis: for matrices you usually get more rows; for batches you often append more items. dim=1 extends the second axis—wider matrices or more features per row. The same x and y below show how the choice changes the layout while reusing identical inputs.

import torch

x = torch.tensor([[1, 2], [3, 4]])
y = torch.tensor([[5, 6], [7, 8]])
z0 = torch.cat((x, y), dim=0)
z1 = torch.cat((x, y), dim=1)
print(z0)  # 4x2: rows stacked
print(z1)  # 2x4: columns appended

torch.cat, torch.concat, and torch.concatenate

In current PyTorch documentation, torch.concat and torch.concatenate are aliases of torch.cat (same behavior). torch.concatenate also documents NumPy-style axis= as an alternative spelling for dim=. Most tutorials and codebases standardize on torch.cat; when you port NumPy snippets, rewrite np.concatenate(..., axis=k) to torch.cat(..., dim=k).

Shape rules and fixing mismatches

For dimension d, every input tensor must have the same size[i] for all i != d. Along d, sizes may differ; the output length on d is the sum of the lengths on d. A quick 3D example: two tensors shaped [2, 3, 4] concatenate on dim=0 into [4, 3, 4].

import torch

a = torch.zeros(2, 3, 4)
b = torch.zeros(2, 3, 4)
c = torch.cat([a, b], dim=0)
print(c.shape)

When you hit a mismatch, reshape or permute so the non-concat axes line up before calling cat. Typical tools are reshape, view when memory layout allows, unsqueeze, squeeze, and permute. Remember that broadcasting rules for + or * do not apply to cat: every tensor must already have compatible concrete sizes.

Examples: two tensors, several tensors, and ranks 1D through 3D

Most day-to-day use is either two tensors or a short list produced by a training loop. The API is the same: wrap the operands in a tuple or list and pass dim.

Two tensors and a list of tensors

Two 1D tensors on dim=0 form one longer vector. Three small matrices concatenate on dim=0 into a taller stack while the column count stays fixed.

import torch

u = torch.tensor([1.0, 2.0])
v = torch.tensor([3.0, 4.0])
w = torch.cat((u, v), dim=0)

a = torch.ones(1, 2)
b = torch.ones(1, 2) * 2
c = torch.ones(1, 2) * 3
d = torch.cat([a, b, c], dim=0)
print(w, d.shape)

Vectors, matrices, and 3D blocks

1D: only dim=0 is meaningful for a plain vector; you are lengthening the axis. 2D: dim=0 appends rows; dim=1 appends columns. 3D: choose dim=0, 1, or 2 based on whether you want more “depth” along batches, rows, or the last slab—the same matching rule applies on the other two axes. The snippet uses fixed arange values so shapes are easy to verify.

import torch

v1 = torch.cat([torch.arange(3), torch.arange(3, 6)], dim=0)
m = torch.cat([torch.zeros(2, 2), torch.ones(2, 2)], dim=1)
u = torch.arange(12).reshape(2, 2, 3)
v = torch.arange(12, 24).reshape(2, 2, 3)
w = torch.cat((u, v), dim=2)
print(v1.shape, m.shape, w.shape)

_ = v1, m

torch.cat versus torch.stack

torch.cat never increases tensor rank. If each input is [B, C], concatenating on dim=0 yields [B_total, C] with B_total equal to the sum of batch sizes.

torch.stack requires every tensor to have exactly the same shape; it allocates a new axis (by default dim=0) whose length equals the number of tensors and packs each input as one slice along that axis. A common pattern is stacking [C] vectors into a [N, C] batch matrix—something cat cannot do without an explicit unsqueeze first.

import torch

a = torch.tensor([1, 2])
b = torch.tensor([3, 4])
stacked = torch.stack((a, b), dim=0)
print(stacked.shape)
print(stacked)

If you unsqueeze each vector to [1, 2] and cat on dim=0, you get the same numbers as stack but arrived at with two steps; stack is the idiomatic shortcut when ranks should grow by one.

Common errors while concatenating tensors

Sizes of tensors must match

PyTorch checks every dimension except the concatenation axis. A frequent mistake is catting along dim=0 when column widths differ: both tensors might have two rows, but if one has three columns and another has four, the operation is invalid because dim=1 does not match. The error string names the dimension, the expected size, and which tensor in the list broke the contract—read it before blindly changing dim.

import torch

a = torch.zeros(2, 3)
b = torch.zeros(2, 4)
try:
    torch.cat([a, b], dim=0)
except RuntimeError as e:
    print(e)

Typical message when you print the caught exception (wording can vary slightly by PyTorch version):

Sizes of tensors must match except in dimension 0. Expected size 3 but got size 4 for tensor number 1 in the list.

Tensors are on different devices

A CPU tensor and a CUDA tensor cannot participate in the same cat until they share a device. The failure mode is a runtime error pointing at device mismatch. In training code, pick device = torch.device("cuda" if torch.cuda.is_available() else "cpu") once, call .to(device) on loaded batches, then concatenate.

# Pattern only — pick one device in real code
# a_cpu = torch.tensor([1])
# b_cuda = torch.tensor([2], device="cuda")
# torch.cat([a_cpu, b_cuda])  # RuntimeError without .to(...)

Empty tensor or wrong tensor list

torch.cat([]) raises because there is nothing to join. On recent PyTorch (for example 2.12.x) you get a ValueError whose message starts with torch.cat(): expected a non-empty list of Tensors. If you build a list of tensors in a loop, check if not tensors: return or provide a default tensor before concatenating. Similarly, passing a nested structure by mistake (for example a list of lists instead of a list of tensors) fails at runtime with a type or shape error—keep the tensors argument as a flat sequence of Tensor objects.

Best practices and quick reference

Treat dim as part of your data model: name variables (batch_dim, time_dim) so code reviews can see why you chose a given axis. During development, assert expected shapes after load and before loss. For performance, prefer a single cat over many small ones when assembling large minibatches from lists. Keep tensors on one device and dtype before concatenating; cast with .to(dtype=..., device=...) when loading heterogeneous sources. Use stack when you truly need a new batch axis; use cat when you are extending an existing axis.

Summary

torch.cat concatenates along an existing dimension; matching sizes on all other axes are required. torch.concat and torch.concatenate are aliases of cat in current PyTorch docs. torch.stack builds a new dimension when every input has the same shape. Watch for shape mismatches on non-concat axes, mixed CPU and CUDA devices, and empty tensor lists. See the official torch.cat and torch.stack pages for argument details.