Position-Wise Feed-Forward Layers: FFN Formula, Parameter Budget, and GeLU vs ReLU

The position-wise feed-forward network (FFN) is the second major sublayer in every transformer encoder and decoder layer. Applied independently to each token position after the attention sublayer, it provides the non-linear capacity that multi-head attention — which applies only linear transformations to value vectors — cannot supply alone.

The Formula

FFN(x) = max(0, xW₁ + b₁)W₂ + b₂

where:

• x ∈ ℝ^{d_model} is the d_model=512 dimensional input for one token position
• W₁ ∈ ℝ^{d_model × d_ff} = ℝ^{512 × 2048} — expands the representation
• W₂ ∈ ℝ^{d_ff × d_model} = ℝ^{2048 × 512} — compresses back to d_model
• max(0, ·) is ReLU; the same sublayer with GeLU is GELU(xW₁ + b₁)W₂ + b₂

The FFN is applied identically and independently to each of the n token positions — it does not mix position information. The network is “position-wise” in the same sense that a 1×1 convolution is channel-wise.

Parameter Breakdown Across Architectures

Hyperparameter	Base Model	Big Model	Modern 4× rule
d_model	512	1024	varies
d_ff	2048	4096	4 × d_model
d_ff / d_model ratio	4×	4×	4×
W₁ parameters (per FFN)	1,048,576	4,194,304
W₂ parameters (per FFN)	1,048,576	4,194,304
Biases (per FFN)	2,560	4,608
Total per FFN sublayer	~2.1M	~8.4M

Where Do the Parameters Go? (Base Model, 65M Total)

Component	Layers	Params per layer	Total
Token embeddings (vocab=37,000)	—	—	~18.9M
Attention blocks (enc + dec)	12	~1.05M	~12.6M
FFN sublayers (enc + dec)	12	~2.1M	~25.2M
LayerNorm + output projection	—	—	~8.3M
Total			~65M

FFN sublayers account for roughly 39% of the base model’s parameters on their own; combined with the attention blocks’ parameter budget (~19%), the 6+6 transformer layers hold ~88% of all parameters. Approximately two-thirds of per-layer parameters are in the FFN.

GeLU vs ReLU

The original transformer uses ReLU. Later architectures switched to GeLU (Hendrycks & Gimpel, 2016), which is defined as:

GeLU(x) = x · Φ(x)

where Φ(x) is the standard Gaussian CDF. Unlike ReLU, GeLU applies a smooth, probabilistic gate that decreases output for negative inputs rather than zeroing them entirely.

Activation	CIFAR-10 error	CIFAR-100 error	Characteristic
ReLU	8.16%	21.77%	Hard threshold at 0; sparse activations
ELU	8.41%	22.98%	Smooth for negative inputs
GeLU	7.89%	20.74%	Smooth gate; weights by magnitude

Shazeer (2020) further extended this with Gated Linear Units (GLU), where the FFN becomes:

FFN_GLU(x) = (xW₁ ⊙ σ(xW_gate)) W₂

This variant and its GeLU-gated form (SwiGLU) are widely used in modern architectures for improved training stability.

See multi-head-attention for the other parameter-dense sublayer in each layer, self-attention-mechanism for the attention formula, and transformer-architecture for how FFN and attention sublayers are composed with residual connections and layer normalization.