Work also done with Cindy Wu, Asic Q Chen, Jean Tarbouriech, Min Ma, Brendan O'Donoghue, João Gabriel Lopes de Oliveira.

Paper here: https://arxiv.org/abs/2606.20560

Overview

In a recent collaboration between the GDM interpretability team and the GDM text diffusion team, we performed a transparency audit of DiffusionGemma, GDM's new text diffusion model.

Overall, we find that DiffusionGemma is not significantly less transparent than Gemma.

Gemma and DiffusionGemma perform similarly on monitorability evaluations.
Although naively DiffusionGemma has a much larger opaque serial depth, we can apply the logit lens to intermediate vectors and ablate non-interpretable information without harming performance. This implies that these intermediate nodes are interpretable, which reduces the opaque serial depth to be similar to that of Gemma.

However, even though the variables that the model uses at different steps are interpretable, this does not necessarily mean that we understand the algorithm that the model uses to reach the final answer. We thus distinguish between variable transparency, which we define as whether we can understand snapshots of the model's computation, and algorithmic transparency, which we define as whether we can use these snapshots to reconstruct the process by which the model arrived at its outputs.

By [...]

---

Outline:

(00:26) Overview

(02:45) Why is this relevant for AI safety?

(03:51) Short summary of main results:

(07:07) Abstract

---

First published:
June 20th, 2026

Source:
https://www.lesswrong.com/posts/zoYXpdaMgFT43Wc24/linkpost-how-transparent-is-diffusiongemma-and-why-it

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Architecture diagram showing DiffusionGemma's denoising steps with attention blocks and information flow paths.

Table showing empirical upper bounds and asymptotic computations of opaque serial depths for Gemma 4 and DiffusionGemma models.

Two bar graphs comparing Top-K and Top-P Ablation across five benchmarks.

Two stacked area charts showing token identity breakdown across diffusion steps with different restrictions.

Bar chart comparing monitorability scores between Gemma 4 26B A4B and DiffusionGemma 26B 4B models across different evaluation categories.

Figure showing three dashboard views for visualizing language model token prediction: summary, line graph, and table.

Summary view of AI model self-correcting from answer 9 to 8.

Two graphs showing probability distributions of

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

″[Linkpost] How Transparent Is DiffusionGemma (and why it matters)” by Josh Engels, Callum McDougall, bilalchughtai, János Kramár, Senthooran Rajamanoharan, Arthur Conmy

LessWrong (30+ Karma)

″[Linkpost] How Transparent Is DiffusionGemma (and why it matters)” by Josh Engels, Callum McDougall, bilalchughtai, János Kramár, Senthooran Rajamanoharan, Arthur Conmy