VaultGemma is a 1B-parameter Gemma 2-based LLM that Google trained from scratch using differential privacy (DP) to prevent the model from memorizing and later regurgitating training data. While still a research model, VaultGemma could enable application cases in healthcare, finance, legal, and other regulated sectors.
Differential privacy is a mathematical technique designed to publish statistical information derived from a dataset without leaking information about individual samples contained in it. This is typically achieved by injecting calibrated noise into the training data in a way that preserves its overall statistical properties while making it more difficult to infer details about specific samples.
A key assumption for this approach to be effective is that the injected privacy-preserving noise significantly outweighs the randomness inherent in the original data, which increases the batch size —i.e., the set of samples sent to the model for training— and thus reduces computation costs.
In the context of a large language model, this approach ensures that the model outputs are statistically indistinguishable from those of a model trained on a dataset that excludes any given individual sample from the original dataset. This, in turn, implies that adversaries cannot infer with confidence whether a particular sample was part of the training set based on the model’s outputs.
While differential privacy provides a rigorous, quantifiable privacy guarantee, it does so at a cost, as the added noise can reduce model accuracy and make training more computationally expensive. Google's research leading to VaultGemma has, in fact, focused primarily on this balance and attempted to identify scaling laws for DP models, or in other words, define the optimal training configuration to achieve the lowest performance loss for a given privacy guarantee and compute budget.
We used the scaling laws to determine both how much compute we needed to train a compute-optimal 1B parameter Gemma 2-based model with DP, and how to allocate that compute among batch size, iterations, and sequence length to achieve the best utility.
Google researchers also devised a new training algorithm using Poisson sampling instead of uniform batches to reduce the amount of noise to inject for a desired privacy guarantee.
Google benchmarked VaultGemma against non-private, non-DP models such as Gemma 3 1B and GPT-2 1.5B, and found that it performs comparably to GPT-2 across HellaSwag, BoolQ, PIQA, SocialIQA, TriviaQA, and ARC-C/E. This comparison provides a relatively objective estimate of the performance cost of differential privacy.
VaultGemma’s weights are available on Hugging Face and Kaggle, subject to acceptance of Google’s terms.
While VaultGemma is not the first foray into differentially private LLMs, Google researchers maintain it is the largest such model to date. More commonly, differential privacy has been applied in the context of large language models for fine-tuning existing models without exposing user data.