Text data has become extremely valuable due to the emergence of machine learning algorithms that learn from it. A lot of high-quality text data generated in the real world is private and therefore cannot be shared or used freely due to privacy concerns. Generating synthetic replicas of private text data with a formal privacy guarantee, i.e., differential privacy (DP), offers a promising and scalable solution. However, existing methods necessitate DP finetuning of LLMs on private data to generate DP synthetic data. This approach is not viable for proprietary LLMs (e.g., GPT-3.5) and also demands considerable computational resources for open-source LLMs. Lin et al. (2024) recently introduced the Private Evolution (PE) algorithm to generate DP synthetic images with only API access to diffusion models.
In this work, we propose an augmented PE algorithm, named Aug-PE, that applies to the complex setting of text. We use API access to an LLM and generate DP synthetic text without any model training. Results on three benchmark datasets demonstrate that Aug-PE produces DP synthetic text that yields competitive utility with the SOTA DP finetuning baselines. This underscores the feasibility of relying solely on API access of LLMs to produce high-quality DP synthetic texts, thereby facilitating more accessible routes to privacy-preserving LLM applications.
We use two private & synthetic samples (reviews for the "restaurant" class) for illustration of Aug-PE.
The prompts are simplified for illustration; see our paper for the complete prompts.
Compared to image generation as in original PE paper, text generation introduces unique challenges: 1) the core components of PE such as the embedding model, RANDOM_API, and VARIATION_API, require domain-specific designs for text; 2) unlike images which have a fixed dimensionality, the length of text can vary; 3) the original PE algorithm yields unsatisfactory text quality. Therefore, in our proposed augmented algorithm, Aug-PE, we explore the design choices for each component, and new algorithmic techniques to increase the diversity and quality of text generation.
We compare Aug-PE to two SOTA methods involving DP finetuning:
We evaluate the downstream model accuracy of Aug-PE and two baselines along different data generators. The highest accuracy across all methods (obtained by Aug-PE) is bolded (underlined).
Leveraging the inherent knowledge within stronger LLM, GPT-3.5, Aug-PE can achieve higher accuracy, especially on challenging datasets OpenReview and PubMed, outperforming DP-FT-Generator by a notable margin.
Many powerful LLMs such as GPT-4, Claude, and Bard are only accessible through inference APIs. DP finetuning them is not feasible. Although standard finetuning APIs are provided for some of the models, DP finetuning requires a special implementation (i.e., DP-SGD) and no model provides this custom API to date.
Using powerful open-source LLMs, such as Mixture-of-Expert Mixtral-8x7B-v0.1, as data generator, leads to improved downstream accuracy for Aug-PE on three datasets.
Finetuning those open-source LLMs with DP is resource-intensive and non-trivial to implement due to the need of calculating per-sample gradient in DP-SGD. The state-of-the-art DP synthetic text approaches are unfortunately still based on GPT-2 (Yue et al, 2023).
With inference API access, Aug-PE is more efficient than DP-FT-Generator that requires DP-SGD finetuning on Yelp for generating 100k synthetic samples (ϵ = 1). The running time of Aug-PE mainly depends on the number of API calls, which is associated with the hyperparameter L.
The text length distribution of synthetic samples generated from GPT-3.5 through Aug-PE more closely matches the original Yelp data distribution across iterations, due to our adaptive sequence length mechanism.
The next word prediction accuracy of the downstream model increases when using larger downstream models for PubMed synthetic texts. Under both ϵ = 1, ∞, the smallest model BERT-Tiny favors the synthetic texts from DP-FT-Generator GPT-2-Large, while larger models such as LLaMA-2 favor synthetic text from Aug-PE GPT-3.5.
This observation underscores the importance of choosing downstream models of a suitable size; employing overly small models could under-estimate the quality of synthetic texts produced by Aug-PE with GPT-3.5. We hypothesize that this is because (1) GPT-3.5 generated texts might already be of higher quality in terms of vocabulary, syntax, semantic coherence, etc., compared to generated texts from finetuned GPT-2-Large; and (2) larger downstream LMs like LLaMA-2 can better understand and utilize the nuances in synthetic texts for improved performance than BERT-Tiny.
For example, larger embedding models such as sentence-t5-xl can more accurately capture the nuances of texts in the embedding space, leading to higher utility for GPT-2 generated texts.
We apply the same API designs and models to the original PE algorithm (Lin et al, 2024) to support text generation. Aug-PE achieves notable improvement over PE for GPT-2 on all datasets, e.g., +22.6% on Yelp rating classification. It shows that the new algorithmic techniques introduced in Aug-PE is effective for diverse and high-quality text generation.
The initial samples (RANDOM_API) or their variants (RANDOM_API + VARIATION_API) exhibit limited utility. However, the synthetic text quality improves notably after just one iteration of Aug-PE when guided by private data, and this improvement continues to amplify with T iterations.
Here are some examples of Aug-PE generated samples with GPT-3.5 under DP privacy budget ϵ = 1. Each row represents an independent sample drawn from the synthetic dataset obtained during the last iteration of Aug-PE.
| Text | Label |
|---|---|
| Example Text 1 | |
| Example Text 2 | |
| Example Text 3 |
Differentially Private Synthetic Data via Foundation Model APIs 2: Text@article{xie2024differentially,
title={Differentially Private Synthetic Data via Foundation Model APIs 2: Text},
author={Chulin Xie and Zinan Lin and Arturs Backurs and Sivakanth Gopi and Da Yu and Huseyin A Inan and Harsha Nori and Haotian Jiang and Huishuai Zhang and Yin Tat Lee and Bo Li and Sergey Yekhanin},
journal={ICML},
year={2024}
}