A Prompt Is Not All You Need: Building a LLM Synthetic Data Pipeline with Open-Source Tools

Today, as the open-source model ecosystem flourishes, developers can easily access top-tier open-source large language models (LLMs) like Gemma4 and Qwen3.5. However, when attempting to implement these technologies in highly regulated fields such as finance and healthcare, they often encounter a significant barrier called "data privacy." Due to restrictions related to personally identifiable information (PII) and regulations, teams cannot access real business data for PoC, performance validation, or RAG system stress testing. Without usable data for validation, even the best open-source models struggle to be integrated into actual business scenarios.

To address the challenge of "data accessibility," synthetic data becomes a key solution.

Many teams initially try manual prompt engineering to generate test data. But when the volume increases to thousands of records, and requirements include diversity, boundary conditions, and logical consistency, this approach quickly faces quality control issues, validation difficulties, and scalability bottlenecks.

This agenda will take Nvidia's open-source tool Data Designer as an example, providing a reproducible pipeline that includes synthetic PII/domain QA seed schema, DAG configuration, validator design, LLM-as-a-Judge rubric, and data quality inspection reports. All sample data and code will be released under open-source licenses, allowing participants to adapt them directly into their own RAG test data or PoC datasets after the session.

Through this practical case study, we hope to assist open-source developers and enterprise IT teams in creating high-quality test datasets "out of thin air" using open-source toolchains, helping the AI implementation journey be more robust at the final leg.