NeMo Data Designer: An Extensible Framework for Multimodal Synthetic Data Generation
Johnny Greco and colleagues at NVIDIA describe NeMo Data Designer, an open-source framework for multimodal synthetic data generation in which humans or agents declare each dataset column in an inspectable configuration.
Ask this paper
Declarative columns: Column types cover model-generated text, code, structured outputs, images, embeddings and statistical samplers configured to control diversity, and a plugin system adds new types.
Preview loop: Users generate a few records, inspect them, refine the specification and then run at full scale; the runtime resolves column dependencies, schedules calls to user-provided endpoints and retries failures.
Nemotron use: About 9K JSON-schema adherence tasks built with NDD were used for RL on Nemotron Nano v3, raising JSONSchemaBench accuracy from 80.2% to 86.9% and StructEval-Text from 64.5% to 72.1%.
Case studies: Other case studies include Nemotron-Personas and enterprise work with Perplexity and CrowdStrike.
Abstract
We present NeMo Data Designer (NDD), an open-source, general-purpose framework for multi-modal synthetic data generation (SDG). Designed to be intuitive to use, NDD provides a declarative configuration format in which human and/or agent users define each dataset column, with column types spanning text, code, structured outputs, images, embeddings, and statistical samplers that are explicitly configured to steer dataset diversity. Additional column types and functionality can be introduced using the framework's flexible plugin system. NDD's configuration is an inspectable artifact, supporting workflow sharing and reproducibility. SDG is an inherently iterative process. NDD therefore builds a preview-and-revision loop into its core workflow, allowing users to generate and inspect a small number of records, refine the specification, and rerun generation at full scale. At runtime, NDD resolves dependencies, schedules calls to user-provided model endpoints, and retries failed requests. We describe NDD's architecture and programming model and present case studies spanning structured, agentic, multimodal, and domain-specialized tasks, including datasets used in Nemotron model development and in production enterprise deployments.