Neura saysSynthetic data — generated by models — augments or replaces scarce real data.
When real labeled data is scarce or sensitive, you can generate it: an LLM produces instruction/response pairs (self-instruct), or a larger model creates training data for a smaller one (a form of distillation). It scales cheaply and avoids privacy issues. Risks: synthetic data can amplify the generator's biases and errors, and training models only on model output can degrade quality ("model collapse") if unchecked.
Power-ups you unlock
Generate training data with models
Self-instruct; teacher → student data
Scales cheaply, sidesteps privacy
Risks: bias amplification, model collapse
The Overfit Ogre attacks — common mistakes
Training only on synthetic data unchecked
Ignoring inherited biases/errors
Assuming synthetic = as good as real
Boss battleName a benefit and a risk of training on model-generated data.