Lifestyle
What Is Overfitting? Acing Past Exam Questions, Failing New Ones
Overfitting means learning even the noise and exceptions in training data: near-perfect training results, poor performance on unseen data. Illustrative error curves explain training, validation and test sets and contrast underfitting. Covers more data, regularization, dropout and early stopping, research on deep networks memorizing random labels, and leaked benchmark questions that make models appear capable.
About 9 min read

Contents
- 1. Training, Validation and Test Sets Each Have a Job
- 2. Example: Where Two Error Curves Diverge
- 3. Distinguishing Underfitting, a Good Fit and Overfitting
- 4. Common Remedies: What Each One Does
- 5. Networks Can Memorize Random Answers: Zhang et al.'s 2017 Experiment
- 6. Leaked Questions: Apparent Competence on Benchmarks
- 7. Applying This When Training, Fine-Tuning or Reading Scores
Overfitting is a common machine learningWhat Is Machine Learning?Machine learning uses data to adjust models to make predictions for unseen inputs or generate content. Breakfast shop prep forecasts show how features, labels, training and testing work together, distinguish supervised, unsupervised and reinforcement learning, and explain the effects of data leakage, overfitting and environmental change. Learn why models need held-out test data and which concrete questions help assess whether a seemingly accurate prediction is trustworthy.Read the full article failure: the model fits training data too closely, learning even noise and exceptions. It gets almost everything right during training but answers incorrectly on new data, like memorizing past exam questions and stumbling on a different exam. Chinese terminology varies: the National Academy for Educational Research's terminology database uses guodu nihe in most fields and in entries tagged artificial intelligence, and also lists guo nihe and guodu peishi; Google's Traditional Chinese materials mostly use guodu peishi. This article uses guodu nihe for overfitting and follows the database in using didu nihe for the opposite case, underfitting.
The following explains how to split data, read the curves and understand the remedies, then discusses research on networks memorizing random labels and leaked benchmark questions.
Training, Validation and Test Sets Each Have a Job
To find out whether a model is overfitting, evaluate it on data it has never seen. Google's Machine Learning Crash Course recommends splitting data into three sets: the training set adjusts parameters; the validation set is used repeatedly during development to compare settings; the test set is reserved for a final check. Each example can belong to only one set.
Chapter 5 of Deep Learning by Goodfellow et al. is more explicit: the test set must not influence any choices, including hyperparameters. The validation set should be split off from the training data; a common split is roughly eighty percent for training and twenty percent for validation.
Google's materials also warn that test examples duplicated in the training set produce unrealistically good scores and must be removed first.
Example: Where Two Error Curves Diverge
Example (invented figures, not actual training results): train a classifier on hotel reviews to predict whether guests will book again. After each training epoch, record the error rates on the training and validation sets:
- Epoch 1: training error 30%, validation error 31%.
- Epoch 8: training error 8%, validation error 12%, the lowest validation error.
- Epoch 20: training error falls to 1%, but validation error rises to 20%.
During the first 8 epochs, both curves fall as the model learns more general patterns. After that, training error keeps falling while validation error rises, indicating that it has begun memorizing particular wording and exceptions. Google's materials describe loss curves that initially fall together and then diverge as a strong signal of overfitting.
Goodfellow et al. define underfitting as being unable to reduce training error sufficiently, and overfitting as having too large a gap between training and test error. Looking only at the 1% at epoch 20 would make that model seem best; considering both curves, the version around epoch 8 is more reliable.
Read the full description
An illustrative chart with invented teaching values, not actual training results. The horizontal axis shows training epochs; the vertical axis shows error rate. Dark green training error falls from 30% at epoch 1 to 1% at epoch 20. Orange validation error falls from 31% to 12% at epoch 8, then rises to 20% at epoch 20. Before epoch 8, both lines fall, indicating that the model is still learning general patterns. After epoch 8, the lines diverge and the gap grows, indicating overfitting. Early stopping would select the epoch 8 version. Notes on the right: both falling means the model is still learning patterns; falling training error and rising validation error mean overfitting; both high means underfitting.
Distinguishing Underfitting, a Good Fit and Overfitting
Google's materials broadly divide the causes into two categories: training data that does not represent real-world data, or an overly complex model. Goodfellow et al. use "capacity" to describe the range of functions a model can fit. Excessive capacity may let it memorize details that are useless on new data.
| State | Training and Validation Error | What to Try First |
|---|---|---|
| Underfitting | Both high, small gap | Change features, train for more epochs, reduce regularization strength |
| Good fit | Both low, small gap | Keep this version; evaluate once on the test set at the end |
| Overfitting | Low training error, high validation error, large gap | Add data, strengthen regularization, stop early |
| Leaked questions | High scores on public questions, substantially lower on new questions | Retest with new, unpublished questions |
Common Remedies: What Each One Does
- More data: Goodfellow et al. consider this the best way to improve generalization—performance on new data—but the data must be representative.
- Data augmentation: create new examples by modifying existing ones without changing their answers, such as shifting an image by a few pixels.
- Regularization: penalize model complexity during training. L2 regularization, also called weight decay, pushes weights toward 0; excessive strength causes underfitting.
- dropout (called "dropout regularization" in Google's Traditional Chinese glossary): randomly and temporarily remove some units at each training step so that units cannot rely solely on particular other units.
- Early stopping: stop when validation error no longer improves, retaining the parameters from the epoch with the lowest error. Google's materials say it is fast but rarely optimal.
- Simplify the model: reduce features or model size to lower capacity.
dropout first appeared in Hinton et al.'s 2012 preprint and was explained fully in Srivastava et al.'s 2014 journal paper. Its purpose is to reduce overfitting in large neural networksWhat Is a Neural Network? A Function Built from Layers and Weights That Learns from DataA neural network is a function of layered simple units: each computes a weighted sum, usually followed by a nonlinear transformation, with weights adjusted from data. A hand-calculated two-switch hallway light example explains forward passes, loss, backpropagation and gradient descent, distinguishes training from inference and parameter count from neuron count and capability, and shows how neural networks relate to deep learning and Transformer.Read the full article. None of these remedies is guaranteed to work. Which ones to use, and at what strength, must still be guided by validation error.
Networks Can Memorize Random Answers: Zhang et al.'s 2017 Experiment
In their paper published at ICLR 2017, Zhang et al. replaced the labels in CIFAR10 (small images in 10 classes) and ImageNet with random labels, then trained standard image classification networks as usual. Training error on CIFAR10 still fell to 0 or nearly 0, and training accuracy on ImageNet reached 95.20%, while test performance on both fell to chance level. On CIFAR10, the networks could even memorize images replaced with pure noise.
The comparison in Table 1 of the paper: on CIFAR10, with data augmentation and weight decay disabled, the same network achieved 100% training accuracy and 85.75% test accuracy with real labels. With random labels, training accuracy was still 100%, but test accuracy was only 9.78%, close to the 10% chance level for 10 classes.
The authors concluded that these networks have enough effective capacity to memorize the entire training dataset, and that traditional complexity measures cannot distinguish good from poor generalization. Explicit regularization such as weight decay and dropout may improve generalization, but it is neither necessary nor sufficient on its own to control the gap between training and test performance.
What the paper did not say also needs to be clear: it did not say large networks always overfit real data—the same network still achieved 85.75% on new images. It did not say regularization is useless, nor did it resolve why networks can generalize. The experiments covered only image classification and cannot be applied directly to large language models.
Thus, "larger models are more prone to overfitting" is not a universal rule. In 2019, Nakkiran et al. observed that test error first fell, then rose, then fell again as models grew larger in settings including CIFAR-10 and CIFAR-100 image classification and IWSLT'14 German-to-English translation. This is called double descent. The peak occurred where the model could just drive training error close to 0. The effect was strongest with noisy training labels, and in some settings, adding examples actually worsened performance.
Leaked Questions: Apparent Competence on Benchmarks
A 2023 position paper by Sainz et al. identifies the most serious form of data contamination as a large language model seeing a benchmark's test questions during training and later being evaluated on that same benchmark, inflating its score. A test set measures real ability only if its questions are new to the model; once questions leak, scores may measure only memorization.
Even without leakage, a model can gradually be tuned to a test set: Goodfellow et al. write that evaluations become overly optimistic when the research community uses the same test set for years.
In 2024, Hugh Zhang et al. commissioned humans to write 1,205 new questions matching the style and difficulty of GSM8K, a benchmark of grade-school math word problems. This new set, GSM1k, was not public when the paper was released. In their settings, model accuracy dropped by up to about 8%. Models more likely to generate original GSM8K questions generally showed larger gaps (Spearman r² = 0.36), suggesting that some models may have memorized the original questions. However, many models showed almost no gap, and all models overall could still solve new questions.
Leakage and overfitting do not have a one-to-one relationship either: some models showed large gaps without appearing to have memorized the original questions. The authors suggested that they might have used data very similar to the benchmark, or selected model versions using benchmark scores. A small amount of leakage does not necessarily cause overfitting, either. In 2019, Recht et al. created new test sets for CIFAR-10 and ImageNet. The various models they evaluated lost 3% to 15% and 11% to 14% in accuracy, respectively. However, they attributed this mainly to insufficient generalization to slightly harder new images, rather than repeated use of the test sets. Lower scores on new questions are a warning; the cause requires separate investigation.
Applying This When Training, Fine-Tuning or Reading Scores
On an existing model, fine-tuningFine-tuning: Adapting an Existing Model to Your TaskFine-tuning continues training an existing model to fit a task or domain. Using support email classification, this article covers data preparation, parameter updates, held-out test sets and version deployment; distinguishes full fine-tuning, LoRA, supervised fine-tuning and RAG; and explains why training gains can mask real-world decline. Learn to judge whether fine-tuning is worthwhile and check format consistency, unknown types, data leakage and loss of prior abilities.Read the full article also continues training using your own data. You still need to set aside validation and test sets first, remove duplicate examples, and pause to compare versions when validation error rises.
When reading someone else's scores, first ask whether the questions were public and whether the model was retested on new questions. The answers cannot prove that the model is problem-free; they only help you judge how much to discount its scores.
To look up other terms, start with the AI Terminology Index.
Found this guide helpful? Share it with someone who needs it
Lifestyle
Agentic Retrieval-Augmented Generation (Agentic RAG): When to Search Again
Agentic RAG lets models make retrieval decisions, choosing sources, rewriting queries or searching further based on the question and available evidence. Using a concrete LangGraph workflow and a comparison of venue-use conditions, this article contrasts it with fixed RAG and explains stopping criteria, permissions, query logs and cost control. Learn to judge whether a task needs dynamic exploration or a predictable fixed workflow is enough.
Lifestyle
Graph Retrieval-Augmented Generation (GraphRAG): Understanding a Document Collection Through Relationships
GraphRAG uses entities and relationships in retrieval to answer questions about cross-document connections and overall themes. Drawing on Microsoft's original paper and official docs, this article explains entity extraction, community summaries, and local and global search, using local event records to show why different questions need different paths. It also shows how missing links, mistaken name merges, and distorted summaries affect answers: a pretty graph is not verified fact.
Lifestyle
Knowledge Graphs: Making Entities and Relationships Clear
Knowledge graphs organize entities and relationships to trace who took part in what, where and when through links. Using community events, this article covers entity identity, relationship direction, sources and time; compares Resource Description Framework (RDF) and property graphs; and explains why graphs are not equivalent to visualizations or reliable facts. Learn when to build one and how it works with tables, vector search and graph retrieval-augmented generation (GraphRAG).
Lifestyle
Hybrid Search: Balancing Exact Terms and Meaning
Hybrid search merges full-text, vector and other retrieval results to cover exact codes and natural-language needs. An equipment-manual query explains candidate retrieval, deduplication, rank fusion and reranking, including how Reciprocal Rank Fusion (RRF) merges results by rank. Hybrid search is not always more accurate. Learn to test quality with code, paraphrase and no-answer queries and spot biases from permissions, filters and duplicate content.
Sources
- Google Machine Learning Crash Course: Overfitting (overfitting, underfitting and generalization curves)Checked:
- Google Machine Learning Crash Course: Dividing the original dataset (training, validation and test sets, and duplicate examples)Checked:
- Google Machine Learning Crash Course: Model complexity (model complexity and regularization)Checked:
- Google Machine Learning Crash Course: L2 regularization (regularization rate and early stopping)Checked:
- Google Machine Learning Glossary (entries for overfitting, underfitting, early stopping and dropout regularization)Checked:
- Google Machine Learning Crash Course, Traditional Chinese edition: Overfitting (official Chinese terminology comparison)Checked:
- National Academy for Educational Research terminology database: overfitting search results (guodu nihe, guo nihe, guodu peishi)Checked:
- Goodfellow, Bengio, Courville: Deep Learning, Chapter 5, Machine Learning Basics (full text on the authors' official website)Checked:
- Goodfellow, Bengio, Courville: Deep Learning, Chapter 7, Regularization for Deep Learning (full text on the authors' official website)Checked:
- Srivastava et al. 2014: Dropout: A Simple Way to Prevent Neural Networks from Overfitting (JMLR 15)Checked:
- Hinton et al. 2012: Improving neural networks by preventing co-adaptation of feature detectors (arXiv:1207.0580; earliest dropout preprint)Checked:
- Zhang et al. 2017: Understanding deep learning requires rethinking generalization (ICLR 2017, arXiv:1611.03530)Checked:
- Nakkiran et al. 2019: Deep Double Descent: Where Bigger Models and More Data Hurt (arXiv:1912.02292)Checked:
- Sainz et al. 2023: NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark (arXiv:2310.18018)Checked:
- Zhang et al. 2024: A Careful Examination of Large Language Model Performance on Grade School Arithmetic (GSM1k, arXiv:2405.00332)Checked:
- Recht et al. 2019: Do ImageNet Classifiers Generalize to ImageNet? (arXiv:1902.10811)Checked:
Compiled from the references and verification dates listed above; fares, schedules, product features, and terms can change, so please confirm the latest details on the relevant official website before travelling, booking, or purchasing.