Lifestyle

What Is Overfitting? Acing Past Exam Questions, Failing New Ones

Overfitting means learning even the noise and exceptions in training data: near-perfect training results, poor performance on unseen data. Illustrative error curves explain training, validation and test sets and contrast underfitting. Covers more data, regularization, dropout and early stopping, research on deep networks memorizing random labels, and leaked benchmark questions that make models appear capable.

About 9 min read

Share on LINE
Two lines beside data points: one bends sharply through every point; the other follows the trend smoothly. A new point at the right lies on the smooth line, far from the jagged one.
Image: Mokaair (© Mokaair)
Contents
  1. 1. Training, Validation and Test Sets Each Have a Job
  2. 2. Example: Where Two Error Curves Diverge
  3. 3. Distinguishing Underfitting, a Good Fit and Overfitting
  4. 4. Common Remedies: What Each One Does
  5. 5. Networks Can Memorize Random Answers: Zhang et al.'s 2017 Experiment
  6. 6. Leaked Questions: Apparent Competence on Benchmarks
  7. 7. Applying This When Training, Fine-Tuning or Reading Scores

Overfitting is a common failure: the model fits training data too closely, learning even noise and exceptions. It gets almost everything right during training but answers incorrectly on new data, like memorizing past exam questions and stumbling on a different exam. Chinese terminology varies: the National Academy for Educational Research's terminology database uses guodu nihe in most fields and in entries tagged artificial intelligence, and also lists guo nihe and guodu peishi; Google's Traditional Chinese materials mostly use guodu peishi. This article uses guodu nihe for overfitting and follows the database in using didu nihe for the opposite case, underfitting.

The following explains how to split data, read the curves and understand the remedies, then discusses research on networks memorizing random labels and leaked benchmark questions.

Training, Validation and Test Sets Each Have a Job

To find out whether a model is overfitting, evaluate it on data it has never seen. Google's Machine Learning Crash Course recommends splitting data into three sets: the training set adjusts parameters; the validation set is used repeatedly during development to compare settings; the test set is reserved for a final check. Each example can belong to only one set.

Chapter 5 of Deep Learning by Goodfellow et al. is more explicit: the test set must not influence any choices, including hyperparameters. The validation set should be split off from the training data; a common split is roughly eighty percent for training and twenty percent for validation.

Google's materials also warn that test examples duplicated in the training set produce unrealistically good scores and must be removed first.

Example: Where Two Error Curves Diverge

Example (invented figures, not actual training results): train a classifier on hotel reviews to predict whether guests will book again. After each training epoch, record the error rates on the training and validation sets:

  • Epoch 1: training error 30%, validation error 31%.
  • Epoch 8: training error 8%, validation error 12%, the lowest validation error.
  • Epoch 20: training error falls to 1%, but validation error rises to 20%.

During the first 8 epochs, both curves fall as the model learns more general patterns. After that, training error keeps falling while validation error rises, indicating that it has begun memorizing particular wording and exceptions. Google's materials describe loss curves that initially fall together and then diverge as a strong signal of overfitting.

Goodfellow et al. define underfitting as being unable to reduce training error sufficiently, and overfitting as having too large a gap between training and test error. Looking only at the 1% at epoch 20 would make that model seem best; considering both curves, the version around epoch 8 is more reliable.

Illustrative error curves: training error keeps falling; validation error rises after its minimum at epoch 8. The gap is largest at epoch 20.
The illustrative figures are invented for teaching. Overfitting appears where validation error starts rising and the curves diverge. · Image: Mokaair (© Mokaair)
Read the full description

An illustrative chart with invented teaching values, not actual training results. The horizontal axis shows training epochs; the vertical axis shows error rate. Dark green training error falls from 30% at epoch 1 to 1% at epoch 20. Orange validation error falls from 31% to 12% at epoch 8, then rises to 20% at epoch 20. Before epoch 8, both lines fall, indicating that the model is still learning general patterns. After epoch 8, the lines diverge and the gap grows, indicating overfitting. Early stopping would select the epoch 8 version. Notes on the right: both falling means the model is still learning patterns; falling training error and rising validation error mean overfitting; both high means underfitting.

Distinguishing Underfitting, a Good Fit and Overfitting

Google's materials broadly divide the causes into two categories: training data that does not represent real-world data, or an overly complex model. Goodfellow et al. use "capacity" to describe the range of functions a model can fit. Excessive capacity may let it memorize details that are useless on new data.

Conceptual comparison based on Google's Machine Learning Crash Course, Chapter 5 of Deep Learning and the GSM1k paper; information current through 2026-10.
StateTraining and Validation ErrorWhat to Try First
UnderfittingBoth high, small gapChange features, train for more epochs, reduce regularization strength
Good fitBoth low, small gapKeep this version; evaluate once on the test set at the end
OverfittingLow training error, high validation error, large gapAdd data, strengthen regularization, stop early
Leaked questionsHigh scores on public questions, substantially lower on new questionsRetest with new, unpublished questions

Common Remedies: What Each One Does

  • More data: Goodfellow et al. consider this the best way to improve generalization—performance on new data—but the data must be representative.
  • Data augmentation: create new examples by modifying existing ones without changing their answers, such as shifting an image by a few pixels.
  • Regularization: penalize model complexity during training. L2 regularization, also called weight decay, pushes weights toward 0; excessive strength causes underfitting.
  • dropout (called "dropout regularization" in Google's Traditional Chinese glossary): randomly and temporarily remove some units at each training step so that units cannot rely solely on particular other units.
  • Early stopping: stop when validation error no longer improves, retaining the parameters from the epoch with the lowest error. Google's materials say it is fast but rarely optimal.
  • Simplify the model: reduce features or model size to lower capacity.

dropout first appeared in Hinton et al.'s 2012 preprint and was explained fully in Srivastava et al.'s 2014 journal paper. Its purpose is to reduce overfitting in large . None of these remedies is guaranteed to work. Which ones to use, and at what strength, must still be guided by validation error.

Networks Can Memorize Random Answers: Zhang et al.'s 2017 Experiment

In their paper published at ICLR 2017, Zhang et al. replaced the labels in CIFAR10 (small images in 10 classes) and ImageNet with random labels, then trained standard image classification networks as usual. Training error on CIFAR10 still fell to 0 or nearly 0, and training accuracy on ImageNet reached 95.20%, while test performance on both fell to chance level. On CIFAR10, the networks could even memorize images replaced with pure noise.

The comparison in Table 1 of the paper: on CIFAR10, with data augmentation and weight decay disabled, the same network achieved 100% training accuracy and 85.75% test accuracy with real labels. With random labels, training accuracy was still 100%, but test accuracy was only 9.78%, close to the 10% chance level for 10 classes.

The authors concluded that these networks have enough effective capacity to memorize the entire training dataset, and that traditional complexity measures cannot distinguish good from poor generalization. Explicit regularization such as weight decay and dropout may improve generalization, but it is neither necessary nor sufficient on its own to control the gap between training and test performance.

What the paper did not say also needs to be clear: it did not say large networks always overfit real data—the same network still achieved 85.75% on new images. It did not say regularization is useless, nor did it resolve why networks can generalize. The experiments covered only image classification and cannot be applied directly to large language models.

Thus, "larger models are more prone to overfitting" is not a universal rule. In 2019, Nakkiran et al. observed that test error first fell, then rose, then fell again as models grew larger in settings including CIFAR-10 and CIFAR-100 image classification and IWSLT'14 German-to-English translation. This is called double descent. The peak occurred where the model could just drive training error close to 0. The effect was strongest with noisy training labels, and in some settings, adding examples actually worsened performance.

Leaked Questions: Apparent Competence on Benchmarks

A 2023 position paper by Sainz et al. identifies the most serious form of data contamination as a large language model seeing a benchmark's test questions during training and later being evaluated on that same benchmark, inflating its score. A test set measures real ability only if its questions are new to the model; once questions leak, scores may measure only memorization.

Even without leakage, a model can gradually be tuned to a test set: Goodfellow et al. write that evaluations become overly optimistic when the research community uses the same test set for years.

In 2024, Hugh Zhang et al. commissioned humans to write 1,205 new questions matching the style and difficulty of GSM8K, a benchmark of grade-school math word problems. This new set, GSM1k, was not public when the paper was released. In their settings, model accuracy dropped by up to about 8%. Models more likely to generate original GSM8K questions generally showed larger gaps (Spearman r² = 0.36), suggesting that some models may have memorized the original questions. However, many models showed almost no gap, and all models overall could still solve new questions.

Leakage and overfitting do not have a one-to-one relationship either: some models showed large gaps without appearing to have memorized the original questions. The authors suggested that they might have used data very similar to the benchmark, or selected model versions using benchmark scores. A small amount of leakage does not necessarily cause overfitting, either. In 2019, Recht et al. created new test sets for CIFAR-10 and ImageNet. The various models they evaluated lost 3% to 15% and 11% to 14% in accuracy, respectively. However, they attributed this mainly to insufficient generalization to slightly harder new images, rather than repeated use of the test sets. Lower scores on new questions are a warning; the cause requires separate investigation.

Applying This When Training, Fine-Tuning or Reading Scores

On an existing model, also continues training using your own data. You still need to set aside validation and test sets first, remove duplicate examples, and pause to compare versions when validation error rises.

When reading someone else's scores, first ask whether the questions were public and whether the model was retested on new questions. The answers cannot prove that the model is problem-free; they only help you judge how much to discount its scores.

To look up other terms, start with the AI Terminology Index.

Found this guide helpful? Share it with someone who needs it

Share on LINE
  • Lifestyle

    Agentic Retrieval-Augmented Generation (Agentic RAG): When to Search Again

    Agentic RAG lets models make retrieval decisions, choosing sources, rewriting queries or searching further based on the question and available evidence. Using a concrete LangGraph workflow and a comparison of venue-use conditions, this article contrasts it with fixed RAG and explains stopping criteria, permissions, query logs and cost control. Learn to judge whether a task needs dynamic exploration or a predictable fixed workflow is enough.

  • Lifestyle

    Graph Retrieval-Augmented Generation (GraphRAG): Understanding a Document Collection Through Relationships

    GraphRAG uses entities and relationships in retrieval to answer questions about cross-document connections and overall themes. Drawing on Microsoft's original paper and official docs, this article explains entity extraction, community summaries, and local and global search, using local event records to show why different questions need different paths. It also shows how missing links, mistaken name merges, and distorted summaries affect answers: a pretty graph is not verified fact.

  • Lifestyle

    Knowledge Graphs: Making Entities and Relationships Clear

    Knowledge graphs organize entities and relationships to trace who took part in what, where and when through links. Using community events, this article covers entity identity, relationship direction, sources and time; compares Resource Description Framework (RDF) and property graphs; and explains why graphs are not equivalent to visualizations or reliable facts. Learn when to build one and how it works with tables, vector search and graph retrieval-augmented generation (GraphRAG).

  • Lifestyle

    Hybrid Search: Balancing Exact Terms and Meaning

    Hybrid search merges full-text, vector and other retrieval results to cover exact codes and natural-language needs. An equipment-manual query explains candidate retrieval, deduplication, rank fusion and reranking, including how Reciprocal Rank Fusion (RRF) merges results by rank. Hybrid search is not always more accurate. Learn to test quality with code, paraphrase and no-answer queries and spot biases from permissions, filters and duplicate content.

Sources

Compiled from the references and verification dates listed above; fares, schedules, product features, and terms can change, so please confirm the latest details on the relevant official website before travelling, booking, or purchasing.

Lifestyle