
A visual knowledge-distillation lab: trains a teacher and a student (with and without distillation) on 2D datasets and compares their decision boundaries, showing what the student actually learns from the teacher beyond accuracy.
Every experiment plots three panels side by side — teacher, student trained on hard labels only, and an architecturally identical student distilled from the teacher — so instead of comparing accuracy numbers you can literally see the boundary each model settled on. On datasets with a real "void" (regions with no training data, like the XOR-blobs case pictured), the no-KD student and the teacher resolve the ambiguity differently, while the KD student — trained on extra teacher-labeled query points sampled from that empty region — reproduces the teacher's exact boundary almost perfectly.
Two further experiments (overlapping blobs, a biased teacher) show the same mechanism from the opposite angle: when there's no void to fill, KD instead transfers the teacher's calibration — including its flaws, like overconfidence or a spurious shortcut feature — which the README argues is structurally the same effect behind why sequence-level distillation works for LLMs, not just a toy analogy.
- Python
- PyTorch
- Knowledge Distillation
- Visualization

