Technical notes · 2026-09
Ten good images are enough for the model, not for the threshold
A few-shot study on line-scan fabric inspection: 8 seeds, 160 operating points, and why none of them passed.
The question
The deployed detector builds its memory bank from 10 good images and sets its thresholds on 50 more. On a 133-image test set (26 defective, 107 good) it catches 24 of 26 defects with 3 false alarms.
Setting up a new product would be much easier if 10 or 15 good images in total were enough, for both the bank and the threshold. This note records what happened when I tried.
Protocol, fixed before scoring
- Data: the Rollo7A roll of the public RAW-FABRID dataset (CC BY 4.0), with the same 133 test images.
- Model: MobileNetV3 pretrained on ImageNet, no fine-tuning. Features from blocks 4 and 6, 448 px tiles, 512 patches per image in the bank.
- Budgets: 10 or 15 good images. Eight seeds (39–46) decide which images. The 10-image set is the start of the 15-image set, so the budgets are nested, not independent.
- Threshold: leave-one-out on the same good images (each one is scored by a bank built without it), then median + 3 × 1.4826 × MAD. No defect image is used to fit anything.
- Gate: at least 25/26 defects and at most 3/107 false alarms.
- 10 scoring or calibration variants × 2 budgets × 8 seeds = 160 operating points. They share one roll and one test set, so they are not 160 independent checks.
Results
| Variant | TP, 10 good | FP, 10 good | TP, 15 good | FP, 15 good |
|---|---|---|---|---|
| Baseline: distance to nearest neighbour | 25–25 | 5–13 | 25–25 | 4–11 |
| Mean of 3 nearest | 24–25 | 4–12 | 23–25 | 4–11 |
| Neighbourhood reweighting (PatchCore-style) | 25–25 | 5–13 | 25–25 | 4–11 |
| Distance ÷ local radius | 25–25 | 6–19 | 21–25 | 6–12 |
| Distance − local radius | 25–25 | 6–45 | 24–25 | 6–34 |
| Context halo 32 px | 21–25 | 4–9 | 23–25 | 6–8 |
| Context halo 64 px | 21–25 | 5–9 | 22–25 | 6–10 |
| Cross-fit threshold, MAD | 25–25 | 5–22 | 24–25 | 4–11 |
| Cross-fit threshold, q95 × 1.10 | 0–24 | 0–5 | 4–24 | 0–5 |
| Cross-fit threshold, max × 1.10 | 0–24 | 0–5 | 1–23 | 0–5 |
No operating point passed. The rules with few false alarms got there by raising the threshold until most defects were missed, not by separating good from bad better.
Is it the threshold or the score?
To find out, I cheated on purpose and picked the threshold with the test labels. Even then, keeping 25 defects cost at least 4–5 false alarms for the baseline and 4–7 for the other scores. So the problem is how the score ranks images, not only where the line is drawn.
Four good images (145, 179, 183 and 218) were flagged in all 8 seeds at both budgets. They contain local structures and horizontal texture bands that the features treat like defects. I kept their labels: looking similar to a defect is not proof that a label is wrong.
What else did not work
- Choosing good images spread across the whole roll instead of from its start: median false alarms 8 against 8.5 on a new 83-image holdout, and none of the 32 banks reached the ≤ 2/83 target.
- Subtracting a per-row background from the score: still 25 defects caught, still 4–13 false alarms.
- Keeping all 11,760 patches per image, a bank 23 times larger: false alarms moved by −3, +1 and 0 on three seeds, while P95 latency rose from about 30 ms to 272 ms on an RTX A4000.
- Ensembles of 9 sub-banks, and comparing each patch with distant regions of the same image: no point passed.
The bottleneck is the threshold
Running the deployed pipeline with only its own 10 bank images for calibration showed why. In leave-one-out, nine of the ten good images scored between 11 and 22. One scored 38.2. A max rule follows that single image: 7/26 defects caught, 0 false alarms. Using the second-highest score instead gives 22/26 with 3/107, close but still below the deployed 24/26.
Adding 5 more images (15 in total) did not change a single image-level decision. Augmenting the good images (brightness, gamma, row bands, shifts) raised their scores 1.05–1.25 times but kept the same order, so it cannot replace independent good images.
Ten good images are enough to build the bank. They are not enough to set the threshold.
A lead, with its caveat
A later probe added a DINOv2 branch next to PatchCore and flagged an image only when both branches exceeded k times their median leave-one-out score. Against a looser target set for the product (at least 22/26, at most 5/107) it passed on 17 of 17 random sets of 5–20 good images.
But the two k values were picked from a 36-cell grid on that same test set. Choosing on one half of the test and checking on the other half (300 random splits) brought the pass rate down to 42%. It is a lead to test on a new roll, not a result.
What I take from it
- With 10 good images the deployed score still ranked images at an AUROC of 0.95, yet no threshold rule fixed in advance passed. For a production line, report counts at a fixed threshold, not only AUROC.
- 3/107 does not prove a false-alarm rate below 3%. Even 0/107 only bounds it at about 2.8% (one-sided 95%). That needs hundreds to thousands of independent good images.
- Fix the protocol before looking at results, and stop tuning on a test set that has already been seen many times.
- Write down negative results so the next attempt does not repeat them.
Limits
All of this is development work on one roll and one test set that was reused many times. It does not show that few-shot methods fail in general, only that these configurations did not pass on this data.