2026-11-13
t50 blind vote: the live model stays
Alex rated 40 side-by-side pairs, live model v39 against the best retrain (T3-s0, the flip-bug fix). v39 won 20, T3-s0 won 14, 2 ties, 4 pairs where both were wrong. The live model stays and T3 does not ship.
Note: the pairs showed raw model output, before the production rules engine. Lines you see here are not what a customer scan would have shown.
What came up in his 17 comments
| Defect | Pairs | Count |
|---|---|---|
| Double-plotted stripes (two lines on one physical stripe, breaks rule #8) | 0f4a, 4e06, 8ab6, c9c1, fa36 | 5 |
| Bottom stripe plotted off the luff / "aggressive tick up at the luff" | 1d81, 33d0, 72eb, 7812 (also fa36 luff/leech accuracy) | 4 (5 with fa36) |
| Photo has no draft stripes and doesn't belong in training | c43a, c453, fdc4 | 3 |
| Fisheye / shot from the mast, poor training photo | 7812 | 1 |
| A stripe missed entirely or half-missed | 0f4a, 178c, 33d0, 783c, 94b1, f1b4 | 6 |
Some comments fall in more than one row. By side: the live model v39 double-plotted in 3 pairs (0f4a, 8ab6, c9c1), T3-s0 in 1 (4e06), and in fa36 he doesn't say which side. So this hits both models, not just the retrain.
Follow-ups from his comments: he thought the luff tick was fixed when a stripe runs off the photo (1d81), and he says the model should pick up battens instead of leaving those photos empty (c43a).
Would prod's rules have caught the double-plots?
Partly, by reading the code only (nothing was run or changed):
- Detection: yes.
checkAllRules(server/inference/rule_checker.mjs) has two rule #8 checks:stripe-overlap(median-spacing test) andstripe-same-stripe-duplicate(neighbour-relative, fromdupLineMerge.mjs::findSameStripePairs). Both mark a hard violation on a set of 3+ lines. The same-stripe check was written for exactly this case, two duplicate pairs in one set that fool the median test. - Removal: only if the flags are on. Dropping the duplicate needs
DUP_LINE_MERGE=1orDEDUPE_SAME_STRIPE=1. Both default OFF in code. I did not check the live Railway values. Without them prod flags the photo (VIOLATED RULES queue, shown with its lines) rather than deleting the extra line. - The first-pass filter in
pipeline/stripeFilters.js(overlap-dedupe) only drops lines closer than 0.2x the median stripe gap (ceiling 2.5% of height). That catches near-identical lines, not a second line a few percent of height away on the same stripe. - I did not run these checks on the 5 pairs' actual points, so "would have caught it" is from the rules as written, not a replay.
All 40 pairs
| Pair | Photo id | His pick | His comment |
|---|---|---|---|
| 0421 | 6f61cdd9-eddf-4af7-8d24-e3de0d2a6813 | v39 (live) | |
| 0608 | 2ca17e83-1dd5-458f-baa9-e90e2a6ef17a | T3-s0 | |
| 0f4a | 35fc0a81-2982-4c5d-b251-866a49f8ae58 | Both wrong | Left is more accurate, although it double plots the bottom stripe and misses the top stripe. The right photo only picks up the mid stripe |
| 178c | 512489c3-71f1-41dd-9524-3b39fcbf35dc | v39 (live) | Left wins only because it picks up all four stripes, however, accuracy is terrible |
| 190a | 5996863e-b974-4d80-96f0-19af7f55ec85 | v39 (live) | |
| 1aa0 | 190d9f80-319f-4c27-8bed-ef92baf42124 | T3-s0 | |
| 1d81 | 7a7f56c8-03db-4532-9c8e-a1933aa1c1b3 | v39 (live) | Left-hand picture is better, but both do the aggressive tech up on the laugh at the bottom stripe. I thought we’d fix this issue when the stripe is plotted off of the photo |
| 33d0 | 5d155571-bab3-41aa-b810-e92e051130f7 | Both wrong | Both wrong, the left photo misses half of the middle stripe and plots the bottom stripe terribly. The right hand photo misses the top stripe and plots the bottom stripe better, but still wrong. too aggressive of a tick at the luff |
| 4e06 | 674b015b-90b4-4669-98de-93fd769752ad | v39 (live) | Write is better but lack accuracy on the bottom stripe is not very good left photo again is double plotting stripes, which breaks our ground truth rules |
| 57fd | 464eb114-460a-456a-b044-8a205e8d8e1e | v39 (live) | |
| 5b39 | 8f498974-2ae2-4734-b3aa-6a88df1dc490 | v39 (live) | |
| 5eae | 607171b7-d115-42ea-a29c-6b81eb4ecffc | v39 (live) | |
| 64b5 | 48e9ec7a-a829-4567-9fe3-4ec196da9084 | T3-s0 | |
| 72eb | 572af92a-2e2f-4df9-9c79-adb1fc45b054 | v39 (live) | Right is better but plots bottom stripe off of luff |
| 7812 | 908215db-865f-48ab-9bde-d6c324f85253 | T3-s0 | Left wins on picking up all the stripes, but plots the stripes off of the luff. However, this is not a great training photo as the user has taken a picture in fish eye mode and from the mast instead of middle of the main sail |
| 783c | 410f4343-fbc6-461f-829c-fe5ddbcc4eb7 | Both wrong | Left photo picks up the top, but totally misses the middle and bottom stripe. Right hand photo picks up the bottom stripe that is terrible on accuracy |
| 8ab6 | 68f95fda-ab6d-45b4-8caf-6f1c8fccef41 | v39 (live) | Left photo wins on picking up all the stripes, but it double plots the top stripe and the bottom stripe accuracy is not very good. The bottom stripe accuracy on the right hand photo is much better |
| 8b36 | 7ddce43e-0625-40e6-bd09-b2db00224196 | v39 (live) | |
| 8ea6 | 4de5f204-4cc9-48ba-a636-f444ae5a5017 | T3-s0 | Left is better, but misses the back drafting slight aspend on the bottom stripe |
| 94b1 | 22fcf683-f5a4-41da-865c-13e828b15de0 | v39 (live) | Left better, but misses top two stripes |
| 9900 | 6ff0668f-6f4b-443f-a9ea-773d57ee85ed | v39 (live) | |
| 9a5c | 2065a1fa-b7f1-42b1-978e-1e4aef33e016 | v39 (live) | |
| a208 | 2bbba290-c602-41b2-a1db-461b1cbadf44 | T3-s0 | |
| a8b7 | 6e077c3c-ed86-4ad2-a157-ad9e0da6d522 | Tie | |
| b160 | 1c036663-7b5d-48c7-a128-d5a8409e473d | v39 (live) | |
| c43a | 60a4604d-5f2d-43ba-a15a-95165913b9c0 | v39 (live) | Left photo marginally wins as it picks up a batten, honestly, not a great picture for the training data set, as there’s no draft stripes; however, the model should be able to pick up battens instead |
| c453 | 226f3f54-ba73-45f5-b63f-8eb2be321c49 | Both wrong | This photo shouldn’t be in the training set. There are no draft stripes on it |
| c9c1 | 78ad37b0-0176-4380-a916-9ddcc0272e45 | T3-s0 | The left photo is much better. Again, the right photo is double plotting stripes on top of each other, breaking our rules |
| dae0 | 69b0dc07-dca6-4eb0-9f00-7b68ec5c5066 | T3-s0 | |
| de70 | 92336393-3a8d-4f9a-b835-0cf64f88d4cf | v39 (live) | |
| ded3 | 4e177a82-0612-4840-a480-3fb2cdfd75ec | v39 (live) | |
| dfbe | 0c86676b-7599-41a5-b4f5-7c6bf7982d93 | T3-s0 | |
| f129 | 15c82afe-cf8d-4c88-a191-4ebb0c055d32 | v39 (live) | |
| f1b4 | 29657778-5bd7-482b-9291-25e3152e52de | T3-s0 | Left better, but both miss top stripe, photo also traces middle stripe off of leech |
| f429 | 42017e33-890e-4c6d-a196-4540b68d8394 | T3-s0 | |
| f470 | 3cad1a93-d14c-4edc-8639-f6b067be641d | Tie | |
| f866 | 911924e8-c9d1-4cd4-84db-047985436271 | v39 (live) | |
| fa36 | 50e49326-2ec8-4c88-a1a9-a2e85ab5415d | T3-s0 | Left is more accurate, but the leech accuracy is not very good nor the luff, on the right the bottom stripe and is wrong. I’m not sure why we’re double plotting stripes that breaks the rules we set |
| fb28 | 8e280018-7d29-4238-84e9-709f921a1aae | T3-s0 | |
| fdc4 | 4a39377f-04c1-457c-9682-f114f64eb2c2 | T3-s0 | Right, more accurate, but this photo probably shouldn’t be in the training set as it doesn’t have draft stripes. However, I think the accuracy is good |