Research results
Other decision tests
Public data and harder cases show where the shortcuts break.
Test results · Decision methods
Public banking questions
On all 3,080 BANKING77 test queries, one trained Qwen classifier identified 2,551 topics correctly. Early stopping had similar total accuracy but failed the stricter error check.
All depths, seeds and timing · Stricter follow-up
Unfamiliar requests and changing rules
A second dataset also failed the early-stop check. Adding an UNKNOWN answer did not reliably handle unfamiliar requests. In a separate test with opposite rules, the full model answered both correctly on only 14 of 100 pairs.
Second dataset · UNKNOWN output · Changing-rule test
A label versus a written reply
A number and a single trained output token can use the same classifier. Our matched test did not establish a reliable speed advantage for the number alone; avoiding transformer work is a different question.
Matched output comparison · Try one-letter and JSON output · Separate implementation replay