Use case · Idea to test
Get an LLM result sooner
Use less model work before returning an answer.
- Read the request
- Check a candidate result
- Return it or continue
A tool request might only need a label or action, while a chat request needs a written answer. Both could benefit from less computation, but they require different stopping methods.
Where the small model fits
For a fixed decision, a head can read an intermediate layer and return an enum. For a written answer, an early-exit decoder or a small draft model could propose tokens for a larger model to check. A confident label head alone cannot supply or validate the rest of a conversation.
What would need to work
Measure time to the first result, time to finish, answer quality and verification overhead. Our layer-exit tests concern classifiers, not a demonstrated faster general-purpose chat model.
Try the related work
This demonstrates classification, not early generation of complete LLM replies.