Reusing Model Probabilities for New Queries
Can one set of model probabilities answer many different questions? This paper studies how fine-grained predictions can be combined into broader categories, keeping answers consistent across label groupings without querying a model again for each grouping.
Experiments examine intent classification, logical queries, and semantic parsing. The practical benefits depend on prompt detail, training, and the questions being asked: consistency is guaranteed by construction, while better predictive performance is not. Reuse was faster in the measured workloads with two or more groupings of the same input.
Cite this paper
Stamate, A., Paraoulaki de Miranda, V., & Wunker, W. (2026). Reusing Model Probabilities for New Queries [Preprint]. Zenodo. https://doi.org/10.5281/zenodo.23584994