The first problem I set out to solve was evaluation consistency, not cost. When a generative LLM serves as the judge, its response format and decision sometimes vary even under the same criteria. Large batches therefore needed a repeatable method of automated evaluation.
Cost still mattered. LLM spending on batch generation of English questions, including evaluation, had reached cumulative tens of millions of won, and the validation stage called a high-end model at roughly KRW 50 (US$0.04) and 30 seconds per item.
Model selection
Reliable access to local GPU resources was difficult at the time, so I excluded self-hosting a small language model (SLM). Instead, I considered lower-cost commercial API models that could be compared directly within the existing pipeline. Costs were calculated by directly tallying input and output token usage across roughly 100 real sample items and applying the API pricing at the time.
Approach
Having the lower-cost model run the same free-form evaluation made evaluation stability even worse. So instead of deciding from the scores and explanations the validation model produced, I built logic that compared the log probabilities of candidate labels. The validation step only needed a pass or fail decision.
I compared the lower-cost commercial model's decisions with those of the existing high-end model, examining cost, processing time, agreement, and reproducibility across repeated runs. To hold conditions constant, I reran the experiment within the Prefect-based generation, validation, and loading pipeline.
PoC results
| Existing judge | PoC candidate | |
|---|---|---|
| Token cost per item | Approx. KRW 50 (US$0.04) | Approx. KRW 5 (US$0.004; −90%) |
| Processing time per item | Approx. 30 sec | Approx. 15 sec (−50%) |
In the PoC, run on roughly 100 real sample items, the candidate model's decisions all matched those of the existing high-end model.