Before I joined, the initial internal review pass rate for generated questions was under 10%. After joining, I took ownership of the generation logic, pipelines, and prompt engineering to raise question quality, while also contributing to backend work for the service. I collaborated with content specialists to codify their tacit knowledge into logic, and worked with the team on the pipelines the service needed.
From batch generation to a real-time service
This was not an experimental notebook, but a production pipeline for generating, validating, and loading questions. Depending on the question type, the system generated batches of 20,000–90,000 questions. The same generation modules also served real-time user requests. Cumulative LLM spending for batch generation ran into the tens of millions of won.
For the service pipeline, the team jointly built the Python and FastAPI microservices, Prefect-based orchestration, and the dbt and PostgreSQL data pipeline. The team's backend was designed with each module split into a layered architecture, with type-specific specifications supplied through dependency injection. In that environment, I contributed to backend module implementation from the initial development phase and worked on type-specific LLM generation and validation logic and prompt engineering.
Some question types used in-house-trained ML models, and I also took part in serving those models and connecting their outputs to the LLM pipeline.
Generation logic
More than ten question types—including topic, main idea, cloze, grammar, vocabulary, and paragraph ordering—differed both in how experts authored them and in the criteria used to judge question quality. Working with content specialists, I codified each type's authoring process step by step and sorted the tacit knowledge into four categories.
- Requirements that must always be satisfied
- Expressions and structures that must not be used
- Review acceptance criteria
- Post-processing rules that can be implemented in code
Instead of delegating every judgment to an LLM, I maintained separate prompts and few-shot examples for each type and moved deterministically checkable conditions into post-processing logic. Questions generated through the Gemini and OpenAI APIs were checked automatically before expert review, and those review results fed back into the rules and examples for the next experiment.
A repeatable review loop
If the way a run executed changed every time a prompt was modified, there would be no way to compare which changes actually worked. I fixed generation, validation, and loading into a Prefect pipeline so the same input always went through the same flow. That made it possible to see in numbers how the review pass rate shifted when a single rule or example changed.
In this pipeline, human review was not a final gate outside the automation. When a specialist judged that “the distractors are too easy,” we examined that feedback together, unearthed the tacit knowledge hidden in it, and sorted it into the four rule categories. Those codified criteria went back into the prompts and logic and re-entered the pipeline, and quality rose as this loop accumulated.
Quality
For raw output before the validation logic ran, the expert review pass rate rose from under 10% to about 70%. In an answer generation PoC, optimizing prompts and post-processing logic improved accuracy from 91.21% to 96.45%. The questions ultimately delivered after validation maintained zero quality complaints over six months. These numbers came less from any single model's performance than from connecting the type-specific generation logic, the evaluation pipeline, and the experts' item-writing logic.
Related projects
- Improving Answer-Choice Combination for Korean CSAT English Questions: restored the valid question pool from about 40,000 to about 90,000 without generating additional answer choices with an LLM after a combination bug reduced the pool.
- Optimizing LLM Item-Validation Costs: used a PoC to test whether evaluation consistency could be improved while reducing token costs and processing time.