Pretraining a 110M-Parameter Korean Language Model

When pretraining a Korean language model from scratch, would presenting the same data from easy to difficult, the way humans learn, help the model learn faster than random shuffling? I designed and ran this personal experiment around that question. The dataset and model size stayed the same, and only the order changed. I trained a 110M-parameter model on a single pass over 2.5 billion tokens and compared a random baseline with three curriculum conditions.

The four orders were:

• random: the baseline, with all data randomly shuffled
• length: divide sentences into ten bands by length and train from shortest to longest
• difficulty: train in ascending order of difficulty calculated from morphological statistics
• mixed: mix 10-15% of the next difficulty band into the current band during training

random, length, and difficulty used the same corpus and changed only its order. mixed replaced the most difficult 10% of tokens with easier data, so it includes a data-selection effect as well as an ordering effect. I treated it as a separate condition from the outset to avoid interpreting that difference as a pure ordering effect. All four conditions used exactly 2.5 billion training tokens.

Controlling everything except order

The model is a 110M-parameter decoder-only Transformer. It uses Llama-family RMSNorm, RoPE, and SwiGLU; I implemented both the model and training loop directly in PyTorch. The tokenizer is a 32k byte-level BPE trained with Hugging Face Tokenizers, and the input and output embeddings share weights.

I cleaned 4.55 million documents containing 6.8 billion characters to create a training pool of 2.94 billion tokens under the custom BPE. Processing included NFC normalization, Korean-character-ratio and length filters, and exact deduplication with BLAKE2 hashes. Content hashes determined the train/validation split, ensuring that a rerun assigns the same document to the same split. I used 8.32 million paragraph-preserving chunks of at most 512 tokens as the ordering units.

The shared corpus C for random, length, and difficulty contained 7.06 million chunks. For mixed, I removed 713,554 chunks from the highest-difficulty band (250 million tokens) and replaced them with 710,017 chunks from the easiest 75% of the data, matching the total token count exactly.

I tested that runs with the same seed started from identical initial weights and generated each data order as a file before training so it remained fixed. Sentence-level shuffling could itself make the baseline a nonstandard training setup, so I chose chunks of at most 512 tokens as the unit of ordering.

Training ran in bf16 on a single RTX 4090 24GB instance in RunPod Secure Cloud. Throughput was approximately 90,000 tokens per second, each run took about 7.8 hours, and the full set of training and evaluation runs cost roughly USD 55-60. I initially planned three seed repetitions. After reviewing the first-seed results, however, I concluded that a potential confound between the learning-rate schedule and data order needed to be addressed first. I therefore prioritized four fixed-learning-rate runs with the same seed and corresponding data order over repeating seeds, comparing eight runs in total: four cosine-schedule runs and four fixed-rate runs.

Reproducible experiment design

I separated data collection, preprocessing, tokenizer training, tokenization, difficulty scoring, order generation, training, and evaluation into the staged CLI main.py prepare → tokenizer → score → orders → train → eval → plot. manifest.json records dataset revisions and the exact data used, while orders/meta.json records the pregenerated order and hash for each condition. I tested that comparison runs began from the same initial weights. For the fixed-LR reruns, I held the seed and order constant and changed only lr = min_lr = 3e-4.

I also fixed checkpoint restoration to recover not only the data position but the CUDA RNG state, preventing loss from changing across an interrupted-and-resumed run. In the process, I found that the checkpoint's map_location moved even RNG-state tensors to the GPU and prevented their restoration. This code path was not covered by CPU tests, so I changed the operating procedure to include a short interruption-and-resumption check on the actual training device before a full run.

Hugging Face Datasets and Hub were used to retrieve corpora and benchmarks, while Tokenizers was used to train the BPE. Transformers was used only in evaluation code that loaded external reference models; the experimental model and its training loop were implemented directly in PyTorch. During KoGPT2 evaluation, I also found that the Transformers GPT-2 wrapper split Korean tokenizer input at the byte level. I added a probe for Korean tokens and loaded the raw tokenizers.Tokenizer instead.

Results

The curriculum conditions required approximately 20-44% more tokens than random to reach the same loss. Their rank order remained unchanged at every intermediate checkpoint, and the validation loss for length even rose slightly from 3.643 to 3.645 near the end of training.

Aggregate loss alone could not explain the ranking, so I split validation results into difficulty bands. The loss gaps between difficulty and random were +0.262 on easy data, +0.080 on medium data, and +0.017 on hard data. The easy-band loss under curriculum training stopped improving after 1.2 billion tokens, consistent with the explanation that performance deteriorated because the model never revisited the easy examples seen early in training.

Fixed-learning-rate reruns

Under a cosine schedule, difficult data always appears in the low-learning-rate portion of training. Before calling the initial result an ordering effect, I needed to examine this confound. I therefore retrained all four conditions with the same seed and data order while holding the learning rate fixed at 3e-4.

With the fixed learning rate, difficulty's validation-loss gap relative to random increased from +0.127 to +0.176, and the easy-band gap grew from +0.262 to +0.369. The condition ranking also remained unchanged. If cosine decay had created the apparent effect, these gaps should have narrowed; instead, they widened. Cosine decay alone therefore could not explain the difference, leaving the failure to revisit easy data as the leading explanation.

Benchmarks

I shuffled outputs for 20 fixed prompts without revealing condition names and asked two LLM judges to compare them. Among cases where both judges chose the same winner, random beat difficulty 12-2; a two-sided binomial test yielded p ≈ 0.013. random also led length 10-5 and mixed 9-6, but neither comparison was statistically significant.

I also evaluated KoBEST, KoBALT-700, and a custom grammaticality minimal-pair set. Every condition reached 96-99% on the minimal pairs within 0.5 billion tokens, while all models remained at chance on KoBALT, so neither benchmark distinguished the conditions. In this experiment, the continuous measures, validation loss and BPC, captured the effect of data order most reliably.

Comparison with external open-weight models

I evaluated the eight final checkpoints and three external models with the same harness. Because their tokenizers differed, all options were scored by log probability normalized by character length.

The external models differ in training-data volume, tokenizer, and tuning method, so they serve as anchors for estimating absolute performance rather than as fair ranking opponents. EXAONE is an instruction-tuned model and may be disadvantaged on MinPair, which compares raw-sentence log probabilities.

Interpretation

Revisiting easy data

In a one-pass curriculum from easy to difficult, difficulty becomes inseparable from training time. Easy texts seen early are never revisited, while difficult texts seen later receive the final parameter updates. In this experiment, the deficit for difficulty was concentrated in the easy band (+0.262) rather than the hard band (+0.017), and the same pattern became stronger under a fixed learning rate. The most likely cause of the gap is that the easy data was never shown again.

A pretraining curriculum should therefore include, alongside difficulty sorting, rehearsal that periodically reintroduces easy data or interleaving across difficulty bands. The fact that mixed lost less ground than the fully sorted conditions is consistent with its preview of the next band making the distribution shift more gradual while also increasing the share of easy data. Because mixed changes both factors, this experiment cannot separate the contribution of previewing from that of data selection.

Data specialization in the external-model comparison

Compared with external open-weight models, this 110M model, pretrained only on Korean, performed comparably to KoGPT2 125M and Gemma 3 270M on the grammaticality minimal pairs and achieved the highest SentiNeg score among the comparison models, including EXAONE 4.0 1.2B. One plausible interpretation is that much larger general-purpose models divide their capacity across multiple languages and tasks, while the smaller model devoted all of its limited capacity to the Korean distribution.

This does not mean a small model understands Korean better than a large one. It showed no consistent advantage on COPA, HellaSwag, or BoolQ, and every model remained at chance on KoBALT. On tasks close to the training distribution, such as grammaticality and sentiment, data specialization partly made up for the smaller parameter count, but it did not on reasoning tasks. Because the models also differ in training-data volume and tuning method, this is not a causal estimate of the effect of Korean-only pretraining.

Human-designed difficulty versus LLM difficulty

Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning (Zhang et al., 2025) reported faster early training for larger English models across several curriculum settings. Model scale, language, data, difficulty metric, and training method all differ, so the experiments are not directly comparable, but I likewise expected an easy-first order to beat random training here, or at least match it. Instead, random led every curriculum condition.

The per-band losses in this experiment show that even the random model ranked hard below mid. The metric intends the difficulty order easy < mid < hard, but a linguistic difficulty measure designed by humans cannot be assumed to mean the same difficulty to an LLM. If the standard of easy and hard differs between humans and LLMs, a human-designed curriculum ends up forcing an inefficient order.

Questions for the next experiment

If forgetting caused the result, then with the total token count fixed, reducing the corpus to one quarter and training for four epochs should make the easy-loss increase disappear under a schedule that revisits the easy band periodically. Comparing reshuffling each epoch, repetition of the full curriculum cycle (spaced), and repetition of difficulty-band blocks (blocked) would test that prediction.

Whether data specialization carries over to other domains is another open question. For example, one could measure how close a small model trained only on code comes to large general-purpose models on code syntax and autocomplete.

A further experiment would replace the human-designed metric with difficulty as the LLM sees it, measured by per-sample loss or forgetting scores on the same corpus, and rebuild the curriculum on that basis. That result is needed to answer the original question of whether an LLM, like a human, learns faster from easy examples first.