LLäMmlein & ModernGBERT at DNB Symposium in Frankfurt

On January 29, we delivered a keynote titled "LLäMmlein and ModernGBERT: A New German LLM Family in Research and Application", at the symposium "Netzwerk maschinelle Verfahren in der Erschließung", hosted by the German National Library (DNB) in Frankfurt.

Abstract

While most large language models focus on English, we introduce two complementary, German-first model families: LLäMmlein, a decoder-only LLM (120M–7B parameters), and ModernGBERT, an encoder-only family (138M–1B parameters). Built for scalability and transparency, both are trained entirely from scratch on a German-only corpus using a custom German tokenizer. We describe our motivation for building fully German models, how we constructed the dataset and training pipeline, and the challenges we overcame in tokenization, corpus curation, and stable scaling across different model sizes and cluster configurations. For evaluation, we introduce Supergleber, our German-specific benchmark, and report results across tasks to analyze how quality scales with model size and architecture. We release the data curation process, tokenizer, training recipes, checkpoints, and evaluation suite to support reproducibility and advance German-language AI.

More information regarding the symposium can be found here.