Master Thesis: Long Context Extension for Public Sector Document Retrieval
AI Sweden
Summary
Extending the context length of dense LLMs beyond their pre-training limits is essential for practical deployment in public administration. Context extension techniques like YaRN or LongRoPE show promise but require rigorous stress-testing on domain-specific, low-resource administrative languages to ensure precision in Retrieval-Augmented Generation (RAG) tasks.
Job description
Introduction
Extending the context length of dense LLMs beyond their pre-training limits is essential for practical deployment in public administration. Context extension techniques like YaRN or LongRoPE show promise but require rigorous stress-testing on domain-specific, low-resource administrative languages to ensure precision in Retrieval-Augmented Generation (RAG) tasks.
Project Background and Problem Statement
AI Sweden is developing context length extensions for the Prelude 9B model. Simultaneously, Skatteverket requires "Labb/experiment" Proof-of-Concepts (POCs) for utilizing LLMs with their internal documents (Väg 1). This thesis investigates:
- How efficiently can Prelude 9B be extended to 32k or 64k (or longer) tokens for Swedish administrative RAG tasks without catastrophic forgetting of its core reasoning and language modeling capabilities?
Outline
The goal is to extend and evaluate the context window of Prelude 9B for real-world utility.
- Literature study: Investigate RoPE scaling, YaRN, LongRoPE techniques, and Needle-in-a-Haystack (NIAH) evaluations.
- Implementation: Apply continued pre-training or SFT on a high-quality Swedish long-context dataset to extend Prelude's window to 32k/64k tokens.
- Evaluation: Deploy the model in a localized RAG environment simulating Skatteverket's "Väg 1" lab setting and evaluate precision, recall, and MFU (Model FLOPs Utilization).
Who we're looking for
We are seeking curious, self-driven MSc students eager to work at the frontier of open-weight European AI research (LLMs). You thrive on empirical discovery, design rigorous experiments, and let data challenge your assumptions.
- Ongoing Master's studies in Computer Science, Data Science, Machine Learning, Engineering Physics, or a related quantitative field.
- Proficiency in Python and hands-on experience with modern deep learning frameworks (PyTorch, Hugging Face ecosystem).
- Familiarity with LLM post-training alignment (e.g., SFT, DPO, RLHF/RLVR) or context-extension, alongside comfort running distributed GPU training in Linux/HPC environments.
Requirements
- Ongoing Master's studies in Computer Science, Data Science, Machine Learning, Engineering Physics, or a related quantitative field.
- Proficiency in Python and hands-on experience with modern deep learning frameworks (PyTorch, Hugging Face ecosystem).
- Familiarity with LLM post-training alignment (e.g., SFT, DPO, RLHF/RLVR) or context-extension, alongside comfort running distributed GPU training in Linux/HPC environments.
Company overview
AI Sweden is Sweden's national center for applied artificial intelligence, established in 2019 with a mission to accelerate the adoption and implementation of AI technologies for societal benefit, economic competitiveness, and national advancement. As a collaborative research institute, it brings together over 170 partners from diverse sectors including government agencies, universities, research institutions, startups, and global corporations. The organization operates as a catalyst for AI innovation, focusing on practical applications across healthcare, energy systems, municipalities, and various industries. With a core team of over 80 professionals comprising AI experts, data scientists, research engineers, linguists, mathematicians, policy experts, and change leaders, AI Sweden facilitates knowledge sharing, talent development, and infrastructure building. The institute operates on a not-for-profit basis, funded by Sweden's innovation agency Vinnova, the Swedish Agency for Economic and Regional Growth, European Regional Development Fund, Västra Götalandsregionen, and partner contributions. Through its AI Labs, talent programs, workshops, and collaborative projects, AI Sweden addresses critical societal challenges while bridging the gap between cutting-edge research and real-world implementation, positioning Sweden as a leader in responsible and impactful AI development.