Master Thesis: Cross-Lingual Safety and System Identity Alignment via DPO

Introduction

Open-weight multilingual models should maintain a consistent system identity and appropriate safety behaviour across languages. However, safety alignment can vary across languages and may introduce over-refusal, particularly in lower-resource settings.

Project Background and Problem Statement

We have developed the openeurollm-model-identity dataset and a multilingual safety refusal suite. This thesis investigates DPO-based alignment in Swedish and Icelandic, focusing on three questions:

• System identity: Does the model reliably identify itself as OpenEuroLLM and resist attempts to overwrite that identity?
• Safety alignment: Does the model appropriately refuse genuinely unsafe requests across languages?
• Over-refusal: Does safety alignment cause the model to refuse legitimate requests?

The broader question is whether these behaviours can be improved without degrading general instruction-following performance.

Outline

• Literature study: Review cross-lingual safety alignment, over-refusal, model identity, and preference optimization.
• Data analysis: Assess the quality and suitability of the preference data, including both preference correctness and the absolute quality of chosen responses.
• Implementation: Fine-tune Prelude 9B using identity and safety preference data with Hugging Face TRL or Megatron-LM. Compare identity-only, safety-only, and combined training where feasible.
• Evaluation: Compare the resulting models against the baseline on identity consistency, safety and jailbreak robustness, over-refusal on benign queries, and general instruction-following performance in Swedish and Icelandic.

Who we're looking for

We are seeking curious, self-driven MSc students eager to work at the frontier of open-weight European AI research (LLMs). You thrive on empirical discovery, design rigorous experiments, and let data challenge your assumptions.

• Ongoing Master's studies in Computer Science, Data Science, Machine Learning, Engineering Physics, or a related quantitative field.
• Proficiency in Python and hands-on experience with modern deep learning frameworks (PyTorch, Hugging Face ecosystem).
• Familiarity with LLM post-training alignment (e.g., SFT, DPO, RLHF/RLVR) or context-extension, alongside comfort running distributed GPU training in Linux/HPC environments.