An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shift

Karouzos, Constantinos; Tan, Xingwei; Aletras, Nikolaos

An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shift

Constantinos Karouzos, Xingwei Tan, Nikolaos Aletras

University of Sheffield
EMNLP 2026

Paper arXiv Code

Adaptation strategy—especially target-domain pseudo-labeling—often matters more than the alignment objective. Its strongest generalization gains, however, come with a loss of output diversity.

Abstract

Preference tuning aligns base language models to human judgments of quality, helpfulness, or safety by optimizing over explicit preference signals rather than likelihood alone. Prior work has shown that preference tuning degrades performance and reduces helpfulness outside the training domain. However, the extent to which adaptation strategies mitigate this domain shift remains unexplored. We address this challenge by conducting a comprehensive and systematic study of alignment generalization under domain shift. We compare five popular alignment objectives and various adaptation strategies from source to target, including target-domain supervised fine-tuning and pseudo-labeling, across summarization, question-answering helpfulness, and safety alignment tasks. Our findings reveal systematic differences in generalization across alignment objectives under domain shift. We show that adaptation strategies based on pseudo-labeling substantially reduce domain-shift degradation but induce mode collapse, revealing a generalization–diversity trade-off.

Problem setup

We study preference tuning when labeled preferences are available only in a source domain. A policy π_θ learns from the labeled source preference set D_S^pref and an unlabeled target-domain corpus D_T, then generates outputs that should remain high quality after the domain shift.

Available supervision

Source domain D_S: prompts with demonstrations, preference pairs, or desirable/undesirable labels, depending on the objective.

Target domain D_T: prompts and responses without preference annotations.

Learning objective

Use source preference supervision and target-domain data to learn a policy that generalizes to D_T. Quality means task performance, stylistic appropriateness, or safety, depending on the testbed.

Adaptation axis

We compare source SFT, mixed source–target SFT, target SFT, and target pseudo-labeling. For pseudo-labeling, a larger teacher supplies preferred target-domain responses that are formatted for each alignment objective.

Alignment axis

We compare five objectives spanning offline and online paradigms: DPO, KTO, ORPO, RLHF-PPO, and GRPO, alongside the SFT baselines.

Study design: source and target domains feed four adaptation strategies and five preference alignment methods, evaluated for generalization and diversity. — Figure 1: Preference optimization under domain shift is decomposed into **adaptation** and **alignment**. Click the figure to open it at full resolution.

SummarizationReddit TL;DR → CNN/DailyMail

QA helpfulnessAskEngineers → AskCulinary

SafetyCybercrime → Violence / Physical Harm

What are the main findings?

Pseudo-labeling dominates summarization transfer. For Llama-3.1-8B, pseudo-labeled SFT achieves the highest target win rate (83.37), while lifting OLMo-3-7B DPO to 72.26, above all non-synthetic methods.
Offline alignment over-specializes to the source domain. Source-only DPO on Llama-3.1-8B reaches 89.87 in domain but 58.09 out of domain; GRPO keeps its gap at 3.79 on the same testbed.
QA helpfulness is comparatively stable under the near-domain shift. Scores remain within a narrow ±8-point gap, but a domain-aware judge reveals that standard helpfulness evaluation can miss epistemic and stylistic mismatch.
Safety transfer is model- and objective-dependent. Pseudo-labeled objectives other than PPO exceed 95% target safety on Llama-3.1-8B. Without pseudo-labeling, KTO achieves the highest OLMo-3-7B target safety score (76.49%).

Full results table with source and target performance for summarization, QA helpfulness, and safety across SFT, DPO, KTO, ORPO, PPO, and GRPO. — Table 1: Source and target performance across the three testbeds. Summarization and QA report LLM-as-a-judge win rates; safety reports safety score. Click the table to open it at full resolution.

Why does pseudo-labeling help?

Teacher quality alone does not close the domain gap. For DPO, source-domain pseudo-labeling raises CNN/DailyMail target win rate from 58.09 to 62.88, while target-domain pseudo-labeling reaches 78.50. Because the teacher is the same in both synthetic conditions, the 15.62-point difference isolates the value of domain relevance.

Method	SFT data	Preference data	Source	Target	Gap
SFT	D_S	—	59.57	36.07	23.50
SFT	D_S^synth	—	72.41	44.15	28.26
SFT	D_T^synth	—	95.70	83.37	12.33
DPO	D_S	D_S	89.87	58.09	31.78
DPO	D_S^synth	D_S^synth	90.52	62.88	27.64
DPO	D_T^synth	D_T^synth	95.79	78.50	17.29

Table 2: Isolating teacher quality from domain relevance on TL;DR → CNN/DailyMail with Llama-3.1-8B. D_S^synth denotes source-domain pseudo-labeled data.

What is the cost of pseudo-labeling?

Despite high win rates, pseudo-labeling sharply contracts semantic and syntactic variety in summarization. Semantic diversity drops to 0.07–0.08 across offline objectives, and syntactic diversity falls from 0.86 for source-only SFT to 0.51 for pseudo-labeled SFT.

Online RL preserves more diversity under pseudo-labeling: PPO and GRPO retain semantic diversity of 0.22 and 0.25, respectively, compared with 0.07–0.08 for the offline objectives.

Bar plots of syntactic, semantic, and logical diversity for six objectives under source-only, target-SFT, mix-SFT, and pseudo-labeled adaptation. — Figure 2: Syntactic, semantic, and logical diversity in summarization with Llama-3.1-8B.

Can a standard judge miss domain mismatch?

Yes. A domain-agnostic helpfulness judge can reward a logically sound response even when it violates AskCulinary’s expected expertise and pragmatic style. A domain-aware judge lowers source-only DPO by 3.83 points, while pseudo-labeled DPO improves by 3.48 points, revealing genuine target-domain calibration.

Method	SFT data	Preference data	Standard	Domain-aware	Δ
SFT	D_S	—	60.08	57.06	−3.02
SFT	D_S+T	—	60.68	60.24	−0.44
SFT	D_T	—	64.94	61.45	−3.49
SFT	D_T^synth	—	76.04	77.60	+1.56
DPO	D_S	D_S	61.45	57.62	−3.83
DPO	D_S+T	D_S	58.80	60.94	+2.14
DPO	D_T	D_S	64.69	61.92	−2.77
DPO	D_T^synth	D_T^synth	75.52	79.00	+3.48

Table 3: Standard versus domain-aware judge win rates on AskCulinary with Llama-3.1-8B. Δ = domain-aware − standard.

How much synthetic data is needed?

Reducing the pseudo-labeled data by 90% causes only small changes. For SFT, KTO, and ORPO, the 10% subset slightly outperforms the full synthetic dataset on the target domain, indicating that the benefit is driven by domain relevance rather than volume.

Method	Dataset size	Win rate (%)
Method	Dataset size	Source	Target
SFT	Full	95.70	83.37
SFT	10%	92.75	83.68
DPO	Full	95.79	78.50
DPO	10%	96.30	77.08
KTO	Full	95.37	83.01
KTO	10%	95.30	84.38
ORPO	Full	96.80	82.38
ORPO	10%	92.26	82.58

Table 4: Data efficiency of pseudo-labeling on TL;DR → CNN/DailyMail with Llama-3.1-8B. The small setting uses 10% of the synthetic data.

Does the order of SFT stages matter?

Strongly. Target SFT before source SFT reaches a 56.40 target win rate, compared with 35.22 in the reverse order. Adding an intermediate source-SFT stage before source DPO raises target performance further, from 56.82 to 65.56.

Method	Source win rate	Target win rate
SFT order
SFT D_T → SFT D_S	67.23	56.40
SFT D_S → SFT D_T	61.00	35.22
Intermediate step
SFT D_T → DPO D_S	67.83	56.82
SFT D_T → SFT D_S → DPO D_S	91.88	65.56

Table 5: Win rates on TL;DR → CNN/DailyMail with Llama-3.1-8B when varying the sequence of SFT stages.

Qualitative case studies

Aggregate win rates hide two different kinds of domain-shift behavior. The examples below reproduce the paper’s appendix outputs in full, without the table layout or ellipses used in the main paper.

1

Epistemic drift: engineering advice in a culinary domain

All responses answer the same AskCulinary prompt with Llama-3.1-8B and DPO. Source-only adaptation frames cooking as heat and fat management, while target adaptation moves toward culinary vocabulary and conventions.

User prompt

Whenever I roast a turkey or turkey parts I always put chopped celery/carrots/onions underneath in the sheet pan because that's what the instructions typically tell me, but does it serve a purpose of somehow flavoring the turkey or is there another reason for this practice? I was thinking it may have something to do with neutralizing the fat that drips down so it doesn't smoke/burn. I have some string beans in the fridge and about to cook some turkey thighs, any reason I couldn't use those instead of the standard misen poix?

Culinary persona Engineering persona

Human reference

If you are making roast potatoes I love placing the spuds under the bird (rack in between) and letting the fat drip down into them

Base model

Thanks. It is not to flavor the turkey; it is to flavor the gravy that you will make from the drippings. A lot of people don't like celery and carrots in their gravy, so they use them as a flavoring agent for the gravy, then remove them before serving the gravy.

Source-only adaptation SFT D_S → DPO D_S

It’s not just for flavor. It’s to keep the bird from sticking to the pan and to help protect the bottom of the bird from burning. When you cook meat, juices (fat) come out of the meat. If you just put raw meat on a bare pan, the juices will run right off the pan and onto the bottom of your oven. This is bad for two reasons: 1) it creates smoke, which is why you get that greasy “smoky” smell when you roast a chicken; and 2) it can cause flare ups in the oven and start a fire (I’ve had this happen once, luckily I caught it early). When you put vegetables underneath the meat, they soak up the fat and help keep it in the pan. They also make for a nice crusty base that protects the bottom of the meat and keeps it from sticking and burning. You’ll see chefs doing this with all kinds of meats, not just poultry. It’s especially important if you’re cooking with a very fatty cut of meat, like a prime rib roast or a pork shoulder.

Mixed-domain SFT SFT D_S+T → DPO D_S

It’s not just about flavor. Carrots, onions, and celery are a great base for making stock from the drippings. It will give you a much better tasting stock. You can also use herbs and spices in the mix for added flavor. The veggies themselves will flavor the drippings. Once you’ve roasted your turkey, you’ll have an excellent base for gravy. There are lots of reasons to use vegetables in the bottom of your roasting pan.

Target-first SFT SFT D_T → DPO D_S

It’s not just about flavor. Carrots, celery, and onions are a great example of the “trinity” in French cooking (aka mirepoix) because they all have a lot of water content. So when you’re roasting a bird, you can use them to soak up some of the fat drippings. That being said, you should *never* use them to clean the pan. You need to leave them in the oven because they will burn to a crisp and stick to the pan. They’re just a flavor booster and moisture absorber. And while we’re here, let’s talk about basting. You should never baste. Basting only moves moisture from the inside of the bird to the outside. This makes the skin soggy and doesn’t make the bird cook any faster. The skin will crisp up when the juices on top evaporate.

Target pseudo-labeling SFT D_T^synth → DPO D_T^synth

Roasting a turkey with vegetables such as celery, carrots, and onions is a common practice, often recommended by cooking instructions. While the purpose of using these vegetables is not always explicitly stated, they can serve multiple functions in the cooking process. One potential reason for using a mirepoix, which consists of diced celery, carrots, and onions, is to flavor the turkey. The combination of these three ingredients provides a savory base that can infuse the meat with flavor. Additionally, the vegetables can also add moisture to the dish, helping to keep the turkey juicy and tender.

Takeaway. A domain-agnostic judge can rate all four answers as helpful because each supplies a logical justification, while missing whether the answer sounds like appropriate culinary expertise.

2

Structural mode collapse under pseudo-labeling

The target pseudo-labeled DPO model maps three unrelated CNN/DailyMail articles onto the same discourse skeleton. The repeated template appears in roughly 10% of its summaries, versus roughly 0.2% for the source-only model and human references, closely tracking the teacher’s rate of 10.5%.

Food trends Lifestyle review

Human reference

Dietitian Sophie Claessens picks the latest ‘miracle’ foods from fads. Charcoal to ‘detox’, while freekah is new supergrain on the culinary scene. Fermented foods like pickled cabbage act like a probiotic yoghurt. And spiralizers to create vegetable spaghetti can help boost weight loss.

Source-only DPO

Some of the latest food fads are worth a try, others are bonkers. The key is to use common sense and ask yourself whether it's sustainable and whether you'll be able to stick with it. Try spiralizing, freekah, fermented foods and bone broth. Avoid activated charcoal and ‘detox’ products.

Target pseudo-labeled DPO

The article discusses the latest food trends and their potential health benefits, including activated charcoal, spiralizing, freekeh, fermented foods, sprouted foods, and bone broth. While some of these trends may have some merit, others lack scientific evidence and can be detrimental to one's health. The author advises against “detoxing” with activated charcoal, which has only been proven effective for acute poisoning, and cautions against replacing complex carbohydrates with spiralized vegetables. Freekeh, a young green wheat, is a nutritious option, but can be expensive. Fermented foods and bone broth are beneficial for gut health and overall wellness.

Kefalonia Travel and leisure

Human reference

Kefalonia is stunning and doesn't play the Captain Corelli's Mandolin card. Island hasn't been tainted by the crowds its Greek counterparts get. It's a quarter of the size of Spain's Majorca but with a 20th of the tourists. The one must-do outing in Kefalonia is to Fiskardo, a lovely harbour.

Source-only DPO

Kefalonia, the Greek island featured in Louis de Bernières's novel Captain Corelli's Mandolin, is still remarkably unspoilt. It's a stunning place with a few surprises including the regular appearance of the stray cat named Arthur.

Target pseudo-labeled DPO

The article discusses the Greek island of Kefalonia, which was the setting for the novel and film “Captain Corelli's Mandolin”. Despite being a popular tourist destination, Kefalonia remains relatively untouched by mass tourism, with only a few hotels and no large resorts. The author describes their experience staying in a villa with a stunning view of Lourdas Beach and meeting a stray cat named Arthur, who became a frequent visitor. They also mention a wildfire that broke out during their stay and the efforts to contain it. The article recommends visiting Fiskardo, a lovely harbor, and Tassia, a waterfront restaurant.

Video technology in football Sports opinion

Human reference

Celtic were denied place in Scottish Cup final by shocking decision. Leigh Griffiths' header was blocked by Josh Meekings' hand. None of the officials managed to spot the game-changing decision. But FIFA will require a higher profile blunder to change their laws.

Source-only DPO

Frank Lampard's disallowed goal in 2010 prompted FIFA to introduce goal-line technology, and Scottish fans must hope that another high profile injustice will force a similar change to allow video review.

Target pseudo-labeled DPO

The article discusses the reluctance of FIFA to adopt video technology to improve officiating in football, despite the growing number of errors and injustices in the game. The author cites examples of goal-line technology being trialed after Frank Lampard's disallowed goal in the 2010 World Cup, but notes that FIFA has been slow to implement it. The article suggests that FIFA only acts when a high-profile incident occurs, and that Scottish football fans may need to wait until the World Cup in Russia to see any changes. The author argues that video technology, such as Hawk-Eye, could help reduce errors and improve the game.

Takeaway. The summaries remain fluent and informative, so the judge rewards them, but the student inherits the teacher’s repeated discourse frame instead of preserving the target domain’s tonal variation.

BibTeX

@misc{karouzos2026preference,
  title         = {An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shift},
  author        = {Constantinos Karouzos and Xingwei Tan and Nikolaos Aletras},
  year          = {2026},
  eprint        = {2601.05882},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL}
}

The arXiv citation is shown until the final ACL Anthology proceedings record is available.

Acknowledgments

We thank Jasivan Sivakumar, Vynska Amalia Permadi, Sam Lewis-Lim, Yanwen Peng, and Atsuki Yamaguchi for their valuable help and feedback. CK is supported by the Centre for Doctoral Training in Speech and Language Technologies (SLT) and their Applications funded by UK Research and Innovation grant [grant number EP/S023062/1]. XT and NA are supported by the EPSRC [grant number EP/Y009800/1], through funding from Responsible AI UK (KP0016) as a Keystone project.

We acknowledge (1) IT Services at the University of Sheffield for the provision of services for high-performance computing; (2) the use of the University of Oxford Advanced Research Computing (ARC) facility; (3) the EuroHPC Joint Undertaking for awarding this project access to the EuroHPC supercomputer LEONARDO, hosted by CINECA (Italy) and the LEONARDO consortium through a EuroHPC Development Access call; (4) the use of resources provided by the Isambard-AI National AI Research Resource (AIRR). Isambard-AI is operated by the University of Bristol and is funded by the UK Government’s Department for Science, Innovation and Technology (DSIT) via UK Research and Innovation; and the Science and Technology Facilities Council [ST/AIRR/I-A-I/1023].

Website acknowledgment

This project page is built from the Academic Project Page Template by Eliahu Horwitz, itself adapted from the Nerfies project page. The template is licensed under CC BY-SA 4.0. Page text, figures, tables, and qualitative examples are adapted from the camera-ready paper by the authors.

An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shift

Adaptation strategy—especially target-domain pseudo-labeling—often matters more than the alignment objective. Its strongest generalization gains, however, come with a loss of output diversity.

Abstract

Problem setup

Available supervision

Learning objective

Adaptation axis

Alignment axis

What are the main findings?

Why does pseudo-labeling help?

What is the cost of pseudo-labeling?

Can a standard judge miss domain mismatch?

How much synthetic data is needed?

Does the order of SFT stages matter?

Qualitative case studies

Epistemic drift: engineering advice in a culinary domain

Human reference

Base model

Source-only adaptation SFT DS → DPO DS

Mixed-domain SFT SFT DS+T → DPO DS

Target-first SFT SFT DT → DPO DS

Target pseudo-labeling SFT DTsynth → DPO DTsynth

Structural mode collapse under pseudo-labeling

Food trends Lifestyle review

Human reference

Source-only DPO

Target pseudo-labeled DPO

Kefalonia Travel and leisure

Human reference

Source-only DPO

Target pseudo-labeled DPO

Video technology in football Sports opinion

Human reference

Source-only DPO

Target pseudo-labeled DPO

BibTeX

Acknowledgments

Website acknowledgment

Source-only adaptation SFT D_S → DPO D_S

Mixed-domain SFT SFT D_S+T → DPO D_S

Target-first SFT SFT D_T → DPO D_S

Target pseudo-labeling SFT D_T^synth → DPO D_T^synth