Final hiring rates can look balanced while the decision path remains uneven. That is the main argument of SCOPED-Hiring, a paper accepted to the EMNLP 2026 Main Conference that examines fairness in large language model multi-agent systems used for hiring.
The paper says outcome-level checks alone can miss what happens during the decision process: who gets questioned more, who receives lower intermediate scores, and who faces extra review before a vote is made. It introduces SCOPED-Hiring as a process-aware fairness diagnosis framework for LLM multi-agent systems, or MAS.
Paper: https://arxiv.org/abs/2609.02092
Open-source code and data: https://github.com/Warren118/SCOPED
Project page: https://scoped-hiring-project-page.vercel.app/
Why the study shifts attention away from final outcomes only
As AI systems move into hiring, credit, and medical screening, the paper argues that fairness is not an optional benchmark. It is a condition for trust and deployment in high-stakes settings.
Most fairness audits still center on the endpoint: who was hired and who was rejected. SCOPED-Hiring asks a different question. It does not stop at who got the offer; it asks how the agents arrived there.
That distinction matters in a multi-agent setup. A candidate may still receive an offer while carrying a heavier burden of suspicion during the process. Another may be rejected, but the issue may have appeared much earlier, during first-round scoring, inter-agent discussion, or the assignment of investigative follow-ups.
From outcome audits to decision trajectories
Traditional audits resemble a check of the final shortlist, comparing hiring rates across groups. SCOPED-Hiring opens the black box further and tracks how candidate information moves through private assessments, public arguments, scores, votes, and stage transitions inside a multi-agent committee.
To do that, the research team built controlled resume variants. Core qualifications, work experience, and job fit were held constant, while selected candidate signals were changed, including employment gaps, educational background, city, socioeconomic proxy signals, and identity-related signals. These candidates were then evaluated by a two-stage hiring committee composed of agents with different roles.
The experiments produced more than 311,000 structured decision trajectories. That dataset let the researchers inspect not just the final hire/reject result, but the burden carried at each intermediate step.
In the paper’s framing, hiring rates can appear balanced at the result layer while hidden fairness risks remain in process, pathway, dynamics, and system design.
The six-part SCOPED diagnostic matrix
SCOPED-Hiring organizes fairness risk in multi-agent decision-making into six complementary views.
- S, outcome view: whether final hiring rates and scores differ.
- C, counterfactual view: whether a decision changes when only one candidate signal is altered.
- O, process view: whether candidates face different levels of negative framing, doubt, or scrutiny.
- P, pathway view: whether risk accumulates or is amplified across screening, review, and discussion stages.
- E, evolution or dynamic view: whether interaction, debate, and vote shifts among agents introduce new unfairness.
- D, design view: how model choice, memory, topology, and workflow change the fairness burden.
Together these views form what the authors describe as a fairness diagnosis matrix. The point is not just to produce a single score. It is to identify which candidate signals trigger higher risk, at which decision point, and through what mechanism.
What the experiments found across GPT, Gemini, and Qwen
The team ran experiments on GPT, Gemini, and Qwen backends. One pattern appeared across all three: risks visible in process, pathway, dynamics, and system design were materially stronger than those visible from final hiring outcomes alone.
Specifically, the paper reports that the average diagnostic significance of process-aware O/P/E/D views was 4.52x, 4.66x, and 2.74x that of outcome-focused S/C views across the three model families.
In other words, calm-looking hiring rates may hide much rougher internal decision trajectories.
Employment gaps drew extra suspicion
Candidates with employment-gap signals did not always show an equally large disadvantage in final hiring rates. But they were more likely to be treated as risk signals during deliberation.
In the GPT hiring committee, agents mentioned gap-related risk in private assessments for 94% of candidates carrying employment-gap signals. The corresponding share for candidates without such signals was 32%, a difference of 62 percentage points. The paper says these differences later fed into skeptical investigation requests and discussion.
That means a candidate may still end up with a similar hiring chance while already absorbing more explanation, verification, and doubt along the way.
Proxy signals shaped judgments about ability
The study also says information that is not directly tied to job ability, such as educational tier, city tier, and lifestyle, can be converted by agents into judgments about qualification, potential, or cultural fit.
For university tier as a proxy signal, final hiring-rate gaps across candidate groups were only 1 to 2 percentage points. Yet across the three model families, Tier 1 candidates still received composite scores 0.14 to 0.32 points higher than Tier 3 candidates.
The warning here is subtle but important. A large gap may not show up at the final outcome level, while proxy-based differences are already accumulating in intermediate scoring and shaping later discussion and voting.
Identity signals changed how scrutiny was assigned
Identity-related signals showed broader trajectory risk. On GPT, Gemini, and Qwen, the paper reports that process-aware risk tied to identity signals was 2.44x to 3.01x that of outcome-based risk.
When other candidate information was held comparable, agents could ask different follow-up questions for candidates carrying different identity signals and form skeptical hypotheses with different intensity and direction. The problem does not always appear as direct rejection of a group. It can also appear as different candidates being placed under different evidence thresholds.
That finding points to a key issue in multi-agent decision-making: information gathering is not automatically neutral. Who gets asked for more explanation, who gets checked more often, and who gets pushed through extra verification are fairness issues too.
The paper presents these patterns in a heat map described as a fairness risk map. The horizontal axis shows where risk appears in the decision chain. The vertical axis shows which candidate signal families are more likely to trigger it. Darker color marks stronger risk. On the summary side of the chart, the process, pathway, interaction, and design layers stand out more clearly than final outcomes alone across GPT, Gemini, and Qwen.
From diagnosis to repair: Fair Skills
SCOPED-Hiring does not stop at identifying the problem. It also proposes a repair mechanism built from the diagnosis itself.
Rather than appending a generic instruction telling all agents to be fair, the framework converts risks identified in the heat map into decision-time interventions called Fair Skills.
The examples in the paper are concrete:
- When an employment gap is treated as a direct signal of ability or stability risk, the agent should return to job-relevant evidence.
- When school, city, or lifestyle proxy signals are used as ability evidence, the agent should separate them from genuine qualification evidence.
- When extra investigation is imposed on a category of candidates, the agent should apply a consistent investigation standard.
The paper frames Fair Skills not as a way to simply relax standards, but as a way to intervene at the point where judgment is actually made, so that uncertainty is not turned into suspicion, proxy signals are not turned into ability evidence, and extra scrutiny is not treated as the default response.
Using the same hiring task, committee structure, and candidate pool, the team compared four conditions. The results show that broad fairness reminders do not automatically help. Under INT1, total risk rose from 8.59 to 10.07. INT2 lowered dynamic-layer risk but did not improve the overall burden. Only INT3, the diagnosis-driven Fair Skills intervention, consistently reduced fairness burden across multiple layers of the decision trajectory.
With the full Fair Skills set enabled:
- total stratified fairness burden fell by 72.3%;
- final hiring rates changed by only 1.86 percentage points;
- output validity remained above 99.7%;
- checks did not indicate a simple pattern of broadly loosening hiring standards.
The paper’s conclusion on repair is narrow and specific. Better fairness does not come from raising pass rates across the board or from adding general reminders. It comes from identifying where risk appears in the trajectory, how it is produced, and then applying targeted intervention.
Why the paper matters for high-stakes AI oversight
The authors do not present SCOPED-Hiring as a replacement for all existing fairness standards. They present it as a different lens.
- It surfaces risks that outcome balance can hide.
- It shifts fairness from a verdict to a diagnosis.
- It links evaluation to repair.
- It is built for an era in which LLM systems act through collaboration, debate, division of labor, and collective decisions rather than one-shot answers.
The paper also draws a connection to real-world hiring AI audits. New York City’s AEDT rule, for example, focuses on selection rates, impact ratios, and scoring rates. Those checks can show whether final disparities exist, but in a multi-agent workflow they may miss extra questioning, proxy-based scoring, or stage-specific thresholds imposed before the final result appears. SCOPED moves the inspection point earlier, into the process that produces the result.
The study’s broader claim is straightforward: an AI system used in high-risk decisions should not only appear fair in its final answer. Its scoring, doubt, discussion, and decision flow should also stand up to scrutiny.
Reference: https://arxiv.org/abs/2609.02092
This article cites material originally published via the WeChat public account "Xinzhiyuan," with author listed as Xinzhiyuan and editor listed as LRST.