vivo AI Lab has introduced CAPO-SLT — short for Confidence-Aware Policy Optimization for Sign Language Translation — as a reinforcement learning method meant to keep sign language translation generation steadier and more faithful to the visual evidence. The paper’s point is simple: even if a model lines up vision and text well, autoregressive decoding can still drift if it chooses a word that feels locally plausible but is not grounded enough in the visual input.
And that mistake does not stay small. Once a token with no visual support slips into the context, it can shape the next prediction, then the next one, and little by little drag the whole sentence away from the source meaning. The team ties this problem to the update rule used in reinforcement learning and puts forward CAPO-SLT without altering either the visual backbone or the reward function.
Why sign language translation demands reliability
The paper describes sign language as a visual language with its own grammar, spatial structure, and non-manual signals — not a gesture-by-gesture copy of spoken language. Automatic sign language translation turns continuous video or pose sequences into natural language so sign language users and non-signers can communicate directly across modalities.
The study lists use cases such as public services, medical communication, classrooms, emergency information, customer service, workplace collaboration, and online video. In those situations, human interpreters are not always on hand. A dependable automatic system can supplement human service and help people get information faster, produce content, and communicate both ways.
That makes the standard tougher. The paper argues that a fluent translation that is wrong can be harder to catch than one that is obviously clumsy. A shift in a keyword. A negation. Even the sentence subject. Any of those can flip the meaning of the entire utterance.
Good alignment does not guarantee stable generation
Automatic sign language translation has to deal with gaps in modality, timing granularity, and language structure. Hand movements may be subtle, similar signs can be mixed up, action boundaries are not always clean, and non-manual cues like facial expression or lip patterns may be incomplete. A lot of recent work has concentrated on visual-language alignment so actions and words can sit in a shared semantic space.
But good alignment by itself is not enough, the paper says. Translation is autoregressive: the model predicts one token, then feeds that token back in with the earlier context to predict the next. Strong language priors can make some continuations sound perfectly natural. Trouble is, if those continuations are not supported by visual evidence, fluency masks the error instead of revealing it.
Reinforcement learning can optimize sentence-level quality directly. Nice idea. But it brings noise too. Rewards are usually given at the sentence level, while advantages have to be spread across individual generated tokens. In that setup, the model may recognize that the sentence is better or worse overall, yet still miss which token helped and which one started the semantic drift.
A single clipping rule does not fit every token
The paper zooms in on policy optimization methods like PPO and GRPO, which limit the probability ratio between the new and old policy so a single update does not jump too far. In the standard setup, all positive-advantage tokens get the same clipping range. The authors say that rule is too blunt and ignores big differences between token states.
A token that already has high confidence under the old policy may just reflect an easy continuation driven by language prior, not the visual signal. Pushing that probability even higher can lock in the wrong path. By comparison, a token with lower old-policy probability but positive advantage may deserve a stronger correction and more exploration, yet the same cap shrinks its room to move.
So CAPO does not treat high confidence as evidence of semantic correctness. Instead, it uses old-policy probability as a control signal for update size, asking two things: how strongly the old model already favors a token, and how much more amplification is actually needed.
How CAPO-SLT changes the update rule
For positive-advantage tokens, CAPO sets the upper clipping boundary dynamically based on the probability assigned by the old policy. The higher the old-policy confidence, the closer the cap stays to a conservative range. The lower the confidence, the more space there is for a positive update. The paper fixes the base upper bound at 0.2 and lets it stretch to 0.3.
The method is not trying to suppress high-confidence tokens across the board. Far from it. Tokens with high confidence and positive advantage are still reinforced, but not needlessly over-amplified. For negative-advantage tokens, CAPO also puts an upper bound on loss magnitude so sentence-level reward noise is less likely to punish individual tokens too harshly.
Training pipeline from pose extraction to RL fine-tuning
In supervised initialization, the model pulls full-body pose keypoints from video and splits them into four groups: face, body, left hand, and right hand. Then it uses ST-GCN to model spatial joint relations and temporal motion information while learning a shared visual-text semantic representation.
During reinforcement learning fine-tuning, the system samples multiple candidate translations for the same sign language input. Sentence-level rewards are calculated with BLEU-1, BLEU-4, ROUGE-L, and BERTScore, and relative advantages are then computed within each group.
In the confidence-aware update stage, CAPO applies confidence-dependent clipping to positive-advantage tokens and limits excessive losses for negative-advantage tokens. The reward function stays the same. So does the backbone network.
On the data side, the Chinese setup uses CSL-News for large-scale pretraining and CSL-Daily for supervised fine-tuning, reinforcement learning, and evaluation. On the English side, the model is pretrained on YouTube-ASL and validated for American Sign Language translation on How2Sign. The paper also tests isolated sign recognition on WLASL2000.
Top reported Chinese results with pose-only input
On the CSL-Daily test set, CAPO-SLT posted the best values in all three reported metrics in the comparison table while using only pose input.
Against Geo-Sign, which also uses pose-only input, the method lifted the three metrics by 2.93, 1.26, and 0.64 points. Against pose-only Uni-Sign, the gains were 4.96, 3.07, and 3.67 points. The comparison table includes methods that use both pose and RGB as well, yet CAPO-SLT still logged the highest values across the three metrics with pose alone.
Cross-language and auxiliary-task results
On How2Sign, CAPO-SLT again relies only on pose input and records 41.4 BLEU-1, 15.2 BLEU-4, and 34.9 ROUGE-L. Compared with pose-based Uni-Sign, those figures are higher by 1.0, 0.7, and 0.6 points.
On WLASL2000, the model reaches 63.77% Per-Instance accuracy and 61.91% Per-Class accuracy. The paper uses those results to argue that the pose representation stays competitive in another sign-language understanding task.
Ablation tests isolate the source of the gains
The research team ran ablations where SFT, GRPO, DAPO, and CAPO all shared the same supervised checkpoint, the same reward, and the same training setup. Only the policy optimization rule was swapped out.
Relative to DAPO, CAPO improved the three metrics by 0.30, 0.50, and 0.67 points. When the upper bound for negative-advantage tokens was removed, the result fell to 58.48/28.03/57.80. When the inverse-proportional confidence mapping was replaced with a linear mapping over the same range, BLEU-4 and ROUGE-L dropped by 0.73 and 0.60 points.
Examples of fluent output that changed the meaning
The paper points to several cases where a model without CAPO produced translations that sounded fluent but were wrong. It translated "I will introduce these colleagues one by one" as "I am a team made up of several people," "It got dark, and I was afraid" as "It got dark, and I fell asleep," and "The actress in this movie is very beautiful" as "This film is about the actress's love story."
In each example, the output stayed readable while changing the predicate, the relation between entities, or the event meaning itself. After CAPO was added, the paper says these examples went back to expressions that matched the reference answer or were semantically equivalent to it. The remaining errors were described as local omissions, shifts in time expressions, or minor wording differences, rather than full sentence-level reversals of meaning.
Training curves point in the same direction
The training curves tell much the same story. According to the paper, GRPO's average entropy falls quickly at first and then climbs again later in training, suggesting renewed policy uncertainty. CAPO drops early and then settles, without a sudden entropy collapse or an obvious late-stage rebound.
The authors say this lines up with the intended effect of confidence-aware clipping: token-level updates become more controlled, which fits the method’s goal of improving stability during generation.
Low-intrusion design and stated limitations
The paper presents CAPO-SLT not as a bigger visual encoder, but as a way to shift attention from whether the model understands the action to how it turns that understanding into stable output. When rewards come from a full sentence but updates happen token by token, using one uniform clipping bound for every token may just be the wrong move. Old-policy confidence provides a signal that can be computed directly and does not need an extra model.
And that gives the method an engineering advantage too. CAPO leaves the visual backbone and reward function alone and changes only the clipping rule inside policy optimization. For sign language translation systems that already have supervised initialization and a PPO- or GRPO-style reinforcement learning pipeline, the paper presents it as a relatively low-intrusion way to stabilize training.
The authors also say that gains on American Sign Language translation are more modest than on CSL-Daily. The current reward still relies entirely on reference translations. Even with BLEU, ROUGE-L, and BERTScore combined, the setup still does not fully capture acceptable paraphrases, discourse-level adequacy, or fine-grained semantic preferences. The team says future work will look at learned preference models or sign-language-specific semantic evaluators.
Paper details
The paper is titled "CAPO-SLT: Confidence-Aware Policy Optimization for Stable Sign Language Translation Generation."
The authors are Ling Zhou, Yuhao Chen, Lin Cheng, Chengwen Yao, Donghui Sun, and Xiaoxin Chen. Ling Zhou and Yuhao Chen are co-first authors, and Donghui Sun is the corresponding author.
Paper link: https://openreview.net/pdf?id=l4bCsrSkYx
The source article says the report came from the WeChat public account Quantum Bit and was written by the vivo AI Lab team.