Zyphra unveils ZHA, a hierarchical attention method aimed at true sub-quadratic attention

Zyphra unveils ZHA, a hierarchical attention method aimed at true sub-quadratic attention

N
News Editor
2026-11-03 03:04:53
Zyphra has introduced Zyphra Hierarchical Attention, or ZHA, a sparse attention method built around a tree-based indexer. The approach is designed to address a limitation in existing sparse attention systems, where the indexer still has to scan the full context. According to the report cited by Techub, ZHA can be distilled into a pretrained frozen model and was trained with about 4 billion tokens. On the Qwen3.6-35B-A3B model, its performance in inference, agentic coding, and 256K retrieval benchmarks was reported to be close to dense attention. ZHA splits keys into contiguous blocks and compresses them layer by layer into summary keys, then uses top-p routing during decoding to adaptively select relevant tokens. In benchmark results, ZHA’s deep tree structure reached 63.4% accuracy on LongBench v2, compared with 58.0% for dense attention, while using only about 1.1% of the keys. Techub attributed the information to MarkTechPost.

Zyphra has released Zyphra Hierarchical Attention, or ZHA, a sparse attention method based on a tree-structured indexer, according to Techub News. The method is intended to solve a problem in existing sparse attention systems, where the indexer still needs to scan the entire context.

ZHA can be distilled into a pretrained frozen model and was trained with about 4 billion tokens. On the Qwen3.6-35B-A3B model, its performance in inference, agentic coding, and 256K retrieval benchmarks was described as close to dense attention.

How ZHA works

ZHA divides keys into contiguous blocks and compresses them layer by layer into summary keys. During decoding, it uses top-p routing to adaptively select relevant tokens.

Benchmark results

Across multiple benchmarks, ZHA’s deep tree structure reached 63.4% accuracy on LongBench v2, above dense attention’s 58.0%, while using only about 1.1% of the keys.

The item cited MarkTechPost as the source of the report.

This article was originally published by Bit.Fan. For more cryptocurrency news and market insights, visit www.bit.fan.
5000

Disclaimer:

The market information, project data, and third-party content displayed on this platform are for industry information sharing only and do not constitute any form of investment advice or return commitment.

Cryptocurrency trading carries high risks. Users should fully assess their risk tolerance and make independent decisions. All profits, losses, and legal responsibilities are borne by the users themselves.

Zyphra unveils ZHA, a hierarchical attention method aimed at true sub-quadratic attention | Bit.Fan