# Why Did TII Build a Quality-First Arabic Leaderboard?

According to the Hugging Face Blog post published April 21, 2026, TII created QIMMA (قِمّة) to address a fundamental flaw in existing leaderboards: they reward models that game metrics rather than produce genuinely useful Arabic text. The post states, 'Existing benchmarks often fail to capture the nuances of Arabic dialects, formal registers, and cultural context.' TII's solution uses a community voting mechanism where users rate model outputs on a Likert scale, with the final score weighted by the rater's historical accuracy—a system borrowed from collaborative filtering but applied to LLM evaluation.

This matters because Arabic LLMs have proliferated without a trusted quality standard. Models like AceGPT, Jais, and others claim state-of-the-art performance, but those claims rest on benchmark-specific optimizations. TII's approach directly challenges that: Falcon2-11B, also built by TII, leads with a QIMMA score of 4.21/5, but the gap to second-place AceGPT-13B (3.98) is smaller than benchmark leaderboards suggest. The methodology introduces a 'quality weight' that penalizes models with inconsistent human ratings, effectively creating a trust score that cannot be reverse-engineered through training data.

# Who Actually Benefits From This New Ranking?

The immediate beneficiary is TII itself: its Falcon2-11B model sits atop the leaderboard, giving it marketing ammunition against rivals. However, the bigger winner is the Arabic NLP research community. According to TII's documentation on the QIMMA Hugging Face Space, the leaderboard includes a 'dialect diversity' metric that penalizes models that perform well only on Modern Standard Arabic (MSA) and fail on Egyptian, Levantine, or Maghrebi dialects. This forces developers to train on more representative data, which benefits end users in the Arab world who speak dialects daily.

Losers include models that optimized exclusively for benchmark performance. For example, AceGPT-13B, which tops some MSA-specific benchmarks, drops to second place on QIMMA due to lower dialect diversity scores. The Jais-30B model, developed by Inception and MBZUAI, ranks fourth, suggesting that larger parameter counts do not guarantee quality. The real test will come when new models submit: if Falcon2-11B's lead holds after six months of submissions, it will validate TII's approach; if it drops, it will expose the leaderboard's own biases.

# What Does the Evidence Actually Support?

The Hugging Face Blog post provides raw data: Falcon2-11B scored 4.21 from 1,247 community votes, while AceGPT-13B scored 3.98 from 892 votes. The confidence intervals overlap significantly—Falcon2-11B's 95% CI is [4.05, 4.37], while AceGPT-13B's is [3.82, 4.14]. Statistically, the difference is not robust. TII acknowledges this in the post: 'We encourage more submissions to increase statistical power.' The evidence supports a tentative lead, not a definitive win.

What remains uncertain is the reliability of community raters. TII uses a 'rater reputation' system that weights votes from users with high historical accuracy, but the system is new and has not been audited independently. According to the QIMMA Space documentation, raters are drawn from the Hugging Face community, which may overrepresent AI researchers and underrepresent everyday Arabic speakers. This creates a sampling bias that could favor technical correctness over natural fluency.

# How Will This Change Arabic LLM Development?

In the short term, developers will rush to submit their models to QIMMA to gain visibility. TII's blog post reports that within 48 hours of launch, 17 models had been submitted, including entries from MBZUAI, Inception, and independent researchers. This pace suggests that QIMMA will become the de facto standard for Arabic LLM evaluation within three months, replacing fragmented benchmark comparisons.

In the long term, the quality-first approach will force a shift in training data collection. Models that score low on dialect diversity will need to incorporate more dialectal text, which is harder to source and clean than MSA data. This benefits organizations like TII and MBZUAI that have invested in dialectal corpora, while disadvantaging smaller teams that rely on web-scraped MSA data. The leaderboard's 'quality weight' also disincentivizes benchmark-specific fine-tuning, which could reduce the prevalence of 'leaderboard-chasing' in Arabic NLP.

1. TII's Falcon2-11B will lose its QIMMA top spot to an MBZUAI or Inception model by October 2026, as dialect diversity requirements favor organizations with broader Arabic data.
2. At least three new Arabic LLMs will be submitted to QIMMA per month through 2026, driven by the leaderboard's marketing value.
3. The QIMMA methodology will be adopted by at least one other language-specific leaderboard (e.g., for Hindi or Swahili) within 12 months, as TII open-sources the evaluation framework.