It turns out that when Call of Duty players play against others at their own skill level, instead of leveling or getting leveled in a disproportionate skill rift, they stick around longer.
This is the conclusion Activision reached when they loosened “skill-based matchmaking,” according to a newly-published whitepaper shared as part of an ongoing blog series about the matchmaking system. The paper, titled “Matchmaking Series: The Role of Skill in Matchmaking,” talks about its matchmaking system, how players’ skill affects its execution, and how the developers adjust accordingly.
The team tapped into “A/B testing,” or putting about half of the population into softened SBMM modes, where the skill bracket for each match was wider, while keeping the other half of the population under the standard SBMM as a control. They used a few measures to record player satisfaction after matches: “how long they continue to play the game, match-level quit rates, player surveys and match outcomes,” citing both the surveys and statistical data often together in their examples.
In short, almost all measures helped support the hypothesis that loosening SBMM is bad, actually.
"The test showed that players with a wider skill gap were more likely to quit matches in progress and did not return to the game at a higher rate than players with unchanged matchmaking," the summary blog reads.
To get into it more, per the white paper, players in lower skill brackets frequently dropped out of the game after a two-week period, and all players across all brackets had greater “blowout” rates, meaning ridiculously high discrepancies between the winning and losing teams, and reported having less fun. These lower-skilled players also performed far worse than their SBMM-adjusted counterparts, with lower kill scores and kill streaks by a long shot. They also flipped the script for some players, tightening SBMM for certain players, and almost all players were more likely to stay in the game, with the unusual exception of the top 10% of players.
The team cited the use of SBMM in other Call of Duty titles as early as the original Modern Warfare in 2007, though the team seems to have integrated more research opportunities into the current iteration, Modern Warfare 3. When they did try the SBMM loosening with the 2019 MW reboot, they said they saw “significant decrease in the number of players” and “and an increase in the overall match quit rate.” They kept the “looser” SBMM for only the top 75% of players to try to readjust and still saw these numbers go down.
They won’t outright say it like this, but when they turn off skill-based matchmaking, players are essentially more likely to ragequit.
Generally, the whitepaper also goes through some interesting discussions about the matchmaking system. For instance, at the highest levels of play, players may experience more ping, as the system has to reach across longer real-life distances to find a suitable match in a reasonable timeframe. (And according to a previous paper, the matchmaking system does take ping into account.)
While Activision is notorious for plenty of questionable practices as a developer, it’s also uniquely positioned to be one of the few gaming companies that can pull off a mass legitimate whitepaper like this. Call of Duty is a game that attracts a massive population of players with diverse skill sets, attitudes and time commitments, meaning there’s less of a need to control for nor consider these factors.
That’s especially compared to a game like League of Legends where even the time commitment factor changes which and how players approach the game. In fact, in these MOBAs, the skill difference is usually more evident; games like LoL and especially Dota 2 have player bases verbally against high-skill players who “smurf,” or take accounts to circumvent SBMM rankings, and Dota will even ban such players (while LoL says they’re more lax unless it’s proven, specifically in competitive modes).
Despite SBMM having been an industry standard for many years, a lot of “YouTubers” and “Tiktokers” have led pushback against SBMM, supposedly because everyone thinks they’re good enough at the game where queueing casually means they’re guaranteed to stomp scrubs, as opposed to the possibility of them being the stomped scrubs.
Now, we have some evidence from the powers that be that that’s not the reality—and that people seem to stick around when they’re toe-to-toe with others at their level.
