If the purpose of the 2023 RWC draw was to prevent the early exit of the favourites, the draw fails dismally.
The RWC 2023 draw was based on the world rankings of January 2020. These rankings reflected the relative strengths of teams at that time that have changed remarkably in the past three years.
Consequently, the top five teams in the current standings – Ireland, France, New Zealand, South Africa, and Scotland – are all grouped on the same side of the draw destined to eliminate each other.
Only two of the top five can therefore make the semi-finals. The draw is especially harsh to Ireland, South Africa, and Scotland which are grouped in the same pool as only two of these teams can progress to the quarterfinals.
The official 2020 draw has been severely criticised by coaches, players, and supporters. World Rugby chief executive, Alan Gilpin, agreed when he stated, that “future Rugby World Cups will follow the example of football, where draws are made less than a year out from the start of a major event’.
(Photo by Mark Kolbe/Getty Images)
This article compares the outcomes of the current draw against the outcomes of a draw that is based on current world rankings using the 2023 RWC Monte Carlo simulation model introduced in my previous article (The winners and losers of the RWC 2023 draw).
This model helps us quantify the impact of the draw by comparing the outcomes of the hypothetical 2023 draw based on current ratings with those of the official 2020 draw (with the latter in brackets).
Will the draw decide the winner? The teams most likely to win the 2023 Rugby World Cup are New Zealand 28.7% (32.3%), then Ireland 22.8% (23.6%), France 16.2% (14.8%), South Africa 13.7% (13.2%) and England 8.9% (10%).
As the winner needs to win three consecutive finals, the draw has very little influence to decide the ultimate winner.
Will the draw decide the finalist? The most likely final in the official 2020 draw is between New Zealand and France (13.1%). This final is played between two teams that the model does not consider the most likely to win the tournament.
For the 2023 draw the most likely final is between New Zealand and Ireland 16.8% (9.6%), the two teams the model considers the most likely to win the tournament. The other likely finals are between New Zealand and South Africa 10.8% (10.7%), Ireland vs France 8.9% (5.9%), New Zealand vs France 7.2% (13.1%), and South Africa vs France 6.5% (3.1%). The draw has a meaningful impact on deciding the finalists.
Does the draw decide the semi-finals and quarterfinals? The chance that any three of the current top six teams – Ireland, France, New Zealand, South Africa, Scotland, and England make up the four semi-finalists increases to 32.3% (from 0.9%).
The chance that the current top eight teams progress to the quarterfinalist also increases to 23.9% (from 0%). The draw has a significant impact on deciding both the quarterfinalists and semi-finalists.
Who are the two biggest winners if the draw was based on current ratings?
The broadcasters, viewers, players, coaches, and the public that prefer to see the finals played between the best sides. The value of the tournament, in my view, diminishes if the favourites eliminate each other early in the tournament. Isn’t the purpose of the draw to prevent this from happening?
Scotland has been dealt a raw deal in the 2020 draw as they are grouped in Pool B with Ireland, South Africa, and Tonga (and Romania). If the draw is based on 2023 rankings, Scotland will be grouped with Ireland, Wales, and Fiji (and Romania) and can more than double their chances to reach the quarterfinals.
Scotland’s Finn Russell (Photo by Paul Devlin/SNS Group via Getty Images)
Who are the two biggest losers if the draw is based on current ratings?
World Rugby contracted a third party to sell the tickets and take the associated risks. It would be very difficult (even unpractical and expensive) to make late changes to the pools as tickets have already been sold.
Wales is grouped in Pool C with Australia, Fiji, Georgia, and Portugal in the 2020 draw. In the 2023 draw, Wales will be grouped with Ireland, Scotland, Fiji, and Romania. The chance that Wales will reach the quarterfinals will plummet from 92% to 69%.
These results are from a “Monte Carlo” simulation model developed by the author that considers the historical outcomes from head-on-head encounters between teams and the relative ratings of the respective teams adjusted for recent form and the impact of the draw.
Monte Carlo simulation is a powerful uncertainty modelling tool that calculates the chance of any particular outcome occurring instead of trying to consider every possible outcome in a complex system.
The model and associated probabilities have been tested against actual outcomes from previous World Cups and recent test matches. I found that the model is well-calibrated and will continue to refine the model as the year progresses.
Although these probability assessments are subjective, they are based on solid evidence, and good reasoning and not a “gutfeel” or what I would like to be true.
This article highlighted that the current draw devalues the semi-finals. In the next article, I will assume that changes to the pools are not feasible and consider how the quarterfinal schedule could be changed to protect the integrity of the tournament.
Bliksem
Roar Rookie
Please see my response to PeterK.
Bliksem
Roar Rookie
This is why I don’t put so much weight on only recent results. Wales, Australia and England are more likely to have a better year after a horrific 2022.
Bliksem
Roar Rookie
I will check if this refinement will help to make the model more realistically to reflect actual outcomes.
WEST
Roar Guru
Wales have been hammered with injuries, also the Wallabies.. why it’s unfortunate that DR became the culprit for all the Wallabies undoing. If Wales had their squad at full capacity I think their results would be different, same goes for the Wallabies.
Mick Jeffrey
Roar Rookie
Whilst historical records should be noted, Wales have also found ways to stuff it up against Pacific Island teams at WCs when least expected.....
Doctordbx
Roar Rookie
The model clearly didn't watch Wales in 2022 or 23 so far.
Lindsay Amner
Roar Guru
I wonder if it would be possible to factor in the ‘lift and hangover’ effect. This is where a team gives its all to beat its perceived primary opponent, sometimes causes an upset, or wins in a thrashing what should have been a tight match, but has nothing left the following week and loses. My article on this should be out tomorrow. RWC winners have often been a team that doesn’t have to lift massively to beat an opponent seen as equal or better than them until the final. NZ in 87 and SA in 07 and 19 are examples of this. We could see this again this year with an ‘easy’ side of the draw. I believe this factor has had an inordinate impact on the outcomes of previous world cups and I wonder whether any modelling could show this.
Bliksem
Roar Rookie
For the 2023 pool rankings Scotland and Wales are grouped in Pool B with Ireland, Fiji, and Romania. The head-on-head encounter in this pool game is key to determining who will go through to the QFs, so you are correct Peter K. As Scotland has beaten Wales by a large margin earlier this year, we may think that it is more likely that Scotland will beat Wales at RWC 2023. Humans have a tendency to make judgments based on a previous outcome that we can easily recall. We then assume that it’s an accurate reflection of reality and use that in making subjective decisions on future probabilities. This is a well-known cognitive bias coined the availability bias. I countered the "Availability bias", by considering the outcome of the last ten head-on-head matches. Of these Scotland only won 3 (see list below). I adjusted the probability by including a form factor that is based on Scotland's ranking and a tournament factor that increases the probability that Wales will win as their minimal error game under Gatland is more suited to knock-out tournaments. The model calculates the probability that Wales will beat Scotland head-on-head at 68%. The last ten matches between Scotland and Wales were almost all close encounters: 1. 15-Feb-15, Murrayfield Stadium, Edinburgh, 23–26, Wales won 2. 13-Feb-16, Millennium Stadium, Cardiff, 27–23 Wales won 3. 25-Feb-17, Murrayfield Stadium, Edinburgh, 29–13 Scotland won 4. 03-Feb-18, Millennium Stadium, Cardiff, 34–7 Wales won 5. 03-Nov-18, Millennium Stadium, Cardiff, 21–10 Wales won 6. 09-Mar-19, Murrayfield Stadium, Edinburgh, 11–18 Wales won 7. 31-Oct-20, Parc y Scarlets, Llanelli, 10–14 Scotland won 8. 13-Feb-21, Murrayfield Stadium, Edinburgh, 24–25 Wales won 9. 12-Feb-22, Millennium Stadium, Cardiff, 20–17 Wales won 10, 11-Feb-23, Murrayfield Stadium, Edinburgh, 35–7 Scotland won
Bliksem
Roar Rookie
Pool C has teams that are rated close to each other in the Rugby World ranking system: Australia (7), Wales (9), Georgia (11), Fiji (13), and Portugal (16). It could be very competitive based on the respective team's current form, however, as Wales and Australia should see a recovery with new coaches, the historical head-on-head record should not be ignored. The model estimates the chance that Australia will progress to the QFs at a very high 97.5% and Wales at 92.4% and the chance that Fiji or Georgia will make the QFs at 6% and 4% respectively. Australia and Wales should consider it a serious failure if they do not progress to the final 8 in my view.
Bliksem
Roar Rookie
I think there is a feasible way to fix the mess by changing the teams that play each other in the quarterfinals which will improve the semi-finals and protect the integrity of the tournament. The next article will address this.
Bliksem
Roar Rookie
Lindsay, I am glad that you enjoy these articles. This model allows us to ask and answer interesting questions based on the probability of different outcomes. The model doesn't get it 100% correct all of the time, (neither do the experts or octopus), however, it increases the likelihood that the answer would be correct. Research in decision science found (overwhelmingly) that humans did not evolve the intuitive ability to accurately assess the probabilities of outcomes from complex and uncertain systems such as a World Cup tournament. As you will notice in the comment section, we find this fact very hard to accept. Even judges, whose job it is to make decisions, are found wanting in research material on decisions as serious as parole applications where their decisions are impacted by irrelevant factors such as the time they took lunch. What makes it so hard to predict the outcome of such a tournament is not only the uncertainty of how each team will perform in general, but it is also the many random factors that can occur in each match on top of the complexity associated with the game schedule. I tested the accuracy of the model and the associated probabilities against previous RWC tournaments. To assess the accuracy of the model I couldn't draw conclusions from a single outcome or observation, but assessed if multiple outcomes aligned with the estimates. For example, if I pick a (prior) probability of 60% that an event will occur and observe that the event occurred 80% of the time then my assessment is underconfident, if it is 40% the assessment is overconfident. I found the model to be well-calibrated - as it reflects reality by minimizing estimation errors. This is what any good risk practitioner should be doing, checking the probabilities and impacts they assigned to events reflects reality. The model includes an element of "match form"; the extent to which a team plays better or worse in a specific match against a specific team than its tournament form and also "tournament form"; the general level of performance over the course of the competition. What I didn't model separately, as the calibration didn't point to the need, is to model "referee form" or "critical injuries". This could be a future refinement.
Lindsay Amner
Roar Guru
Great articles Blikzem. Your modelling really adds to the mystique of the RWC and rather than predicting winners, I suggest it highlights that imponderables and unpredictability are actually what define the tournament. I would be really interested to see if your model, when applied to previous tournaments, actually predicted correctly. I believe modeling all struggles with the fundamental effect that leads to wins and losses at a RWC which I call the lift and hangover effect. This is based on a lift to beat an arch enemy followed by a hangover from the lift in the next game when a team does not perform so well. This effect is seen throughout world cups and regularly leads to the cup winner being a team which is not necessarily the best in the tournament. Writing this I think I need to write a full article so will go do that now…
JD Kiwi
Roar Rookie
Good stuff, looking forward to next time! What a mess they've made.
WEST
Roar Guru
Most world cups there’s always that one pool they call the pool of death.. it’s unfortunate, but it seems difficult to spread the best teams evenly. Wales.. I don’t think anyone is giving them much of a chance this year. But Fiji in the Wallabies pool won’t be easy beats, the Fijian players working with Mick Byrne will have a huge advantage. Byrne is a great coach who relates extremely well with these guys.
PeterK
Roar Guru
it seems your model has wales with more chance than scotland of making the qf's with 2023 rankings. This is despite scotland winning their last encounter 35-7 in february and being ranked 5 v wales 9. I say this because you have wales chance at 69% of making the qf's, so for it to be above 50% and assuming ireland are favorite to make the qf's from a 2023 pool it would mean scotland has less chance than wales to make it.