A few months ago, I wrote an article that shocked some on The Roar when I predicted the All Blacks were the favorites to win the 2023 Rugby World Cup in France.
This prediction came from a “Monte Carlo” simulation model I had developed for the tournament (Monte Carlo simulations are a powerful uncertainty modelling technique that calculates the chance of any particular outcome occurring within a complex system).
The model used historical outcomes from head-to-head encounters between each team, the recent form of the teams as expressed in the relative World ranking and the impact of the draw on the outcome of the tournament.
After I published the article, I added a tournament factor to account for the performance of some teams during knock-out phases that had a positive impact on team like the Springboks, All Blacks and England and a negative impact on teams like Ireland, Fiji, Japan and Tonga.
Aaron Smith celebrates with his All Blacks teammates. (Photo by Cameron Spencer/Getty Images)
So how accurate was the model?
Across the tournament, the model correctly predicted the winner for 41 out of 48 games, more than 85 per cent accuracy. The model was especially good at predicting the likely winner during the pool stages and only got 4 from 40 games incorrect. These were the Wallabies losses against Wales and Fiji, Portugal’s upset win against Fiji (all in Pool C) and Japan’s win against Samoa in Pool D.
The model correctly predicted the winner and the runner up for Pool A, Pool B and Pool D, however predicted that the Wallabies would perform much better by winning Pool C with Wales second. The model didn’t foresee the shocking outcomes of the Wallabies during the pool stages. Seven of the eight finalists were correctly predicted, with 6 in the correct order. I think that was a fantastic outcome.
The real test came in the quarterfinals as most of these games were judged by rugby experts as 50/50. This was the stage when the knock-out factor also came into play. The model predicted:
– A 70% chance that Argentina will knock out Wales in QF1
– A 55% chance that Ireland will knock out New Zealand in QF2
– A 68% chance that England will knock out Fiji in QF3
– A 54% change that France will knock out South Africa in QF4
The model managed to correctly predict only 50 per cent of the games – a pleasant disappointment for All Black and Bok supporters. At the semi-finals the model estimated the probabilities as:
– a 90% chance that New Zealand will beat Argentina in SF1 and a
– 78% chance that South Africa will beat England in SF2.
Wallabies Angus Bell and Robert Leota celebrate a try. (Photo by Chris Hyde/Getty Images)
The model predicted the SF winners correctly. For the Bronze medal, the likelihood was 80% that England win beat Argentina and the final the likelihood was slightly in the All-Blacks favour (51%). Bok supporters will be delighted that model got that prediction also wrong.
The biggest surprises were the Portugal win over Fiji that had less than a 1/50 chance followed by Australia’s loss against Fiji that was considered only 8% likely. The Boks also surprised by recording wins against expectations against both France and New Zealand (these games were almost 50/50).
Before the tournament, the model considered the All Blacks as is most likely to win. After the application of a tournament factor, the probability was estimated at more than 30% for an All Black win, 20% for the Boks followed by Ireland (18%) and France (15%). The model also correctly predicted England would come 3rd (more than 35% likely) with Australia and Argentina most likely in 4th position (33% and 25% respectively).
What do you think – was that a good, reasonable or bad performance?
Muglair
Roar Rookie
Very impressive Bliksem, but your analysis does also show just how robust the rankings actually are. I had a look after the pools and 34/40 went with the pre RWC ranking. Your 4 games, the draw and England beating Argentina in the first game. Given the rankings are a slow burn, they could not take into account the dramatic improvement of Wales and England since the Six Nations and the further decline of you know who. It would be interesting to look back now and see which games did not go according to the post RWC rankings. Note to self. Sometime soon.
Bliksem
Roar Rookie
Coaching Northern Transvaal was his dream job - never got the opportunity to do it. Christie was sacked as Bok coach while in hospital undergoing treatment for cancer. He never had the opportunity to loose a test, won 14 in a row including tests against the All Blacks,
JD Kiwi
Roar Rookie
Exactly the same record as The Needle! No world cup in the 60s of course. Terrible how N Transvaal treated him.
JD Kiwi
Roar Rookie
Japan overachieved in the two previous world cups though... but they have an aging squad and have lost the advantage of the three year residential qualification. I don't think the All Blacks have tended to overachieve in world cups compared to the complete cycle.
Bliksem
Roar Rookie
I think Kitch Christie still have the best record of all. Hard to beat 100%.
JD Kiwi
Roar Rookie
Fantastic work - far more accurate than I was! So much for everyone's judgement - maths wins every time!
JD Kiwi
Roar Rookie
I think it's not just that South Africa so prioritise the World Cup knockouts. They rely so much on physicality and defence rather than skilful attack. It's difficult, in the international windows of a normal year, to consistently rise to the sort of effort levels needed for the very best punishing rush defence. Therefore, most of the time attack has more of a chance to succeed than it does during the RWC knockouts. It's so much easier to taper and get yourself to a physical peak for the pinnacle event. Attack oriented teams will therefore face a far tougher standard of defence and it will be harder for them to succeed, especially under such high stakes pressure. This is part of the reason that South Africa can win so many world cups (& England do so well in them) despite being not so great between them. Fun fact: Ian Foster has the worst win percentage of any All Black coach since Laurie Mains 1992-5. Yet it's a better win percentage than all the Springbok coaches since Nick Mallett over twenty years ago.
JD Kiwi
Roar Rookie
And the exact opposite of his approach against England, where he kept on far more older players than most nations after the 2015 and 2019 World Cups.
praetorian
Roar Rookie
Great work Bliksem Mooi so .
Bliksem
Roar Rookie
You can add Fiji, Japan with Ireland and France to teams that disappoint against expectations in knockout tournaments. Boks, England, All Blacks, Argentina and even the Wallabies play a game that suits knockout tournaments.
BleedRedandBlack
Roar Rookie
I never realised what a horrific record France have in KO games in France. Three games, three loses, 1991, 2007, 2023. All their KO victories in RWC's they have wholly or partly hosted have come outside France, 2 in 1999, 1 in 2007. That's as bad as Ireland's "never past the quarters". Stats can be very cruel.
Pisda Chastna
Roar Rookie
I see you did the predictions before Eddie selected that horrendous 33 man squad. You should have amended it after he announced who was going and who he cut. You would have got 38 out of 40 instead of 36 out of 40 :laughing:
BleedRedandBlack
Roar Rookie
I read it, was pleased but sceptical, happy to be proved wrong, am now extremely impressed by your accuracy. In my rather more hit and miss way I picked all the big games at group level, including Aussie v Fiji, thought I was pretty good, then went to pieces in the quarters, got three out of four wrong. Only rescued by England. Lordy, Ireland and France are horrible at tournament rugby. Got the last four right, but that was relatively easy. So on the whole I got one up on you over OZ and FJ, but behind you on NZ, SA, FR, IR, WA and AR. Fairly obviously, you win.
Bliksem
Roar Rookie
Doing that in my day to day job…decision and risk analysis…
Muzzo
Roar Rookie
Interesting read Bliksem, but to be fair it just shows again, where the strength of the game is!! In the SH!!
Lem
Roar Rookie
Time to monetise that IP mate...
mzilikazi
Roar Pro
I echo that, Ken. No joy at all.
mzilikazi
Roar Pro
That is honest, Blik. I guess I, and others here, esp. in Qld. have long memories of just what damage Jones can do. But in all honesty, I never saw him being quite so bad. The dropping of the experienced group just came out of left field.
Bliksem
Roar Rookie
What was interesting when I did a back calculation was how much the probability of winning improved for the Boks in the knockout phase compared to the periods between RWC. The Boks were a staggering 33% more likely to win during knockouts that between RWC. It was clear that Rassie-Nienaber put a much higher premium on the RWC than others and was more inclined than others to use the tests between RWC tournaments to prepare for the next RWC. EJ got flack for extending that to use a RWC tournament to prepare for the next RWC. I don’t like that four year cycle but it is clear that it works. The Boks had the experience on the field when it mattered that allowed then to win close games.
Bliksem
Roar Rookie
I overestimated Jones, that is sure. I even backed his approach to pick a young squad.