What the hell happened to DialIdol?  Once the source for predictive rankings, DialIdol has gone from dead-on to vague to inaccurate.  Three weeks ago, Matt was predicted as safe from the bottom three.  He was in the bottom two.  And Megan was a DialIdol cellar dweller three weeks in a row before even falling into the bottom three (the week she was voted off).  Kris has had the lowest DialIdol score two weeks in a row and has not been in the bottom three either week.  Yet countless message board denizens swear the fix is in because the bottom three results didn’t jibe with DialIdol, not taking into account the possibility that DialIdol has shit the bed.  So, what happened?

 

Before we get to that, keep in mind that any DialIdol prediction is inherently inaccurate because it does not include text votes.  There were 78 million text votes cast last year – including the prelims (3 weeks of two performances a week), that’s 17 shows or an average of 4.6 million text votes an episode.  If the demographics of the text voters skew differently than the dial-in voters and that demo favors one contestant over the other, that can have a major effect on the real vote totals.  So right off the bat, DialIdol is not gospel when it comes to the bottom three.

 

But the real problem with DialIdol is its methodology.  It assumes that if a contestant gets enough calls, the line will reach capacity and start returning busy signals.  Thus, the contestants with higher busy signal ratios should be receiving more votes (there’s more statistical analysis applied to get down to a comparative number, but that’s it in a nutshell).  But this method assumes something that doesn’t appear to be true anymore – that each contestant has their own phone number with its own individual capacity.

 

Let’s go back to season 5, when they first started tracking with DialIdol.  For the top 10 show year, 32% of the calls, or nearly one out of three calls resulted in busy signals.  In season 6, that number dropped to 21%, a bit more than one out of 5 (presumably because they added more capacity per contestant).  But last year, in season seven, the call busy percentage plummeted to 3.5%.  This year, the call busy percentage dropped a bit more, to 3.3%.  So what happened? 

 

Presumably, the Idol producers installed a new call system last year to gather votes, one that probably pools all of the capacity together so that busy signals are effectively eliminated.  If no one is voting for Matt, why let his capacity sit there unused when callers are desperately trying to log votes for Adam?  If you chart the drop in viewers against the rise in votes per week (they broke 36 million votes a few weeks ago even though they have around 10 million fewer viewers since season 5), it clearly shows that more people are getting their votes logged than ever before, even though the number of people dialing has dropped 30%.  So why do he busy percentages match at all?  And why are there any busy signals anyway? 

 

Well, back in season 5, someone at MIT made an interesting observation – not all busy signals are created equally.  If you live in an area where there is high phone capacity, say, NY or LA, then your local exchange has the capacity, especially after non-business hours, to route a larger number of calls than a local exchange in a smaller city anywhere else.  So if you live in a smaller town and all of your neighbors are trying to reach Idol at the same time, you may encounter busy signals, but many of them are busy from the swamped local exchange and not because the Idol phone banks have reached capacity.  But the DialIdol software cannot distinguish between the two – a call busy is a call busy is a call busy.

 

The reason the busy signal measure still sort of works even with all these changes is that if a local exchange gets swamped in Podunkville, USA and most of the people there are trying to log a vote for Danny Gokey, then Danny will appear to have a lot of call busies.  So there is still some correlation between call busies and voting trends.  However, these trends are skewed – places with less phone capacity are over-represented in this model while dialers in larger, metropolitan areas will not see many call busies to begin with.  Still, if there are enough votes separating contestants and there just happens to be consistent voting patterns throughout the country, then you will get results like the one for Top 9 week where the bottom three DialIdol scorers were the bottom three (although Anoop was bottom two and not Allison).  However, that’s not actually the correct way to read the results – there really was no “bottom three” that week, since five of the nine contestants were in a statistical dead heat to be voted off, and seven of the nine could have been in the bottom three (statistically speaking).

 

But wait a minute – isn’t DialIdol still well over 80% accurate?  Well sure – if you only call one or two contestants as safe and then bunch everyone else together as in danger, it’s not really much of a feat to be correct.  As a matter of fact, due to the bottom two going home this week, all of the Idols are in danger of being eliminated according to DialIdol.  Which means, perversely, that DialIdol will be 100% correct no matter what.  The vagueness problem started last year when the plunge in call busies made the sample size too small.  The only way to correct it was to increase the margin of error, but all that did was make all but one or two contestants in danger of being voted off.  The rankings and DialIdol scores themselves are immaterial if you are reading the results correctly – it only matters that they are green (safe), yellow (in danger of being voted off), or red (predicted to be voted off).  The rookie mistake when interpreting DialIdol is to assume that since Kris, let’s say, is ranked last, he’s more likely to go home.  The truth is, all the yellows are equally likely to go home (statistically speaking).

 

So the next time the voting results don’t jibe with DialIdol, it’s not a conspiracy on the producers’ part – it’s just another exposure of the flaws in the way DialIdol works.  Michael Johns (who was voted out even though he was supposed to be safe last year) knows this better than anyone.

(added 4/23/09)

 

Why was Lil ranked #1 at DialIdol and then voted off?

 

Note that Lil has consistently been getting the highest dial busy percentages. But the sample sizes are so small (she had the lowest support among DialIdol users) and are so concentrated in a few states (smaller ones in the South), that they statistically correct for it every week and drop her score (even so, she was ranked number two during the first top 7 week and still ended up in the bottom two). I suspect the lion's share of her support is in states with relatively low phone capacity which is why she gets so many call busies but not a correlated quantity of actual votes (the Vote For The Worst campaign for the past two weeks probably drove up the call busies as well).  This led to the illusion that Lil was getting a lot of votes.  She was actually getting a lot of votes (or more correctly, a lot of attempted votes) in a concentrated geographical area, and it did not correlate to the country at large.

Also keep in mind that the rankings are immaterial, statistically speaking. If everyone’s status is yellow, it means everyone has an equal chance of going home.  So there isn’t really a #1 ranking – all the contestants were #1 (or 7 or every number in between).  The unintentionally hilarious part is that the site is taking credit calling the bottom three correctly.  It’s like saying, “I think it could be anyone going home tonight” and then the next day saying, “I was right!”  Technically right, but practically useless.