'People lie, internet stats tell the truth'
4th Mar 2009 | 15:46
Interview: Stats guru and author Bill Tancer talks trends
Lies, damn lies and search data
"Someone's not telling the truth," laughs internet stats guru Bill Tancer as he tells TechRadar of his latest stunt at conferences – asking for a show of hands on who accesses adult sites.
Tancer, the author of CLICK: What We Do Online and Why It Matters and General Manager of Global Research for web stats company Hitwise, certainly knows what he's talking about.
"I ask by a show of hands how many people access adult entertainment and sites," he says when TechRadar asks him about the disparity between what we admit to publicly and what we actually do on the internet.
"Out of thousands of people I've only ever got one person raise his hand and that was the sound guy at one of conferences I was doing.
"And then you go online and look at the percentage of the online population that accesses adult entertainment sites - here in the UK I think it's 7.7 per cent and in the US it's about 11 per cent, so clearly someone's lying."
It's not all about peope's embarrassment over their porn habits however, with Tancer admitting that 'cognitive dissonance' or saying one thing while doing another is a common trait.
"People want to appear in the best possible light," he adds. "They don't want to appear unsavoury in an audience of other people so they might change their answer.
"Adult sites are an extreme example, but it can come into play with much smaller things.
"Sometimes it's a matter of recall, but often people just want to answer in a way that makes them look intelligent."
Tancer spends much of his day seeking patterns in a sample of 25 million taken worldwide by HitWise through ISPs, although he hastily points out that the data is anonymised and collated. Hitwise simply doesn't have any interest in tracking individuals.
TechRadar asks if it's sometimes easy to start seeing patterns where there are none, or add two and two together and make five.
"One thing I've written about since finishing the book is what I call narrative fallacy – so when you have a massive dataset you might fall in love with your hypothesis and really look at the data that supports it.
"So there is a little bit of intellectual honesty needed, and you have to be pretty diligent.
Lipstick on a pitbull
"I was writing a piece for Time and there's a theory in the states which as been propagated by the Chairman of Estee Lauder that in tough economic times sales of lipstick goes up.
"The theory is that lipstick is an affordable luxury – when times are tough it doesn't cost much to buy lipstick.
"So I charted the search term lipstick over a three year period to see if I could find a correlation and get this flat line up until September 2008 and then a massive increase in the searches on lipstick.
"I then superimposed the Dow Jones Index over the top and got that perfect negative correlation. I'm so excited; I write my column and I'm about to submit and then I decided to take a step back and looked a little deeper.
"One thing I can do with our data is to look at all the search terms that contain the work lipstick and when I did that the entire column was shot because the number one search term was 'lipstick on a pitbull'.
"The whole discussion between Sarah Palin and Barack Obama and that one phase had been a huge topic of conversation; people were looking for videos or articles. It just happened to coincide perfectly with a drop in the Dow."
Globalisation of search
Tancer, has noticed some key differences between the United States audience that he knows so well and the UK's internet users.
"There is an optimisation or globalisation of what people search for but there are key differences between cultures," he adds.
"I do see differences and if you look at the stocks and share sites in recent months there was a massive spike that week [of the crash] in September and it's maintained a high level.
"In the UK, however, there was a spike in September but surprisingly a lot of that traffic has calmed down. Is that the hysteria dying down or are people simply not wanting to look at the problems? I can't answer that just yet.
"We certainly see more diversional activity in the States – we found games sites are getting more popular, more movies are being rented and they are visiting celebrity blogs more – it's escapism."
It is clear that the way in which we use the internet offers fascinating glimpses into our changing lives, and Tancer is clearly a man keen to keep tracking those changes - be it predicting the winner of Celebrity Come Dancing or telling us what internet trend is going to hit the mainstream next.
Tancer's book 'CLICK: What we do online and why it matters' is out now in paperback published by Harper Collins.