The technology press can be a strange beast sometimes. We’ll quibble over facts and quotes that were mentioned in jest or passing (oftentimes in less than 140 characters). But, inexplicably, we’re willing to regurgitate statistics that are nebulous at best and sometimes just plain misleading — usually because they sound good in a headline. Today, I’m going to single out what’s probably the most frequently abused statistic: the cumulative user count.
You see it all the time. Hell, I’ve probably written hundreds of posts that mention the total number of users a startup has without batting an eye. This often refers to the number people to ever register for a site or the number of total downloads of an app. But in reality this number doesn’t say much because it fails to take into account how many people have stopped using a service. Put another way: going by total user count, MySpace is still kicking butt.
The total user count can be meaningful for a startup’s first few months of life, but the older a startup is, the more useless it becomes. A startup could have a gangbusters launch with two million signups only to morph into a ghost town over the ensuing six months. But you can bet its press release would still boast about its millions of users.
This was brought back to my attention this morning when I saw the news that Evernote had passed 5 million users. I love Evernote — I use it all the time, and the fact that the service added the most recent million in less than three months indicates that it’s doing well. But it also reminded me of a talk that Evernote CEO Phil Libin gave earlier this year where he give entrepreneurs some insight on how to track their users. Here’s what he had to say about total cumulative users:
“That’s, by the way, the only chart that’s guaranteed by the way it’s put together to always be up and to the right… You should always have one of these charts in your pitch deck.”
Again, I’m not saying Evernote is doing anything sinister here — releasing this stat is par for the course in the industry, and Evernote actually tends to release data that other companies keep private, like how many of its users are paying customers. What I am saying is that technology journalists need to be more critical and careful when it comes to stats like these.
Further exacerbating the problem is how loosely journalists will throw these terms around. Words like ‘Members’ and ‘Users’ are often used interchangeably, when they can mean totally different things. Case in point: last month Mashable published an article proclaiming that Foodspotting had both 400,000 users and 400,000 members (Foodspotting then tweeted that it had 400,000 members). This didn’t seem to add up, because when I signed up for the service after the post was published my user ID was around 80,000. Update: Foodspotting says that its tweet came from a community manager who was just repeating what they’d read in the article.
I reached out to Foodspotting CEO Alexa Andrzejewski, who said that they had been very clear with Mashable that they’d had 400,000 app downloads, and that you don’t need to become a registered member to explore Foodspotting content, which is obviously an important distinction. Again, I don’t mean to insinuate that Foodspotting has purposely done anything misleading — it sounds like a reporter simply didn’t pay attention to how the startup was defining these terms.
The answer to this is pretty simple: tech reporters need to push startups to talk about how many active users they have, where the meaning of active isn’t up for debate (I think Facebook’s definition is a good one: an active user is one who’s logged in within the last thirty days). Many startups won’t want to release this stat, which is absolutely their prerogative. But treating the cumulative user count as if it’s a meaningful measure of popularity or engagement just doesn’t cut it for a startup when it’s no longer in its infancy.
Comment