Uncovering the Discrepancies in Unique User Data Generated
by
Panel and Server-Based Online Measurement Techniques
A Point of View By Bill Perry, Director, Special Project, Audit Bureau of Circulations, Schaumburg, Ill.
Part two in a three-part series.
This article is the second in a three-part series intended to address the confusion surrounding online audience and traffic metrics, including differing methodologies and the future of the online measurement industry. The first white paper in the series examined the two primary methods used to measure a Web site’s traffic and audience and the role of third-party auditing in providing independent verification for either method.
Because ABC audits both panel and server-based data for the U.S. daily newspaper initiative Audience-FAX, I’d like to shed some light on how each of these measurement methods can generate different numbers for two of the most common metrics—unique users and page impressions—and how both of those numbers may be calculated correctly, based on their underlying methodologies.
Calculating Unique Users
One of the more popular Web metrics is “unique users” or “unique visitors.” These terms are typically used interchangeably, although the IAB is expected to further define them in its forthcoming audience measurement guidelines. The first step in understanding the discrepancies between the unique user statistics generated from a panel and those generated from a Web site’s server log files is to recognize that they are really two different sets of data. Even though both may report unique users or unique visitors, the methodology used to arrive at the data is so fundamentally different that the resulting statistics can’t be considered equivalent.
The Panel Method
The panel-based measurement method recruits a sample of Internet users and applies their habits and behaviors to the larger Internet population. Participants on the panel are typically required to install software on their computer (preferably on each computer they use) and are provided a unique login to use each time they access the computer. This helps ensure that all of the Web activity is tied to an individual. Because the panels track the habits of actual people participating on the panel, not machines, they also have the advantage of providing user demographic data.
In a recent interview published in Editor & Publisher, Nielsen Online’s President of Global Services and U.S. Sales Manish Bhatia said that their random-digit-dial panel (RDD) was comprised of approximately 30,000 people in the U.S. Of those 30,000 people, about 3,000 are part of the at-work panel that measures the panelists’ Internet usage while at work. For the panel companies to generate representative numbers for your Web site, the Internet usage habits of those 30,000 panelists need to be typical of your Web site’s visitors. If not, the measurements can skew greatly from month to month.
Let’s examine a couple of different scenarios that may affect panel measurements.
- Minimal Panel Coverage for Regional and Niche Web Sites
Web sites that are targeted to a unique or regional market and those that diverge from the broad Internet population may not be accurately represented on a panel. This is especially relevant for ABC’s business-to-business publication Web sites and regional and community newspapers. If there are only a few panel participants who visit your Web site, then the panel may not accurately represent your typical traffic. This lack of representation may result in large swings in unique user and page impression counts from month to month.
- Large Volume of At-Work Traffic
One of the most controversial elements surrounding the panel companies is their ability to accurately track the Internet usage habits of the at-work population. The at-work market segment is an active and important part of the Internet community. Industry estimates project that the average at-work user generates 17 sessions per person and visits 41 domains per week. The average at-home user only generates 11 sessions per person and 26 domain visits per week. This is why Web site publishers are so eager to receive precise numbers for their traffic activity during normal business hours. If a significant amount of your Web site traffic is generated during work hours, there is the potential that the panels are not accurately representing your Web site’s unique users.
- URLs and Domains Included in Your Web Site Definition
This third scenario is not influenced by the composition of the panel, but it is something that affects a significant number of our members. When panel companies gather data for a Web site, they must decide on what domains to include in the final overall number for that Web site. Upon first glance, this may seem simple; the panel companies should include the homepage of the Web site and any other URLs that contain the homepage domain. But in reality, many Web sites use third-party companies to deliver content throughout their sites. Often, these third-party companies design the pages to mimic the overall “look and feel” of the Web site’s homepage and the visitors never realize they’ve left their original destination. But if the panel company doesn’t include the third-party vendor’s site as part of the Web site definition, that traffic is credited to another site. In the newspaper industry, this is especially relevant for Web sites that use third-party vendors to deliver obituaries, classifieds, etc.
Let’s use a fictitious newspaper Web site as an example—www.anytownnews.com. This is the main homepage of the newspaper and is how the majority of the visitors access the site. Now let’s theorize that the Anytown News uses a popular service to deliver its obituaries to readers via www.obits.com/anytownnews. When the visitor of www.anytownnews.com clicks on the link to access the obituaries, they are seamlessly transferred to www.obtis.com/anytownnews. The obituary page has the same branding as the Anytown News’ Web site, but the domain is different. Even though that traffic should arguably be credited to the Anytown News, the panel company may not include www.obits.com/anytownnews in the Web site’s definition and instead may credit www.obits.com.
The Server Method
While the server method does not provide any demographic information for your site, it does have the distinct advantage of tracking almost 100 percent of the traffic on your Web site, regardless of visitors’ locations, including at-home, at-work and mobile devices. The server method however, does not actually track the Internet usage of a human like the panel method; instead it equates a unique user with a unique cookie set on a browser. All of these factors are key in determining an accurate unique user count. If any of these elements change, a new unique user may be erroneously flagged.
Let me give you some examples of how these changes might affect the server’s unique user count.
- Change in Unique Cookie Set
Cookies are small text files stored on a computer that collect a variety of data to identify the user. Some people manually delete their cookies on a regular basis as a security or privacy measure. When they visit your Web site again, new cookies are issued that identify them as new unique users.
The rate of cookie deletion is debatable, although some research suggests it may be as high as 30 percent. Others argue that users who completely reject cookies or multiple users accessing the same computer may actually offset the inflation caused by cookie deletion.
- Change in Unique Computer
For many of us, a change in the device we use to access the Internet is a daily occurrence—home computers, work computers, Blackberries, etc. Because the server method recognizes computers and mobile devices based on cookie sets, not the person using them, this event registers a new unique user and can cause you to be counted as one unique user on your work computer, another unique user on your home computer and yet another unique user on your mobile device.
On the other hand, multiple people accessing the same computer may not be counted as individual unique users. For example, if you, your spouse and your two children all use the same browser, the four of you may only count as one unique user.
Calculating Page Impressions
Despite the challenges of defining a page impression in the Web 2.0 world of Ajax, widgets and other evolving technologies, many ABC members still find this metric relevant and include it as part of their audited Web metrics. Page impression data is important because it represents the interaction of a visitor with a Web site. This interaction is the foundation of all other Web metrics, including unique users, time spent, and visits.
The controversy surrounding page impression data is more muted than that surrounding unique user data. Why? Because by eliminating the “unique” factor, you’ve effectively removed the angst of much of the unique user debate. Page impressions counted by a panel are counted just like a unique user. The same panel and the same methodology used to generate unique user data are also used to generate page impression data. This is also true for server data. The log files that generate the unique user data generate the page impression data. The only difference on the server side is that the “unique” element is removed, rendering it irrelevant if cookies are deleted or if the same person accesses the page on several computers.
Which Methodology is Better?
Panel measurements and server measurements have their own distinct advantages and disadvantages. An independent third-party audit can help limit the discrepancies associated with each type of measurement.
Since both methodologies have strengths and weaknesses, it’s impossible to endorse one over the other; nor is it ABC’s role as an independent auditor to do so. Rather it is important that both publishers and advertisers understand the inherent differences of the methodologies and also understand how to best apply the data generated from either measurement method to buy and sell online advertising.
Next in the Series
In the third and final article in this series, we’ll discuss what’s in store for online audience metrics including the soon-to-be released IAB audience measurement guidelines, which ABC has participated in drafting. We’ll also evaluate an innovative Web site measurement company named Quantcast. Quantcast has developed new measurement technology that combines both panel and server-based methodologies to provide online publishers and advertisers yet another way to measure Web sites.
For More Information
For more information on our interactive audit services, members are encouraged to submit an online request form or contact ABC’s
manager, marketing and sales, at (720) 324-0558.
About ABC
With more than 4,000 members in North America, ABC is a forum of the world’s leading magazine and newspaper publishers, advertisers and advertising agencies. The organization provides credible, verified information essential to the media buying and selling process. ABC maintains the world’s foremost online database of audited-circulation information and a growing array of readership, audience and Web site usage data. To learn more, visit http://www.accessabc.com.
Bill Perry
joined ABC in 1984 as a newspaper field auditor. He joined ABC Interactive (ABCi) in 1997 as manager of interactive auditing and was promoted to director, special projects in 2000. Perry’s current responsibilities include leading a team of auditors in a variety of non-traditional auditing projects including Web activity, readership verification, insert verification and centralized audits. Perry earned his bachelor’s degree in accounting from Indiana University.
|
