Wednesday, February 4, 2004

   Go directly to:
   ShowPrep Starters

"An education isn't how much you have committed to memory, or even how much you know. It's being able to differentiate between what you know and what you don't."

—Anatole France
French novelist, Nobel Prize winner

Go to year-end charts

  2005 Year-End Charts
> AC
> "Best Of" Reporter Poll
AC Comprehensive Chart
> AC1: Expanded Chart
> AC2: Chart Breakouts
> AC Top Spins
> AC Top Impressions/BDS
CHR Comprehensive Chart
> CHR1: Expanded Chart
> CHR2: Chart Breakouts
INS Comprehensive Chart
> INS1: Expanded Chart
> INS2: Chart Breakouts
Station Playlists
> BDS Monitor
  2004 Year-End Charts
> AC
> "Best Of" Reporter Poll
  PDA Home
  About The PDAdvisor
  About The PDA Website
  Subscribe to Free PDA Emails
  AvantGo: PDA on your PDA!
  PDA Reporting Stations
  BDS Monitored Stations
  The PDA 2004 AC Radio Seminar:
  2004 Seminar Recap
  2004 Seminar Pictures 1
  2004 Seminar Pictures 2
  Contact Us

Registration is fast and free!

2005, Brad Burkhart Christian Media, Inc. All Rights Reserved. Materials may not be reproduced without permission.

AC Top Impressions chart data 2005, Christian Music Trade Association. All Rights Reserved.

The PDAdvisor
12860 Bucksport Dr.
Roswell, GA 30075-6314

Click here to no longer receive The PDAdvisor


10 Online Research Keys
by Roger Wimmer, Ph.D.
Roger Wimmer, president of Colorado-based Wimmer Research, is considered by many to be the foremost authority on media research in the world.

Over the past two weeks, The PDAdvisor has presented the first two parts of Roger Wimmer’s in-depth response to several questions posed to him about Internet-based research. In Part 1 (read>>), Roger discussed the standards for "valid and reliable" research and began looking at several issues related to developing a good sample.

Part 2 (read>>) examined the sample size fallacy that "big = good" and the sample control problems created by online research, while the sidebar looked at the challenges of testing new music online. Roger concludes the series this week by reviewing 10 keys to consider when hiring an online research company or deciding to conduct Internet-based music research in-house.

Online research can be done correctly. The Internet has opened the doors to a variety of new and exciting research collection methods. However, from what I have seen, the research is not being conducted correctly and radio station people are using data that are, at best, questionable.

If you do decide to hire a vendor to collect information via the Internet, or if you conduct the research yourself, there is one requirement you must satisfy:

1. Know your respondents. I understand that 100% respondent verification isn't possible in any type of research. However, you must have some idea of who is answering your online research questions. If you don't, then you shouldn't use the data.

When you have respondent verification (whatever amount you have), you can check the validity and reliability of your data with a few simple statistical tests. If you don't know how to conduct these tests, find a researcher or statistician to help you run things like a t-test, z-score comparisons, correlations, and standard deviation comparisons.

If you satisfy the first requirement, then here are a few other mandatory items:

2. If you hire a company to conduct your research, make sure that the company has a researcher or statistician (minimum Master's degree in research or statistics; Ph.D. preferred) on staff, or at the very least, that the research method was designed and tested by a researcher or statistician. If the company doesn't have a researcher or statistician involved in the methodology, then don't use the company.

If you conduct the research yourself, the same rule holds — you should either have a Master's or Ph.D. in research or statistics, or hire someone who does to develop and test your methodology.

3. Make sure that the measurement scale uses at least 5 points. Anything less than 5 points will create "factor fusion," in which the small scale "crunches" the data into too few points and doesn't show enough variance. Although a 7-point scale is good, I prefer to use a 10-point scale for music ratings and perceptual ratings.

4. Before you begin to use your data for decision-making, you need to run several statistical tests to determine the validity and reliability of the data. You don't need to get carried away here — t-tests, z-score comparisons, correlations, and analysis of variance will do.

5. Do not fall for the large sample scam. A large sample does not guarantee that the sample is correct.

Stay away from any company that sells its sample as valid and reliable because the sample size is large. Hire a company that sells its sample as valid and reliable because the respondents go through appropriate screeners and are verified.

6. Do not use online research to test new songs unless the respondent hears the entire song. We know if respondents hear an entire song in an auditorium setting. We don't know how to do that on the Internet.

If you figure out a way to verify that the respondents did actually hear the entire song, please let me know how you accomplished the task.

7. Do not use pure volunteers for your research. All respondents must pass through your screener. This is true for all respondents, regardless of the source of the list (e.g., your radio station's database).

8. Rotate your sample. As detailed in Part 1 (read>>), for specific types of research including listener advisory boards/panels, replacing 25% of your sample for each study will eliminate the problems of relying on only one group of people for your data.

9. Use z-scores to compare your online data to auditorium tests, callout, or one online test to another. Do not compare raw scores. That isn't valid or reliable. It's also wrong.

10. Do not lend your data to another radio station or take data from another radio station and use the information for your radio station. There is no guarantee that the data from one market will relate to any other market. (If you know statistics, there is a way to determine if you can share data with other radio stations.)

As I mentioned, online research presents many great opportunities, but you must know what you're doing. If you use online research, do it right.

If you have a research question for Roger, email him at

>> Back to Top



The PDA office has received quite a bit of feedback on the first two parts of Roger Wimmer's online research article, ranging in tone from professional to personal. Many have expressed curiosity and a desire for deeper understanding of the subject, some satisfaction that the articles affirmed what they already believed, and a few skepticism and even anger that their initiatives would be questioned.

Dr. Wimmer's knowledge in the field of media research is unmatched by anyone working in Christian radio, but this isn't (or shouldn't be) an issue of who is right and who is wrong. This is about understanding objective, scientific truth and ultimately desiring to do what is best for our stations.

And of all of the formats, those of us in Christian radio should be the most willing to sift out ego in our quest for doing what is right, for finding truth.

Following are Roger's responses to some of the new questions he received during the past week:

> Question 1: We simply can't afford "real" callout. Isn't online research, regardless of the lack of control over the situation or how poorly it's conducted, better than no research at all?

Is bad or poorly-conducted research better than no research at all? That's ridiculous. What would you say to a doctor who gave you a bottle of pills and said, "Although we don't know if these pills are any good, take them anyway because they are better than nothing."

Research is an attempt to discover something. However, to be valid and reliable, you must know what the "something" is and where the data came from. In the case of online music research, the something isn't known for sure.

Decisions made with bad or poorly-conducted research can only be bad or poor. Case closed.

> Question 2: From what you've written, our online research isn't really accurate and the way we're doing our listener panels isn't exactly right. But when you combine these and look at the big picture, don't things more or less even out? Even thought the findings are inaccurate by themselves, don't we have a much better view of what our audience really thinks?

If bad or poorly-conducted research is useless, how can a combination of inaccuracies be of any value? Contrary to some opinions, research doesn't simply involve collecting data. Research involves collecting good data from good respondents. There is no way that combining inaccurate findings to make decisions can make the data valid and reliable.

> Question 3: I know online music research isn't perfect, so wouldn't it be okay if I just look at it out of curiosity and then don't make any decisions based on the data?

Sorry, but that's not human nature. It's the same thing PDs said when Arbitrends (monthlies) were first produced. In reference to Arbitrends, Arbitron says, "The accuracy of Arbitron audience estimates cannot be determined to any precise mathematical value or definition." Even with this warning, programmers now interpret Arbitrends as "real" numbers.

If you know that online music (or listener panel) research isn't valid and reliable, then there is no reason to look at the data for any reason.

>> Back to Top

Monday, 2/9:
Topic: The Passion Of The Christ movie (in theaters 2/25)
Anniversary (40th): The Beatles' U.S. TV debut on "The Ed Sullivan Show"

Tuesday, 2/10:
Topic: Valentine's Day (Saturday)
The cost: The average person is expected to spend $100 on Valentine's Day gifts this year.
– What's the most creative Valentine's gift you've ever given?
– What's the most creative Valentine's gift you've ever received?

Wednesday, 2/11:
National Shut-In Visitation Day
– How do you make a visit to a shut-in meaningful?
– How do you involve your children in a visit to a shut-in?
Birthday: Clay Crosse

Thursday, 2/12:
Lost Penny Day
Donate those pennies you've been collecting to a good cause.
– What's the most you've collected in coins and what have you used it for?
Anniversary: Birth of Abraham Lincoln

Friday, 2/13:
Presidents' Weekend
National Day of Purity
Topic: Valentine's Day (Saturday)
Christian perspective:
Interesting facts:
10 ways to show your love for your kids:
– What was your most memorable Valentine's Day?
– What was your most humorous Valentine's Day?
– Do you do anything special for your kids on Valentine's Day?

This Weekend:
Saturday, 2/14: Valentine's Day
Sunday, 2/15: Daytona 500

Looking Ahead...
2/16: President's Day
3/17: St. Patrick's Day
3/20: Spring Begins

ShowPrep StartersTM is compiled by PDA publication manager/editor Sherri Hull.

>> Back to Top

Powered by Cahill Interactive