Wednesday, January 21, 2004

   Go directly to:
   ShowPrep Starters

"Some drink deeply from the river of knowledge. Others only gargle."

—Woody Allen

Go to year-end charts

  2005 Year-End Charts
> AC
> "Best Of" Reporter Poll
AC Comprehensive Chart
> AC1: Expanded Chart
> AC2: Chart Breakouts
> AC Top Spins
> AC Top Impressions/BDS
CHR Comprehensive Chart
> CHR1: Expanded Chart
> CHR2: Chart Breakouts
INS Comprehensive Chart
> INS1: Expanded Chart
> INS2: Chart Breakouts
Station Playlists
> BDS Monitor
  2004 Year-End Charts
> AC
> "Best Of" Reporter Poll
  PDA Home
  About The PDAdvisor
  About The PDA Website
  Subscribe to Free PDA Emails
  AvantGo: PDA on your PDA!
  PDA Reporting Stations
  BDS Monitored Stations
  The PDA 2004 AC Radio Seminar:
  2004 Seminar Recap
  2004 Seminar Pictures 1
  2004 Seminar Pictures 2
  Contact Us

2005, Brad Burkhart Christian Media, Inc. All Rights Reserved. Materials may not be reproduced without permission.

AC Top Impressions chart data 2005, Christian Music Trade Association. All Rights Reserved.

The PDAdvisor
12860 Bucksport Dr.
Roswell, GA 30075-6314

Click here to no longer receive The PDAdvisor


Online Research by Roger Wimmer, Ph.D.
Roger Wimmer, president of Colorado-based Wimmer Research, is considered by many to be the foremost authority on media research in the world.

I have received many questions about using the Internet for music testing or perceptual research. I think the best thing is to address all these questions at one time. Here is a sample of some of the questions I have received:

> Question 1: Our station is about to enter into a relationship with a research company for online "callout" research. The company says that the research is more representative and "valid" than regular callout research because the sample size will be so large it will overcome any discrepancies. Is this true?

> Question 2: We use one of the largest (they claim) Internet music research services. This company says we can test currents before we add them and be "assured" that the songs we add will be hits to our target audience. However, I don't understand how much people can really tell about a song they hear for the first time from a hook, even a long 20-second hook. The company boasts that we can make all, final, music decisions like this. How valid is testing previously unexposed new songs online?

> Question 3: My PD uses music research where listeners sign up to take music tests online, as well as answer some perceptual questions. He takes the data very seriously. My opinion is that this research method is relatively worthless because: 1. It's not a random sample; and 2. It polls only active, tech-savvy listeners.

My PD's take is that it allows us to super-serve the P1s. My take is that by super-serving the active P1s, you're allowing a very, very small percentage of the listenership to determine the direction of the radio station — kind of like polling request-line callers and making decisions based upon what they say. How much weight would you say this type of research should carry?

> Question 4: We don't have much of a budget to do research, so one idea I had was to let listeners go to our website and rate what they hear coming from our station. Do you think it's a good idea to include this on a radio station's homepage, or should we rather not mess with this?

To answer these and similar questions, it is important to understand some of the complex issues involved in online research. Let's look at some of these issues.

Valid and Reliable

In research, "valid" is defined as "testing what is intended to be tested." For example, determining a person's favorite radio station by asking if the person likes chocolate is not a valid measurement. With music testing, the validity question is very simple — "Does the method really test the respondents' likes, dislikes, or perceptions of songs (or parts of songs) they hear?"

Why is this important? Well, regardless of the music testing methodology (callout, auditorium, online), it's important to make sure that the procedure is scientifically correct.

This includes the rating scale used by the respondents. What type of scale is used? How many points are used — 3, 5, 7, 10, or something else? Do respondents understand the scale? Does the scale actually measure a person's feeling toward a song? (Note: Using anything less than five points may hide the respondents' perceptions about a song. In research, this is known as "factor fusion" — using too few rating points "squeezes" the data and hides fine distinctions among the ratings.)

Just because a research methodology is sold/pushed by someone does not make it valid. Don't blindly accept a new methodology (or any methodology) without questioning the validity of the method.

"Reliable" is defined as "consistently testing the same thing." A measurement instrument is unreliable if it produces different results each time it's used with a sample selected in exactly the same way. If the music test data "bounce around" — a song tests well in one test and poorly in another — the measurement instrument may not be valid. (The bouncing scores may also be due to other things, such as bad samples, or samples selected in different ways, or simply because of naturally occurring sampling error.)

Random Sample

In most online research situations, radio stations either have their music test on their website (addressed later) or an outside vendor uses a sample of volunteer respondents who rate the music. These are not random samples. This is a big point that must be discussed, but first I need to address the concept of a random sample.

A random sample is a sample in which everyone in the population/universe under study has an equal chance of being selected. In reality, there are no truly random samples in behavioral research because the respondents volunteer to participate in any study. To be truly random, we would need to force each randomly-selected person to participate in our study. We obviously can't do that, so we must hope that each randomly-selected person will volunteer. The problem is that not all of the selected respondents volunteer to participate.

If only one of the randomly-selected respondents refuses to participate in a study, the sample no longer matches the definition of a random sample and the sample becomes a volunteer sample. In other words, there are no truly random samples used in radio research. In fact, there has never been a true random sample used in radio research — every sample for every radio study ever done has used a volunteer sample.

Okay, it's a given that we can never expect a truly random sample in radio research. But, there are things that can be done so that the sample will be as random as possible. First, you should never accept pure volunteers in your sample — people who call in, write in, or in any other way ask to be involved in your study — unless these folks pass the screener designed for the research project.

But some leeway is often accepted in sampling, particularly in reference to radio station databases. Many radio stations have good databases of people and it makes sense to use them. However, it is not acceptable to blindly accept the respondents in the database for a research project. These people must pass the screener designed for the study.

Advisory Boards/Listener Panels

In the past few years, many radio stations have developed a listener "Advisory Board" or a "Listener Panel" to gather information about what the listeners like and don't like about the radio station. There is nothing wrong with using an Advisory Board or Listener Panel to gather research information if:

1. The board or panel is selected randomly.

2. The board or panel includes at least 100 respondents.

3. Twenty-five percent of the board or panel is replaced each time a study (or any data collection procedure) is conducted. That is, if the group is convened (or called) once each quarter, 100 respondents take part in the first quarter project. In the second quarter, 75 of the original sample are reused and 25 new respondents are added. When next year's first quarter project comes around, the entire original sample will have been replaced.

Replacing 25% of your sample for each study eliminates the problem of relying on only one group of people for your data.

In next Wednesday's PDA, Roger continues his in-depth evaluation of online research as he examines the effectiveness of testing new, unfamiliar music online, the issue of sample size, and "the big problem" with online radio research.

If you have a research question for Roger, email him at

>> Back to Top



The radio community has quickly accepted online (Internet) research for two main reasons: cost and speed. While PDs and others see the online research as a way to gather information cheaply and quickly, most don't question if the data are valid and reliable.

In addition, in many cases, the information is being gathered by people who don't have a research background.

Much of my nearly 30 years of experience in research has been spent explaining what research can and can't do. One recurring theme I hear often is that conducting research is easy — anyone can conduct research.

In addition, I have found that many people accept a research methodology or product simply because it's new (New = Good). These folks don't take the time to find out if the new methodology/product is right (valid and reliable).

What many people don't understand is that research is a complicated process that involves an understanding of many different things — sampling, questionnaire or instrument design, data collection, data analysis, statistics, and interpretation.

All of these areas include many strange-sounding words and terms and because of this, it's easy for someone to sell research to someone who has no research background. (I believe it's commonly known as "Pulling the wool over someone's eyes.")

For too many years, non-researchers have developed methods that look and sound right, but are patently wrong. These salespeople (and I'm talking about some of the "big" names who have the word "research" in their company's name) sell their products to people who have no research experience.

In many cases, the products are not correct. Programming consultants or former PDs develop the products, not researchers or statisticians. That, as some people say, "Don't be right."

The problem with radio research, specifically new methodologies like online research, is that not enough people ask questions about the research they believe in, buy, and use.

Instead of asking questions related to the correctness of the products and background of the salespeople (usually consultants or former PDs), many broadcasters simply ask, "How much does it cost?" and "How quickly can I get the information?" What happened to simple questions such as, "Do you have a research background?" and "Are the data valid and reliable?"

—Roger Wimmer

>> Back to Top

Monday, 1/26:
Topic: National Creative Frugality Week (1/25-31)
– What's the most frugal thing you do?
Bubble Wrap Appreciation Day
Anniversary: TV premiere of "The Dukes of Hazzard" (1979)
Theme Song:
Birthday: Kirk Franklin

Tuesday, 1/27:
Topic: National Take Back Your Time Week
– What's your biggest timesaver?
– What's your biggest timewaster?

Wednesday, 1/28:
National Compliment Day
– What's the best compliment you've ever received?
– What's the worst compliment you've ever received?
Anniversary: Space Shuttle Challenger explosion (1986)

Thursday, 1/29:
Topic: Sunday's Super Bowl XXXVIII
New England Patriots vs. Carolina Panthers in Houston (televised on CBS, 6:25pm EST on Sunday)
– What do you do if you don't watch the game?

Friday, 1/30:
Topic: Sunday's Super Bowl XXXVIII
Football 101: documents/articles/rth_art_035288.cfm
Fun facts:
Anniversary: Birth of Franklin D. Roosevelt

This Weekend:
Saturday, 1/31:
Birthday: Eliot Torres (Salvador)
Sunday, 2/1: Super Bowl
Anniversary: Space Shuttle Columbia disaster (2003)
Birthdays: James and John Katina (The Katinas)
Toby Penner (Jake)

Looking Ahead...
2/14: Valentine's Day
2/16: President's Day
3/17: St. Patrick's Day

ShowPrep StartersTM is compiled by PDA publication manager/editor Sherri Hull.

>> Back to Top

Powered by Cahill Interactive