| Online Research
by Roger Wimmer, Ph.D.
Roger Wimmer, president of Colorado-based
Wimmer Research, is considered by many to be the foremost authority
on media research in the world. www.rogerwimmer.com.
I have received many questions about
using the Internet for music testing or perceptual research. I think
the best thing is to address all these questions at one time. Here is
a sample of some of the questions I have received:
> Question
1: Our station is about to enter into a relationship with a
research company for online "callout" research. The company
says that the research is more representative and "valid"
than regular callout research because the sample size will be so large
it will overcome any discrepancies. Is this true?
> Question 2: We use one
of the largest (they claim) Internet music research services. This company
says we can test currents before we add them and be "assured"
that the songs we add will be hits to our target audience. However,
I don't understand how much people can really tell about a song they
hear for the first time from a hook, even a long 20-second hook. The
company boasts that we can make all, final, music decisions like this.
How valid is testing previously unexposed new songs online?
> Question
3: My PD uses music research where listeners sign up to take
music tests online, as well as answer some perceptual questions. He
takes the data very seriously. My opinion is that this research method
is relatively worthless because: 1. It's not a random sample; and 2.
It polls only active, tech-savvy listeners.
My PD's take is that it allows us to super-serve
the P1s. My take is that by super-serving the active P1s, you're allowing
a very, very small percentage of the listenership to determine the direction
of the radio station — kind of like polling request-line callers
and making decisions based upon what they say. How much weight would
you say this type of research should carry?
> Question
4: We don't have much of a budget to do research, so one idea
I had was to let listeners go to our website and rate what they hear
coming from our station. Do you think it's a good idea to include this
on a radio station's homepage, or should we rather not mess with this?
To answer these and similar questions, it is important
to understand some of the complex issues involved in online research.
Let's look at some of these issues.
Valid and Reliable
In research, "valid" is defined
as "testing what is intended to be tested." For example,
determining a person's favorite radio station by asking if the person
likes chocolate is not a valid measurement. With music testing, the
validity question is very simple — "Does the method really
test the respondents' likes, dislikes, or perceptions of songs (or parts
of songs) they hear?"
Why is this important? Well, regardless of the music
testing methodology (callout, auditorium, online), it's important to
make sure that the procedure is scientifically correct.
This includes the rating scale used by the respondents.
What type of scale is used? How many points are used — 3, 5, 7,
10, or something else? Do respondents understand the scale? Does the
scale actually measure a person's feeling toward a song? (Note: Using
anything less than five points may hide the respondents' perceptions
about a song. In research, this is known as "factor fusion"
— using too few rating points "squeezes" the data and
hides fine distinctions among the ratings.)
Just because a research methodology is sold/pushed
by someone does not make it valid. Don't blindly accept a new methodology
(or any methodology) without questioning the validity of the method.
"Reliable" is defined as "consistently
testing the same thing." A measurement instrument is unreliable
if it produces different results each time it's used with a sample selected
in exactly the same way. If the music test data "bounce around"
— a song tests well in one test and poorly in another —
the measurement instrument may not be valid. (The bouncing scores may
also be due to other things, such as bad samples, or samples selected
in different ways, or simply because of naturally occurring sampling
error.)
Random Sample
In most online research situations, radio stations
either have their music test on their website (addressed later) or an
outside vendor uses a sample of volunteer respondents who rate the music.
These are not random samples. This is a big point that must be discussed,
but first I need to address the concept of a random sample.
A random sample is a sample in which everyone
in the population/universe under study has an equal chance of being
selected. In reality, there are no truly random samples
in behavioral research because the respondents volunteer to participate
in any study. To be truly random, we would need to force each randomly-selected
person to participate in our study. We obviously can't do that, so we
must hope that each randomly-selected person will volunteer. The problem
is that not all of the selected respondents volunteer to participate.
If only one of the randomly-selected respondents refuses
to participate in a study, the sample no longer matches the definition
of a random sample and the sample becomes a volunteer sample. In other
words, there are no truly random samples used in radio research. In
fact, there has never been a true random sample used in radio research
— every sample for every radio study ever done has used a volunteer
sample.
Okay, it's a given that we can never expect a truly
random sample in radio research. But, there are things that can be done
so that the sample will be as random as possible. First, you
should never accept pure volunteers in your sample —
people who call in, write in, or in any other way ask to be involved
in your study — unless these folks pass the screener designed
for the research project.
But some leeway is often accepted in sampling, particularly
in reference to radio station databases. Many radio stations have good
databases of people and it makes sense to use them. However, it is not
acceptable to blindly accept the respondents in the database for a research
project. These people must pass the screener designed for the study.
Advisory Boards/Listener Panels
In the past few years, many radio stations have developed
a listener "Advisory Board" or a "Listener Panel"
to gather information about what the listeners like and don't like about
the radio station. There is nothing wrong with using an Advisory Board
or Listener Panel to gather research information if:
1. The board or panel is selected randomly.
2. The board or panel includes at least 100 respondents.
3. Twenty-five percent of the board or panel is replaced
each time a study (or any data collection procedure) is conducted. That
is, if the group is convened (or called) once each quarter, 100 respondents
take part in the first quarter project. In the second quarter, 75 of
the original sample are reused and 25 new respondents are added. When
next year's first quarter project comes around, the entire original
sample will have been replaced.
Replacing 25% of your sample for each study eliminates
the problem of relying on only one group of people for your data.
In next Wednesday's PDA, Roger continues
his in-depth evaluation of online research as he examines the effectiveness
of testing new, unfamiliar music online, the issue of sample size, and
"the big problem" with online radio research.
If you have a research question for Roger, email
him at rogerwimmer@thePDAdvisor.com.
>> Back
to Top
|
|
 |
|
ASK THE RIGHT
QUESTIONS
The radio community has quickly
accepted online (Internet) research for two main reasons: cost and speed.
While PDs and others see the online research as a way to gather information
cheaply and quickly, most don't question if the data are valid and reliable.
In addition, in many cases, the
information is being gathered by people who don't have a research background.
Much of my nearly 30 years of experience in research
has been spent explaining what research can and can't do. One recurring
theme I hear often is that conducting research is easy — anyone
can conduct research.
In addition, I have found that many people accept a
research methodology or product simply because it's new (New = Good).
These folks don't take the time to find out if the new methodology/product
is right (valid and reliable).
What many people don't understand is that research is
a complicated process that involves an understanding of many different
things — sampling, questionnaire or instrument design, data collection,
data analysis, statistics, and interpretation.
All of these areas include many strange-sounding words
and terms and because of this, it's easy for someone to sell research
to someone who has no research background. (I believe it's commonly
known as "Pulling the wool over someone's eyes.")
For too many years, non-researchers have developed methods
that look and sound right, but are patently wrong. These salespeople
(and I'm talking about some of the "big" names who have the
word "research" in their company's name) sell their products
to people who have no research experience.
In many cases, the products are not correct. Programming
consultants or former PDs develop the products, not researchers or statisticians.
That, as some people say, "Don't be right."
The problem with radio research, specifically new methodologies
like online research, is that not enough people ask questions about
the research they believe in, buy, and use.
Instead of asking questions related to the correctness
of the products and background of the salespeople (usually consultants
or former PDs), many broadcasters simply ask, "How much does it
cost?" and "How quickly can I get the information?" What
happened to simple questions such as, "Do you have a research background?"
and "Are the data valid and reliable?"
—Roger Wimmer
>> Back
to Top
|
