Online Research Comment
Dr. Wimmer: You’ve got a great thing going with this column. Useful and entertaining—what a concept! I just wanted to throw down some major kudos (not the candy bar either) to you regarding the online research debacle. I know that there are several of "us valid researcher folk" who are waiting in the wings with online product. However, until there’s an ideal way to verify the sampling criteria, there’s absolutely now way of knowing who's participating in the test. I think I'd have a hard time getting my clients to buy a music test in which respondents could just call in (from any state or province) and rate the music with no verification whatsoever. That’s a scary thought (or in your terms, it just don't be right)! Yet, that’s what many PD’s are subscribing to with the whole online thing. Use your heads folks! If I can join a chat room as a buxom blonde supermodel, then just imagine who' is in your research panel! Doc, you're a champion of common sense and a purveyor of truth. Keep up the great work! - Mike
Mike: Thanks for the comments about the column. I appreciate that.
The main problem I have had during my career in research is trying to get people to see what research can and can’t do. Many people immediately subscribe to a research methodology or product just because it’s new. They don’t take the time to find out if the new method is right.
The problem is that research is a complicated process that involves many different things—sampling, questionnaire/instrument design, data collection, data analysis, and interpretation. Because of this complexity, it’s easy for someone to sell things to someone who has absolutely no research background. (I believe it’s commonly known as "Pulling the wool over someone’s eyes.")
For too many years, research salespeople (they are not researchers) have developed methods that look and sound right, but are patently wrong. These salespeople (and I’m talking about some of the "big" names in research) sell these ideas to those who don’t have any research experience at all. The salespeople throw around a few research terms to make things sound complicated and correct, and the unsuspecting PDs and GMs think they are correct. They take the words of these salespeople as gospel and just assume that what they’re buying is scientifically correct. But in most cases, the methods are not correct. They are merely methods developed in the sales department, not research department.
Online research in all businesses has great potential, but as you say, we just don’t know enough about who is answering the questions. We don’t know enough about sampling. We don’t know enough to compute a simple sampling error percentage.
I’m reminded of one group PD who said, when I asked him why he switched to 100% online research, "They [the research company] told me that the results are the same as a an auditorium music test and a telephone study." I asked him to see the data that compared the two methodologies. That was over a year ago. I haven’t received a thing. I haven’t received a thing because he doesn’t have it—the comparison data don’t exist. He took the word of the research company. No data, no proof, no scientifically conducted comparison—just their word. Quite frankly, in scientific terms, that sucks.
Would he (or anyone else) accept a medical prognosis from an electrician who learned medicine because of so many visits to the doctor? Would he (or anyone else) allow a bank teller to install a transmission in his vehicle because the teller had the job done so many times? I think not.
So why do radio people (TV people are just as bad) accept the word of non-researchers to provide data for their multi-million dollar operations? Why? I don’t get it.
If such a study exists, bring it on! Send it to me. I will donate my time to review it. If the data show, and can be replicated by me, then I will change my mind. That’s the advantage of following the scientific method—it’s self-correcting.
Merely saying, "My auditorium music tests scores are pretty close to my online music research scores." will not cut the mustard. I want to see valid statistical tests that compare the two methods (the same thing with comparing online research with telephone perceptual studies). Show me the data! That’s all I ask.
If I can replicate findings, I will be happy to change my tune and I’ll be the first to admit that online research is "OK."
Sometimes the adage "You get what you pay for" doesn’t fit. But in the case of online research, it’s perfectly appropriate. If you have data to prove me wrong, I already said, "Bring it on." And let’s get ready to rummmmmblllllllle!
(Warning: Don’t write to me and say that your data "look" the same to you. Prove it statistically. I will not accept personal opinion.)
Online Research - Website
Hi Doc! I just had a meeting at our radio station, and out of the financial problems we have, somebody proposed we should care more about what our listeners actually wanted to hear. (Some don’t—after all it’s public radio, they say, so the DJs and talents should not lose the freedom of what they do and play.)
Now since we don’t have really much of a budget to do research, one idea I had was to let people go to our website and rate what they hear coming from our station. However, I don’t know if that’s a good idea because I’ve yet to see this at any terrestrial radio station. There are some stations here who display what’s playing on their website, but none of them lets the web site visitors rate those songs (or other content). I have only seen that form of rating on some Internet radio stations, like Radio Paradise, or Flashback Radio. Do you think it’s a good idea to include this on a radio station’s homepage, or should we rather not mess with this? And do you know any terrestrial radio stations that have a rating feature like that (for the current content) on their homepage? With greetings from Austria. - Kurt
Kurt: Hi to you again. You really keep me busy.
First, the comment that the “DJs and talents should not lose the freedom of what they do and play” and therefore you shouldn’t ask the listeners what they want makes no sense. In fact, it’s a ridiculous approach to follow. The radio station—public or not—is for the listeners, not the DJs or other talent. The role of the on-air folks is to provide the listeners what they want, not what they (DJs and talent) think they need. That’s goofy.
OK, on to your question.
You can include song ratings on your website if you understand/do/know these things:
You don’t know who is rating your songs. The people could be children, pranksters, people from out of Austria, or other things. You’ll have data, but you won’t know the source.
If you do decide to rate songs on your website, then you should run several statistical tests to determine the validity and reliability of the data. You still won’t know the exact identity of the respondents, but a few statistical tests will help determine if you can use the data at all. If you don’t know how to conduct these tests, find a researcher to help you run things like a t-test, correlation, and make sure to check the standard deviations for the scores.
I don’t know any radio stations using this technique that I can mention. I’m sure there are many that do use this approach, but I don’t know if they do it correctly. My guess from past experience is that they do not conduct any follow-up statistical tests and, therefore, don’t know anything about the quality of the data.
Online Sample Size
I was at the Conclave ‘Internet Research’ panel on Saturday. The issue of sample sizes was a topic of discussion. My question is: Is it REALLY true that an on-line sample size of 1,000 is better than a telephone sample of 400-500? – Need to know
Need: Your task is to read this answer 100 times.
There is absolutely no relationship between sample size and sample
quality.
None. Nada. Zip. Zilch. Goose egg. Zero. (Need others?) A sample can be poor
quality (invalid and unreliable) regardless of its size.
You and your career will end up in a huge pile of cow leavings if you conduct
on-line research merely because the sample sizes are bigger than a typical
telephone perceptual study.
I’ll say it one more time: Sample size has no relationship to sample quality.
This cannot be debated. Listen to me now and believe me later.
Online vs. Telephone Line
What's up Doc? Tell us the scoop on using the standard callout method versus online stuff like Bill Richards Rate the Music. I have seen the two side-by-side and see big variances in rank placement, a higher fatigue level on the online stuff, and other things that make me go hmmm. What's your take? - Anonymous
Anon: I want to say upfront that I have no vested interest in either
procedure since my company doesn’t conduct (and never will) either type of
test.
The problem I have with your question is that I don’t have any data to look
at. My only information is what you say . . . "big variances in rank,"
"higher fatigue," and "other things" with the online
procedure. I believe the "rank" statement is self-explanatory in that
I’m guessing that you’re saying that the same songs rank differently on both
methods.
However, I don’t know what you mean by "higher fatigue." How did you
determine this? I "gots ta know" in order for me to comment
Next, I don’t know what the "other things" are that make you go
"hmmm." Like what?
OK, but let’s assume that your comments are valid. What’s my take? My take
is that you are dealing with two separate and distinct testing procedures and by
virtue of this, you must expect different results. In addition, the two
procedures differ in reference to control of the experimental situation. Let me
explain.
In an auditorium music test, all of the respondents are in the same testing
location at the same time, and you know (within limits) that the people who are
they are the correct respondents. The researcher controls the situation and the
respondents rate the songs under the same conditions at the same time. In other
words, most intervening (confounding) variables are eliminated. You know, for
the most part, that the respondents are actually listening to the music and
scoring the songs they here. They aren’t eating dinner, playing with kids or
pets, or anything else.
This is not true with callout. Although there is a human being (supposedly) on
the other end of the phone, and it may be the correct respondent, you don’t
know what other activities the person is engaged in while rating the songs.
Callout introduces several uncontrolled intervening variables.
Now go to the online system. Not only do you have the same confounding
influences as in callout, but you also don’t know for sure who is on the
computer rating the hooks. An adult? A child? Who?
So my "take" is that there are many unknowns with online testing. What
is needed is an independent, carefully controlled scientific test to validate
the online procedure. The companies that conduct online testing may have data to
support the procedure, but I haven’t seen anything.
Technology doesn’t always mean "good." In some cases, technology can
be a giant leap backwards. I do not know if online testing is good or bad. But I
do know that if I were in your shoes (I’m assuming you’re a PD), I would
absolutely demand to see research (more than one study by the way) that verifies
the validity and reliability of the procedure. If you ask for such information
and get a study that you don’t understand, then it is your duty to have an
independent researcher or statistician look at the data for you.
One more thing . . . if I developed a new "high tech" music testing
procedure that involved respondents using e-mail or chat rooms (hey, it’s new
and high tech) would you automatically assume that the methodology was valid and
reliable? I hope not. If you said "yes," then I have beach front
property in Denver you might be interested in.
Open door
Can a door be open and closed simultaneously? – Anonymous
Anon: Is this a test? I think you must have read the book The Nature
of the Impossible by Davis and Park.
The answer is: Yes, a revolving door meets your qualifications.
Open-Ended Questions
I’m new to research. In a conference call the other day, our researcher asked if I wanted to have any "open-ends" in the questionnaire. I didn’t want to seem stupid, so I didn’t say anything. What’s an "open-end?" – John
John: There are two broad types of questions: open-ended and closed-ended. An
open-ended question allows a respondent to answer a question without any
prompting, such as, "What do you like most about the weekday morning show
on WAAA?"
In a closed-ended question (also known as a forced-choice question), a
respondent picks an answer from a pre-determined list, such as, "Which of
the following radio stations has the most entertaining weekday morning show . .
. WAAA, WBBB, or WCCC?"
Keep in mind, though, that you can also add "Don’t know/none" and
"Other" to the closed-ended question responses.
Open Notorious...Legal Question
Doc: I hope you can help me here. I have lived in my house for six years (a long time for a radio person). One thing I didn’t pay to much attention to is my property boundary lines. My neighbor, who has lived in his house for four years, has an elaborate garden on his yard. What I didn’t know is that his garden is actually a few feet on my property. (I found this out when I hired a surveyor to mark our lot.)
In a discussion with the neighbor, I mentioned that his garden was on my property and I wanted him off. He said that the land now belongs to him because of something called open, notorious, adverse, possession. He said that because he has had the garden on my lot for four years, that he could now claim ownership to it.
Is this right? Does my neighbor really own my property where his garden is? - Anonymous
Anon: Hey, a legal question I don’t have to send to my twin brother (a retired trial attorney.) I know this answer because I had to deal with it myself a few years ago. But, before I make my comments I need to say that I’m not an attorney and what I say should not be construed as a formal legal opinion. I suggest that you contact an attorney where you live who deals with land disputes. (Although you might be able to get the answer by calling the District Attorney in your city. It won’t cost you anything. Just ask how many years are required for the state’s Open, notorious, adverse, possession law.)
With that said, here are my comments. Open, notorious, adverse, possession is a real legal term. It refers to a person (just as your neighbor suggested) making claim to someone else’s property if the property has been used by the “taker” for a period of time.
The period of time of adverse possession in most states in the U.S. is 10 years. In other words, your neighbor would have to have his garden on your property for 10 years without your objection. Since he has lived in his house for only four years, I doubt whether he will meet the law in your state. Assuming this is correct, you have the right to tell him to take his garden off your property. (I searched the Internet and can’t find any state where four years is considered possession.)
There are many complicated legal cases involving open, notorious, adverse, possession. I set up a search for you for more information. Just click here: Hey moron! Get off my property!
Optimum Effective Scheduling
Dr. Wimmer: I am sure you have heard of the OES formula before. It's the equation that can supposedly tells you how many times you should play something to reach most of your listeners. CUME/AQH Persons = Turnover and Turnover x 3.29 = OES (the optimal number of spins per week to reach the average listener).
If I knew who came up with this, I would just ask them. I don't though, so the buck is passed to you! Do you know what the significance is of the number 3.29?
Also, do you think that the OEM equation is a reasonable way to determine how often to spin things? Thanks! - Mike in burr cold Michigan
Mike: No problem with passing the buck to me. First, you'll notice that I changed your question. You asked about the OEM formula, but it's actually OES, which stands for Optimum Effective Scheduling, and that's what I used to rename your question.
The OES formula was developed by Steve Marx, now chairman of The Center for Sales Strategy and included in a book co-authored with Pierre Bouvard (now president of Arbitron. I have known Pierre for many years, so I wrote to him and asked why they used the 3.29 multiplier. Here is what he said:
The formula developed by Steve Marx was designed to maximize "3+" reach, meaning folks that heard a message three or more times. A lot of memory, recall, and persuasion theory says hitting someone three or more times is key to advertising success.
So, Steve looked at various schedule sizes and what the "3+ reach curve" looked like. Steve observed there was a point where the curve growth flattened. He found that at 3.29 times the station turnover generated the optimal amount of 3+ reach (so called "effective reach"). Some have told us the "3.29 times turnover" is the minimum floor, and they observe amazing results running even more spots that 3.29 times turnover. Our response? You cannot argue with success!
You said, " If I knew who came up with this, I would just ask them." Now you know.
If you're interested, the book is Radio Advertising's Missing Ingredient: The Optimum Effective Scheduling System, by Steve Marx and Pierre Bouvard (1993). You might be able to get it from Amazon or another source, although I think it's out-of-print, so getting a copy might be a bit difficult. However, there is a lot of information on the Internet about the formula — click here.
Do I think the OES formula is a reasonable way to determine how often to spin music? Although the OES is designed primarily to determine the number of spots to run, I don't see any problem using the formula to determine spins. It all depends on your goals in reference to how many times your listeners will hear a song.
(I want to thank Pierre for his information. He has always been willing to help.)
Optimum Effective Scheduling (Spin Formula)
Doc: A while back I read an article about a formula using Arbitron numbers to determine how many times per week a record should be spun. It seems like it was cume divided by average persons x 3.88 but I can't remember if that is correct. Ever heard of it? Not even sure if it works, but was interested in running some numbers for the hell of it. Any help? Anon please! - Anonymous
Anon: I would first like to make a comment about your request for "Anon please." When anyone sends a question to me, the AllAccess.com server strips the sender's email address from the question. I have no idea who sends a question to me unless the person includes his/her name and/or email address. That's why there are no many Anons in my answer—people don't include their name or email address.
Therefore, I don't know who you are. On to your question . . .
What you are referring to is a formula called, Optimum Effective Scheduling (OES) that is discussed above. However, one of the best explanations I have seen for OES is located here. Keep in mind that OES is only an estimate. No one on the planet knows for sure how many exposures to a message or spins of a song are necessary to reach a certain percentage of people, persuade or convince them to do or try anything, or hear a song. But, an estimate in this case is better than nothing.
Owner Problem with Programming Suggestions
I’m a new PD at a CHR radio station in a medium-sized market. This is my first job as a PD and I love it. But I have one problem and I hope I can explain it clearly. My radio station is one of a handful of radio stations owned by a husband and his wife (there are still a few small owners!). By the way, the GM is new too. He started two weeks before I did.
Here’s the problem. About two weeks ago, I received a handwritten note from the wife (owner). In the letter, she had a list of 15 songs she wanted me to add to the playlist. I couldn’t believe it and showed the letter to my GM. The GM’s eyes rolled to the ceiling and he said something like, “Give me a break.” Well, I didn’t add the songs because they didn’t fit the radio station. I’m not being critical here, but the wife is about 150 years old and her suggestions for adds just don’t make any sense.
I just received another letter with more suggestion and at the end of the letter was the comment, “I haven’t heard any of the songs I suggested before. Please take care of this.”
I’m so confused with this situation that I don’t know what to do. Can you help? I know it’s not a research question, but this is driving me crazy! - Anonymous
Anon: You’ll notice that I did some heavy editing to your question. I don’t think I changed the meaning of your question, but let me know if I did.
OK. First, I can’t stop laughing. I really can’t. I’m not laughing at you, but rather the situation because I have seen this happen before. And I have seen it in large markets too, so don’t feel that it’s a symptom of small or medium-sized markets.
Here’s my suggestion, but before you do anything, go to your GM and explain your plan to see if he has any other suggestions. Make sure he signs off on the approach you want to use.
I think it would be best to call the wife-owner. Don’t write to her. Your conversation should be something like, “Hey, lady, life is short and I’m going to be on the planet longer than you will, so get off my case.” No. Don’t say that. Just teasing.
You should say something like: “Mrs. Owner. I received your letters with the suggestions for songs to add to the radio station. I always appreciate opinions from others because it helps me develop the best radio station I can. However, as you may know, we conduct music tests on a regular basis and before I add the songs you have suggested, or songs that anyone else suggests, I need to find out what our target listeners think about them. I know your goal is to have a successful radio station and the best way to do that is to ask our listeners.”
If you don’t conduct music tests, then exclude that and include the process you follow to add songs. The goal here is to explain to the owner that her opinions, while appreciated, cannot take precedence over the target audience. Let me know if you need additional help, and good luck.
By the way, you might want to ask the GM to call the wife. That’s up to you. (But I think you should do it to get the experience.)
Ownership Rules
What are the current laws regarding the numbers of or percentage of a market any one company can own? Does this include radio, TV and print media? - Anonymous
Anon: This USA Today article is a good explanation of the current ownership rules.
In case you need additional information, here is a link to the FCC’s Ownership Rules. There are two links at the bottom of that page to go to information about recent changes.
All Content © 2009 - Wimmer Research All Rights Reserved