Online Research Comment
Dr. Wimmer: Youíve got a great thing going with this column. Useful and entertainingówhat a concept! I just wanted to throw down some major kudos (not the candy bar either) to you regarding the online research debacle. I know that there are several of "us valid researcher folk" who are waiting in the wings with online product. However, until thereís an ideal way to verify the sampling criteria, thereís absolutely now way of knowing who's participating in the test. I think I'd have a hard time getting my clients to buy a music test in which respondents could just call in (from any state or province) and rate the music with no verification whatsoever. Thatís a scary thought (or in your terms, it just don't be right)! Yet, thatís what many PDís are subscribing to with the whole online thing. Use your heads folks! If I can join a chat room as a buxom blonde supermodel, then just imagine who' is in your research panel! Doc, you're a champion of common sense and a purveyor of truth. Keep up the great work! - Mike
Mike: Thanks for the comments about the column. I appreciate that.
The main problem I have had during my career in research is trying to get people to see what research can and canít do. Many people immediately subscribe to a research methodology or product just because itís new. They donít take the time to find out if the new method is right.
The problem is that research is a complicated process that involves many different thingsósampling, questionnaire/instrument design, data collection, data analysis, and interpretation. Because of this complexity, itís easy for someone to sell things to someone who has absolutely no research background. (I believe itís commonly known as "Pulling the wool over someoneís eyes.")
For too many years, research salespeople (they are not researchers) have developed methods that look and sound right, but are patently wrong. These salespeople (and Iím talking about some of the "big" names in research) sell these ideas to those who donít have any research experience at all. The salespeople throw around a few research terms to make things sound complicated and correct, and the unsuspecting PDs and GMs think they are correct. They take the words of these salespeople as gospel and just assume that what theyíre buying is scientifically correct. But in most cases, the methods are not correct. They are merely methods developed in the sales department, not research department.
Online research in all businesses has great potential, but as you say, we just donít know enough about who is answering the questions. We donít know enough about sampling. We donít know enough to compute a simple sampling error percentage.
Iím reminded of one group PD who said, when I asked him why he switched to 100% online research, "They [the research company] told me that the results are the same as a an auditorium music test and a telephone study." I asked him to see the data that compared the two methodologies. That was over a year ago. I havenít received a thing. I havenít received a thing because he doesnít have itóthe comparison data donít exist. He took the word of the research company. No data, no proof, no scientifically conducted comparisonójust their word. Quite frankly, in scientific terms, that sucks.
Would he (or anyone else) accept a medical prognosis from an electrician who learned medicine because of so many visits to the doctor? Would he (or anyone else) allow a bank teller to install a transmission in his vehicle because the teller had the job done so many times? I think not.
So why do radio people (TV people are just as bad) accept the word of non-researchers to provide data for their multi-million dollar operations? Why? I donít get it.
If such a study exists, bring it on! Send it to me. I will donate my time to review it. If the data show, and can be replicated by me, then I will change my mind. Thatís the advantage of following the scientific methodóitís self-correcting.
Merely saying, "My auditorium music tests scores are pretty close to my online music research scores." will not cut the mustard. I want to see valid statistical tests that compare the two methods (the same thing with comparing online research with telephone perceptual studies). Show me the data! Thatís all I ask.
If I can replicate findings, I will be happy to change my tune and Iíll be the first to admit that online research is "OK."
Sometimes the adage "You get what you pay for" doesnít fit. But in the case of online research, itís perfectly appropriate. If you have data to prove me wrong, I already said, "Bring it on." And letís get ready to rummmmmblllllllle!
(Warning: Donít write to me and say that your data "look" the same to you. Prove it statistically. I will not accept personal opinion.)
Online Research - Website
Hi Doc! I just had a meeting at our radio station, and out of the financial problems we have, somebody proposed we should care more about what our listeners actually wanted to hear. (Some donítóafter all itís public radio, they say, so the DJs and talents should not lose the freedom of what they do and play.)
Now since we donít have really much of a budget to do research, one idea I had was to let people go to our website and rate what they hear coming from our station. However, I donít know if thatís a good idea because Iíve yet to see this at any terrestrial radio station. There are some stations here who display whatís playing on their website, but none of them lets the web site visitors rate those songs (or other content). I have only seen that form of rating on some Internet radio stations, like Radio Paradise, or Flashback Radio. Do you think itís a good idea to include this on a radio stationís homepage, or should we rather not mess with this? And do you know any terrestrial radio stations that have a rating feature like that (for the current content) on their homepage? With greetings from Austria. - Kurt
Kurt: Hi to you again. You really keep me busy.
First, the comment that the ďDJs and talents should not lose the freedom of what they do and playĒ and therefore you shouldnít ask the listeners what they want makes no sense. In fact, itís a ridiculous approach to follow. The radio stationópublic or notóis for the listeners, not the DJs or other talent. The role of the on-air folks is to provide the listeners what they want, not what they (DJs and talent) think they need. Thatís goofy.
OK, on to your question.
You can include song ratings on your website if you understand/do/know these things:
You donít know who is rating your songs. The people could be children, pranksters, people from out of Austria, or other things. Youíll have data, but you wonít know the source.
If you do decide to rate songs on your website, then you should run several statistical tests to determine the validity and reliability of the data. You still wonít know the exact identity of the respondents, but a few statistical tests will help determine if you can use the data at all. If you donít know how to conduct these tests, find a researcher to help you run things like a t-test, correlation, and make sure to check the standard deviations for the scores.
I donít know any radio stations using this technique that I can mention. Iím sure there are many that do use this approach, but I donít know if they do it correctly. My guess from past experience is that they do not conduct any follow-up statistical tests and, therefore, donít know anything about the quality of the data.
Online Sample Size
I was at the Conclave ĎInternet Researchí panel on Saturday. The issue of sample sizes was a topic of discussion. My question is: Is it REALLY true that an on-line sample size of 1,000 is better than a telephone sample of 400-500? Ė Need to know
Need: Your task is to read this answer 100 times.
There is absolutely no relationship between sample size and sample quality. None. Nada. Zip. Zilch. Goose egg. Zero. (Need others?) A sample can be poor quality (invalid and unreliable) regardless of its size.
You and your career will end up in a huge pile of cow leavings if you conduct on-line research merely because the sample sizes are bigger than a typical telephone perceptual study.
Iíll say it one more time: Sample size has no relationship to sample quality. This cannot be debated. Listen to me now and believe me later.
Online vs. Telephone Line
What's up Doc? Tell us the scoop on using the standard callout method versus online stuff like Bill Richards Rate the Music. I have seen the two side-by-side and see big variances in rank placement, a higher fatigue level on the online stuff, and other things that make me go hmmm. What's your take? - Anonymous
Anon: I want to say upfront that I have no vested interest in either procedure since my company doesnít conduct (and never will) either type of test.
The problem I have with your question is that I donít have any data to look at. My only information is what you say . . . "big variances in rank," "higher fatigue," and "other things" with the online procedure. I believe the "rank" statement is self-explanatory in that Iím guessing that youíre saying that the same songs rank differently on both methods.
However, I donít know what you mean by "higher fatigue." How did you determine this? I "gots ta know" in order for me to comment
Next, I donít know what the "other things" are that make you go "hmmm." Like what?
OK, but letís assume that your comments are valid. Whatís my take? My take is that you are dealing with two separate and distinct testing procedures and by virtue of this, you must expect different results. In addition, the two procedures differ in reference to control of the experimental situation. Let me explain.
In an auditorium music test, all of the respondents are in the same testing location at the same time, and you know (within limits) that the people who are they are the correct respondents. The researcher controls the situation and the respondents rate the songs under the same conditions at the same time. In other words, most intervening (confounding) variables are eliminated. You know, for the most part, that the respondents are actually listening to the music and scoring the songs they here. They arenít eating dinner, playing with kids or pets, or anything else.
This is not true with callout. Although there is a human being (supposedly) on the other end of the phone, and it may be the correct respondent, you donít know what other activities the person is engaged in while rating the songs. Callout introduces several uncontrolled intervening variables.
Now go to the online system. Not only do you have the same confounding influences as in callout, but you also donít know for sure who is on the computer rating the hooks. An adult? A child? Who?
So my "take" is that there are many unknowns with online testing. What is needed is an independent, carefully controlled scientific test to validate the online procedure. The companies that conduct online testing may have data to support the procedure, but I havenít seen anything.
Technology doesnít always mean "good." In some cases, technology can be a giant leap backwards. I do not know if online testing is good or bad. But I do know that if I were in your shoes (Iím assuming youíre a PD), I would absolutely demand to see research (more than one study by the way) that verifies the validity and reliability of the procedure. If you ask for such information and get a study that you donít understand, then it is your duty to have an independent researcher or statistician look at the data for you.
One more thing . . . if I developed a new "high tech" music testing procedure that involved respondents using e-mail or chat rooms (hey, itís new and high tech) would you automatically assume that the methodology was valid and reliable? I hope not. If you said "yes," then I have beach front property in Denver you might be interested in.
Can a door be open and closed simultaneously? Ė Anonymous
Anon: Is this a test? I think you must have read the book The Nature of the Impossible by Davis and Park.
The answer is: Yes, a revolving door meets your qualifications.
Iím new to research. In a conference call the other day, our researcher asked if I wanted to have any "open-ends" in the questionnaire. I didnít want to seem stupid, so I didnít say anything. Whatís an "open-end?" Ė John
John: There are two broad types of questions: open-ended and closed-ended. An open-ended question allows a respondent to answer a question without any prompting, such as, "What do you like most about the weekday morning show on WAAA?"
In a closed-ended question (also known as a forced-choice question), a respondent picks an answer from a pre-determined list, such as, "Which of the following radio stations has the most entertaining weekday morning show . . . WAAA, WBBB, or WCCC?"
Keep in mind, though, that you can also add "Donít know/none" and "Other" to the closed-ended question responses.
Open Notorious...Legal Question
Doc: I hope you can help me here. I have lived in my house for six years (a long time for a radio person). One thing I didnít pay to much attention to is my property boundary lines. My neighbor, who has lived in his house for four years, has an elaborate garden on his yard. What I didnít know is that his garden is actually a few feet on my property. (I found this out when I hired a surveyor to mark our lot.)
In a discussion with the neighbor, I mentioned that his garden was on my property and I wanted him off. He said that the land now belongs to him because of something called open, notorious, adverse, possession. He said that because he has had the garden on my lot for four years, that he could now claim ownership to it.
Is this right? Does my neighbor really own my property where his garden is? - Anonymous
Anon: Hey, a legal question I donít have to send to my twin brother (a retired trial attorney.) I know this answer because I had to deal with it myself a few years ago. But, before I make my comments I need to say that Iím not an attorney and what I say should not be construed as a formal legal opinion. I suggest that you contact an attorney where you live who deals with land disputes. (Although you might be able to get the answer by calling the District Attorney in your city. It wonít cost you anything. Just ask how many years are required for the stateís Open, notorious, adverse, possession law.)
With that said, here are my comments. Open, notorious, adverse, possession is a real legal term. It refers to a person (just as your neighbor suggested) making claim to someone elseís property if the property has been used by the ďtakerĒ for a period of time.
The period of time of adverse possession in most states in the U.S. is 10 years. In other words, your neighbor would have to have his garden on your property for 10 years without your objection. Since he has lived in his house for only four years, I doubt whether he will meet the law in your state. Assuming this is correct, you have the right to tell him to take his garden off your property. (I searched the Internet and canít find any state where four years is considered possession.)
There are many complicated legal cases involving open, notorious, adverse, possession. I set up a search for you for more information. Just click here: Hey moron! Get off my property!
Optimum Effective Scheduling
Dr. Wimmer: I am sure you have heard of the OES formula before. It's the equation that can supposedly tells you how many times you should play something to reach most of your listeners. CUME/AQH Persons = Turnover and Turnover x 3.29 = OES (the optimal number of spins per week to reach the average listener).
If I knew who came up with this, I would just ask them. I don't though, so the buck is passed to you! Do you know what the significance is of the number 3.29?
Also, do you think that the OEM equation is a reasonable way to determine how often to spin things? Thanks! - Mike in burr cold Michigan
Mike: No problem with passing the buck to me. First, you'll notice that I changed your question. You asked about the OEM formula, but it's actually OES, which stands for Optimum Effective Scheduling, and that's what I used to rename your question.
The OES formula was developed by Steve Marx, now chairman of The Center for Sales Strategy and included in a book co-authored with Pierre Bouvard (Executive Vice President, Cross-Platform Services at Arbitron). I have known Pierre for many years, so I wrote to him and asked why they used the 3.29 multiplier. Here is what he said:
The formula developed by Steve Marx was designed to maximize "3+" reach, meaning folks that heard a message three or more times. A lot of memory, recall, and persuasion theory says hitting someone three or more times is key to advertising success.
So, Steve looked at various schedule sizes and what the "3+ reach curve" looked like. Steve observed there was a point where the curve growth flattened. He found that at 3.29 times the station turnover generated the optimal amount of 3+ reach (so called "effective reach"). Some have told us the "3.29 times turnover" is the minimum floor, and they observe amazing results running even more spots that 3.29 times turnover. Our response? You cannot argue with success!
You said, " If I knew who came up with this, I would just ask them." Now you know.
If you're interested, the book is Radio Advertising's Missing Ingredient: The Optimum Effective Scheduling System, by Steve Marx and Pierre Bouvard (1993). You might be able to get it from Amazon or another source, although I think it's out-of-print, so getting a copy might be a bit difficult. However, there is a lot of information on the Internet about the formula ó click here.
Do I think the OES formula is a reasonable way to determine how often to spin music? Although the OES is designed primarily to determine the number of spots to run, I don't see any problem using the formula to determine spins. It all depends on your goals in reference to how many times your listeners will hear a song.
(I want to thank Pierre for his information. He has always been willing to help.)
Optimum Effective Scheduling (Spin Formula)
Doc: A while back I read an article about a formula using Arbitron numbers to determine how many times per week a record should be spun. It seems like it was cume divided by average persons x 3.88 but I can't remember if that is correct. Ever heard of it? Not even sure if it works, but was interested in running some numbers for the hell of it. Any help? Anon please! - Anonymous
Anon: I would first like to make a comment about your request for "Anon please." When anyone sends a question to me, the AllAccess.com server strips the sender's email address from the question. I have no idea who sends a question to me unless the person includes his/her name and/or email address. That's why there are no many Anons in my answerópeople don't include their name or email address.
Therefore, I don't know who you are. On to your question . . .
What you are referring to is a formula called, Optimum Effective Scheduling (OES) that is discussed above. However, one of the best explanations I have seen for OES is located here. Keep in mind that OES is only an estimate. No one on the planet knows for sure how many exposures to a message or spins of a song are necessary to reach a certain percentage of people, persuade or convince them to do or try anything, or hear a song. But, an estimate in this case is better than nothing.
Owner Problem with Programming Suggestions
Iím a new PD at a CHR radio station in a medium-sized market. This is my first job as a PD and I love it. But I have one problem and I hope I can explain it clearly. My radio station is one of a handful of radio stations owned by a husband and his wife (there are still a few small owners!). By the way, the GM is new too. He started two weeks before I did.
Hereís the problem. About two weeks ago, I received a handwritten note from the wife (owner). In the letter, she had a list of 15 songs she wanted me to add to the playlist. I couldnít believe it and showed the letter to my GM. The GMís eyes rolled to the ceiling and he said something like, ďGive me a break.Ē Well, I didnít add the songs because they didnít fit the radio station. Iím not being critical here, but the wife is about 150 years old and her suggestions for adds just donít make any sense.
I just received another letter with more suggestion and at the end of the letter was the comment, ďI havenít heard any of the songs I suggested before. Please take care of this.Ē
Iím so confused with this situation that I donít know what to do. Can you help? I know itís not a research question, but this is driving me crazy! - Anonymous
Anon: Youíll notice that I did some heavy editing to your question. I donít think I changed the meaning of your question, but let me know if I did.
OK. First, I canít stop laughing. I really canít. Iím not laughing at you, but rather the situation because I have seen this happen before. And I have seen it in large markets too, so donít feel that itís a symptom of small or medium-sized markets.
Hereís my suggestion, but before you do anything, go to your GM and explain your plan to see if he has any other suggestions. Make sure he signs off on the approach you want to use.
I think it would be best to call the wife-owner. Donít write to her. Your conversation should be something like, ďHey, lady, life is short and Iím going to be on the planet longer than you will, so get off my case.Ē No. Donít say that. Just teasing.
You should say something like: ďMrs. Owner. I received your letters with the suggestions for songs to add to the radio station. I always appreciate opinions from others because it helps me develop the best radio station I can. However, as you may know, we conduct music tests on a regular basis and before I add the songs you have suggested, or songs that anyone else suggests, I need to find out what our target listeners think about them. I know your goal is to have a successful radio station and the best way to do that is to ask our listeners.Ē
If you donít conduct music tests, then exclude that and include the process you follow to add songs. The goal here is to explain to the owner that her opinions, while appreciated, cannot take precedence over the target audience. Let me know if you need additional help, and good luck.
By the way, you might want to ask the GM to call the wife. Thatís up to you. (But I think you should do it to get the experience.)
What are the current laws regarding the numbers of or percentage of a market any one company can own? Does this include radio, TV and print media? - Anonymous
Anon: This USA Today article is a good explanation of the current ownership rules.
In case you need additional information, here is a link to the FCCís Ownership Rules. There are two links at the bottom of that page to go to information about recent changes.
All Content © 2015 - Wimmer Research All Rights Reserved