Callout Questions Continued

 

Rating Scale - Type

I apologize in advance if this is a dumb set of questions.  Some stations view their callout results using the %Positive, %Neutral, and % Negative numbers.  Others view results through the average score.  So…

- Anonymous

 

Anon:  As I have said many times, there are no dumb questions.  Don’t worry about that.

Sample

We've just started conducting call out research on our current songs. Our target audience is females, 25-49. My question is, can a listener screen out of the research by listening too many hours? Also, can a listener screen out by listening too few hours? How would you determine how many hours is too many and how many is too few? Our TSL is about 9 hours per week. Thanks! Look forward to your response! - T. Greene


T: You can design the screener any way you want to. How do you determine the minimum and maximum number of listening hours? Here's an approach:


First, remember that you are collecting ratings of the currents you play. Do you want these ratings to come from people who listen to your radio station only occasionally, or from people who listen frequently?


I'm going to take a guess that you'd rather have answers from people who listen frequently. So what does "frequently" mean? Well, there is no real answer to this question, but one way is to define it as "at least one hour per weekday" (forget the weekends). The one hour requirement will eliminate the people who tune in only once in a while when they're scanning the dial. (You can set the minimum to any amount of time you would like, but remember if you go too high, you'll be interviewing only your fans or P1s—but that may be your intent, I don't know.)


Now, about a maximum number of hours . . . I don't believe that a person can listen to your radio station "too much." I believe you would be safe to write your screener to address only a minimum amount of listening. If the person meets the minimum, they're in.

Sample Accuracy

We're beginning an in-house callout project to research our currents. My main concern is with the sample of people we plan to call. We've had several people who have agreed to participate. They originally screened into our auditorium test but couldn't participate for some reason, so we know they are P1 listeners. However, I'm not comfortable calling this same list of people every week (it's about 75 people). What are your suggestions for making sure we get an accurate sample for our music callout? - T. Greene


T: First, it's good that you plan to use the same people in your callout. Your results will be more stable than if you used a new sample each week. Plus, you'll also save money and many headaches.


I have two suggestions for you. However, both require that you have a pre-qualified sample of about 200 people. It's best to have respondents screened in advance than to try to "cold call" people each week.


Suggestion 1: This procedure requires some bookkeeping to keep things straight. It also follows a procedure where a person participates for a total of 4 weeks and then is put on hold for a while.


First, divide your 75 people into four somewhat equal groups and assign each person in each group a number (1 through 4). Therefore, all people in Group 1 are assigned a "1," all people in Group 2 a "2," and so on. (Make sure that each group is a "stand alone" group in reference to your quotas. That is, each group should have the same percentage of ages splits, listening requirements and so on.)


Conduct your first callout with 75 people. In week 2, replace Group 1 with new people; in week 3, replace Group 2; week 4 replace Group 3; and week 5 replace Group 4.


As you can see, the longest participation is 4 weeks from Group 4. You can bring back Groups 1, 2, and 3 until they also have participated for 4 weeks. Whenever a person has participated for four weeks, put them aside for a month or two and then bring them back later (if they still qualify in your screener).


However, keep in mind that you must be prepared to replace more people than those who are in each group. You won't always be able to contact all of the people in a group (for example, all of the people in a group may not be home).


Also, if you want people to participate for more than 4 weeks, roll your sample over in a smaller percentage, such as 20% new people each week.


Suggestion 2: This procedure requires less bookkeeping.


Start with a database of about 200-300 pre-qualified respondents. Each week, randomly select 75 people. Be sure that you watch your quotas if you have them (such as age or sex).


Keep track of each person's participation and limit each one to 4 participations (or whatever number you select).


If you don't like either of these approaches, make up your own. However, as I mentioned, it is best to pre-qualify respondents for your callout—cold calling each week will kill you. The best way to have a database of respondents is to include pre-recruiting as part of your callout each week. After your callout is done for the week, have your interviewers get on the phone to build your database.

Sample Size

What would you recommend as the minimum number of responses for weekly music callout?  How do you feel about 45 per week versus 50 or 60 per week with a rolling two-week average? - Anonymous

 

Anon:  Callout research is a univariate (one dependent variable) statistical procedure.  However, callout is also a “repeated measures” design, which means that respondents are doing the same thing repeatedly (rating several songs using the same rating scale), which increases the reliability of the methodology.

 

Now, if you consult the advice of univariate statisticians, most will say that a sample of 30-35 respondents is adequate to collect reliable (repeatable) information.  This is further supported by the repeated measures design.

 

So what does that mean for you?  If your callout involves only one small cell, say females 25-34, then you can probably get away with a sample of 35 respondents.  I say, “probably get away with,” because you really need to analyze your data from one week to the next.  If the results are consistent, within statistical limits, then you know 35 respondents is good.  If you don’t know how to do this, ask for help from someone who knows statistics.

 

The reason most callout and auditorium tests use samples of 75-100 is that the data are broken out into smaller cells, such as 25-29 and 30-35.  If you use 35 respondents, you must look only at one cell—you can’t break the 35 respondents into two or more smaller cells.

 Sample - Willingness to Participate

Hi again. I told you I would probably follow up based upon your answer to "which music research is better, callout or online?" You basically answered that given one or the other, go with callout. But, if you have the luxury of both then average them together.


I still question the validity of callout in today's society. How willing will participants be to take a 5-15 minute phone survey at a time when telemarketing is at an all-time high? Wouldn't you prefer someone who has sat down on their own time to consciously fill out a survey they want to take about a radio station they want to help?


And, if the old 80/20 rule is correct, wouldn't you want your research based on an active core audience such as what you would get online than a passive audience that may pass your screener but rarely listen to the station. - Anonymous


Anon: The problem of getting cooperation from people to participate in callout is the same problem facing all research. Telemarketers have ruined the legitimate research business. This means that you have to make a lot of calls to complete a study. There are two options here: stop conducting research or develop new procedures. Most of the researchers I know are concentrating on the second option.


I said that I prefer callout to online testing because there is more control over callout. However, given the choice, I wouldn't do callout. I'd prefer an auditorium setting because of the control over the situation. But if you are determined to use callout, at least interpret the trends rather than an individual report.


Finally, in reference to your 80/20 rule, it depends on the goal of your test—test your core or test your cume to help convert them to your core. Finally, if you want your core audience in your callout, then change the screener. You aren't forced to interview only occasional cumers (as you suggest).

Screener & Artists and Titles

Hi Doc. I asked a few of these questions a couple weeks ago and you suggested that I ask you again when I have some options from which you can choose. So, here we go:

 

1) Callout Screener. In situations where specific competitive stations are not mentioned during the screening process, a music screener may include or exclude potential respondents. A montage presents relevant songs that may or may not appeal to the participant and allows songs by non-recognizable artists to be included. Simply listing artists may skew opinions based on the overall popularity of an artist and precludes the inclusion of "one-hit wonders" who name may not be familiar. Which do you think is best?

 

2) In the callout survey, should the title and artist of songs be listed? Doing so may help speed along familiarity or burn, but it might also taint the results based on that information alone and not the song itself. Which do you prefer? - Anonymous

 

Anon: I have no opinions, but I will tell you what controlled tests of your questions show:

 

1.  By "montage," I assume you mean playing a 2 or 3-hook sample of music to use as your screening question. Here's what I know . . . If you play hooks to screen people into your research, whether it's for callout or any other research project, you can lose up to 65% of your P1s. The reason is that you may include one or more songs that people don't like. Several replicated (repeated) studies have shown that recruiting via a list of artists that are read to the respondents is much more effective in recruiting.

 

You say that "simply listing artists may skew opinions…and precludes the inclusion of one hit wonders." The list of artists that are read should represent your radio station or a specific type of music. "One-hit wonders" generally don't qualify and, therefore, should not be included in the first place.

 

The purpose of reading a list of artists is to represent a specific type of music or a specific type of radio station. The artists should be immediately recognizable and should collectively provide a "feel" of music.

 

2.  Once again, replicated studies show that providing artist and title information for callout or for an auditorium music test will have an effect on your scores. The reason is that some people…the number varies…may rate an unfamiliar song only by the artist's name (the song title doesn't have an equal effect). Callout and auditorium tests are intended to collect listeners' ratings of songs, not artists' names. The studies I have conducted show that playing the hook alone is the best procedure—leave out the name and title.

When to Ask the Listeners

I think I have an interesting question for you here.  I just read an article by Guy Zapoleon, which you can find by clicking here.  There are several points discussed in this article, however, there is one where I am wondering if they really follow (or suggest) the correct approach there.

 

The point is about when to call and ask your listeners about a song.  This article suggests that you have to play a song on your stations many times before you are able to get a real scientifically valid rating from your listeners.  If you ask them early on when the song’s not really familiar, you might get a rating that consists only of respondents listening to another radio station playing this song more often or for a longer time.  This implies that they might have a different music taste than the average listener of your own station, and thus their rating wouldn’t be correct.

 

However, this would mean that you’d have to play a song often on your station to make it familiar among your own listeners before you ask them if they like it. But they may NOT like it, and if you do this for several songs at once (playing them in high rotation without being sure if they like it), the listeners may tune away from the station because you play “too much crap.”

 

I’d think the better approach would be to ask your listeners and take the rating of those who already know the song, even if the sample may be somewhat skewed here. That way, you wouldn’t have to waste that much air time for the song if it’s not really liked by the listeners. If it’s over-rated by the group of people already know it, there is no harm done because following the other approach, you would have to rotate the song heavily as well. If it’s underrated, you could still correct the rotation if it receives more requests than expected after some time.

 

Who do you think is right here, Zapoleon or me? - Kurt

 

Kurt:  I read Guy’s article, and since I have known him for several years, I thought I’d ask him to address your question.  I’d like to thank him for his answer, which is:

 

“Roger, I believe Kurt is on to something.  My intention is not to use non-P1s who don’t listen enough to your station to rate a song you don’t play. I had some bad experiences with looking at response for songs my Top 40/Hot AC clients weren’t playing, but were getting big scores on the Alternative station or the R&B and Hip Hop station.  So we could be judging an Urban song that does great with Hip Hop listeners or a Rock song that appeals only to Alternative listeners.  That is misleading information.  Or, the converse is where we are trying to judge a Pop song that a Rhythm Top 40 is playing and we get weak scores because their audience isn’t into it.

 

I do support Kurt’s idea of looking at a healthy sample of P1s to extrapolate potential on a song that you may not be playing, but generally prefer for a PD to network with other stations similar to his for success on new songs that your station might not be playing.”

 

My response:  Your main question is whether to play new music a lot and then test it, or test the music while it’s in a normal rotation on your radio station while you know there will be a high unfamiliarity percentage.  So, I’ll address that.

Zaploleon and Callout

Guy Zapoleon makes broad assumptions about callout (in “Consultant Tips”) and the respondents who participate based on what seems to be just his opinion and not any research to back it up.  He says:

 

“Now, because of the onslaught of telemarketing and other calls to the household -- as well as the development of caller ID -- it is nearly impossible to reach your average listener.  Instead, often-times traditional callout reaches the person who has time to talk to anyone who calls, and that is not average these days!

 

Today most callout participants are light and medium light listeners and the methodology does not capture enough medium and heavy listeners.”

 

Do you agree with these conclusions since you actually do research and Guy is a consultant who has a vested interest in online research?

 

Keep up the good work. - Anonymous

 

Anon:  I’ll try to keep up the good work.  Thanks.  If you haven’t read the article in “Consultant Tips,” then click here.

 

OK.  What do I think about the article?  In any discussion of research (and probably most other things), it’s important to address both the advantages and disadvantages of a methodology.  This is important because there isn’t one perfect methodology—there isn’t one methodology that is a panacea for answering all research questions.

 

While there are problems with telephone callout, the evidence that I have seen is that if the research is conducted correctly, telephone callout continues to be a valid and reliable way to collect information.  I know there are problems with acceptance rates (people who agree to participate), but that has been true for as long as callout has been conducted (since about 1985) and it will continue to be a problem in the future.

 

Good callout research requires constant attention to the sample.  If you, or anyone else who conducts callout, is willing to put in the time to ensure that the sample is good, then there is no problem with the method.  In other words, I don’t see any evidence to suggest that callout is “dead.”

 

Now, about online research…As I said, it’s important to consider both the advantages and disadvantages of a methodology.  Sure, online music research is fast and cheap.  But there is still the major problem of control over the research situation.  Control of the research situation is important in research because control eliminates or reduces the affects of confounding variables that may create error.  For example, control over the research situation is great in an auditorium music test—we know that respondents are exposed to the hooks they rate because they are physically in front of the researcher.  Control over the situation isn’t as good in callout because the respondents may take the phone away from their ear when hooks play, but at least there is an interviewer who can (somewhat) document that the respondent was exposed to the hooks.

 

This is not true in online research because there is no control—none—over the research situation.  We don’t know if the respondents actually heard the hooks and we don’t know who entered the ratings for the hooks.  The ratings could have been entered by anyone in the household—a young child, a 95 year-old grandparent, or even an employee of one of the competitors.

 

I totally support the pursuit of new research methodologies because new methods (or refinements of old methods) may collect better information.  I also support the idea of using the Internet because: (1) It is the best information source ever invented; and (2) It follows Occam’s Razor (the easiest approach is always the best).

 

What I don’t support is a blind acceptance of the Internet as THE answer to data collection.  I also don’t support the assumption that the Internet is free of problems.  New methodologies, whether they involve the Internet or not, must be carefully tested and retested.  I know many GMs, PDs, OMs, and others, who use the Internet for music testing and perceptual information.  When I ask these people if they know who actually responded to their Internet research, they look at me as a deer looks in headlights.  These people are using Internet-collected information to make decisions about their radio station(s) and they don’t know who answered the questions.  Gag me with a beaker.

 

I believe the Internet has great potential as a data collection tool.  However, I also believe that the people who use the Internet for data collection should be qualified researchers.

 

A Arbitron B C D E F G H I J K-L M
Music-Callout Music-Auditorium N O P Q R S T U V W-X-Z Home

All Content © 2012 - Wimmer Research   All Rights Reserved