Rating Scales

What is the best type of rating scale to use in a research study—5 points, 7 points, or 10 points, or something else? – Mike

Mike: Psychometric statisticians (peeps who develop theories and techniques concerning human behavioral measurement) have been investigating your question for a long time—back to the early 1950s. Without boring you to death, here is a summary of what these statisticians have found:

The advantage is always to use more points (sometimes called "steps") rather than fewer points because the amount of discriminating information increases with the number of points (up to about 20 points). The reliability (stability) of measurements increases very quickly from 2 to 10 points, with a leveling at about 7 points. After 11 points, there is little gain in reliability.

Should a scale use an even number of points (where there is no "real" middle score) or an odd number of points (where there is a "real" middle score)? There is no universal agreement here, but research shows that people who rate something as "neutral" on an odd-numbered scale do, in fact, show differences in ratings when they "re-rate" the same items on an even-numbered scale. In other words, people who rate something as "neutral" on an odd-numbered scale do show differences (variance) when asked to use an even-numbered scale.

So what does this mean? Considering all information, it is best to use a scale of 1-10. The reason a 10-point scale works so well is that people of all ages and backgrounds are very accustomed to using it and seeing it (such as "he/she is a 10," and the ratings in the Olympics and other sports events). My vote is for a 10-point scale.

Ratings and Variables

Doc: In your experience, with respect to radio listening data, what are the variables that have shown the highest correlation to: (1) Cume; (2) TSL; and (3) Turnover?  Thanks for your time. - Anonymous


Anon: Each market is somewhat different in reference to what drives a high cume and TSL and a low turnover.  That's why it's necessary to conduct a research study to find out what your listeners want.  It's not a good idea to use the research from another market because there will always be differences.


With that said, there are several variables that tend to collect at the top of the list in terms of what's important to listeners.  Once you find out what things (variables) your listeners consider important, you can concentrate on those items and have a direct affect on cume, TSL, and turnover.


Some of the variables that increase cume and TSL and decrease turnover include, but are not limited to:

  1. Good music or Talk shows (both involve several characteristics).

  2. Information elements such as news, weather, and traffic.

  3. Good jocks or Talk Show hosts.

  4. Telling the artists' names and titles of songs.

  5. Commercials that aren't obnoxious.

  6. Uniqueness (not the same thing every day).

  7. Programming aimed TO the listeners, not AT them.  (Programming that gives listeners the feeling that they are "part of" the radio station, not someone on the outside just listening in on people having a good time.)

The key to attracting listeners (Cume), keeping them listening for a long time (TSL), and reducing the number of times they leave the radio station (Turnover), is to find out what things are important to them.  If you give them those things (and tell them that you did), then you will have a high cume and TSL, and a low turnover.

Ratings (vs. Share)

Doc:  Can a radio or TV station's rating number ever be larger than its share? - Anonymous


Anon:  No.  A rating is the percentage of people (radio) or households (TV) listening or watching to a radio or TV station of the Total population or universe.  A share is the percentage of people (radio) or households (TV) listening or watching to a radio or TV station of only those who are using radio (PUR — Persons Using Radio) or watching TV (HUT — Homes Using Television).


Now, theoretically, a radio or TV station's rating might equal its share if PUR or HUT were 100%, but that would never happen.  Let's look at a radio example — Let's say there are 1,000 people in a market, and ALL 1,000 were listening to the radio.  If 100 people were listening to WAAA, the radio station's rating and share would be 10.  However, as I said, I can't imagine that this situation would ever happen.

Ratings - Projecting for a New Format

How many sacrificial chickens do the Arbitron deities want in order to produce ratings projections for a new format?


Seriously though, realizing that it is essentially anybody's guess as to what kind of ratings a new station will produce, is there any sound method to arrive at a decent “best guess” that will pass the BS test?  I can think of various ways that involve looking at where we think that station's target listeners are currently camping, but I'm having a difficult time settling on one way I'm comfortable defending without saying, "well...ummm... it's better than voodoo."  I appreciate your input. - Al


Al:  This is a tough question for me to answer because it took me a few years to develop a valid and reliable way to predict Arbitron numbers for a new radio station.  Since my livelihood comes from conducting research (and using the methodology I developed), I’m not really ecstatic about giving it away for free…but I’ll do the best I can.


First, I know you have research experience, so I’ll answer your question using some technical terms that other readers may not understand (I apologize to anyone who finds this answer too complicated.)


Second, I’ll say that you’re on the right track when you say that you need to look at where the radio station’s target audience is “currently camping.”  However, you can’t look only at Arbitron numbers because those numbers represent what was, not what is or what might be.


In light of the amount of work I did developing the methodology, here is what I’ll say…


You must conduct your own research to collect information to use in developing a valid and reliable ratings prediction methodology.  Arbitron numbers can provide a minor indication, but there isn’t enough information to use in a predictive model.  I have tried dozens of formulas, but there just isn’t enough information in an Arbitron book to help.


In your research, you must collect: (1) Cume and P1 (listen most often) information; (2) Amount of time listening to the P1 radio station; (3) Radio station ratings (overall performance ratings); (4) Format ratings of radio stations already in the market as well as formats that may be available (format holes); and (5) Radio station format ownership data.


After you collect the data for these five points (I suggest 1-10 rating scales), you can produce a linear combination to develop a “Ratings Potential Score,” or whatever you want to call it.  If you have specific knowledge of the market, you may want to weight one or more of the variables, which obviously produces a weight linear combination.


The final step is to convert the linear combination scores to Z-scores because you’ll get a better look at the data and you’ll reduce the influence of outliers.


I apologize if my answer is somewhat obtuse, but you sound like a smart person and should be able to figure out a good way to predict Arbitron numbers.

Ratings Service Reliability

Hi, Dr. Wimmer:  This is a tough one...


I live in Spain, and we don't have Arbitron in our market. The research company used for ratings by most of the ad agencies and media brokers is called EGM (Estudio General de Medios or General Media Research). Let's say this is the “official” ratings service.


This ratings service is giving us some results that present radical variations from one sweep to another without significant format changes in the market.  For example, (the results are given in number of people who listened to radio yesterday):


One radio station in a market with a population of 7,500,000 grew from 140,000 to 309,000 listeners in a six months period (growth of 121%).


In the second market of the country, with a population of 2,500,000, one station grew from 20,000 to 139,000 (586%), and other station in that market, from a completely different format, decreased it's audience in 96,000 in the same period of time (-39%).


This happens in many markets, and it has been happening from study to study.  As I said, there were no significant format changes in those markets.  Do you think this is logical?


For your information: EGM uses a methodology of recall, with deep personal face-to-face interviews, asking for all media habits (Radio, television, newspapers, billboards, internet, magazines.)  Each interview takes about 45 minutes to complete.


It's made during a month, and concerning radio, they especially ask about “Last Day Listening” habits. First, they ask if you listen to radio, then, if you do, at what times of the day, and finally they ask you about every station you listened to yesterday at those particular time slots.


Thanks in advance for your help.  I am a big fan of yours. - Anonymous


Anon:  A few things before I get to your questions:

  1. Thanks for being a fan.  I now have one and will alert my wife.

  2. I eliminated some of the information from your question because I didn’t need it for your answer.

  3. I didn’t know anything about EGM, so I checked a few sources on the Internet.  I didn’t find much, so I’ll go with your information.

Although I don’t know much about EGM, the key to the problem is EGM’s methodology—door-to-door interviews with “day-after” recall.  That’s really all I need to provide some perspective.  I can’t guarantee that I’m 100% correct with my comments, but I think a personal example of what happened in the United States should be enough to solve this situation.


In the 1960s and early 1970s, there was a ratings service in the U.S. called Pulse.  I was in college in the early 1970s, and one day while visiting a friend, I answered the doorbell and a young person introduced himself as a “survey taker” for Pulse Radio.  Here is roughly how the discussion went:

(He then asked if I listened during different dayparts and then asked about several other radio stations.  When he left, he had recorded that I listened to several radio stations at a variety of times.  All of it was made up—100%.  Another problem is that he never asked if I lived in the area or if I was in my own home.  He didn’t know I was a visitor at that house.  The interview lasted about 15 minutes.)


Now I’m not claiming that EGM does the same thing with their interviews, but here are some of the other problems with door-to-door interviews and day-after recall research.

  1. Most people who conduct door-to-door interviews are not supervised.  This means that many are often very flexible when it comes to their responsibilities.

  2. Most people who conduct door-to-door interviews are paid according to their completed interviews.  An attempted interview doesn’t count, so they do all they can to complete as many interviews as possible.  This pressure to get a “complete” often creates situations like the one I faced with Pulse—the answers are made up by the interviewer.

  3. Door-to-door sampling isn’t very good because it’s extremely difficult (even in Spain) to get a decent sample.  Too many people refuse to participate and the sample is usually significantly skewed to a particular type of person.

  4. Average people are notoriously bad in remembering what they did yesterday.  You can verify this by asking your friends what they had for lunch yesterday.  Most people are involved in so many things that they don’t pay attention to which radio stations they listen to and they can’t remember much about them when they’re asked.  Remembering radio stations listened to yesterday is difficult enough, but it becomes even less valid and reliable when they are asked how much time they spent listening to these radio stations.  Day-after-recall is not a good data collection method because people can’t remember things very well.

  5. Door-to-door (or personal) interviews create a situation in research known as “prestige bias,” where respondents will make up answers to questions so they won’t look stupid, dumb, unaware, or any other word you want to use.

That’s enough for a start.  Now, EGM may have good controls over interviewers and that may not be one of the company’s problems.  However, EGM can’t control the memories of the respondents they contact, and that may be the only problem (although I really don’t think so).


People’s habits, including radio listening, don’t usually change much unless there is some type of dramatic situation (a major tragedy, weather problems, and so on).  The audience size swings you give in your examples don’t seem logical unless something dramatic happened in the market.  I don’t know because I’m not there.


In summary, without knowing the entire story, my guess is that the numbers aren’t real.  Instead, the numbers appear to be an artifact (characteristic) of the methodology.

Ratings Terms

If you were to explain how ratings for radio work to a person who isn't in the business, how would you explain what TSL, Cume, etc. mean? - Anonymous


Anon:  Here are definitions for the two items you mention.  I’m not sure what you mean by “etc.”


Cume: The number of people estimated to listen to a radio station during a given time period.


TSL:  The average amount of time the listeners spend listening to a radio station in given time period.


In reference to your “etc.” definitions, this search will help you find definitions for Ratings Terminology.  Another source is Arbitron’s Purple Book that explains just about everything you want to know about radio ratings.  There is a good glossary of terms in this PDF downloadable book.

Razor - Cleaning in Alcohol

I know this isn’t a research question, but as you say, “I gots ta know.”


A friend of mine said that he read somewhere that if you soak a razor (a typical 2 or 3 blade razor) in rubbing alcohol it will stay sharper longer.  Do you know anything about this? - Dave


Dave:  It’s interesting that you mention this topic since my twin brother mentioned the same thing to me a few months ago.  He can’t remember where he read it, but the information is the same—soaking a razor in rubbing alcohol extends the life of the blade(s).


I have been doing this for several weeks now and it seems to work.  There is obviously something to it because the Braun Electric Razor has an automatic cleaning feature that involves immersing the razor in alcohol.


Here are a few things I learned about this procedure (I’m referring now to a safety razor with disposable blades, not an electric razor):


1.  Like many people, I don’t enjoy the smell of denatured (rubbing) alcohol, so I use “wintergreen” alcohol (it’s green) that doesn’t smell like a doctor’s office.  You can find wintergreen alcohol at almost any pharmacy or grocery store.


2.  Use the plastic top from your shaving cream for a convenient soaking cup.


3.  After you shave, clean your razor, and place it in the cup with an amount of alcohol that covers the blades.  Leave it in there until the next time you shave (rinse the razor before you use it and discard the alcohol).


My informal test shows that the life of a disposable blade is extended by two or three times (the blades stay sharp for several weeks).  It’s a neat little gimmick that will save you money.  By the way, it also works for my wife’s disposable razors.  She’s convinced too.


My guess is that the alcohol “super cleans” the blades, which makes them last longer.


I’ve only been in radio for a year and keep seeing “RCS experience helpful” in ads for jobs. What is RCS?  Thanks. – Radio-Man


Radio:  RCS (Selector) is music scheduling software.  You can read all about it by clicking here:  Selector.

Reading Materials

Doc: Don't you have some articles or reading materials about several research topics?  A friend told me he saw them, but doesn't know where they're located.   Are they included in the Archive? - Anon

Anon: Yes, I have them on my business website, but I'll make it easy for you.  These articles have appeared in radio trade publications in one form or another.  Let me know if you have any questions after you read any (or all) of the articles.

Advertising Frequency
Research in Advertising
5 Stages of Persuasion
It Seems Like . . .
Intro to Media Research
Logic and Music Tests
Radio Ratings - Where They Come From
Research Validity

Reading Tables

Is there any quick way to learn how to read tables that are produced for perceptual studies? – Anonymous

Anon: That’s a rather broad question since I don’t know what you mean by "read." If you asking whether there is a quick way to learn how to understand what the tables mean, my answer would be "no."

Now, that doesn’t mean that you can’t quickly learn what the numbers mean in a typical cross-tab table, but it takes some time to learn how to synthesize the information from several banner points (the column headings) and several tables.

I usually see people who are new to research look at a table and come to some type of conclusion. The conclusion, however, is generally wrong or misplaced because they haven’t considered all the alternatives. A well-written questionnaire asks significant questions in several ways. It is important to consider all of the tables before coming up with conclusion.

If you’re new to research, I suggest that you do this:

Keep in mind that the quality of perceptual study tables relates directly to the quality of the questionnaire. If the questionnaire sucks, so will the tables. You will know when you reach a level of expertise in research when you can "see" the tables while you’re designing the questionnaire.

Reach and Frequency without Arbitron

Is there a way a small market station without a Arbitron can calculate reach and frequency? - Anonymous


Anon:  Well, let’s see here…


What is Reach?  The number of people who listen (are reached) during a given daypart or week.


What is Frequency?  The average number of times the audience reached by an advertising schedule is exposed to a commercial.  The formula for calculating Frequency is: Gross Impressions/Reach = Frequency.


What is Gross Impressions?  The Average Quarter-Hour Persons audience for all spots in a given schedule, or AQH persons x The number of spots in a schedule.


As you can see, reach and frequency require audience numbers.  If you don’t have Arbitron numbers, you’ll have to collect your own listening data (when the respondents listened and for how long).  You can do this with a very short and simple telephone study a few hundred respondents, probably no less than 300.

Record Label Addresses

I am trying to get the addresses of labels that may be accepting demo submissions.  How can I get a free listing without having to purchase a book or buy into a website? - Anonymous


Anon: I don't know of any list that shows which record companies are (or are not) accepting unsolicited materials.  However, there are some other materials on the Internet that may help.  For example, click here for a site that lists the addresses of about 5,000 record companies.  In addition, I set up two searches for you that may help.  Not all of the references are relevant, but you may find something good—click here and here.

Recruiting Questions

How can you ensure a room full of quality participants for a research project, especially when the radio research company hired to do your project farms out the task of recruiting to an other research company that does not specialize in radio research? Am I going to end up with a room full of research pigs (similar to radio prize pigs) who will say anything to participate in any research project to make a quick $50? Do most radio research companies recruit using outside sources? Are radio research firms who do their own recruiting any better? Should I even be concerned? - Anonymous


Anon: First, for some reason, you’re making the assumption that the quality of the respondents in a sample recruited by a local field service (subcontracted by the research company) is inferior to a sample recruited by the research company. This is entirely incorrect. While some research companies may do their own recruiting, most do not. And there are simple reasons for this.


Recruiting respondents in another city is usually referred to as a "foreign recruit." For example, a research company in Denver may recruit music test respondents in Kansas City. Experience has shown many times that foreign recruits are very difficult. A research company usually doesn’t have a good database of names and therefore must do cold calls. Respondents who are recruited via cold calls have a notoriously low show rate (that is, a low percentage of cold call respondents actually show up for the project). Because of this, the research company has to over-recruit by 100% or more to guarantee (hopefully) that enough people will show up. This raises costs and these costs are passed on to you.


Then there are also problems with reminder letters, hiring hosts or hostesses to oversee the project, arrangements for co-op payments, and cancellations. The cancellations are an enormous headache in a foreign recruit.


Because of all these problems, most research companies subcontract a local field service to handle the recruiting. The field services have local databases, know the area, and solve all the other problems with foreign recruiting.


Finally, a field service doesn’t have to specialize in radio research to be a good field service. What the company must specialize in is the ability to do large projects. Recruiting a sample for a radio project is no different that recruiting respondents for a medical project. The only difference is the screener that is used. What is most important is that the field service has the experience in recruiting large sample projects. There is no reason for a field service to specialize in radio research.


So, claims that "we do our own recruiting" mean nothing to me. I have been the CEO or companies that did and did not have their own field service, so I have seen both sides. An equal amount of control and supervision is possible by the companies that subcontract field services to do the job. A radio research company’s own field service can be as good or bad as an independent field service. The fact that a research company owns its own phone room does not guarantee that the respondents are "better than" respondents recruited by a subcontracted field service.


Now, you also make a comment about "research pigs." I’m not going to say a lot about your comment other than you need to keep in mind that these "pigs" (as you call them) are the same "pigs" who volunteer for a variety of things, including Arbitron ratings. Behavioral researchers have known for years that people who volunteer for research projects are different from those who do not. Volunteers tend to be more active in the community, interested in current events, and much more.


The odds are very high that your research project "pigs" are the same "pigs" who fill out your Arbitron diaries. They may be "pigs" to you, but they are also the people who can make or break your radio station.


So . . . you can be concerned about this problem to whatever degree you feel necessary. You can also subscribe to as many urban legends about sampling as you wish. However, I don’t consider research respondents as "pigs." I consider them to be the people who provide information to businesses concerning how the company’s products or services can be made better. Without such feedback, we would all have products and services that the companies think we need.


By the way, the same goes for your contest "pigs." Many research projects have shown that your "pigs" are probably P1s, and more than likely, exclusives. (The exceptions are those people who just scan around looking for contests.) However, most of your contest "pigs" fall into the same category as your research "pigs," and, therefore, are very likely to participate in Arbitron if they are asked. Get out the slop.


What are recurrents? Are there different kinds? Do different formats have different definitions? - Anonymous


Anon: The word recurrents is "radiospeak" for a song that used to be a current (a new song) but isn’t anymore. The song was once in the highest rotation on the radio station, but is demoted (or brought back in some cases) to a lower rotation.


Most radio stations usually use three broad categories of songs: currents, recurrents, and gold and the rotation of songs follows that order…currents are played most often, followed by recurrents, and then the gold songs.


There are as many definitions of recurrents as there are PDs in radio. The definitions are not tied to format.

Recurrents - Freshen Up

We need to freshen up our Recurrents, Power Golds, and Gold.  I gotta believe that NO ONE wants to hear Sugar Ray, Barenaked Ladies, Smashmouth, and the Rembrandts on Top 40 Radio in 2003!  It seems to me the best sounding music seems to be leaning Urban and I’m not sure what to do.  I do know that the above-mentioned bands are being played heavily on AC stations, which makes me shudder even more!  Where should I turn to get a list?  (Everything I have found on line regarding the past four years is full of wimpy AC crap!)  Help.  Thank you. - Anonymous


Anon:  Well, I’m not sure if I can help you here.  Here are my problems: 

  1. You say that “We need to freshen up…” and “I gotta believe that NO ONE wants to hear…”  You didn’t provide any supporting data here, so I have to assume that these are your feelings and not the feelings of your listeners.  You need to ask them.  You can’t assume that your ideas about “freshening up” the music and your opinions about certain artists are shared by your listeners.  Assuming your listeners agree with you is a mistake.  If you fix something that isn’t broken, you’ll break it.  Understand?

  2. You also say that, “It seems to me the best sounding music seems to be leaning urban…”  Ahhhhh (game show buzzer).  The telltale sign here is the word, “seems.”  Just because something “seems” to be, doesn’t mean that’s the way it is.  The best sounding music may seem to be Urban leaning to you, but your listeners may not agree.  It may seem to them that the best music is by Sugar Ray, Barenaked Ladies, Smashmouth, etc.  I don’t know.  And you don’t either.  You need to ask your listeners.

  3. I don’t know why you shudder because AC radio stations are playing the same artists.  So what?  A format doesn’t own artists or titles.  The goal of any radio station is to try to attract as many listeners as possible.  If that includes playing Sugar Ray et.al., then so be it.  If you think your format “owns” certain artists or titles, then you need to look around.

  4. Where should you turn to for a list?  A list of what?  “Urban leaning” songs?  If you think your audience wants to hear Urban leaning music on your radio station, then you need to look at Urban/Hip-Hop radio stations and test some of the titles.

Music is your product.  Before you mess with it, you better ask your listeners how they “lean.”  It’s the same thing I have been saying for 20+ years:  Find out what they want, give it to them, and tell them that you gave it to them.  If you give them what you think they need, you’ll probably be wrong.

Recycling - Horizontal and Vertical

Would you please give a very basic example of horizontal and vertical recycling. It came up in a recent discussion to better promote events on our station. Thanks. Once a radio student, always a radio student. - Anonymous


Anon: Hey, I’m older than you are and I’m still a student of radio. People who say they know everything about radio are legends in their own minds.


Simple explanation, eh? OK…how’s this:


To visualize audience recycling, picture a sheet of paper with seven columns, where each column represents a day of the week and is labeled Monday, Tuesday, Wednesday, and so on. Under each day of the week is a list of the programs for that specific day. The list can be by hour or daypart. Got it? Now, using that mental picture of the sheet, let’s consider your terms, horizontal and vertical, and use define them literally:


Horizontal (across columns) recycling: Relates to the fact that listeners tend to listen at the same time each day (such as morning or afternoon drive).


Vertical (up & down a column) recycling: Relates to listeners tuning in to various dayparts in a given day.


So what? Well, to achieve a high frequency of exposure to your promo, you would concentrate on horizontal recycling; if you wanted to achieve a high reach (contact as many listeners as possible), you would concentrate on vertical recycling.


There are your definitions in 10 words or less plus a few extra.


Click Here for Additional R Questions


A Arbitron B C D E F G H I J K-L M
Music-Callout Music-Auditorium N O P Q R S T U V W-X-Z Home

Roger D. Wimmer, Ph.D. - All Content ©2018 - Wimmer Research   All Rights Reserved