If a tree falls in the forest, and less than 100 people hear it, can we scientifically prove it made a measurable sound that will remain familiar over time?

They key to research of course is numbers. Without numbers, or sample size, we cannot reliably extract meaningful data, much less take action on it. So for many stations, one of the hardest parts of music research is getting enough responses to a survey to meet that “good data” threshold. In this article we’ll share some best practices to not only get better numbers, but develop a better database as well.

First, we should define what we are looking for when it comes to responses. According to research experts, a qualifying sample size for the type of research we are doing is 100. That means you need a minimum of 100 people to participate in each survey before you can scientifically say you have a quality survey. Now, if you plan to dissect your results among a particular demo, such as by age or gender, then you will need a minimum of 100 returned surveys within that demographic. So getting that email that says your 100th survey has been completed always feels great, but your work may not be done just yet. By having at least 100 in your sample size, you will have enough returns to statistically smooth out anomalies such as suspect data, extreme opinions and the like. In Songscore, you can quickly see how many people have graded any particular song by looking at the results table’s far right column. A song is counted as surveyed when it is familiar to the user, so one song may have 100 completed surveys, while another song might have 89, since 11 people may have marked that song as unfamiliar. It’s ok to have unfamiliar songs in your total sample count (because it’s an honest answer) but in an ideal world, you would have enough margin to have 100 opinions on every song in your target demos. This might mean going for 150-200+ responses total. If you see excessive unfamiliar numbers, you may be testing new music too soon.

At Songscore, we have dozens of clients in a variety of formats ranging from Contemporary Christian to Alternative Rock. Some of our clients are in top 10 markets with hundreds of thousands of listeners, while others are in small markets with tens of thousands of listeners. However, we see success and struggles in all size markets. Market size is not the issue. We have clients in top 5 markets who struggle and we have clients in very small markets to do quite well with results. It really comes down to recruiting and maintaining your user base and how you communicate with them during the process.

How the top stations get results:

1. Start Big. If you are just starting out with your surveys, or are starting over, show your users you really, really care about their opinions by offering a very large incentive. You won’t have to do this every time, but by starting with something large, like sold-out tickets or a backstage experience, you will be letting your listeners know this is important you and there is something in it for them (just remember not to survey the artist you are using for the incentive or you’ll get skewed results on that artist).
2. Stay Big for the first few surveys.If you drop off from a free cruise on survey #1 to nothing on survey #2, your audience will get the message that it’s not as important as it was before, so keep up some nicer incentives. Give away normal concert tickets, or some smaller experience. If you can make it something that is not offered for sale or on the radio, even better. Being able to say to your database “you guys are insiders, we trust you and need your opinions, so we have an experience you’ll only hear about here..” will make them feel special and will motivate them to act over time (i.e. they will do more surveys even if they do not win). Remember, we are building connection here.
3. Which brings us to the 3rd, somewhat obvious tip- always incentivize! Your users are busy and sometimes it takes a nudge to get them to share their opinion. It’s also true that if you don’t get “normal people” into your surveys then the “music lovers” will take over and skew results to reflect an uber-fan that might not represent your normal audience. By giving away 5 random t-shirts, or a 5-pack of cds on every survey, you will give the average person at least some sort of incentive to take the time to score your songs. We also recommend annual prizes. Send a note to everyone at the end of a season, tell them “song of the summer” for your market and then announce that John Doe has won a cool prize for no other reason than that he represents all the cool people who have taken surveys over the summer. Surprise and Delight these valuable users/listeners.
4. Use Rewards. A couple of users have experimented with the Rewards System. This is a very basic system created at the request of one of our original users, but you might find it useful. You upload images that are “coupons” or rewards for your users. After they take xx surveys, they earn that reward. Maybe a station t-shirt for 10 surveys, etc. The reward will not display until they have earned it.
5. Share the Results. Each email you send out should start with a recap of the top songs in the previous survey. Tell them how they influenced your playlist and how they made the station better. Most stations recap the top 3 songs (you can easily copy and paste the bar graph into an email), but go a step further and tell them something like “hey, we didn’t know you like that new Johnny Jones song so much! We’ve started it playing it more often because of your feedback.” Pulling back the curtain and letting your audience know they have a real impact on the station is a huge motivator. If someone writes back and says they hate the Johnny Jones song, great! Tell them to keep taking surveys and their opinion will count. You can also use this tactic on the the air with a special top 5 song countdown feature or other way of letting people know you really do listen.
6. Don’t fatigue your users. We recommend surveying your audience no more than once every 2 weeks. It can be fatiguing if you ask them to help with almost an identical list of songs that often. If you have a very large database, you can segment it and send half your users a survey each time, thus getting results more often but not burning out your users. Just be sure to shuffle your user groups once in a while.
7. Email your panel. It seems obvious, but in helping clients we’ve found sometimes stations simply forget to take the extra step to send out an invitation o the user base.
8. Constantly recruit new users. Imagine if the Gallup organization used the same 200 people for every poll they ever conducted on any topic. The results would be predictable and over time, no longer representative of your larger, constantly changing audience. You have a big advantage over Gallup- you have a micriphone! So be sure your on-air team personally invites people to join the panel. Invite people song by song. Rather than say “hey go tell us about 20 songs we want to know about..” make it about one song at a time. “Love that new Muse song, or can you do without? Tell us at wxyz.com on our songscore survey.” Another tool at your disposal is to sort results by number of surveys taken or last survey taken. This powerful tool let’s you see what your heavy survey-takers think vs your newest survey-takers think.
9. Use Social Media. Go where your listeners are already sharing opinions and ask them there to take your survey. They are already online and talking about things they are passionate about, so the jump to a survey is not a large one.
10. Email past users who have not participated in a while. We have a couple of clients who are masters at this, sending out a personal note to everyone who has not completed a survey in the past 6 months. It’s not a heavy handed email, it’s a carefully worded, friendly connected. “I noticed you have not completed a survey in a while, so I just wanted to see if you still wanted to be a part of this. If you don’t, let me know and I’ll make sure you don’t get these emails anymore. If you do, we’d love to know what you think…”
11. Remind users during the survey. We’re really excited about this one! Our next version of Songscore is going to include the ability to automatically send reminder emails to users, and only those who have not taken the survey yet! So right now you can send a blanket email to everyone on your list and you can schedule that ahead of time- say 5 days after the initial survey goes out. You can schedule a couple of reminder if you want. However, in our new version you’ll be able to set reminder1, reminder2 etc. and we’ll send that out to only those who have not completed a survey. Nifty, huh?
12. Be conversational. In your emails, on-air messaging, social media platforms- anywhere you are inviting people to participate, be conversation. No one likes to be approached by a clipboard-baring survey taker at the mall, so don’t communicate in the way. Engage people about the songs and artists you play and then let them know their opinion matters. Also, attach a well-known personality to your emails. Listeners are more loyal to people than brands, so if Joe Coffee from the Morning Brew is asking, it might do better than a generic “thanks from WXYZ”.
13. Add extra questions. Throw in a question about something you know is on your listeners mind. You may not even care about the results. “Are you for or against daylight savings time?”. It will make the survey memorable, more personal and give your recruiting tools more to work with. Now the morning show can invite people to sound off on the top by taking the survey and oh by the way, rate some songs too.
14. Use your streaming assets to recruit. People listening online via computer or via an app are only a small step away from being able to complete a survey. They don’t have to remember to get online or do it later, they are already online. So add a link to the survey to your app menu and your website. If you can customize your desktop player, add a button to the survey. At kxoj.com, they put a thumbs up/thumbs down graphics under the now playing widget. Both graphics lead to the survey, so they don’t actually register a vote but they do lead people to the survey.
15. Don’t survey too many songs. We mentioned survey fatigue earlier from too many surveys, so it’s also good to mention song fatigue. Our recommendation is to not test more than 20 songs in a survey, and 15 is even better. If you have not yet played a song on-air for at least 200 spins, or whatever your OEM scheduling number is, then those songs are not likely to be familiar, so those can wait a few weeks. The same might be true for some of the oldest songs on your list. Keeping the survey easy to take in the mind of the listener is going to make them want to come back and do it again with less incentive.

If you have other tips to share, we’d love to hear them and pass them along to other users in our next blog post. Happy surveying!