What is a 'Pair Rank' question on OpinionX?
Pair Rank is a survey question format on OpinionX which is used to turn a set of options into a ranked list by using a series of head-to-head pair votes. For example:
When should I use a Pair Rank question?
Pair Rank questions are great for ranking lists of 10+ options that are too long for standard drag-and-drop ranking (eg. the Order Rank block). By only having to vote on two options at a time, the Pair Rank list requires much less mental work from participants.
How are the Pair Rank scores calculated?
Pair Rank on OpinionX is based on a format called "Win Rate Analysis". For each option, this takes number of wins and divides by its total number of pair comparisons, giving a score between 0-100 (ie. the percentage win rate).
What do the Pair Rank exported results look like?
For more information about exports (including examples of exported spreadsheets), please visit the Knowledge Base page on exporting.
How many Pair Votes per participant will generate robust results?
There is a small ( ℹ️ ) icon that appears beside the box where you can decide how many votes each participant should vote on. Click that icon to show the Pair Rank calculator.
How the Pair Rank calculator works:
There are two formulas you can use to make sure you have a good minimum sample size:
Both are explained below...
Formula 1: Limited pool of participants
This formula tells you how many votes each participant will need to cast so that every possible pair combination has appeared at least 3 times in total across the whole survey. The formula to calculate this is as follows, where "n" is the number of options you're ranking:
3n(n-1)/2
----------
participants
Here's a breakdown of how this formula works:
For example, if you have 16 options to rank and know that you'll only have 30 people to complete the ranking survey, our formula would be (3(16x15)/2)/30 = 12 votes per person. In this case, when deciding how many votes each participant must cast, I should opt for at least 12 votes or more per person.
Formula 2: Large pool of participants
When you know that you've got a good plenty of participants you can engage, you'll want to get as much data per person without making your survey overwhelming. We'll use a slightly different formula:
3n(n-1)/2
---------------
votes per person
Here's a breakdown of the meaning of this formula:
Whereas before our limiting factor was the number of participants, now our limiting factor is the number of pair votes we will ask each participant to cast.
Let's say we have 40 options we want to rank using the Pair Rank format. I usually use 10 votes per participant as my starting point. That formula would then look like: (3(40x39)/2)/10 = 234 participants. If that seems like a lot more than I'll be able to reach, I can try increasing the number of votes to 15, which adds up to 156 participants. As you can see, the number of votes per participant changes your required sample size a lot!
Full Formula Explanation:
Total number of possible pairs is
n(n-1)/2, where n is the number of ranking options.Multiply the total possible pairs by 3 to calculate your target number of total votes.
Divide your target number of total votes by your minimum estimate of participants you expect will complete the survey.
The answer you get is the minimum number of pair votes you should require from each participant.
Some Caveats:
Generally, if the formula results in a number below 10, I would just leave the default 10 votes in place as it is a very reasonable amount to ask from each participant.
If your survey has multiple Pair Rank questions, you should consider the total number of votes a participant will have to cast across your entire survey. The higher your total number of pair votes, the lower your completion rate will be. Generally, anything above 40 pair votes within a survey is a lot to expect of any participant (unless you provide a strong participant incentive).
If segmentation is going to be an important part of your analysis, you should substitute the "
participants" variable for your estimate of the total number of people you expect to reach from your smallest key segment. Here's an example:I have 14 ranking options and expect to reach 150 participants, but I plan to segment my results to see the differences between customers on my four different pricing tiers. The segment I'll get the least responses from is Enterprise Customers, but I think I can get at least 16 participants from them. So my formula would be
3(14*13)/2/16= 17 votes per participant.
Can I use multiple Pair Rank blocks in my survey?
All OpinionX users can have an unlimited number of Pair Rank blocks in their survey, regardless of whether they're a free or paying customer. This apples to all block types on OpinionX.
How many ranking options can I include in a Pair Rank question?
There is no limit to the number of Pair Rank options you can include. OpinionX has facilitated surveys with hundreds of ranking options in a single Pair Rank block.
Do Pair Rank questions work on mobile devices?
Yes. OpinionX surveys are designed to be suitable for participants on any web browser, whether that's a desktop monitor, laptop, tablet or smartphone.
Can I upload images for my Pair Rank block?
You can rank images using the separate "Image Rank" block, which uses the same voting and scoring format as the Pair Rank block.
Can I use emojis in my Pair Rank options?
You can use emojis in any text-based field on OpinionX, such as questions and ranking statements.
I have a question about Pair Rank blocks that wasn't answered :(
Send us a message using our live chat or email [email protected] to get a quick answer from the OpinionX team.