From the Item Bank

The Professional Testing Blog

Developing Nudges for Your Exam Program

By Cynthia G. Parshall and Brittany Johnson

Introduction

Behavioral design is the process of identifying points in a user process where behavioral tools, such as nudges, usability, and more, can help the user take good actions. Nudges are small changes to an environment that can have a powerful effect. Educators, corporate offices, and governmental institutions are formally incorporating nudges into their interactions with the public and their staff. Similarly, testing programs can use them to support examinees, subject-matter experts, staff, and employers in taking actions in their own best interests.

However, behavioral design is not a “magic pill” and simply implementing a nudge doesn’t guarantee a desired outcome or a positive change in the intended behavior. The context has an impact and to implement a nudge effectively you will need to conduct research within the context of your exam program, with your target audience. This discussion of research is part of the Action Design series for  Instruction and Assessment.

To find the most effective nudge for your context, start with this iterative research process – Diagnose, Experiment, and Learn.

Step 1: Diagnose

In this first step, you will diagnose where to focus your efforts by completing three activities – identifying the key behavior, analyzing friction points, and selecting a behavioral design principle to apply. By taking the time to complete this first step, you ensure your efforts are aligned to both your test takers’ best interests, as well as your exam program’s goals.

Identify the key behavior

To identify the key behavior, start with your program’s goals.

• What problem is most important to solve right now?
• What goal are you trying to achieve?

Write down your program’s goal. Now, you can more easily identify the key behavior, that is — what behavior or action you want your test takers to carry out. The key behavior should be specific and measurable.

For example, staff at a certification program, STAR*, had an ongoing issue of test takers waiting too long to register for their exam to renew their certification. Not registering on time had several negative consequences that directly affected their test takers and their business. Their goal was to increase the number of test takers who registered by the renewal deadline. They had an automated email reminder sent out a month before a test taker’s renewal date, letting them know it was time to renew. STAR researchers selected “register for the exam after receiving an automated reminder email” as their specific and measurable key behavior. (This example is summarized in Table 1.)

Analyze friction points

The next activity is to consider all the friction points that could prevent a test taker from completing the key behavior. This is often called a behavioral audit and it’s a detailed, step-by-step analysis of every action in the process. Your goal is to determine all the barriers that might possibly keep your test takers from action, listing each step a test taker needs to do to complete the key behavior. Reviewing existing data from your exam program related to your key behavior and each step can be a helpful guide to friction points. Additionally, for each step, ask yourself:

• Is this step necessary?
• Could there be points of confusion (about what to do, the information presented, etc.)?
• Is there anything about this step that could be a distraction?
• Is there anything about this step that could cause a negative or unmotivating reaction?

After reviewing all friction points, choose the one you want to address first; this is often the biggest barrier.

Continuing with the previous example, when STAR completed their behavioral audit, they reviewed existing data, including the number of recipients who opened their reminder email, how many clicked the button in the email to get to the registration page, and how many actually registered from the reminder email. They noticed a low open rate  (the percent of users that opened the reminder email). This suggested that a substantial number of emails were never opened. After a complete analysis of all potential friction points, they made the decision to target the barrier of an uninformative and/or unmotivating email subject line for their first experiment. (Further analysis details are provided in Table 1.)

Craft your nudge

Once you have selected the friction point to address, you can start thinking about all the behavioral design principles that could be helpful. Choose the one you think is most likely to nudge your test takers to action.

For example, STAR decided to design a nudge by incorporating loss aversion messaging into the subject line of their email. The subject line could, truthfully, point out that test takers must begin the renewal process on time, or be subject to additional fees. (Further possible changes are provided in Table 1.)

Step 2: Experiment

Now that you’ve diagnosed the problem and crafted a nudge to influence the key behavior, it’s time to test the effectiveness. The best way to test this is to design an experiment. A good experimental design should include what you are testing, how you will test it, and how you will measure it.

What are you testing?

Clearly distinguish a control and test condition. In behavioral design, your test condition will typically be based on the nudge you are planning to implement, while the control condition can be the existing, unchanged element. It is essential that your test condition has only one variable or change. If you have more than one variable, you won’t know which variable caused a change if you see a difference.

For example, STAR researchers are testing the new email subject line; this will serve as the test condition. They will use the original (unchanged) subject line as their control condition.

How will you test it?

Two important details to think about when planning your experiment are the research sample and the timeframe. Luckily, there are best practices that can guide your decisions. When deciding who should participate in the experiment, it’s best to use random selection within a representative user group.

You also want to make sure you have a sufficient and equal number of participants in both the control and test group. Testing with too few participants or people who are not representative of your user population may not provide you with results you can reliably use to predict how your full population will respond.

Once you’ve decided on your sample, you can determine how long you’ll need to run the experiment to get the number of participants you want. It’s also a best practice to choose a timeframe that is fairly consistent and won’t be affected by external events, such as seasonal or promotional effects.

STAR had been targeting a sample size of 200 users in each group and historically sent around 50 reminder emails per week. Thus, they needed to run the experiment for 4 weeks. They verified that the upcoming timeframe wasn’t during a peak and no promotions would be underway.

How will you measure it?

This is arguably the most important aspect of running an experiment. As with item creation, if you cannot sufficiently measure your experiment you will have no way to interpret the results. This is why the key behavior you select needs to be specific and measurable.  Choose your target metric early. Your metric will measure the outcome you want to achieve. Once you’ve nailed down your target metric, you can set up the software or reporting needed to track your target metric for each group.

In STAR’s example, the target metric was the email open rate.

At this stage you are ready to execute and field your experiment!

Table 1: The primary elements in a behavioral design research study are listed in the table below, along with the elements included in the example used in this post.

Step 3: Learn

When your experimental timeframe has ended, you can begin to analyze the data. At this stage your goal is to learn from your data, in order to make an informed decision about how to move forward. Start by looking at the target metric of each group, and then ask yourself some additional questions.

• What is the outcome for each group? Is the target metric for one larger than the other, or are they roughly the same?
• Is there anything surprising about the results? What is it and why it is surprising? Does this suggest other research paths you should pursue?
• Are there any additional patterns?

After reflecting on the results of the data, you can decide on the best course of action – Implement or Iterate.

1. Implement: If the results are satisfactory (typically meaning the test condition was more successful), congratulations! Your nudge is effective. You can implement the test condition to your entire test taker population. Then you can move on to the next barrier and repeat this process.
2. Iterate: If the results show no difference between test groups, or the test condition is less successful than the control, the nudge needs some refinement. Try a different behavioral design principle to address your barrier, or try a different implementation approach.

STAR was pleased to see their nudge email subject line increased the open rate by 7%. Furthermore, the increased open rate, on its own, lead to a 1% increase in registering for the exam. As a result, STAR implemented the improved subject line for all renewal emails. Researchers then proceeded to the next point of friction in the list they had prepared.

Summary

The subtle encouragement provided by a targeted nudge can have a substantial impact. Research has shown that  well designed nudges have helped people in numerous ways. As Thaler and Sunstein note, “By knowing how people think, we can make it easier for them to choose what is best for them, their families, and society.” (2008).

However, the field is also full of initial nudge attempts that had no effect whatsoever. The difference between an unsuccessful application of nudges and a successful one is research. Fortunately, the process for behavioral design research is clear and manageable. A thorough behavioral audit will typically reveal a number of friction points which provide opportunities for improvement. And a well planned experiment, including iterative revisions, will usually result in important and cost-effective improvements to the program.

*The exam program “STAR” is a fictitious name used for the purposes of illustrating the research process.

Additional posts

For more information on similar topics go to Behavioral Design.

References

Go to Additional Resources for Action Design for references.

Categorized in: Industry News, Marketing, Test Development