Applying behavioural insights to improve COVID vaccination uptake: a guide for councils

Applying behavioural insights to improve COVID vaccination uptake
This publication focuses on the work councils can undertake to improve vaccine uptake.

1. Understand your challenge

It is important to take some time to reflect on the behaviour(s) you are trying to change and the contexts in which they take place. More specifically it can help to begin by narrowly defining your behaviour(s) of interest. These can be ‘ultimate’ (i.e., the behaviours that you ultimately want to change) or ‘proximate’ (i.e., behaviours that influence the likelihood of the ‘ultimate’ behaviour occurring).

You can then move on to identifying the groups whose behaviour you want to influence, and to undertaking research that uncovers key barriers to change. For those under time pressure this can be completed relatively quickly through desk research. For those who have more time and resources, it may also be valuable to undertake some primary research (e.g., surveys or interviews).

This guide focuses on the work councils can undertake to improve vaccine uptake. However, you can use the steps outlined below to change other behaviours related to COVID-19.

1.1. Determine which behaviours to change

When designing a behavioural solution, it is crucial to define the behaviour(s) that you are trying to change. For example, do you want to:

Increase the number of residents who get vaccinated?

  • Change perceptions about vaccinations?
  • Increase the number of people who adhere to social distancing after getting vaccinated?
  • Reduce the number of missed vaccination appointments?

The final ‘behaviour’ you are looking to change, increasing vaccine take-up, can be thought of as the ‘ultimate’ behaviour. To understand how to achieve this it can be useful to break down this outcome into constituent ‘proximate’ behaviours which influence your ‘ultimate’ behaviours. When doing so, it is important to outline a clear Theory of Change that shows the relationships between ‘proximate’ behaviour(s) causally affects the ‘ultimate’ behaviour.

An example of this approach would be to try to encourage vaccinations (‘ultimate’) by first changing attitudes toward vaccines (‘proximate’). However, while changing attitudes is likely to causally affect vaccine uptake, it may not be sufficient. For example, individuals may not know how to get vaccinated. Thus, in this case, it may be worthwhile to design solutions that also provide residents with information.

This process is also linked to how you might evaluate the impact of your work. Understanding what behaviour(s), proximate or ultimate that you can measure. This is covered in section 3 ‘Evaluate your solution’.

1.2 Identify your target audience

Once you have defined the behaviour(s) you want to change you then need to identify your target audience. You can follow the two-step process below.

1.2.1. Review evidence on vaccine hesitancy

Conducting a quick literature review on vaccine hesitancy can be very insightful. It can help you identify the socio-demographic groups that are more likely to be vaccine hesitant.

Once you’ve identified key groups, check whether you have any local data to challenge or support your assumptions. Draw on local vaccination data if available or feedback from your community. You will need to continuously reassess your target groups as more data becomes available. Questions to consider include:

  • Are there any socio-demographic groups or communities that have lower vaccination rates in your area?
  • Are the groups you identified lagging in terms of vaccination rates? Are there other groups that you have to consider?
  • What is the feedback from community representatives or public health teams?

List of groups which have been identified as likely to be vaccine hesitant

1.2.2. Prioritise target groups

After having identified groups that are more likely to be vaccine hesitant you should prioritise them based on the national vaccine delivery plan, your area’s socio-demographic composition and your own local knowledge. Doing so will help you form an achievable target.

  • Which cohorts are due to be vaccinated? Are there any groups within these cohorts that are more likely to be vaccine hesitant?
  • What are the socio-economic and demographic characteristics that are most represented in your area?
  • Are there any other groups you should consider based on your local knowledge? Are there communities that are particularly isolated?
Examples: The London Borough of Havering work around the national vaccine delivery plan to identify and prioritise groups to target in their area. They first focus on the next cohort that is due to get vaccinated. This gives them time to start addressing people’s concerns before they become eligible to vaccinate. They then conduct a quick literature review and engagement sessions to identify specific groups within that cohort that are more likely to be vaccine hesitant. Read the full case study: The London Borough of Havering: Using the COM-B framework to develop a vaccine take up strategy

Other councils have adopted strategies which first identify a set of challenges or groups that are likely to be vaccine hesitant in the future. They construct this list by reviewing the vaccine priority lists and reviewing local data and insights. They then look to address the three or four most pressing problems. By addressing three or four challenges instead of trying to tackle all of the challenges they are facing this ensures the workload is reasonable and not overwhelming. Doing so allows the council to be forward looking and agile.

1.3 Understand group-specific barriers to vaccine take-up

It is important to work with communities to understand what their concerns and barriers to take up are. To do this you need to both engage with your target audience and to try to map and identify their specific needs and challenges. We have expanded on these below.

1.3.1. Engage with your target audience

During the pandemic councils are using a range of tools to engage with, and gain insight from, specific groups or populations. The most common methods used include social media monitoring, organising FAQ sessions, webinars, meeting with community forums, as well as using surveys and focus groups.

It’s important to consider the strengths and weaknesses of the different approaches as well as the time and resources needed to carry them out.

List of some of the common tools councils are using to listen to, and engage with, their residents during the pandemic

1.3.2. Identity barriers to vaccine take up

To help you structure your thinking, we present two helpful behavioural science frameworks. You can use these frameworks to identify barriers to vaccine take-up. Note that you will need to adjust these frameworks to fit your local context.

Once you have identified the key barriers to take-up, move on to the next step and start developing your behaviourally informed solutions.

Confidence, complacency, convenience model of vaccine hesitancy

Report of the Sage Group on Vaccine Hesitancy (PDF, WHO, 2014)

Vaccine hesitancy is a complex issue. The ‘3C model’ is a useful tool to understand the main reasons that drive vaccine hesitancy. It breaks them down into three main factors:

  • Confidence: People are vaccine hesitant because they have low confidence in the vaccine’s effectiveness and safety and distrust scientists, policymakers and health professionals.
  • Complacency: People are vaccine hesitant because they don’t perceive themselves to be at risk and view the vaccine as not necessary.
  • Convenience:  People are vaccine hesitant because there are a number of barriers (physical, logistical or economical) that hinder them from getting a vaccine.

We developed a template based on this model to help you understand the reasons driving vaccine hesitancy among your target audience. You can use it when conducting interviews, focus groups or surveys.

Download the template to understand vaccine hesitancy

COM-B model of Behaviour Change (Michie et al., 2011)

The COM-B model is a behaviour change model and is a useful tool when identifying barriers and opportunities for behaviour change.  It suggests that any behaviour is the result of three main factors. These can be used as levers to achieve behaviour change:

  • Capability: People have the skills, ability and knowledge to perform in a behaviour
  • Opportunity: There are external factors which make a certain behaviour possible, these can be provided by the social or physical environment (social norms, peer support, resources).
  • Motivation: People have the desire, impulse, habit or the intention to perform a behaviour. 

Numerous councils use the COM-B model to identify barriers to vaccination and to develop solutions to address these barriers. We developed a template to help you apply the COM-B model.

COM-B template to understand vaccine hesitancy

2. Apply Behavioural Insights

After you have mapped the barriers to the takeup of vaccines, you can begin developing solutions that either remove––or push people to overcome––these barriers.

Councils have a range of tools that they can use to support vaccine uptake, such as communications, policies, and programmes. The following sections provide concrete recommendations that you can take into account when designing your solutions. 

2.1. Design your behavioural solution

In this section, we list a set of behavioural insights and recommendations that councils can use to try and change or shift behaviour.

The first set of behavioural insights help to encourage individuals to form intentions to vaccinate. These can be applied specifically when designing communications.

The second set of behavioural insights help to get people who have the intention to get the vaccine to take action and go and get the vaccine. This is called bridging the intention-action gap. These insights can be applied to communications, or to designing services, the physical environment or be linked with how you enforce policies.

You do not need to apply all of the recommendations at once; instead, choose the ones that best suit your context.

2.1.1. Encourage vaccination intentions

Keep communications brief

Recommendation: Keep your messages short and clear.

Why: Shorter messages are more likely to be read in full. They are also more easily recalled and perceived as more credible (Godinho et al., 2016; Lawes-Wickwar et al 2020). Adding more information, even if it is useful, may distract from the key message.

Audience: The general public

Example: The London Borough of Hounslow designed a social media post to correct vaccine misinformation among minority communities. The message is a short and includes a clear quote from a local GP from an ethic background stating “You can’t get COVID-19 from the vaccine”.

Read the full case study: London Borough of Hounslow: Used the messenger principle to address vaccine misinformation

Inform people about the seriousness, and risk, of contracting COVID-19

Recommendation: Communicate the health consequences and risk of contracting the virus, and frame the vaccine as a protection against these. When targeting groups that are especially vulnerable to COVID-19, be specific about the severity of the risk that they face (Butter et al. 2020).

Why: People are more likely to get vaccinated if they believe COVID-19 is serious, they risk contracting it and if they view the vaccine as risk-reducing  Brewer et al., 2017; Godinho et al., 2016; Williams et al., 2020). 

Audience: Individuals that are vaccine hesitant (specifically targeting messages to at-risk groups).

Example: The London Borough of Hackney has designed a communication that targets residents suffering from diabetes, emphasising the risk and the seriousness of COVID-19 in their specific case.

Provide balanced information about vaccine effectiveness and safety

Recommendation: When communicating about the vaccine provide transparent and factual information on its effectiveness and safety. Messages should: 1) convey that the vaccine is effective (both in terms of reducing the risk of infection and serious illness) and safe and that it has undergone the same rigorous testing as other vaccines, and 2) be transparent about potential side effects.

Why: People are likely to get vaccinated if they believe the vaccine is effective and safe. Balanced communications that set out the costs and benefits of vaccination are more credible and trustworthy (Rubinstein et al., 2015; Godinho et al., 2016 ; Freeman et al., 2020).

Audience: The general public, specifically Individuals that are vaccine hesitant with concerns over the effectiveness and safety of the vaccine (e.g., BAME groups and pregnant women).

Example: The London Borough of Hackey conducted a focus group with local residents to test different communications. They found out that people want to know the real risk they would be exposed to if vaccinated. Communications should be transparent rather than “sugarcoating” the vaccine. Read the full case study: Hackney Council: Used resident focus groups to test vaccine hesitancy messaging

They also designed a poster addressing people’s concerns over the novelty of vaccine by emphasising that it works just like any other vaccine.

Highlight the pro-social benefits of vaccination

What: Emphasise the collective, societal importance of getting vaccinated in achieving herd immunity and protecting vulnerable people. Appeal to the common good and elicit empathy towards those that are vulnerable.

Why: Highlighting the social benefits of vaccination is an important motivation for getting a vaccine (Betsch et al., 2017). Combining informational content about herd immunity with emotional content about protecting those that are vulnerable can be particularly effective (Pfattheicher et al., 2020).

Audience: For both vaccine acceptant and hesitant groups, especially groups that perceive that they have a low risk of contracting the illness (e.g., younger generations) (Isler et al., 2020; Pfattheicher et al., 2020).

Example: Wirral council has produced a video campaign to encourage self isolation among BAME communities. The primary message is to self isolate to protect those that cannot self-isolate, such as healthcare professionals and key workers. Similarly, the social benefit of vaccination can be highlighted.

Apply social norms supporting vaccination and make them salient

Recommendation: Emphasise the number of people that already got vaccinated, that there are growing intentions to get vaccinated, that that the majority of people approve vaccinations, and make vaccination visible to others as much as possible.

Why: People are more likely to get vaccinated if they know that most other people have done so as well and if others approve of it (Moehring et al., 2021; Palm et al., 2021 ; Bish et al., 2011; WHO, 2020; Bavel et al 2020).

Audience: The general public, especially individuals that do not perceive themselves to be vulnerable to COVID-19 (Betsch et al., 2015).

Example: The London Borough of Barking & Dagenham is focusing on establishing a social norm around vaccination among faith groups by setting up a vaccination center close to a mosque.

Further, the London Borough of Hackney designed a poster using social norms to encourage vaccine take-up.

View another example of how social norms can be used in direct messages to promote vaccination: Behavioral emails increase flu vaccine uptake among city employees

Use role models to endorse vaccination

Recommendation: Promote vaccination by using people that got vaccinated as role models. Choose people that are relatable to your target audience.

Why: People are more likely to vaccinate if others, who are similar to them, have done so (Bish et al., 2011; Cialdini & Trost, 1998).

Audience: The general public

Example: The London Borough of Hounslow are distributing posters and bumper stickers to elderly people to show that they got vaccinated. Similarly, they are developing social media assets, such as Instagram story. Read the full case study: London Borough of Hounslow: Used the messenger principle to address vaccine misinformation
 

Provide positive reinforcement

Recommendation: Provide positive messages about the vaccine, linked with the hope it provides. Praise those that get vaccinated.

Why: Positive messages about COVID-19 vaccines lead to more positive attitudes towards vaccines and such messages more likely to be shared with others. They can also be an effective strategy to counteract the negative emotions usually employed by anti-vaccine communications (Chou et al 2020; Petersen et al 2020, Attay & Mercier, 2020).

Audience: The general public, especially individuals that are pro-vaccine or individuals that feel overwhelmed by the negativity around COVID-19 (Chou et al 2020, Attay & Mercier, 2020).

Example: To encourage vaccination within care homes, The London borough of Havering writes to care homes that have particularly high rates of vaccination to praise them for their efforts. They also use them as positive examples for other care homes. Read the full case study: The London Borough of Havering: Using the COM-B framework to develop a vaccine take up strategy

Build on past COVID-19 behaviours

Recommendation: Draw a direct connection between getting a vaccine and complying with other COVID-19 preventive behaviours (e.g., lockdown compliance, social distancing, masks).

Why:  Building on past experiences or efforts related to COVID-19 gives people confidence in getting vaccinated. It also frames vaccination as an action that is consistent with their past behaviours (Williams et al. 2020).

Audience: Vaccine acceptant individuals.

Example: Norfolk Council designed a communication aimed at encouraging its residents to stay at home by building on their past experience during the first lockdown. The poster states “we’ve done it before, so this time we’re better prepared”. The goal was to increase people’s sense of confidence and consistency.

2.1.2. Bridge the intention-action gap

Help people plan for their vaccination

Recommendation: Encourage people to create plans about what they need to do to get vaccinated: How they will travel? How can they anticipate and plan for problems?

Why: Making plans helps people commit to their intentions and can encourage vaccine uptake (Milkman et al., 2011; Brewer et al., 2017).

Audience: Individuals who already have positive attitudes toward vaccines.

Example: During the first lockdown, Norfolk council designed a communication aimed at helping people use public transportation safely. It lists the key steps people should take to encourage them to plan their commute and comply with preventative behaviours.

Similarly, the London Borough of Hackney designed a communication that prompts people to plan their vaccination.

Remind people about their vaccination

Recommendation: Remind people about their vaccination. Reminders are particularly effective if 1) they come from a trusted source, 2) people can choose their preferred channel of communication, and if 3) they provide information on how to take action.

Why: Reminders are an effective way to help people act upon their vaccination intentions (Briss et al., 2020; Brewer et al., 2017).

Audience: Individuals that are due to get their vaccine.

Example: Sandwell council is repurposing its existing phone service (previously used for signposting residents to council services) to remind people that are overdue for vaccination. Read the full case study: Sandwell Council: Creating an army of vaccine advocates.

Remove practical barriers to vaccination

Recommendation: Address practical or logistical barriers that can get in the way of vaccination. Do people know where vaccine centers are located and how to reach them? Can they take time off work and get vaccinated? Are vaccination center staff trained to address concerns that patients might have?

Why: reducing barriers and making it easy to get vaccinated increases vaccine uptake (WHO, 2020; Brewer et al., 2017)

Audience: The general public

Example:  In order to reduce physical barriers, The London Borough of Barking & Dagenham has set up a vaccination site close to a well-known mosque in the area, an important reference point among the Islamic community.

Leverage social networks

Recommendation: Get connected individuals that already got vaccinated to encourage their peers to do the same and support them throughout the process.

Why:  Social networks can amplify the impact of an intervention. Getting people that are well connected to encourage others to vaccinate can be effective. (Bavel et al., 2020)

Audience: The general public

Example: The London Borough of Havering is planning on launching a volunteer ambassador programme. The goal of the programme will be to provide peer-support on how to book a vaccine and address any related concerns people might have. The council is also encouraging residents that have already received their vaccine to encourage their peers to do the same. Read the full case study: The London Borough of Havering: Using the COM-B framework to develop a vaccine take up strategy

2.2. Find the right messenger

If your solution is to send a communication or to provide information, consider who delivers the message. People are not only influenced by the content of a message, but also by the individual or organisation that conveys it.

Consider whether your council would be an influential messenger to communicate to your target audience. If the council is not the most influential communicator, consider working with partners.

  • Who might be most effective at delivering the message?
  • Councils have access to individuals in a range of roles. Are there any specific figures within your council that would be more influential (e.g. public health leads or councillors)?
  • Are there any influential external figures that could support or endorse council communications (e.g., local GPs or community leaders)?
  • Are there any influential external figures that could become direct messengers (e.g., community leaders or third sector organisations)?

Below are some recommendations on how to identify influential messengers: (Martin & Marks, 2019).

'Hard' messengers:

'Soft' messengers:

  • Recommendation: Leverage community ties and relatable figures to disseminate vaccine communications, such as community champions, volunteer organisations, social media influencers. 
  • Why: Soft messengers are influential because they have similar characteristics to the intended recipient and are seen to be trustworthy.
  • Examples: Wirral Council is using its existing ‘community connectors’ programme to encourage vaccination. Community connectors are a group of local residents with strong links within their community. They regularly engage with their network to understand and address their concerns about the vaccine. Read the full case study: Wirral Council: Listened to the ‘voice of the resident’ and engaged with hard-to-reach communities. Similarly, several councils are working with sports clubs, ambassadors and other organisations to promote social distancing and vaccine uptake.

Things to avoid:

Avoid involving figures that can be politicised. They can create resistance among part of the population, undermining vaccination intentions.

2.3. Choose your channel

If your solution is communication-based, consider the channel(s) that can most effectively deliver your message to your target audience. 

In some cases, the most effective communication channel will be dependent on the ‘messenger’ you choose. In other cases, you won’t need to find a channel as the ‘messenger’ will deliver the message through their network.  For example, community leaders can work as messengers and channels to disseminate messages.

The channel you choose will affect your ability to evaluate your solution. Consider whether there are any outcome metrics you can record to assess the impact of your solution. For example, if you are sending out bulk emails you will be able track whether recipients open them or click on links.

Here are a few examples of channels used by other councils:

Social media (Twitter, Facebook, TikTok, Instagram)

Example: The London borough of Hounslow launched a series of messages via Facebook that address concerns about vaccine trials being rushed. Using Facebook, they were able to track engagement and determine the effectiveness of their messages.

Read the full case study: London Borough of Hounslow: Used the messenger principle to address vaccine misinformation

Similarly, the Hertfordshire council posted videos on TikTok to reach younger generations. The videos they designed received over 40,000 views.

Webinars

Example: Stockport Council organised a webinar with care home staff and the council’s public health team. The goal was to have a wider conversation about vaccination. They used this channel to engage directly with their target audience and address their concerns.

Example: The London Borough of Havering organises online FAQ sessions on vaccination for its residents, working with local health authorities. They leveraged their existing links to different community forums and specifically used this channel to engage with BAME residents and women. Read the full case study: The London Borough of Havering: Using the COM-B framework to develop a vaccine take up strategy

One to one - personal networks

Example:  Wirral Council use ‘community connectors’, which are local residents with strong community ties, to disseminate vaccine communications within the community. Before COVID-19 they operated door to door in neighbourhoods. During lockdown they have shifted their work online but continue to build on local networks. Read the full case study: Wirral Council: Listened to the ‘voice of the resident’ and engaged with hard-to-reach communities

Posters

Example: The London Borough of Hackney designed a series of physical posters to encourage vaccine take up

Poster on COVID-19 risk for people with diabetes

Poster using social norms

Local radio

Example: To reach out to its South Asian community, Sandwell Council has leveraged its community links and transmitted messages via the Punjab Radio. Local radio with specific groups and demographics is an excellent way to communicate with those where English is a second language. Read the full case study: Sandwell Council: Creating an army of vaccine advocates

Direct messaging (email, letters, flyers)

Direct messaging that utilise behavioural science principles––such as personalising texts and using commitment devices (e.g., asking individuals to pledge to get vaccinated or asking people to set up a booking)––have been effective in many contexts.

Example: The behaviour change consultancy Ideas42 designed emails aimed at encouraging vaccination that combine various behavioural techniques: social norms, loss framing, and an ‘enhanced active choice’, which prompts people to consider the consequences of their actions.

Read the case study: Behavioral emails increase flu vaccine uptake among city employees

3. Evaluate your solution

The importance of evaluation cannot be understated. Where possible we would advise you to dedicate some time to tracking the outcome(s) of your work.

Evaluating the impact of your solutions enables you to understand if you have achieved your goals, lets you build on what works, and helps you direct resources to activities with maximum impact. Furthermore, it is particularly valuable to share these learnings across the LGA network.

That said, it is not always feasible (or desirable) to conduct a full-scale evaluation. Most councils are working with limited resources and are having to adapt to changing policies and circumstances. The following section will help you to set a reasonable evaluation plan and choose appropriate methods at this time.

3.1. Define your outcomes and metrics

3.1.1. Set your outcomes

When setting up an evaluation, first identify your desired outcomes. Earlier in the guide [1.1. Determine which behaviour to change] we outlined the difference between an ‘ultimate’ outcome, which is the final behaviour you wish to change and the intermediate or ‘proximate’ outcomes. We suggest drafting or using a Theory of Change to help understand how you expect your solution will remove barriers, address proximate behaviours to change your solution.

Ultimate outcomes

Think about the ultimate outcome that you aim to achieve: Are you looking to increase vaccination rates within a specific group? Correct misinformation around vaccines? Encourage people to take their second dose? Or prompt them to maintain social distancing once they have got their vaccine?

Proximate/intermediate outcomes

Once you have identified your ultimate outcome, think of any intermediate outcomes that you need to change in order to achieve your main goal. Your solution might lead to an increase in vaccine take-up, but might do so by first changing attitudes, beliefs, or intentions around vaccines.

There are a number of factors that are strongly associated with vaccination that can be considered as intermediate outcomes:  1) having positive attitudes towards vaccination, 2) perceiving that its a social expectation, 3) feeling anticipated regret if they were not to vaccinate, and 4) believing that COVID-19 is serious and that they are at risk of contracting it (Godinho et al., 2016; Bish et al. 2011).

Read more about How to develop a Theory of Change.

3.1.2. Determine your metrics

Once you have defined your outcomes identify metrics that allow you to track them.

First consider whether you can measure your ultimate outcome. Then examine whether you can measure any other proximate or intermediate outcomes. In order to assess the impact of your solution, keep in mind that you should be able to directly link your solution to the outcome.

  • Are you able to measure behaviour, such as actual vaccination?
  • Are you able to measure attitudes, intentions, beliefs or self-reported vaccination through surveys?
  • Can you directly link your solution to the outcome? In other words, can you determine whether the person who was exposed to your solution later went on to get vaccinated?
  • At what level can you capture your metrics? At an individual-level? household-level? Neighbourhood-level?
  • Can you capture repeated measures of your outcomes to monitor change overtime?

Using proxy metrics

When tracking your ultimate or intermediate outcomes is not possible, consider finding proxy metrics. Proxy metrics are data points that can be used to represent other metrics.

Examine the extent to which your proxy metric provides data specifically about your target audience. Consider whether it is granular or representative enough to be meaningful.

Below are some examples of proxy metrics that could be used as substitutes for tracking vaccination intentions:

  • Click through rates on links providing information on vaccines
  • Social media reactions to posts (likes, comments, shares, views)
  • Number of downloads or delivered leaflets.
  • Number of people attending to vaccine related online events.
  • Number of scheduled vaccination appointments

Example: The London Borough of Hounslow tested different behaviourally informed messages on Facebook. Their goal was to address concerns about the vaccine trials being rushed and ultimately encourage vaccination among minority groups. They added links to the Facebook posts that directed residents to the council’s COVID-19 webpage. They tracked click throughs and used them as a proxy metric to evaluate the effectiveness of their messaging.

Read the full case study: London Borough of Hounslow: Used the messenger principle to address vaccine misinformation

3.2. Choose your evaluation method

Deciding on what evaluation method to use will largely depend on your time, resources, and practical feasibility. Below you will find a list of commonly used methods that you can use to evaluate your interventions:

A/B tests (also known as randomised controlled trials)

About: A/B tests are a robust evaluation method. They rely on randomly assigning people to either a control group or one or more treatment groups and then comparing outcomes between these groups. People in the treatment group(s) will receive the solution that you have designed, whereas people in the control group will not.

Strengths:  It provides accurate results. If done well, they ensure that any impact you record will be directly attributable to the effectiveness of your solution and not to external factors.

Limitations: Setting up A/B tests can require resources and planning. You will have to ensure that you can control who receives your solution in order to randomly assign them into either group. However, most social media platforms have in-built tools to do this. 

Example: The London Borough of Hounslow has conducted an A/B test on Facebook to test the effectiveness of different messages aiming at addressing people’s concerns about the vaccine being rushed. They targeted people within a 10 mile radius from Hounslow and used click through rates as a metric for engagement.

Read the full case study: London Borough of Hounslow: Used the messenger principle to address vaccine misinformation

Resources: Learn more about A/B testing and design your experiment in this A/B testing tool by ideas42 and access a guide on RCTs by Nesta

Pre-post surveys

About: You can measure the impact of your solution by getting your target audience to respond to a survey before and after rolling-out the solution. By comparing outcomes before and after you will be able to assess the effectiveness of your solution. You can track different outcomes in the survey, such as self-reported vaccinations, intentions to vaccinate, attitudes or beliefs.

Strengths: It is an effective method to use when you cannot randomise who will be exposed to your communication or solution.

Limitations: It is less accurate than an A/B test or randomised control trial. There might be other external factors that explain changes in outcomes before and after the implementation of your solution.

Example: Wolverhampton Council have developed an online survey to track changes in attitudes. They deliver the survey before and after launching campaigns to understand which messages work and with which groups.

Read the full case study: Wolverhampton City Council: Applying behavioural science principles as quick wins to COVID work

One-off survey experiments

About: You can launch one-off surveys where you randomly assign survey participants into a number of groups. Each group will be shown a different communication. You can record respondents’ beliefs, attitudes or intentions as outcome metrics. Doing so will allow you to compare the effectiveness of different types of communications and identify the most impactful ones.

Strengths: One-off survey experiments require less planning than an A/B test can can provide accurate results.

Limitations: Getting your target audience to take the survey might be challenging. Consider using incentives to ensure participation.

Qualitative feedback

About: You can conduct a focus group to test your behavioural solution or communication. Gathering qualitative feedback from residents can help you understand if it might work and fine tune your solution before a full roll-out.

Strengths: Qualitative feedback provides rich input that can help to answer why a particular solution might (not) work.

Limitations: Feedback based on qualitative data is hard to quantify and organising focus groups might be time consuming. This method might be best used during the solution development phase but not as a means of conducting a final evaluation.

Example:  The London Borough of Hackney conducted a series of focus groups with residents to test the effectiveness of a set of COVID-19 communication. This allowed them, to improve their design and pick the best one.

Read the full case study: Hackney Council: Used resident focus groups to test vaccine hesitancy messaging

Monitoring clickthroughs, downloads, or queries

About: You can monitor how your target audience interacts with your solution. For example, you can track the number of views, clicks, or downloads of a social media post or a an online resource.

Strengths: Often quick and easy to implement (the data are readily available).

Limitations: Unless the data you monitor closely represents your ultimate or intermediate outcomes, the ability to evaluate the impact of your solution is limited.

Example: Norfolk Council developed a behaviourally informed COVID-19 prevention tool-kit for small businesses. They were able to estimate the impact of their solution by monitoring the number of times the tool-kit was downloaded from the council’s website.

Read the full case study: Norfolk County Council: Designed toolkits for small businesses to encourage preventative behaviours
 

3.3 Learn, adapt and scale

Once you have selected your evaluation method and analysed your data, you should be able to tell which of the solutions you implemented worked particularly well and which were low performers.

You might have also collected valuable feedback throughout the evaluation phase. For example, if you engaged in focus group discussions, you might have received comments from residents when they reacted to your proposed solutions.

Use these learnings to improve your solutions and, once you have the capacity, evaluate them again. This iterative process will ensure that you systematically build on what works and implement behavioural solutions that have the strongest positive impact.

Scale and disseminate

If you have conducted research and found relevant learnings, it is immensely valuable to share these internally within your council and with practitioners across the LGA network. 

Please share your learnings with us! We would like to promote and disseminate them across our network.