{
    "version" : "https://jsonfeed.org/version/1",
    "content" : "news",
    "type" : "single",
    "title" : "How To Navigate Challenges to Successfully A/B Test Your Emails |Digital.gov",
    "description": "How To Navigate Challenges to Successfully A/B Test Your Emails",
    "home_page_url" : "/preview/gsa/digitalgov.gov/bc-archive-content-3/","feed_url" : "/preview/gsa/digitalgov.gov/bc-archive-content-3/2021/05/21/how-to-navigate-challenges-to-successfully-a-b-test-your-emails/index.json","item" : [
    {"kicker" : "A/B Testing","title" :"How To Navigate Challenges to Successfully A/B Test Your Emails","summary" : "The article will go over some best practices of implementing A/B testing on your emails.","date" : "2021-05-21T11:44:00-05:00","date_modified" : "2025-01-27T19:42:55-05:00","authors" : {"jenn-wingerberg" : "Jenn Wingerberg"},"topics" : {
        
            "customer-experience" : "Customer experience",
            "design" : "Design",
            "research" : "Research",
            "usability" : "Usability"
            },"branch" : "bc-archive-content-3",
      "filename" :"2021-05-21-how-to-navigate-challenges-to-successfully-a-b-test-your-emails.md",
      
      "filepath" :"news/2021/05/2021-05-21-how-to-navigate-challenges-to-successfully-a-b-test-your-emails.md",
      "filepathURL" :"https://github.com/GSA/digitalgov.gov/blob/bc-archive-content-3/content/news/2021/05/2021-05-21-how-to-navigate-challenges-to-successfully-a-b-test-your-emails.md",
      "editpathURL" :"https://github.com/GSA/digitalgov.gov/edit/bc-archive-content-3/content/news/2021/05/2021-05-21-how-to-navigate-challenges-to-successfully-a-b-test-your-emails.md","slug" : "how-to-navigate-challenges-to-successfully-a-b-test-your-emails","url" : "/preview/gsa/digitalgov.gov/bc-archive-content-3/2021/05/21/how-to-navigate-challenges-to-successfully-a-b-test-your-emails/","weight" : "1","content" :"\u003cp\u003eHow many emails do you generally get in a single day? The answer is probably at least 20-50. In today’s digital world, consumers receive so many emails – which is why it is a great medium to use for A/B testing. As a reminder, A/B testing (also sometimes called split testing) is a process of showing two or more variants of the same digital media (website or email) to different segments of visitors at the same time and comparing which variant drives more engagement. Email A/B testing can really help you learn more about your audience and tailor your communications to their needs. In this blog we will outline our method, best practices, and some lessons learned while email A/B testing at the Consumer Financial Protection Bureau (CFPB).\u003c/p\u003e\n\u003cp\u003eTo begin we will discuss our method. When starting any A/B test, whether that’s on our website or via email, we want to have a well-defined test strategy. Our strategy must include a clear understanding of WHAT we’re testing and WHY. Always keeping in mind \u003cstrong\u003ewho\u003c/strong\u003e our audience is and the \u003cstrong\u003egoal\u003c/strong\u003e that we want them to achieve. Some of the lessons learned from past email A/B testing have been: \u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003etest the right variants, \u003c/li\u003e\n\u003cli\u003edon’t have too many variants, \u003c/li\u003e\n\u003cli\u003eand help yourself out by using a template. \u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eIf you’re interested in increasing open rates, you probably will not want to test adding a picture into the email content since the user has to actually open the email first to view the picture. Instead, testing subject lines and how they affect open rates, or testing pictures to see if one affects engagement with a specific call-to-action would be better tests.\u003c/p\u003e\n\u003cp\u003eWe also learned that narrowing down the number of variants to test is very important for statistical significance. If you test too many variants and spread your segmented visitors too thin, you may never reach statistical significance and therefore cannot interpret the data accurately. \u003c/p\u003e\n\u003cp\u003eFinally, having a process and capturing past research is helpful when developing new tests to know what has and has not worked already. Past research can also lend to the development of best practices when sending out future emails – example emails sent from a personal looking email account perform better than emails sent from an agency.   \u003c/p\u003e\n\u003cp\u003eHaving a grand idea is always great, but in addition to our strategy, we have to ensure that the test can be executed successfully. Questions we may ask ourselves: \u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eDo I have the right tools? \u003c/li\u003e\n\u003cli\u003eIs what we are interested in testing actually feasible? \u003c/li\u003e\n\u003cli\u003eAre there technology limitations? \u003c/li\u003e\n\u003cli\u003eWill my listserv generate enough traffic to even reach statistical significance? \u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eEvery test should be thoughtfully planned and given enough time to make sure the test can actually be set up and run successfully. Speaking of enough time, when thinking about statistical significance, there are several calculators online that can help you estimate how much traffic you’d need to reach statistical significance. We have been fortunate enough to work with our data scientists at the bureau to create our own calculator, but you can also search online for “A/B Split Multivariate Test Duration Calculator” and see which one will work best for you. Keep in mind: if you have a test that fails to reach statistical significance, you are still learning. Learn from the hiccups and keep trying more tests!\u003c/p\u003e\n\u003cp\u003eIf we are able to successfully run a test, the last step is analysis and applied learning. The goal of every A/B test is to find what variant drives more engagement, and then implement that change. However, it should be noted that not every successful test should be thought of as “one and done.” What you learned in one email A/B test may not be as helpful in another email catering to an entirely different audience. Over time, your audience could also evolve; what resonates with them today may not in 6 months. That’s why it is important to continually test. Be curious—but also keep track of all of the different email A/B tests that you have done and review them. When you are looking at setting up new tests for the quarter or for the year, think about what you did already (what worked and what didn\u0026rsquo;t work), or what could you refine and test again? There is always room to keep learning.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eFor more information on A/B testing, join our \u003ca href=\"https://digital.gov/communities/a-b-testing-community/\"\u003eA/B Testing Community of Practice\u003c/a\u003e, and check out our \u003ca href=\"https://digital.gov/topics/a-b-testing/\"\u003eA/B testing resources and videos\u003c/a\u003e.\u003c/em\u003e\u003c/p\u003e\n"}
  ]
}