Lessons Learned: Evaluating Video Content
If you were to perform research on the value proposition of training videos, you would notice that opinions are split on their efficacy. Despite all the tools that are out there that can help you evaluate video quality, views, and drop-off, there are some things that should be considered in the analysis of your organization’s videos.
As a member of the Service Design practice at the Consumer Financial Protection Bureau (CFPB), I was tasked with a research project evaluating how non-consumers interact with the CFPB in regards to complaint data. This audience included companies, federal and state level government agencies, and congressional offices. Each group accesses a “portal” to either respond to complaints (companies) or to have access to complaint data (congressional offices and government agencies). The tasks that are performed in each respective portal vary dependent upon responsibilities.
Each portal was comprised of an interface that was not highly intuitive to navigate, but was supported with a manual of how to use the portal, FAQs (which somewhat mirrored the manual), and “training videos.” When it came to evaluating the videos, I did not have access to any analytics on usage or if they were viewed completely, and surveys from a year prior did not capture feedback on them either. As I began analyzing, I started having some thoughts on videos in general.
Determine why the videos were created in the first place
Was it something users even wanted? Is your UI and content too hard to understand in the first place and that’s why you created a video? It turns out that on our end, one business unit would train all new portal users once they were on-boarded. This practice was not scalable at a growing agency, so they decided to create videos. It was not something that the CFPB had ever received feedback on, but rather a business desire.
I would say think holistically — make sure that the video is not trying to solve another problem your users are having that will still exist once you create the video.
If possible, leverage any end user interviews
With my timeline and Paperwork Reduction Act (PRA) limitations, I was limited to nine interviews. I took a sample that varied in entity size, complexity, and regulatory-facing experience. The interviews encompassed a variety of topics and probed about what the user does when he/she does not know how to utilize the portal. It turns out that every single entity was using the manual, but would have preferred screenshots in it. If the manual was unclear, they ended up calling that business unit anyway. Each user said he/she didn’t have time for videos, and had never even bothered viewing them.
Consider the cost of maintenance
Depending on the content, there is potential that the video could become outdated quite quickly. When I was reviewing the videos, I noticed that they were done at a time prior to some incremental enhancements, and therefore were not 100% accurate. At a relatively new bureau where things can evolve quickly, the potential for further change was high. In terms of the investment of maintaining these, we realized it would be much more cost-effective to just include screenshots in the manual, and republish it more quickly than a video.
Overall, evaluation of training videos really depends on your project and the limitations you have, but I cannot stress the importance of understanding of why it was created in the first place and what problem the video is trying to solve. The solution may not be in the video, but could lie elsewhere.
This article is the result of a thread on our User Experience Community listserv. See how other agencies are working with content and video, and find out how to join our Communities, like Digital Audio/Video Production and Strategy and User Experience. If you’re interested in writing for DigitalGov, learn about the Open Opportunities program and check out our assignments (for feds with .gov or .mil email accounts only). If your agency’s team has tech-related lessons learned, case studies, digital or mobile product news, strategy success stories, etc. to share, please review our guidelines and contact DigitalGov to submit an article idea.
