Webinar: Baking Evaluation Into Your Technology Projects

Laura Quinn - Consulting Principal

Zach Zarnow - Illinois Equal Justice Foundation

Brandon Thomas - Florida Justice Technology Center

Think of evaluation as something that should be a continuous evaluation project. Not something that happens at the end of a large project. The longer you wait the harder it will be to make improvements based on input.

Evaluation vs Baking in Metrics

When you “bake it in and repeat” basically you are incorporating evaluation into your projects as an ongoing process. You design a small chunk (system AND METRICS), build that piece, measure that piece and then report it.

Benefits:

No scramble to figure out how the system is doing
No big one-time outlay of expenses for an evaluation
You can refine your metrics as well as your designs over time
You can accumulate more date before you have to officially report

Case Study: Florida Name Change

Floridanamechange.org

What is Florida Name Change?

Website that helps Floridians legally update their name and gender market
Was built to serve the transgender community as your cornerstone user
Can also serve anyone else that just needs to change their name

Why did we focus on the transgender community? There are over 100k that live in Florida. 68k of those haven’t updated their identification documents and they are an “at risk” group that faces discrimination.

Items to Find Out:

What do we need to know from website reporting?
How many visitors coming to site?
How often to people come back?
Where do visitors come from?
What our our visitor demographics?
What type of browser and device?

Enhanced items:

How popular is each page and how to visitors interact with them (entrances, exits, how many scrolled?)
How many legal form packets are being completed?
How effective is our interview process for each automated form?

Withing Google Analytics you can see a list of websites with pageviews BY pageviews so you can see that if people exit on a middle step page there is a problem there that needs to be corrected.

Case Study: Illinois Armed Forces Legal Aid Network

This law was started by a state law and is funded by a $2 filing fee on legal documents. Basically this is a statewide network of providers to help military and veterans with free and reduced cost legal services.

Things We are Asking:

What are the civil legal needs of vets, active duty and their spouses and dependents
To what degree are those needs being met
Which organizations are meeting those needs?>
Where in Illinois are those in need located?
What unique or complicating factors exist that need to be considered for effective service delivery?

We want to get all our partners and affiliates who receive funding to coordinate, the organization has about 13 right now. How can we standardise all the date we are receiving? We created a group intake document to be used by all organizations. We tried to make it as simple and applicable as possible. We have questions about rank, ID etc. We also wanted to talk about outcomes. What are you doing? What are you not doing? What would be too hard to do? We asked about VA benefits. How many applications filed, accepted, denied etc. You may get medical care instead of financial benefits and we would keep track of that in the form too.

We built a CMS using our salesforce date and turned this into a website. We can see accept and reject rates for the different organizations. This allows us to create custom reports, acceptance rate, rejection rate, referrals by location, hotline calls, referrals by problem code etc.

Case Study: The “New” Ohio Legal Help

There is actually a site for Ohio Legal Help already but we are just loosely affiliated with that site. We have a new state-wide site that is finished design and is entering development for launch around Q2 of 2018. It is carefully designed using data and research and the site will be used to gather data for future decision making.

What We are Looking to Track:

Number of people who visited
% of people who teek key actions that imply engagement
% who took a next action
% who found it helpful
% who took actual action in the world down the road

This document shows requirements, not a finalized set of what they were tracking. For example we might later take on veterans organizations but that isn’t planned out yet.

So we have been thinking about what is actually feasible to build vs what you can actually USE and what is the best overlap there.

For the Topic Page:

# of visitors
Referral path (quiz, search engine etc)
Bounce Rate (including by referral path)
% who opened an expandable header
% who use any Send To
% who click through on any content link within the main page content
% who say it’s helpful (thumbs up / down)
% who saw this page who then said they took a subsequent action in SMS follow up

For the Home Page:

Number of visitors
Number who start each mini-quiz
Bounce Rate

(throughout: standard Google Analytics device metrics, geo codes, new vs repeat etc)

Forms:

# visitors
% visitors
% who start assembling, % who finish
% who say it's helpful (thumbs up / down)
% who use any send To
% who click through on any content link within the main page content

Last updated on January 08, 2022.

Data & Evaluation