6. Data & Evaluation Toolkit: Building a Strong Data Culture

How can organizations improve their ongoing use of data?

The use of data to routinely inform decisions and operations is referred to as a data culture. There are a few core components to data culture including a commitment to using data, staffing for data analysis work, and practices to maintain and make use of organizational data regularly.   

Legal aid providers fall across a wide spectrum in terms of their current use of data. Some organizations are beginning to think about how to use data, while others may have staff dedicated to preparing and analyzing data. No matter where a provider is, there are many actions they can take to strengthen their commitment to making better, more informed use of data.

 Some examples include the following suggestions:

• Encourage Staff Buy-In: Staff at all levels participate in data efforts, from paralegals collecting intake data to grants managers reporting on grant data, to supervisors reviewing team caseloads. For many organizations, it can be a big shift from under-utilizing data to routinely relying on data. To help encourage staff throughout this transition, make sure to stress the how and why behind the change. How will data be used moving forward? How will processes change as a result? Why is this change needed and how is it expected to help the organization? Answering these questions clearly and repeatedly will demonstrate to staff the benefits of data-driven work and the organization’s commitment to this practice. Consider utilizing change management strategies such as those discussed in this Harvard Business Review article. 
  • Another practice to encourage staff buy-in is to reward those who are adhering well to data practices. Identify which staff are maintaining accurate, timely, and complete data and which staff are using data to inform their work. Rewarding good performance on data efforts can encourage greater participation throughout the organization, as well as acknowledging staff contributions to this change. 
     
• Make Data More Prominent: If staff can’t access data, they can’t use it. Minimize the burden on staff to seek data out by making the data more accessible in the first place. This transparency demonstrates an organizational commitment to utilizing data routinely. Efforts to make data more prominent will take different forms, such as broadly disseminating analysis results or reports, discussing metrics in staff or team meetings, including data summaries in internal newsletters, and building dashboards for ongoing use. 
   
• Address Data Literacy: No matter how accessible data is, staff are likely to ignore or misuse it if they do not understand how to properly interpret data. Start by assessing data literacy within the organization. To inform this discovery work, consider asking staff to take a data literacy assessment such as this test offered by the Data Literacy Project. From there, tailor trainings, webinars, and other educational events for staff to help boost knowledge around data analysis vocabulary, techniques, and meanings.
  • Staff who are more familiar with data analysis can be resources for these educational activities. Providers could also turn to volunteers, local academic institutions, or external partners for training support and resources.   
     
• Ask More Questions: Recognize that data analysis work is not a one-time-only activity. Legal aid providers face operational and programmatic decisions all the time that can be informed by data.  Don’t stop with just one project; consider other ways that data can be used or what new analysis questions could be asked based on existing results. 

The above actions all help to foster a culture of data use and encourage data-informed, continuous improvement. Providers can also strengthen a data-driven culture by maintaining clean, quality data for routine use; strategies relating to data quality maintenance are discussed in the next section.

How can organizations maintain high quality data?

Legal aid providers typically have a large quantity of data on hand from case management systems (“CMS”) and other data repositories. A strong data culture is bolstered by frequent review and upkeep of this data, to help ensure that the data being collected routinely aligns with organization’s needs. This data can be used for ongoing analysis but is only as useful as it is clean. Consider asking the following questions when reviewing the quality of case management system data, one of the most common data sources for legal aid providers: 

• Is there missing or outdated data? 
  • Example: Case handlers are behind on closing their old cases, so it is not possible to get an accurate count of active, open cases by staff member.
     
• Are there structural issues with the way any data is collected?
  • Example: A mandatory intake field “Do you have a disability” only has response options of “yes” and “no”. Intake workers select “no” to continue with the intake when the answer is unknown to them, because a response is required. This clouds the meaning of “no”, to include those who do not have a disability AND those who did not answer the question. 
     
• Are respondents able to select values that are accurate and true to them?  
  • Example: A client identifies as “non-binary” but the only options in the gender field are “Male”, “Female”, and “Unknown”. There is not a value that accurately captures their gender identity. 
     
• Is there a shared understanding of the definitions behind fields and values by those responsible for data entry? 
  • Example: A staff member selects “Brief Service” for their case’s Closing Level of Service after providing advice to a client for two hours via phone. A different staff member selects “Advice and Counsel” for their case’s Closing Level of Service after providing the exact same service. The same work is captured differently in the Closing Level of Service field, because there is not a common understanding of the difference between “Advice and Counsel” and “Brief Service”.
     
• Addressing the above questions is a great starting place towards improving the quality of data that gets collected on an ongoing basis. To help with this, consider implementing routine data cleaning practices to help maintain quality data. These practices could include quarterly reports to review data completion rates for new intakes or cases with no recent updates that may need to be closed or automated reports pulling cases from the CMS with incomplete data for staff to address. Also schedule regular administrative reviews of data collection processes throughout the year, such as intake and closing forms, to make sure these are set up to facilitate complete and accurate data collection of key data points. 

How can organizations ensure they have adequate staffing for data analysis work? 

When thinking about staffing for data work, keep in mind that this includes not just those conducting the analysis, but also staff who enter or review data into case management systems and other data repositories. Building a strong data culture can involve staff time and effort from all across the organization, and it is important to acknowledge this work as part of data analysis efforts. 

In terms of hiring for data analysis work, it is not always feasible for legal aid providers to hire a full-time staff member or team dedicated solely to data analysis. For organizations that find themselves in this situation, here are a few alternative approaches to staffing for data analysis:   

• Combine data analyst responsibilities with another related position, such as a grants coordinator, case management system administrator, or technology manager. 
• Bring on part-time support or interns for this work.
• Talk with community partners to see if there is a shared need for data analysis staff and consider splitting a position across multiple organizations. 
• Connect with volunteers in the community who may be willing to engage in data analysis work for low or no cost. Ask board members to identify individuals in their network who may be willing to volunteer their data analysis services.
• Connect with local academic institutions or hire external consultants for discrete analysis projects.