Watch for Data Patterns
- The data analyses you undertake are going to reveal patterns in your data.
- Watch for patterns that are unexpected and explore further when you find them.
Example: Unintended Declining Services:
Trend analysis of case management system data from one legal aid revealed a sudden a sizeable decrease in both intakes and cases for Black clients over the most recent 2-year period. Further analysis revealed that these decreases coincided with the organization’s new focus on domestic violence work, for which there were known barriers preventing people of color from seeking DV assistance. This legal aid quickly increased the number of outreach events and developed new partnerships within the Black community. Follow-up data analysis revealed that the organization’s efforts had almost immediately changed the trend: intakes and cases for Black clients were trending back up.
Example: Verifying Senior Trends:
One legal aid became concerned about a drop in both the number of intakes and cases opened for seniors. Analysis of census data revealed that both the number and proportion of seniors in poverty in the service area had likewise decreased indicating that the intake and case data were mirroring need among this population. While the staff concern was ameliorated, the legal aid remained committed to assisting seniors and decided to continue its significant outreach efforts to this population.
Data Integrity
- Reliable data allows organizations to understand clients’ needs, assess and improve effectiveness, assess and improve efficiency, define strategic goals and measure progress towards them, measure and describe impact, impress funders, target outreach efforts, step back and see the big picture, and celebrate success.
- Analysis of erroneous or incomplete data is counterproductive as a gauge of client service effectiveness and efficiency.
- Before starting to analyze your data, it’s important to assess the quality and reliability of the data.
- Audit your data and identify the fields that consistently show missing data, erroneous data, or data that are hard to interpret when you run a report analyzing the field.
- Clean up the data issues you uncover and develop standard data entry procedures to ensure data integrity moving forward.
- See “Clean Data, Clear Insights,” a presentation by Rachel Perry of Strategic Data Analytics & Susan Vincent of the Legal Aid Foundation of Los Angeles about data integrity hosted by LSNTAP on Dec. 9, 2024. https://youtu.be/06_jSTCrWdQ.
- See Perry, Rachel, “Data Integrity: The Untapped Treasure of Legal Services Data,” MIE Journal, Summer 2014, pp. 22-26. MIE Journal 2014 Summer-Data Article.pdf
Dealing with Difficult Data
- Uncollected data: Sometimes you do not or cannot collect data about something that is important to your clients.
Example: Inconsistent Disabilities Data:
One legal aid wanted to analyze data about the disabled clients it served, but it wasn’t collecting disability information consistently. Until the legal aid could get staff to enter the data more consistently, it decided to use proxy information as the best alternative. The proxy variables they used were: 1) Income = Disability Financial Assistance or SSI, and/or 2) Legal Problems = SSDI, SSI, Mental Health, or Physically disabled rights.
- Incompatible data format: Sometimes your internal data and the external data you want to compare it to are in different formats or measurements, which makes them difficult to compare.
Example: Differences in Domestic Violence Data:
One legal aid wanted to compare internal data about its domestic violence clients with external data about domestic violence survivors in the state. It gathered information internally about DV clients, but the only external DV data available from the state courts did not include poverty information. Thus, there was some mismatch between the populations in the two data sources.
Still, trends in both sets of internal and external data could be considered together for deeper understanding about DV cases and survivors, as long as the legal aid noted that its data was limited to the below poverty population.
Factors that Can Skew Data
- Case Acceptance Guidelines/Organizational Priorities: Limited resources mean that legal aids are not able to help everyone in need, so they are forced to prioritize their service offerings, usually via case acceptance guidelines. That prioritizing means that legal aid case data will be skewed in favor of the prioritized legal issues and/or client groups.
- Not a Random Population: The clients who come to legal aid are not a random sample of an area’s poverty population. Their decision to come to legal aid may be influenced by location and/or transportation options, a legal aid’s reputation and/or visibility, economic or even popular trends, etc.
- Example: Spike in clients bringing eviction cases to legal aids because of the affordable housing crisis and its rising rents.
- Example: An NFL player abuses his wife, and it is caught on camera. There is public outcry, and legal aids see a spike in clients with domestic violence issues.
- Even Imperfect Analyses are Useful: You can still analyze skewed data and get a deeper understanding of your clients and their legal needs. Randomization and zero skew are not required for non-academic analyses. Just make sure to understand these factors and how they influence your data and explain them whenever you share your findings.
Run Data Findings by Staff
- A statistical analysis of legal aid data will never, by itself, provide the whole picture.
- Involve staff in the process early on by asking for their assistance interpreting findings because they know the clients best.
- Staff input will guide your analysis in the right direction.
Example 1: Pre-Consolidation Lingering Habits:
One legal aid noticed that a larger than expected proportion of its domestic violence cases were originating in one of the counties in its five-county service area. The staff wondered if that indicated a troubling prevalence of domestic violence in that county, which would require an increased regional focus on DV issues. But organizational leaders realized that the office in that county used to be its own independent organization focused almost exclusively on DV issues and wondered if the community near that office assumed it still had that primary focus. The legal aid decided to conduct increased outreach in that county to get the word out about all the other services available.