Building a Culture of High-Quality IDEA Data in Wisconsin

00:00:01.54  >> You're listening to "A Date With Data," with your host, Amy Bitterman.

00:00:07.38  >> Hey, it's Amy. And I'm so excited to be hosting "A Date With Data." I'll be chatting with state and district special education staff who, just like you, are dealing with IDEA data every day.

00:00:19.53  >> "A Date With Data" is brought to you by the IDEA Data Center.

00:00:24.49  >> On today's episode, we're joined by Seth Bishop, data analysis and grant evaluation consultant at the Wisconsin Department of Public Instruction, to talk about data culture.

00:00:34.55  Seth, can you tell us about what you do, and how long you've been doing it?

00:00:38.11  >> Thanks, Amy. I've worked at the Wisconsin Department of Public Instruction for about seven years now, five of which have been on the special ed team. And I am essentially the core data analyst for special ed in Wisconsin. I handle all our federal reporting, be they calculations or the reports we send to districts, and I am the go-to data expert on all the different identifications, and all the little different ways that they vary. There is someone who keeps track of all of these things, and I am the lucky/unlucky person who's responsible for remembering all the ins and outs of those indicators.

00:01:23.96  >> Tell us how you share data internally and externally. And what are some things that have worked well, and maybe areas you'd like to improve.

00:01:31.41  >> In building a data culture in Wisconsin, I think one of the biggest struggles that we had to overcome is making sure that everyone was speaking the same language, which might seem obvious, but it has proved to be a very challenging thing for people across all the different state and federal calculations that have to be made, to actually get people at the same table and understand data in a sort of comprehensive and coherent manner. We have a lot of data experts in Wisconsin, a lot of very qualified people, but historically, we've also been a little bit too siloed, we're all in our own little worlds of IDEA identifications, ESSA Title I identifications, and that has been the first step, is getting them speaking the same language, because that's part of what a culture is, having people together, understanding each other.

00:02:42.20  In order to do that, we've created something that's relatively unique in Wisconsin, and that is what we call our Joint Federal Notifications. And that combines our ESSA and Title I identifications with our IDEA LEA determinations and significant racial disproportionality calculations. So all of these come to the districts and the local education agencies as a coherent package that they evaluate together. So it's not just special ed directors sitting at a table and talking with us, it's also the school principal or the superintendent and other staff. And that has been a huge step forward for us. We do this through these joint federal notification packets that are reports that show the data broken down across all the different measures, which we send securely to every LEA, and then we meet with those districts and talk through the reports with them. And these reports have been a pretty big game changer for us in transparency and communication with districts around continuous improvement, and really aligning those conversations across all the different areas of identification that are possible.

00:04:19.55  >> Wow. That sounds like such an amazing process that you've put in place. How did you get to the point where you were able to come together and produce those reports jointly?

00:04:30.77  >> A lot of work and a lot of communication. It pre-dated my time joining the team. I came in on the last legs of that effort. But it was a multi-year effort, in which we reviewed timelines and flexibility with timelines, whether or not -- which areas we could sort of give to get these released at the same time. It required a lot of coordination, and it continues to require a lot of coordination. We meet once every three weeks year-round, even though we only release the reports in a narrow window of a couple of months, because it requires that sort of attention. And speaking the same language requires us to know what's going on in each other's worlds. So those monthly updates are essential for that. And it took a willingness for everyone involved to come to the table and compromise, that when you work together and when you're unsiloing, you've got to give. You can't have things the way you always have had them because how you've always had them isn't sustainable in a world that you're trying to bridge these chasms between data teams and outcomes. That has been the big thing that I think can be challenging for a lot of people, is knowing that you're going to have to give some things up, but the value added of the whole process more than makes up for that. I don't know of anyone involved in our joint federal notifications who would want to go back to the way things were, even though it was painful to get things started.

00:06:27.79  >> That makes a lot of sense. And what about the districts, who are the recipients of these reports? How have these reports helped to build their capacity and using the data?

00:06:37.99  >> Yeah, so one of the things we do is, we allow districts to -- not only do we provide all the data to the districts for them to review, but we also have what we call our WISEdash secure dashboard, which would allow districts to go in and drill down to student-level data to sort of really understand how these things are getting calculated and the consequences of things. We've actually seen far more engagement with this WISEdash secure system, because it existed for a long time because of the reports that we're now producing, because districts want to understand, from the ground up, how these calculations are made, and how they earned the score that they did, and then how they can improve, moving forward. It's been incredibly positive for me to see so many districts begin to use these tools to understand their data, because now they have the instructions and incentive and resources to do so.

00:07:54.37  >> Can you talk a little about how the data culture has shifted, if at all, during the pandemic?

00:08:00.84  >> COVID certainly affected the data. There's no way around that. But it hasn't negatively affected our data culture. If anything, I've found that people are more engaged and -- yeah, I think overall it's been non-disruptive from a data culture perspective, but immensely disruptive from a data quality perspective, because no one was anticipating this.

00:08:33.22  >> One thing I've heard from other states is that being able to come together virtually via conferences and meetings allowed for a wider range of individuals to come together maybe, than before when things were done in person.

00:08:48.58  >> I have absolutely seen that happening. Although our meeting process with LEAs pre-dated COVID, and we were using virtual meetings for those, the necessity of everyone getting more comfortable with virtual meetings and associated technologies has made those calls much easier, because now we're not the only people they're meeting virtually with. Half of their meetings, at least, are virtual. That increased in familiarity with the virtual meeting tools has made our JFN -- Joint Federal Notification -- booking calls much more effective and easier to implement.

00:09:37.63  >> Seth, can you share some of the different virtual meeting tools that you've been using that you've found successful?

00:09:43.06  >> It very much depends on the meeting. We make it interactive by sharing screens, and walking through the reports themselves. And when it comes to digging into the data, there's very little more successful than just sharing screens and using the dashboard tools and other reports that we already have created in a manner that we're all interacting with them simultaneously.

00:10:13.42  >> I'm curious, these are very, especially with significant disproportionality and the other equity indicators are very complex, intricate calculations. Within the reports themselves, when you are having these conversations with them, can you kind of share how you sort of walked through those calculations?

00:10:31.57  >> This is actually one of the areas in which I hope to improve our reporting outputs in the coming years, and that is to provide more intuitive visualizations and data in the reports that don't necessarily dig into the complications of how to calculate a risk ratio, and whether it's a standard risk ratio or an alternate risk ratio, or the number of years used and the minimum cell size, and all these different moving parts to these calculations which are incredibly complicated. It's too easy for districts to get in the weeds with that. And although that is how we're required as a state to identify districts with significant disproportionality, it doesn't have to be the way that we engage districts in their improvement planning. And that's been something that we're working on more and more to translate what is a very complicated and unintuitive measure into something that is more accessible and comprehensible, and widely available already in a variety of formats. I don't need a district to worry about the calculation of the rate at which black students in their district are getting identified as emotional behavioral disabilities compared to non-black students. I've got that calculation down. I've already made that determination. What I need the district to do is to look at that rate and say, okay, how do we address this? How do we stop identifying students who are black with emotional and behavioral disabilities at such a high rate? What are we doing as a district that can right this ship? You don't need a risk ratio to answer that question. A risk ratio may even complicate things, and a risk ratio may look like it's getting worse for a couple of years before it starts to improve, because it's just a function of mathematics, you know? You would have to reduce a risk at twice the rate of their peers in order to just show a level risk ratio. To actually improve it, you would have to improve more than double, depending of course on what your risk ratio threshold is. And that's all stuff that I do not want districts worrying about. I don't want districts thinking about how they can game the system to move the risk ratio. The risk ratio will change if they pursue best practices eventually, and that's what we want to see.

00:13:32.59  >> I love that, because so many districts do really get so caught up in the numbers, and trying to explain that, this is what they are. We need to accept it, and now how are we going to fix it? How are we going to improve?

00:13:46.21  >> Yeah.

00:13:46.41  >> So looking forward to what you develop in the future.

00:13:50.03  >> And I understand, you know, for some people, a number doesn't become real to them until they can walk through the math themselves. And I get that. But it's also a huge headache to reproduce something that ultimately, at the end of the day, isn't what you should be looking at on a day-to-day basis. Trying to understand how the calculations are made are one thing -- I totally understand that. But it's not the bulk of your work. That's not the challenge. If that is your focus, that is an incorrect focus. Just focus on improving things for students, and that will carry over into the risk ratio as the years go by.

00:14:35.19  >> So one final question. Can you share a success story, or something you're really proud of, related to your state's data culture?

00:14:42.03  >> Sure. Recently I was on a call with a district, a new special ed director, and some things fell through the cracks the first year that ideally we would have liked to see addressed in their snapshotted data -- so there were data errors. And through the Joint Federal Notification packets and WISEdash secure and a half hour meeting -- so three resources -- that special ed director went from feeling overwhelmed and not understanding how they could have possibly been identified in that area, to fully understanding, knowing exactly what went wrong, how to fix it, moving forward, and really sort of empowered to bring this back to her staff and say, this is what we did, this is how we fix it. It wasn't even an issue of negative outcomes happening. It was an issue of incorrect reporting, that the dates didn't align for graduation exiters, and therefore students were getting counted as transferring to regular ed rather than exiting special ed with a diploma, because the exit dates were two days off. And showing that special ed director how to get there, how you can see it in the WISEdash dashboard, how to quality assure the data, moving forward in advance so it's fixed in the future -- we were able to do all of that in a half hour meeting, and the district staff was incredibly happy and pleased that they were able to understand this and piece it together. Those sorts of "aha" moments for districts, when they figure out how to use the tools that we've provided, are my success stories. That's what I like to see, because it isn't about creating something new for people. It's getting people to understand the resources that we've already put out there and how to use them. And that's what I want to see.

00:16:49.48  >> Thank you so much for joining us today, Seth. The resources and tools that you have available in Wisconsin sounds amazing, and I know other states have really learned from you, and been able to use the tools and resources you've developed, and then the TA that you're providing to your districts all in combination has really made an impact on your districts and stakeholders, and other states are learning so much for you as well. So thank you.

00:17:14.00  >> I would also add, if I may, that if any other state is interested in recreating similar reports, I have recreated reproducible examples that I am happy to share.

00:17:28.02  >> Thanks, Seth. We will make sure to have those resources available with this podcast.

00:17:33.45  >> To access podcast resources, submit questions related to today's episode, or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content. Or connect with us via the Podcast page on the IDC website, at IDEAData.org.