Editor's Note: Whenever we talk about PBIS, we talk about implementation. We can't talk about implementation without talking about fidelity. So, what is implementation? How do we know when we're doing it right? How can we do it better? This is the second in our series exploring the basics of implementation and how to assess your own to get the outcomes you want to see. Check out the first article exploring seven keys to every implementation here.
Two years ago, I took every piece of clothing I owned off the hangers, out of the drawers and flung everything on my bed in a giant pile. I measured that pile. It was shocking-feet-wide by embarrassing-feet-tall. Large. There were shirts with the tags still attached to the sleeve. I owned six non-descript gray sweaters. I found three pairs of jeans held together by what I can only assume were angel’s whispers and magic. If it had a stain, hole, or tear, it went in the garbage. The rest went to my friends or donated to one of two local shelters. In the end, not only did I have clothes that fit me, I had 22 things I loved and really wanted to wear all the time.
Fast forward to January 1, 2019, Netflix released a brand new show called Tidying Up with Marie Kondo, based on her New York Times best-selling guide, The Life-Changing Magic of Tidying Up. Marie Kondo has devoted her life to turning the typically painful process of decluttering our homes into a joyful art called the KonMari method. Here’s how it works:
In her Netflix show, we meet people whose lives have become chaotic, or confused, or otherwise controlled to some extent by the clutter in their homes. Marie Kondo comes in without judgement, shares her excitement for tidying and her method for accomplishing it. She leaves their homes cleaner, more organized, and full of the things they love than ever before. It’s life changing. Having been through my own process of decluttering, I know just how empowering downsizing can be.
What does all of this have to do with measuring implementation fidelity? Clutter, of course.
Open up PBIS Assessment to take a fidelity survey and you see nine different options. If you’re measuring Tier I, you have your choice of the
- School-wide Evaluation Tool (SET)
- Benchmarks of Quality (BoQ)
- Self-Assessment Survey (SAS)
- Early Childhood Benchmarks of Quality (ECBoQ)
- Team Implementation Checklist (TIC)
Teams measuring their Tier II and III implementation can take the
- Individual Student Systems Evaluation Tool (ISSET)
- Benchmarks for Advanced Tiers (BAT)
- Monitoring of Advanced Tiers Tool (MATT)
- Self-Assessment Survey (SAS)
Collectively, these surveys measure each tier and every system of your school’s PBIS implementation. That’s just it, though. Getting the full picture of your implementation requires a collective set of surveys. Schools never take every survey, but you might find they take three or four of them during the year. With an array of surveys, which one do you take? How do you know which ones are right for your setting? What if you want to assess more than one tier at the same time? Enter: the School-Wide PBIS Tiered Fidelity Inventory (TFI). Yes, it’s the ninth survey on the list, but this one is different.
School teams along, with guidance from their coach, take the TFI together to measure a school’s PBIS implementation across all three tiers, all in one survey. You use it to check out your initial efforts, create action plans for improvement, and monitor your progress moving forward. The TFI takes the eight other surveys and unifies them into a single assessment. Ms. Kondo would be proud.
Before releasing the TFI to the masses, researchers tested it three different ways.[1] They had 12 experts review the survey’s content. (The experts think it’s great.) They had 15 school teams test it to see how long it took to complete the TFI (It takes 15 minutes per tier) and whether it was easy to do. (It is.) They distributed the survey to 789 schools across seven states to find the difference between taking the TFI with and without a coach (Coaches make everything better) and how closely it correlates with other fidelity measures (Really closely).[2]
Great! So, experts and schools agree: The TFI is a solid survey. It takes a short amount of time to complete and the results aren’t so different from other, longer, more resource-intensive surveys. I bet schools across the country have held their last BoQ in their hands, thanked it for its years of service, carefully set it in a discard pile, and replaced it on the shelf with the TFI…right?
Not so fast. Remember the clutter? Of the 11,785 schools who took a TFI on PBIS Assessment last school year, 5,734 of them also took another fidelity assessment.[3]
Sometimes we’re suckers for doing things the way we’ve always done them. Assessing fidelity is no different. In fact, when researchers asked schools what they thought about the TFI, team members said they’d definitely use it, but they’d probably keep on using another fidelity measure, too. Schools hang on to older surveys for the same reason I still have a hand mixer and a stand mixer:
- I don’t know when I’ll use a hand mixer, but I know I’ll probably need it someday.
- Both appliances do the same thing, but what about those times when I need to do that thing just a little differently?
- Having a hand mixer doesn’t really hurt anything and it makes me feel good knowing it’s there if I need it.
Except, an extra appliance takes up space, and all those extra surveys take time.
If your school has been on the fence about transitioning to the TFI, or you haven’t been able to let go of your favorite fidelity measures just yet, ask yourself: WWMKD – What Would Marie Kondo Do?
Pick a Day and Do It
Would you clear out your closet by getting rid of one t-shirt at a time, every day? Probably not. Do it that way and you might finish tidying by 2020. Marie Kondo suggests we start all at once, completely. If you haven’t taken the TFI and you don’t know when to do it, you should know that now is a good time. Connect with your district’s PBIS coach and schedule a day to work through the survey together. Their expertise and perspective on your implementation is critical to getting results you can use for action planning. Based on research, taking the TFI this way lets you reliably use it in place of other fidelity assessments you planned to take at the end of the year. The first time you take the TFI, take it across all three tiers, even if you’re not implementing all of them right now.
Here’s a tip for PBIS Assessment schools: If you have more than one team implementing different tiers, each team should assess their tier on paper first. Give the final scores to one person and have them enter each tier, all together as one TFI survey on PBIS Assessment.
Organize Yourself by Category: Fidelity vs Outcome
Our next step to decluttering is to stop cleaning room-by-room and start tidying by category. In the case of surveys, we have two categories: Fidelity measures and outcome measures. One fidelity measure you could stop taking right now in favor of an outcome measure is the SAS. Lots of schools take the SAS because they value the input from the entire school staff. The SAS measures fidelity of implementation – something with which most school staff are unfamiliar. Stop asking them about fidelity and start asking them about something they’re more comfortable talking about – school climate. The School Climate Survey is an outcome measure giving you a new way to solicit feedback from staff members in your building about how your implementation affects their classrooms. Spoiler alert: We're going to help you with this survey in our next Teach by Design article...Stay tuned!
Keep What Sparks Joy
In this part of the process, we pick up an item and ask ourselves whether it brings us joy. It’s a difficult thing to describe, but you’ll know it when it happens. It’s the difference between the feelings you get when you write with your favorite pen compared with the free one you picked up at a conference.
Keeping our favorite assessments simply because we like them only maintains the clutter. Instead, consider which data you need the most as you monitor your team’s progress. School’s taking the BoQ, I’m looking at you. The BoQ has 53 items to score, seven of which relate to the classroom setting. The TFI only has two questions about classroom implementation. If your implementation is super focused on those classroom settings, the BoQ might spark all the joy for your team. On the other hand, if your team is strapped for time lately, the TFI and its 15 questions might be just what you need to get excited about evaluating your Tier I fidelity this year.
We know the number of resources available for checking your PBIS implementation is overwhelming. If you don't know where to start, or the amount of time you spend taking surveys is unmanageable, the TFI is a perfect tool to try. Starting new routines can be tricky. These are just three ways to help you transition away from using many surveys toward using just one. Pick a day and commit to taking it. For more information about the TFI, check out all the resources we've put together to help you submit your first one before the end of the year.