What Do We Know About CBE Through Evaluations?

This is an updated article originally published in 2019.

Where’s the beef?

Looking Under the Hood of Competency-Based Education

Are you old enough to remember the Wendy’s slogan? It’s now a more general question about the value of any idea. Although I am an advocate for personalized, competency-based education, I also think we should be asking Where’s the beef? . We need feedback to know if we are designing a system that is benefiting students. We need feedback to understand what a high quality modern school looks like.

After nearly a decade of effort transitioning out of the traditional system’s distorted structure, which was developed at the turn of the 20^th century, toward a system that is designed for all students to successfully learn, we have little formal research to support that this is a better way of educating students. There are lots of reasons for this. I don’t offer them as excuses but as a necessary step in reflecting on our work together in order to correct this situation. Please note: At the bottom of this article is a list of research that has been completed to date.

Why We Are Operating Without Research to Guide or Affirm Us

Foundation Designed: Too often, foundations develop frameworks internally and then evaluate schools according to those frameworks. Thus, the research is only being used to validate their framework, not necessarily seeking to understand what the district or school is doing to help improve learning and/or achievement.

Foundations also repeatedly make the mistake of evaluating schools that have been operating or using a new model for less than three years. We all know that it takes a while to get a new school up and running, especially if it is using a new model. Schools that are transitioning have the challenge of getting all the staff on board and moving in the right direction. I would doubt that research on schools with only one or two years of seeking to implement competency-based education would ever show consistent evidence of improvement and learning throughout the school or district.

Measuring What Matters: There are often disconnects between what researchers are measuring and what competency-based schools are trying to do. It is important that the researchers look at all three sets of student outcomes: application of knowledge and skills; lifelong learning skills; and traditional academic content knowledge. However, there is more complexity in trying to evaluate all three sets of outcomes, and I don’t believe any of the research to date has tried to do so. This raises the question: Have we simply underfunded the research and evaluation so that we evaluate what we can without asking the important questions?

Walk the Talk or Formative Before Summative: Competency-based education is an enormous transition. First, you’ve got to shake out the beliefs of the traditional system. Second, you have to align everything with the research on learning and development. We’ve had a ton of quality issues when districts and schools jump into the structural changes without thinking through the implications for culture and pedagogy.

What we’ve needed are formative evaluations that could help districts improve and for the field to better understand the process of the transition. But we are always jumping into a “does it work” mode rather than a “what are we learning” mode.

Recommendations

For us to move through this next stage and develop high quality models in every state, we need to do the following:

Co-design: All research and evaluation should be developed with districts and schools.
Walk the talk: Invest in formative evaluation.
Value learning and achievement: It’s important to look at whether students are developing the keys to learning (meta-cognition, growth mindset, self-regulation, agency) and whether they are learning to apply knowledge and skills, in addition to test scores.

Examples of Research to Date About CBE

Implementation of Proficiency-Based Learning in Maine: Understanding Implementation of Proficiency-Based Education in Maine by Students at the Center (2018) follows up on the 2012 Maine legislation requiring all students to graduate from high school with a proficiency-based diploma. Education Development Center partnered with ten districts in rural Maine to investigate exposure to proficiency-based education, including links to engagement and academic outcomes as well as the nature of implementation.

Implementation has largely focused on identifying graduation standards and implementing new proficiency-based grading practices, with traditional classroom practices still fairly commonplace.
This raises a question: If Maine schools had spent more time improving instruction (and culture) rather than just focusing on setting standards and changing grading, would Maine still be on a statewide pathway toward proficiency-based diplomas?

Other reports about Maine’s implementation include:

Looking at Instructional Practices: Looking Under the Hood of Competency-Based Education: The Relationship Between Competency-Based Education Practices and Students’ Learning Skills, Behaviors and Dispositions by AIR (2016). This is one of the strongest pieces of research. However, its focus on CBE instructional practices means that there were many practices used in non-CBE schools. Effective instruction is the same in a CBE school or in a traditional one. Check out the CBE 360 Survey Toolkit (2017).

MRL produced an interesting study, Measuring student progress and teachers’ assessment of student knowledge in a competency- based education system that indicated that 43–47 percent of the students who were behind their traditional grade levels completed their performance levels in three or fewer quarters, less time than it would take in a traditional education system.

Personalized Schools: Rand’s Informing Progress (2017) was a study on the early implementation of new personalized schools as part of Gates’ efforts to evaluate its own investments. The schools had only been operating two years or less. Some of the schools had some elements of competency-based education. However, CBE was never well-defined or thought about consistently across schools or the study. Based on my own conversations with staff at NGLC and some of the schools, it is likely that competency-based education was thought of as adaptive educational technology that differentiated learning. It is difficult to disentangle implementation/execution issues from design and very, very hard to gain insights about CBE.

Pilot CBE Programs: Rand also produced Competency Based Education in Three Pilot Programs on behalf of Gates Foundation based on their internally developed definition. They studied three very different competency-based initiatives in very different stages of development. One of the initiatives was a brand new pilot program in a school. The school wasn’t transforming itself and the district that had no intention of becoming competency-based. There is very little reason to believe that this type of a pilot, without being seated within a schoolwide competency-based structure, would be able to generate many benefits to students within their first year of operation.

RISC Model: RISC vs. Non-RISC Schools: A Comparison of Student Proficiencies for Reading, Writing, and Mathematics (2010) by Marzano Research Laboratory found that the RISC framework significantly impacts student achievement. Researchers compared 7 RISC districts and 8 Non-RISC districts in three states. Study findings:

The odds of a student in a RISC School scoring proficient or above on state tests are 2.3 times greater for reading, 2.5 times greater for writing, and 2.4 times greater for mathematics than the odds of a student scoring proficient or above on state tests at a Non-RISC School.
Compared to students in Non-RISC Schools, students in RISC Schools are 37% more likely to score proficient or above on state tests for reading, 54% more likely to score proficient or above on state tests for writing, and 55% more likely to score proficient or above on state tests for mathematics.
In addition, the degree of RISC model implementation was found to relate to the number of students scoring proficient or above on state tests in reading and writing. The odds of a student in a High-RISC School scoring proficient or above in reading and writing were found to be about two times larger than the odds of a student in a Medium-RISC School scoring proficient or above.

Research on Earlier Model of Mastery Learning: In the 1980s, there was an interest in an instructional approach called Mastery Learning. There are common themes in this earlier version and today’s competency-based approach. However, we now believe that the entire school has to change to fully support learning, not just instruction.

It should also be noted that in John Hattie’s work, Mastery Learning falls in the middle of his comparison of effect size of different educational strategies. Yes, there was positive impact from the approach but other strategies are much, much more important.

Below are some of the findings from a review of 27 evaluations of mastery-based learning in Synthesis of Research and the Effects of Mastery Learning in Elementary and Secondary Classrooms by Thomas Guskey and Sally Gates published in 1986:

Achievement results are overwhelmingly positive, but vary greatly from study to study.
Although students at all levels appear to benefit from mastery learning, effects are somewhat larger in elementary and junior high school classes than at the high school level.
Although applicable across subject areas, effects in language arts and social studies classes are slightly larger than those attained in science and mathematics classes.
Students tend to retain what they have learned longer under mastery learning both in short-term (2-3 weeks) and longer-term (4 months) studies.
Students are engaged in learning for a larger portion of the time they spend in mastery classes and require decreasing amounts of remedial (corrective) time over a series of instructional units.
Students in mastery classes develop more positive attitudes about learning and about their ability to learn.
Teachers using mastery learning develop more positive attitudes toward teaching, higher expectations for students, and greater personal responsibility for learning outcomes, but may experience diminished confidence in their teaching skills.

Research and Reports from Students at the Center

Students at the Center, an incredible resource hub, is developing research on a four-part framework of student-centered learning. Competency-based progression is one of the elements in the framework. However, it is not clear if this is solely meant as an instructional strategy or if it suggests a school- or district-wide approach of understanding competency education. As far as I can tell, effective instruction is always valued. However, if schools pass students on at the end of the year without a plan on how to repair gaps and ensure they learn all of the learning goals, we are still allowing the sorting and ranking functions to dominate.

Learning With Others: A Study Exploring the Relationship Between Collaboration, Personalization, and Equity by AIR (2018) explored collaboration as a strategy to personalize learning for diverse student groups. Focusing on grades 9-12 in four high schools, the study looked at connections between collaboration in student-centered learning classrooms and student outcomes and how these differed by race and ethnicity.

Student reports of having opportunities for high-quality collaboration were strongly associated with positive classroom experiences and higher engagement, motivation, and self-efficacy.

Maximizing Student Agency: Implementation of Student-Centered Learning Approaches (AIR 2018) is a partnership with New Tech Network schools to create multiple opportunities to design, test, and revise teacher practices as part of a Networked Improvement Community (NIC). The NIC’s goal was to develop a menu of effective teaching practices in support of student-centered learning geared toward student ownership of learning or agency. The study also looked at how student agency was related to academic outcomes and how these varied for students from different racial and ethnic backgrounds.

Abolishing the Phrase “I’m Not a Math Person” by Students at the Center investigates how to improve student agency and learning outcomes in math — particularly for students from traditionally marginalized groups. With an improvement science framework, the collaborative tested, refined, and spread “high-leverage” practices that reframed mathematical struggle as learning and engaged students in collaborative problem-solving.

There are seven student-centered practices that can make a difference for students. One important point is to make sure that students understand that there are different ways to solve problems, not just one that makes the others wrong.

– – –

I know there are many more studies on personalized learning. Many have a very strong focus on educational technology. I’m not sure anything has been done or could be done to understand the overall approach to personalization (is it only technology? does it includes strategies such as voice and choice to improve motivation? does it include differentiated instruction? does it embrace the idea of active learning and agency?) and recognize which aspects are making a difference. If you know of studies that are valuable please let me know. In the meantime, my recommendation in making decisions is to always check John Hattie‘s list of effective practices along the way. (Just Remember: He includes mastery learning but its a very discrete and rigid instructional practice of making sure students understand before they move on to the next step. That’s different than ensuring students are learning as an overall school design. However, it’s not high on the list and we should all pay attention to that.)

Do you know of other research that informs our work? Let me know!