by Laura Leeson (Director of M&E and Strategic Development, Projet Jeune Leader)
Back in 2018, we embraced the power of beans — yes, you read that right… beans — to collect and respond to feedback from our most important constituents: our adolescent students.
Every week, students attend a timetabled class with athe Projet Jeune Leader Educator embedded in their public middle school. These classes are one of the most important components of our program — providing essential knowledge about health and puberty, and equipping adolescents with life skills around leadership, healthy relationships, self-confidence, and more.
During the 2018 school year, we adopted a new participatory evaluation tool. At the end of class, students would “vote” with beans in response to a question, placing their vote in one of three possible response “buckets.”
Using this participatory approach helped us systematically check in on our work from our students’ point of view. Did they like and understand their lessons with Projet Jeune Leader? Had they talked to their parents about what they learned in class? Did they feel that they could approach their PJL Educator if they had a concern or needed advice?
Little did we know that our bean-voting would reveal a serious issue in our program.
When we asked students if they understood that day’s lesson, we saw that comprehension varied widely across schools. We were definitely surprised… and mostly disappointed.
When we dug a little deeper, we found two contributing issues: our curriculum, and how PJL Educators were applying it.
Projet Jeune Leader’s original curriculum was developed in 2013. Structured 54 lessons (27 lessons per grade over two school years), each lesson plan shared the topic’s key messages and offered several options for game-based and interactive activities to transmit the content. Key messages followed UNESCO’s learning objectives for sexuality education. Similar to teachers of core subjects, our Educators would write their own lesson plans using the key messages and choose which activities to use.
When we developed our curriculum in 2013, we were working in just 4 partner schools. By 2018, we were working in 12 schools. As our program grew in reach, it became increasingly difficult for supervisors to assess exactly what our Educators were explaining to their students (i.e., ensure quality and non-biased information) without attending the class in-person.
The results from the bean-voting reaffirmed that a change was needed to our curriculum.
We started with an in-depth review of over 30 sexual health education curricula from renowned global health organizations around the world. We found a lot of great resources, but a few hurdles remained:
- Only a handful of curricula were adapted to PJL’s target age group (young adolescents, 10-15 years old), and the few that did exist for this age group were too complex for the corresponding educational level in Madagascar.
- Many of the activities were designed for groups of 15-20 students, not 50+ students as is typical in the public schools where Projet Jeune Leader works. Many required printed handouts — a material cost and logistical requirement which would be unfeasible at scale.
- 90% of curricula were in English.
- A large subset was designed to be delivered to one gender, but not mixed-gender groups as was the case for Projet Jeune Leader-taught classes.
We were determined to find the right solution — so we built our own. We drew ideas from a combination of external sources (citing them, where appropriate), and drew on our team’s expertise and previous lessons to create new, detailed, and scripted lesson plans that were age-appropriate and locally relevant. Like our original curricula, we made sure the courses followed a logical, chronological sequence for building knowledge, attitudes, and skills amongst adolescent students. We also took the opportunity to make our lessons gender-transformative, integrating the exciting research we had read on the need to mainstream discussions of power and gender in sexuality education to improve its effectiveness.
We wrote the lessons in the local language, Malagasy. They clearly stated instructions for how our Educators should conduct the participatory activities contained within. The handbook also provided our Educators with specific, clear, evidence-based, non-biased language and information to use when teaching their students.
Promising Results in Effectiveness and Consistency
Scripted lesson plans are still rare in sexuality education programs, even though they are increasingly recognized as a successful feature of educational curricula that improve learning outcomes.
Throughout our first year with scripted lesson plans in place, we brought back the beans to understand the impact of the change in our context.
What we found: Improved results. Bean-voting with 4,852 students throughout the year showed that 79% understood the lesson taught that day well, 19% understood it somewhat well, and only 1% did not understand it at all, a significant improvement across the board from before. There was also significantly more consistency in results across our schools.
Our Educators also expressed that the scripted lessons allowed them to focus more on their tone and energy for participatory teaching because they did not to need to come up with detailed explanations on the spot — another promising outcome.
Furthermore, bean-voting showed that 85% of 1,498 students said that what they learned in the lesson that day would help them in their lives.
We then tested the influence of the lessons on students’ knowledge, attitudes, and behavioral intentions. We saw that these factors improved among students who had received the lessons when compared to comparison students who did not receive the lessons. We have continued to see these positive effects of the lessons as we work in new partner schools.
The Unexpected Benefits of Scripted Lessons
In the years since we made the change, our scripted lesson plans have helped improve our organization’s credibility and leadership in sexuality education in Madagascar.
In 2020, we were able to gain quick authorization to expand our program from the national Ministry of Education; through the scripted lesson plans, they were easily able to understand the exact purpose and content of Projet Jeune Leader’s curricula.
In 2022, we signed an expanded three-year partnership agreement with the National Ministry of Education which established Projet Jeune Leader as the preferred technical partner for sexuality education in Madagascar. Through this partnership, we launched a new pilot — implemented in 2022-2023 — to mainstream Projet Jeune Leader’s program in additional rural public schools in direct collaboration with the National Ministry of Education and its teacher training institute.
During the pilot, government-recruited teacher-trainees charged with delivering PJL’s program expressed that they valued the scripted lessons plans because they were easy to use and cut down on their preparation time for teaching (unlike their core subjects, which required intensive preparation). Teacher-trainees also appreciated how participatory activities in the lessons enabled them to form stronger connections with their students, which had spillover effects on other classes and subjects. As it turned out, the scripted curricula became one of the key drivers of teacher-trainees’ motivation to deliver the program.
Creating Buy-In With Beans
Now, when we first launched the pilot with the teacher training institute, not everyone was convinced by the scripted lessons. Nearly all of the teacher trainers (those preparing the trainees for their core subjects) told us that they believed the scripted lessons were poor pedagogy, especially if trainees looked like they were just reading the information out loud. They thought trainees should be able to add their own stories and experiences and examples to the lesson content, and not be restricted by the pre-determined explanations.
It was only when we sent the trainers out to observe the trainees in-person and collect results through — you guessed it —bean-voting, that we saw their opinions begin to change.
Trainers told us that the favorite part of their class observations was doing the bean-voting with students at the end of the class. Through this, trainers saw first-hand what students understood and valued about their Projet Jeune Leader courses. Students’ positive feedback convinced most of the teacher trainers that scripted curricula were appropriate and effective in the context of the program and students’ learning.
The Future of Participatory Evaluation
Collecting and responding to our constituents’ feedback has become an integral part of our program. It helps us learn what’s working and what’s not, and course correct (literally and figuratively!) — making sure we are doing the best work we can and responding to what our constituents want. Bean-voting is just one of many feedback mechanisms we have embraced to do this, and we expect that we will always rely on those beans as we strive to effectively serve every student in Madagascar.