FINDINGS OF TWO STUDIES VALIDATING THE GLOBAL CITIES EVALUATION TOOL
The Need for Global Competency Education
“Being born into a global world does not make people global citizens. We must deliberately and systematically educate our children in global competence,” said Andreas Schleicher, Director for Education and Skills at OECD which co-published the Global Cities Codebook for Global Student Learning Outcomes. “It’s fair to ask how you can teach something if we can’t assess it. We now have concrete evidence that the Codebook serves as a practical tool for educators to develop curriculum and measure what their students are learning.”
How to Make it Easier for all Educators to Teach and Assess Global Competency
The Codebook for Global Student Learning Outcomes developed by Global Cities in partnership with Harvard Graduate School of Education Project Zero’s Out of Eden Learn (now known as The Open Canopy), is based on data from the Global Scholars program operated by Global Cities. This evaluation tool (the Codebook) is designed for researchers to examine data sets of posts and replies from virtual exchange e-classrooms, and for program designers and classroom educators to teach global competency and evaluate whether students are learning. The Codebook captures 55 observable indicators of four core Global Student Learning Outcomes: Appreciation for Diversity, Cultural Understanding, Global Knowledge and Global Engagement.
Validating the Tool (the Codebook) Through Two Studies
Global Cities commissioned two studies to test whether the tool (the Codebook) could be used to evaluate global competency learning in other global competency virtual exchange programs. One study looked at curricula created by AFS Intercultural Programs, specifically Global You Changemaker and Global Up Teen. The other study looked at Harvard Graduate School of Education Project Zero’s The Open Canopy curricula known as Planetary Health and Remembering the Past. The research validated Global Cities global competency evaluation tool, demonstrating that it can be used to identify evidence of global competency learning in samples of work created by students in virtual exchange programs with different models, curricula, and student populations.
Access to Full Reports
Here is the link to the AFS report, Measuring Global Learning Among Students in AFS Virtual Exchange Programs. This research was conducted by Ester R. Fuchs, Professor of International and Public Affairs and Political Science at Columbia University School of International and Public Affairs (SIPA), and Rebecca Casciano who studied sociology and demography at Princeton University and after earning her doctorate there, has led the evaluation research organization Glass Frog Solutions. They prepared the report with assistance from Linda Stuart, Director of Global Education Innovation at AFS Intercultural Programs, and three principals at Global Cities, Inc., Anissa Bazari, Executive Director for Strategy, Marjorie B. Tiven, Founder and President, and Megan Wilhelm, Chief Program Officer.
Here is the link to the Open Canopy report, Analyzing Student Work from The Open Canopy Remembering the Past and Planetary Health Learning Journeys Using the Global Cities Codebook. Research for the report was conducted by Dr, Liz Dawes Duraisingh, Lecturer on Education, Co-Director Project Zero at the Harvard Graduate School of Education; Dr. Sarah Sheya, Project Manager, Project Zero at the Harvard Graduate School of Education; Dr. Christina Smiraglia, Learning Researcher and Instructor, Project Zero at Harvard Graduate School of Education; and Rebecca Casciano who studied Sociology and Demography at Princeton University and after earning her doctorate has led the evaluation research organization Glass Frog Solutions. The report was written by Dr. Rebecca Casciano, Glass Frog Solutions; Dr. Liz Dawes Duraisingh, Project Zero; Dr. Ester R. Fuchs, Professor of International and Public Affairs and Political Science at Columbia University School of International and Public Affairs (SIPA); Dr. Sarah Sheya, Project Zero; and Dr. Christina Smiraglia, Project Zero. Technical assistance for the research and report preparation was provided by Anissa Bazari, Executive Director for Strategy at Global Cities, Inc.; Marjorie B. Tiven, Founder and President of Global Cities, Inc.; and Megan Wilhelm, Chief Program Officer at Global Cities, Inc.
Selected Findings from Measuring Global Learning: Among Students in AFS Virtual Exchange Programs
By using the Codebook to identify the types of curriculum content and activities that promote global learning, this research also provides evidence of the effectiveness of AFS’ global learning model. Important findings include:
Many students were able to discuss their own and others’ cultures. Modules that explicitly prompted such discussions of culture were clearly more likely to elicit more detailed cultural descriptions. However, simply asking students to describe their communities also encouraged some students to talk about their cultures or other cultural groups in their communities. In both programs, we saw many more examples of cultural “awareness” than we did cultural “understanding.” There were cases in which students may have had a more nuanced understanding of their own culture, but their posts lacked sufficient detail to be coded for the more advanced indicators.
Structured opportunities for exchange led to students listening to and interacting respectfully with each other. The virtual discussion boards that are part of both the Global Scholars and AFS program models created a learning environment conducive to demonstration of Appreciation for Diversity indicators. However, we observed that prompts that explicitly told students to respond to two or three of their peers’ posts were more likely to promote respectful interactions among students. The GYC curriculum modules contained more of these prompts, and consequently we saw many more examples of respectful dialogue in the GYC sample.
Two of the most frequently coded indicators in the GYC curriculum related to students’ recognition of their capacity to effect local, regional, or global improvement and willingness to do so, attitudes critical to Global Engagement. This is likely a result of the curriculum having a clear focus on students creating change in their communities by developing a prototype that would address a social or global issue. This project-based, action-oriented design may have prompted students to recognize their own capacity and show a willingness to take action. However, the activities in this portion of the curriculum also gave students clear guidance and templates to describe their project’s goals and benefits. These templates may have helped students articulate both their goals and plans for reaching them.
While the GYC curriculum included projects that provided explicit opportunities for students to take action on global issues, the GUP Teen curriculum demonstrates that even a hypothetical discussion of future actions can help build student efficacy. Students’ recognition of their capacity to impact local, regional, or global issues was generally lower in the GUP curriculum, which is not project-based. However,, one-third of all posts in the Taking Action module in the GUP Teen curriculum showed evidence that students recognized their capacity to make change. This module attempted to bridge the gap between the personal and interpersonal skills developed in the prior modules and a student’s next steps as they prepared to complete the program. Asking students to consider how they might get involved in the future was effective in encouraging them to recognize their own capacity to make changes in their community and resulted in a higher prevalence of global engagement indicators than other modules of the curriculum.
The research also points to ways AFS can improve their students’ global learning through simple modifications in program and curricular design. These include:
Foster Respectful Dialogue Through Peer Interaction. Even in virtual environments, interaction is essential for practicing respectful listening and dialogue. Encouraging students to reply to each other’s posts provides more opportunities for them to practice and demonstrate these essential skills by fostering organic interaction where diverse perspectives can be both exchanged and valued.
Encourage Deeper Reflection. When providing prompts for writing or discussion, ask students to explain the significance of what they are sharing, not just to describe or provide facts. This encourages them to delve deeper into the origins and meanings behind the concepts they discuss, promoting a more thoughtful engagement with the material and ultimately a better understanding of global or cultural issues.
Provide Scaffolds and Templates for New Concepts. Since students may be encountering unfamiliar ideas and concepts, providing explicit scaffolds and templates can support their ability to engage with the material effectively. These tools help students organize their thoughts and structure their ideas, ensuring they can better navigate complex topics.
Empower Students by Asking for Involvement. Project-based curricula that provide structured opportunities for student action can be effective in developing student recognition of their capacity and willingness to contribute to local, regional, or global improvement. In global competency programs that do not include projects, it can be beneficial to explicitly ask students how they would like to contribute or how they see themselves getting involved in addressing global challenges in the future. This inquiry can help them develop a sense of personal efficacy and ownership over their role in tackling global problems.
Selected Findings from Analyzing Student Work from The Open Canopy Remembering the Past and Planetary Health Learning Journeys Using the Global Cities Codebook.
By using the Codebook to identify the types of curriculum content and activities that promote global learning, this research also provides evidence of the effectiveness of Open Canopy’s global learning model. Important findings include:
The findings from this analysis validate the existing Open Canopy program design in many ways. Students exhibited many different indicators of global learning in their work, often in ways that reflected the curriculum content and goals of the respective learning journeys. Many of the Remembering the Past (RTP) posts were particularly rich and were coded for multiple global learning indicators, demonstrating that students can exhibit many indicators at once. The students participating in RTP were older than those participating in Planetary Health (PH) which accounts, to some extent, for their more sophisticated and more detailed posts. Specifically, across the 1,200 student discussion board posts in the sample from both Open Canopy programs, 98% were coded for at least one indicator of global learning, including 97% of PH posts and 99% of RTP posts.
The three most prevalent indicators in the data are skills and behaviors fundamental to the learning that can take place in virtual exchange (7-AD Ability to listen to others and discuss issues in a respectful and unbiased way , 15-AD Interacting with people of different backgrounds positively and respectfully, and 48-GE Using digital tools to learn from and communicate with students from cities around the world), supporting the efficacy of virtual exchange for providing students with critical opportunities for learning and demonstrating global competency. These virtual exchange competencies relate to students’ ability to interact with diverse peers respectfully and positively, so it is encouraging that we found many examples of students exhibiting these skills. Another important aspect of the virtual exchange context is the ability to directly interact with peers around the world. This learning environment makes it possible for students to exercise their ability to listen, respectfully discuss, and interact positively with people of different backgrounds. This suggests that the virtual exchange opportunity helped support students’ ability to interact with peers of different backgrounds positively and respectfully.
Different topics and activities elicit demonstration of different student learning outcomes and indicators. For instance, the two learning journeys had a noticeable difference in the prevalence of indicators related to culture and differences between people, with some indicators more prevalent in RTP compared to PH. We hypothesize that this difference in prevalence across the two learning journeys is likely related to their topics. RTP is entirely focused on cultural practices, while PH is focused on environmental health, so it would make sense that students would demonstrate more knowledge and attitudes related to culture in the RTP learning journey and students would demonstrate more knowledge and attitudes related to global issues in the PH learning journey.
The analysis highlights specific areas where Open Canopy students are developing global competence. For example, students did well in terms of the interactive element of the program, with 7-AD Ability to listen to others and discuss issues in a respectful and unbiased way, identified in 63% of posts in the sample and 15-AD Interacting with people of different backgrounds positively and respectfully identified in 59% of the posts. The curricula are designed to facilitate meaningful and respectful intercultural exchange among diverse youth, and the high prevalence in the sample is validation of their success in building these skills. Likewise, the high prevalence (61%) of 1-AD Awareness of how one’s life and the lives of others are influenced by broader cultural and historical contexts among students participating in RTP suggests that the curriculum has been successful in encouraging students to think about the roles of culture and history in their own and others’ lives.
The research also points to ways The Open Canopy can improve their students’ global learning through simple modifications in program and curricular design. These include:
The Open Canopy platform could take greater advantage of virtual exchange to boost students’ performance on some of the indicators. Some of the knowledge indicators can be promoted through the very experience of interacting with peers from around the world. For instance, 31-GK Understanding that global issues are borderless and affect everyone, could be promoted by having students reflect on the similarities and connections they are seeing across student posts in different geographic locations.
The global learning indicators could also be very useful in helping to design strong learning objectives for new curricula or to clarify the priority objectives for existing curricula. This is where the comprehensive nature of the Global Cities framework is helpful in that it offers teachers and other curriculum designers an array of indicators to choose from, some of which may not have initially been in the curriculum, but which could explicitly be added to enrich students’ learning. For example, The Open Canopy might want to design a new learning journey that prioritizes developing students’ understanding and appreciation of different perspectives by designing activities that are likely to encourage those understandings, for instance, by incorporating interviews and community-based surveys.
The research findings also point to ways curriculum assignments can help move students from weak to strong demonstrations of a global learning indicator. It could be helpful to build time into the learning design for students to unpack what different quality student examples look like or to assess their own work or that of their peers in terms of a particular indicator. They could be invited to explore ideas for strengthening their own and others’ work.
Implications for Measuring Student Global Competency Learning
The analysis of AFS and Open Canopy programs shows how the Global Cities framework and Global Competency Evaluation Tool (Codebook) can be used to assess student global learning for different programs. It can then help teachers and program designers iterate on or fine tune their existing curricula to promote global competence or consider ways to boost their students’ performances from weak to strong demonstrations of various global learning indicators. This research also provides helpful guidance to those seeking to develop new curricula by enabling them to intentionally select an array of indicators that could allow for a well-rounded learning experience related to each of the four main global learning outcomes in the Codebook’s framework.