DEBUGGING FAILURE

"One of the American midfielders,
I think it was Michael Bradley, made a bad pass.
A lazy, diagonal pass.
One of the Turkish players picked it off.
Now Turkey’s on the attack.
They’re moving down the left side, they’re moving the ball in.
I’m thinking to myself, This all began with a bad pass.”

— George Vecsey (quoted in Grantland)

When significant events happen, whether good or bad, we devote considerable time to understanding why they occurred. This happens publicly through newspapers, film, and television, and personally through private thoughts and interpersonal dialogue. Gottschall‎ (2012), in his book, The Storytelling Animal, lightheartedly refers to this as Sherlock Holmes Syndrome. We create stories that explain the outcomes in our lives, reasoning backwards. Studied under the umbrella of attribution theory in academia, these stories can shape how hard we work, how much we persist, and what we do differently in the future.

Debugging Failure in Computer Science

Through an NSF-funded collaboration between 9 Dots, UC Berkeley (Dor Abrahamson), and UCLA (Noel Enyedy and Francis Steen), the Debugging Failure project revolves around the design, implementation, and evaluation of a computer science education workshop aimed at fostering a culture of productive failure practices among elementary and middle school students. In order to learn how students can make the most of productive failure, we are studying how a community of teachers, students, software developers, and researchers understands and shapes its practices around telling stories, assigning fault, and fostering agency during the common experience of encountering bugs in computer code. Our team is implementing cycles of design-based research around four elements of the coding workshop: setting new norms around encountering, interrogating, and practicing expert debugging practices; designing arts-based inquiries into failure and success; leading instructor education workgroups focused on noticing the structure of failure stories and planning discourse-based responses; and building coding software that gives students metadata on their struggles and provides authentic debugging resources. Our data sources stretch across students' ways of participating in coding, their reflections on their coding experiences, and the artifacts they produce along the way. Through micro longitudinal case studies, content analyses, and ethnomethodological conversation analyses, we are making progress understanding how affect, storytelling, play, peer interactions, arts-based reflections, and instructor support carve out a generative debugging culture. 

DeLiema, D., Hufnagle, A. S., Ovies-Bocanegra, M. (2024). Contrasting stances at the crossroads of debugging learning opportunities. British Journal of Educational Psychology. Special issue on learning from errors. DOI:10.1111/bjep.12666

Fong, M. M., DeLiema, D., Flood, V. J., & Walker-van Aalst, O. (2023). Contesting sociocomputational norms: Computer programming instructors and students’ co- operative stance-taking around refactoring. International Journal of Computer-supported Collaborative Learning. DOI: 10.1007/s11412-023-09392-2

Ryan, Z., & DeLiema, D. (2023). Reflections on sustained debugging support: Conjecture mapping as a point of departure for instructor feedback on design. Instructional Science. 10.1007/s11251-023-09629-5

DeLiema, D., Kwon, Y., Chisholm, A., Williams, I., Dahn, M., Flood, V., Abrahamson, D., & Steen, F. (2023). A multi-dimensional framework for documenting students’ heterogeneous experiences with programming bugs. Cognition & Instruction, 41(2), 158- 200.

Dahn, M., & DeLiema, D. (2020). Dynamics of emotion, problem solving, and identity: Portraits of three girl coders. Computer Science Education, 30(3), 362-389.

DeLiema, D., Dahn, M. Flood, V. J., Abrahamson, D., Enyedy, N., Steen, F. F. (2020). Debugging as a context for collaborative reflection on problem-solving processes. In E. Manolo (Ed.), Deeper Learning, Dialogic Learning, and Critical Thinking: Research-Based Strategies for the Classroom (pp. 209-228). New York, NY: Routledge. (pdf)

Dahn, M., DeLiema., D. & Enyedy, N. (2020). Art as a point of departure for storytelling about the experience of learning to code. Teachers College Record, 122(8). (pdf)

DeLiema, D., Abrahamson, D., Enyedy, N., Steen, F., Dahn, M., Flood, V. J., Taylor, J., & Lee, L. (2018, April). Measuring debugging: How late elementary and middle school students handle broken code. In D. A.-L. Lui & Y. Kafai (Chairs & Organizers), Measuring making: Methods, tools, and strategies for capturing learning, participation, and engagement in maker activities. Symposium conducted at the annual meeting of the American Educational Research Association, New York City.

Kafai, Y., DeLiema, D., Fields, D. A., Lewandowski, G., & Lewis, C. (2019). Rethinking debugging as productive failure for CS education. In Proceedings of the ACM Special Interest Group on Computer Science Education. Minneapolis, MN: ACM.

Flood, V. J., DeLiema, D., & Abrahamson, D. (2018). Bringing static code to life: The instructional work of animating computer programs with the body. In J. Kay & R. Luckin (Eds.), "Rethinking learning in the digital age: Making the Learning Sciences count," Proceedings of the 13th International Conference of the Learning Sciences (Vol. 2, pp. 1085-1088). London: International Society of the Learning Sciences.

Flood, V. J., DeLiema, D., Harrer, B. W., & Abrahamson, D. (2018). Enskilment in the digital age: The interactional work of learning to debug. In J. Kay & R. Luckin (Eds.), "Rethinking learning in the digital age: Making the Learning Sciences count," Proceedings of the 13th International Conference of the Learning Sciences (Vol. 3, pp. 1405-1406). London: International Society of the Learning Sciences.

Aalst, O. W-V., DeLiema, D., Flood, V., & Abrahamson, D. (2018, May). Peer conversations about refactoring computer code: Negotiating reflective abstraction through narrative, affect, and play. Paper presented at the Jean Piaget Society Annual Meeting, Amsterdam, The Netherlands.

During summer 2018, two of my collaborators, Angela Kwon (UC Berkeley) and Andrea Chisholm (UC Berkeley), were looking at video data of students coding, with the goal of understanding whether students had addressed the “cause” of the bug in their coding solution. Angela and Andrea found themselves puzzled by the example covered in the video above. We took this as a sign that something was awry about our understanding of the causes of bugs. Since then, I have been working with Jeff Bye (UMN) and Vijay Marupudi (UMN) to build and analyze a collection of these moments. What we’ve learned so far is that responding to broken computer programs is often characterized as finding the bug causing the problem. However, even with “simple” program breakdowns, there are a wide range of factors that could be positioned as failure. As such, teachers and students notice, represent, and explore only a subset of possible discrepancies, causal factors, and interventions during a given debugging session. We have been unpacking this active dynamic of partial discrepancy noticing, causal modeling, and intervening by weaving together research on causal reasoning (disjunctive vs. conjunctive causation) and debugging. Our argument is empirically rooted in moment-to-moment, multimodal interaction analyses of teachers and middle school students working together to debug. Our first paper on this topic was published in ACM’s Transactions on Computing Education (see below). The analysis is starting to raise actionable considerations for pedagogy aimed at robust learning following failure.

DeLiema, D., Bye, J., & Marupudi, V. (2024). Debugging pathways: Open-ended discrepancy noticing, causal reasoning, and intervening. Transactions on Computing Education. DOI: http://dx.doi.org/10.1145/3650115

We are working with elementary and middle school CS educators to unpack the pedagogical implications of these findings in a small research-practice partnership, and have started to share some of the work coming from this collaboration.

Wilson Vasquez, A., DeLiema, D., Goeke, M., & Bye, J. (2023). Debugging debugging pathways: A research-practice partnership in K-8 computer science education. In 17th International Conference of the Learning Sciences (ICLS) 2023 (pp. 2276-2279). Montreal, Canada: International Society of the Learning Sciences.

Routing around moments of struggle in math tutoring

Some of my work focuses on what happens in natural conversation when students encounter and work though difficulties on math homework with the help of tutors. What do students blame, what chances do they think they have of succeeding, what do they try next, and how are these thoughts organized in sequences of conversation turns with tutors? Below, a tutor catches a mistake, points it out in the visual field, describes the flow of events up to the sticking point, and then hands the floor over to the student.

DeLiema, D. (2017). Co-constructed failure narratives in mathematics tutoring. Instructional Science. DOI: 10.1007/s11251-017-9424-2. (link to full article)

DeLiema, D. (2014, June). Attributions and epistemology in conversation: How math tutors and students co-construct accounts of failure and knowledge. For ICLS Doctoral Consortium. In proceedings of the International Conference of the Learning Sciences, Boulder, Colorado.

Impasses in the Wild

Moments where children encounter problems in their chosen activities represent potentially generative sites for learning, particularly when supportive adults are present to scaffold the learning process. How do parents decide when to offer support and what kind of support to offer? How do these considerations play out in the course of naturalistic outdoor play? In two studies, we explore these questions, first by looking at how parents and their young children respond to moments in which children get stuck during play, and second by examining how video can serve as a point of departure for weighing the complexity of these timing decisions.

Baker, J., DeLiema, D., Hufnagle, A. S., Carlson, S. M., Sharrat, A., & Williams Ridge, S. (2022). Impasses in the wild: Autonomy support in naturalistic, parent-child outdoor play. Frontiers in Education, 7, 1-28.

DeLiema, D., Hufnagle, A. S., Rao, V. N. V., Baker, J., Valerie, J., & Kim, J. (2023). Methodological innovations at the intersection of video-based educational research traditions: Reflections on relevance, data selection, and phenomena of interest. International Journal of Research & Method in Education, 46(1), 19-36.

Playful Problem Solving

Play provides an exceptionally fertile context for learning (Zosh et al., 2017). Failure in particular, which can often lead students to cease pursuing an activity, is treated during play as a core, even intrinsically motivating part of the process (Juul, 2009; Steen & Owens, 2001). Even further, the more game-players fail before success, the more likely they are to understand the ideas covered in the game (Anderson et al., 2018). Broadly speaking, efforts that improve incrementally across each failure moment increase their likelihood of success (Yin et al., 2019). Because learning to problem-solve requires working through impasses, there are substantial benefits that play can provide with respect to lowering the cost of failure and fostering iteration. However, there is not a solid body of research to inform how adults can effectively scaffold problem-solving learning when youth are playing video games. In addition, it is non-trivial to consider what problem solving actions players can effectively take when they reach impasses. As adults who grew up playing video games start to raise their own children, it will be far more likely that families play video games together in their home environments. For all of the above reasons, developing knowledge of how to scaffold young players’ problem solving practices during play could be particularly impactful. This project will focus on Baba is You, a challenging, elegant, and fun puzzle game, as a context for developing a problem solving measurement tool and a general purpose dialogic scaffolding framework that can be used by instructors and parents to understand and deepen students’ problem solving practices. Baba Is You provides an auspicious context for this work because participants visibly interact with the rules on the screen, have to question/re-shape the goal (a real-world problem-solving constraint), rewind without a cost when the avatar dies, and advance through the game building on past skills.

Anderson, C. G., Hussein, B., Carpenter, Z., & DeLiema, D. (2024). Show or tell? A comparison of direct instruction tutorial and learn by doing increased impasse versions of initial levels of a puzzle game. In proceedings of the conference on Foundations of Digital Games 2024. Massachusetts, USA: ACM.

Anderson, C. G., Goeke, M., Hussein, B., Carpenter, Z. L., Salehi, S., DeLiema, D. (2022). Baba is hint - Designing a scaffolding guidebook for game-based learning. In Proceedings of the Games+Learning+Society Conference 2022. Irvine CA.

Carpenter, Z., & DeLiema, D. (2023). Learning through play at the intersection of problem- solving, epistemic (un)certainty, and emotion. In 17th International Conference of the Learning Sciences (ICLS) 2023 (pp. 1178-1181). Montreal, Canada: International Society of the Learning Sciences.

Carpenter, Z., Wang, Y., DeLiema, D., Kendeou, P., & Shaffer, D. W. (2023). Using multi- modal network models to visualize and understand how players learn a mechanic in a problem-solving game [Poster presentation]. In companion proceedings of The 13th International Learning Analytics and Knowledge Conference (pp. 99-101). Arlington, TX.

DeLiema, D., Goeke, M., Hussein, B., Valerie, J., Anderson, C., Varma, K., Chen, B., Salehi, S., & Bernacki, M. (2022). Playful learning following deviations: A mixture of tinkering, causal explanations, and revision rationales. In C. Chinn, D. Tan, C. Chan, & Y. Kali (Eds.), 16th International Conference of the Learning Sciences (ICLS) 2022 (pp. 1421- 1424). Hiroshima, Japan: International Society of the Learning Sciences.

Interested in participating in one of our Baba Is You research projects?

If you are a parent of a middle schooler and have access to a windows computer, you may be eligible for one of our studies. Please have a look at this research advertisement, and if you are interested in participating, share your email with us at the link in the research ad.