2023 + 2024
Ai Literacy
Fostering AI Literacy with LuminAI through Embodiment and Creativity in Informal Learning Spaces
Context
This study focuses on enhancing AI literacy through LuminAI, an interactive art installation. LuminAI enables participants to engage with an AI dance partner in an embodied, creative experience. Recently redesigned for informal learning spaces, the installation aims to demystify AI by providing interactive experiences that explain concepts such as movement tracking, co-creation, and data grouping.
Method
The redesigned LuminAI features three panels, each with specific learning objectives. Panel 1 teaches participants how movement data is represented by AI, Panel 2 focuses on co-creative improvisation with an AI dance partner, and Panel 3 visualizes how AI clusters similar movements. This modular design ensures accessibility for middle-school-aged participants and broader public audiences, with simplified terminology and structured prompts tailored for informal learning environments.
Results
The redesign addresses usability challenges, enhances engagement, and promotes understanding of AI concepts. Early evaluations show promise in making AI more approachable, particularly for underrepresented groups. Future work includes iterative improvements based on feedback and expanded evaluation to assess knowledge gains and learning outcomes.
Co-Creative AI
LuminAI: World’s first collaborative improvised dance performance
How it all began
Dance class retrospective
Exploring Collaborative Movement Improvisation Towards the Design of LuminAI—a Co-Creative AI Dance Partner
Context
This research explores the co-creative potential between humans and AI in dance improvisation, focusing on embodied, dyadic interaction. It aims to understand how dancers’ movement choices are influenced by the self, partner, and environment, providing insights for designing LuminAI—a co-creative AI dance partner.
Method
Focus groups with 24 university-level dance students examined improvisational dynamics through thematic analysis. The study synthesized findings into an Interconnected Model of Improvisational Dance Inputs, addressing in-the-moment influences, generative strategies, and heuristics for collaboration. This human-centered approach ensures LuminAI aligns with dancers’ needs and practices.
Results
The study revealed that improvisational success depends on awareness of self, partner, and environment, supported by strategies like mimicry, transformation, and repetition. These insights informed recommendations for enhancing LuminAI’s design to facilitate natural and meaningful human-AI co-creative interactions in embodied domains.
AI Meets Holographic Pepper’s Ghost: A Co-Creative Public Dance Experience
Context
The research showcases LuminAI, an AI-driven dance improvisation tool that uses motion capture and holographic technology to foster embodied co-creativity. The system enables performers to engage with an intelligent AI partner in real-time, blending technology and improvisation in an immersive public installation.
Method
LuminAI employs a five-module pipeline for perception, movement segmentation, learning, transformation, and response generation. Using motion capture and Laban Movement Analysis, the AI analyzes and responds to human movements, creating a dynamic co-creative experience. The design integrates lightweight, portable components, including Hologauze projection technology, to ensure accessibility and adaptability for public exhibitions.
Results
Initial tests at public events revealed challenges in audience engagement and perceptions of AI capabilities. Observations highlighted the need for improved transparency in the AI’s decision-making and more inclusive designs for audience interaction. Future iterations aim to refine these aspects, enhancing the co-creative and participatory experience.
Observable Creative Sense-Making (OCSM): A Method For Quantifying Improvisational Co-Creative Interaction
Context
Observable Creative Sense-Making (OCSM) addresses the challenges of analyzing co-creative improvisation in dance. By focusing on observable dimensions—participation, newness, and appropriateness—it offers a framework grounded in embodied social cognition.
Method
OCSM uses qualitative coding and a web-based tool for analyzing interactions. A study with 16 dancers guided by the Laban Movement Analysis framework validated the framework through motion capture and video analysis.
Results
The study showed strong correlations among OCSM dimensions, with high agreement on participation and appropriateness. OCSM offers a reliable method for analyzing co-creative processes, with applications in human-computer interaction and generative AI.
aiDance: A Non-Invasive Approach in Designing AI-Based Feedback for Ballet Assessment and Learning
Context
The Challenge of Tradition and Feedback in Ballet Training
Classical ballet training has remained largely unchanged for centuries, relying on tradition-bound practices such as mirrors and instructor-led verbal feedback. However, these methods often fall short in providing effective, timely, and standardized feedback for learners. Mirrors, while ubiquitous, have demonstrated negative effects on performance and self-awareness, and the subjective nature of instructor feedback creates inconsistencies in assessment. This dissertation addresses these issues by examining how AI and machine learning (ML) can augment traditional ballet pedagogy to provide objective, non-invasive feedback, ultimately transforming teaching and learning in this highly codified art form.
Method
A Mixed-Method Approach Across Three Phases
The research was conducted in three interconnected phases. The first phase explored how augmented feedback, delivered via an AI-enabled mirror prototype, could support remote ballet learning. Using a mixed-method approach with 32 participants (16 novices and 16 experts), the study evaluated the effects of visual and verbal feedback on performance. The second phase focused on the challenges faced by teachers and dancers in traditional settings, identifying barriers to technology adoption and deriving design principles for future tools. The third phase culminated in the design and evaluation of aiDance, an AI-based feedback system that uses human pose estimation and ML to assess movement quality and provide personalized, data-driven feedback. The evaluation included teacher interviews and prototype testing, leading to insights on integrating AI into ballet pedagogy.
Results
Advancing Ballet Training Through AI
Key findings revealed that augmented feedback significantly improved performance for both novice and expert dancers, with remote feedback proving particularly effective for skill acquisition. Field observations and interviews highlighted the need for tools that are non-invasive, promote critical thinking, and align with ballet’s traditional workflow. The aiDance system demonstrated its potential to democratize learning by providing accessible, objective assessments tailored to individual needs. This research not only enhances ballet education but also contributes broadly to Human-Computer Interaction (HCI) by introducing a scalable framework for motor skill learning in other domains.
2021
Show Me How You Interact, I Will Tell You What You Think: Exploring the Effect of the Interaction Style on Users’ Sensemaking about Correlation and Causation in Data
Context
Findings from embodied cognition suggest that our whole body (not just our eyes) plays an important role in how we make sense of data when we interact with data visualizations. In this paper, we present the results of a study that explores how different designs of the ”interaction” (with a data visualization) alter the way in which people report and discuss correlation and causation in data.
Method
We conducted a lab study with two experimental conditions: Full body (participants interacted with a 65” display showing geo-referenced data using gestures and body movements); and Gamepad (people used a joypad to control the system).
results
Participants tended to agree less with statements that portray correlation and causation in data after using the Gamepad system. Additionally, discourse analysis based on Conceptual Metaphor Theory revealed that users made fewer remarks based on FORCE schemata in Gamepad than in Full-Body.
Current Use, Non-Use, and Future Use of Ballet Learning Technologies
Problem
Learning ballet is a complex motor task that can be effectively enhanced by technology. Learning technologies, however, are not typically used for the assessment of ballet technique due to a lack of adequate and non-invasive tools that can be pragmatically adopted.
Method
We conducted an interview-based qualitative study with seven expert ballet teachers and six pre-professional dancers to examine their current and future technology use in a ballet technique class.
results
Through inductive and deductive analysis, we identified reasons for technology non-use and derived seven requirements that can inform the design and implementation of ballet assessment technologies including designing for: adaptation to multi-skill/multi-method environments, teacher/dancer skill augmentation, agency, non-invasive design, feedback for gross/fine movements, trust, and proprioception by supporting transformative assessment. We discuss barriers for technology acceptance and unintended consequences that should be considered when designing future technologies for ballet.
2020
Exploring Casual COVID-19 Data Visualizations on Twitter: Topics and Challenges
context
Social networking sites such as Twitter have been a popular choice for people to express their opinions, report real-life events, and provide a perspective on what is happening around the world. In the outbreak of the COVID-19 pandemic, people have used Twitter to spontaneously share data visualizations from news outlets and government agencies and to post casual data visualizations that they individually crafted.
purpose
Our study explores what people are posting, what they retweet the most, and the challenges that may arise when interpreting COVID-19 data visualization on Twitter.
Data collection and analysis
We conducted a Twitter crawl of 5409 visualizations (from the period between 14 April 2020 and 9 May 2020) to capture what people are posting. We analyzed data through a bottom-up inductive approach and four rounds of open-coding and affinity diagrams until we reached saturation. See figure below.
results
Our findings show that multiple factors, such as the source of the data, who created the chart (individual vs. organization), the type of visualization, and the variables on the chart influence the retweet count of the original post. We identify and discuss five challenges that arise when interpreting these casual data visualizations, and discuss recommendations that should be considered by Twitter users while designing COVID-19 data visualizations to facilitate data interpretation and to avoid the spread of misconceptions and confusion.
Designing AI-Based Feedback for Ballet Learning
Problem
Since its codified genesis in the 18th century, ballet training has largely been unchanged: it relies on the word of mouth expertise passed down generation to generation and on tools that do not adequately support both dancers and teachers. Moreover, top-tier training comes at an exceptional price and only found in a few locations around the world.
Design
In this context, artificial intelligence (AI)-based video tools such as the use of OpenPose could represent an affordable and non-invasive alternative: they would allow dancers and teachers to quantitatively self-assess as well as enable skilled ballet teachers to connect with a wider audience. In my dissertation research, I study how to design and evaluate AI-based tools for ballet dancers and teachers to quantify performance and facilitate learning.
“Alexa is a Toy”: Exploring Older Adults’ Reasons for Using, Limiting, and Abandoning Echo
Problem
Intelligent voice assistants (IVAs) have the potential to support older adults’ independent living. However, despite a growing body of research focusing on IVA use, we know little about why older adults become IVA non-users.
Purpose
This paper examines the reasons older adults use, limit, and abandon IVAs (i.e., Amazon Echo) in their homes.
Method
We conducted eight focus groups, with 38 older adults residing in a Life Plan Community.
results
Thirty-six participants owned an Echo for at least a year, and two were considering adoption. Over time, most participants became non-users due to their difficulty finding valuable uses, beliefs associated with ability and IVA use, or challenges with use in shared spaces. However, we also found that participants saw the potential for future IVA support. We contribute a better understanding of the reasons older adults do not engage with IVAs and how IVAs might better support aging and independent living in the future.
Move your body: Engaging museum visitors with human-data interaction
Problem
Museums have embraced embodied interaction: its novelty generates buzz and excitement among their patrons, and it has enormous educational potential. Human-Data Interaction (HDI) is a class of embodied interactions that enables people to explore large sets of data using interactive visualizations that users control with gestures and body movements. In museums, however, HDI installations have no utility if visitors do not engage with them.
Method
In this paper, we present a quasi-experimental study that investigates how different ways of representing the user (“mode type”) next-to a data visualization alters the way in which people engage with a HDI system. We consider four mode types: avatar, skeleton, camera overlay, and control.
Results
Our findings indicate that the mode type impacts the number of visitors that interact with the installation, the gestures that people do, and the amount of time that visitors spend observing the data on display and interacting with the system.
2019
Towards ai-enhanced ballet learning
Problem
Since its codified genesis in the 18th century, ballet training has largely been unchanged: it relies on the word of mouth expertise passed down generation to generation and in tools that do not adequately support both dancers and teachers. Moreover, top-tier training is only found in a few locations around the world and comes at an exceptional price.
design
In this context, artificial intelligence (AI)-based video tools might represent an affordable and non-invasive alternative: it would allow dancers and teachers to self-assess as well as enable skilled dance teachers to connect with a wider audience. In my research, I study how to design and evaluate AI-based tools to improve ballet performance for dancers and teachers.
Designing for Ballet Classes: Identifying and Mitigating Communication Challenges Between Dancers and Teachers
Problem
Dancer-teacher communication in a ballet class can be challenging: ballet is one of the most complex forms of movements, and learning happens through multi-faceted interactions with studio tools (mirror, barre, and floor) and the teacher.
Method
We conducted an interview-based qualitative study with seven ballet teachers and six dancers followed by an open-coded analysis to explore the communication challenges that arise while teaching and learning in the ballet studio.
Results
We identified key communication issues, including adapting to multi-level dancer expertise, transmitting and realigning development goals, providing personalized corrections and feedback, maintaining the state of flow, and communicating how to properly use tools in the environment. We discuss design implications for crafting technological interventions aimed at mitigating these communication challenges.
2018
Takes Tutu to Ballet: Designing Visual and Verbal Feedback for Augmented Mirrors
Problem
Mirrors have been a core feature in ballet studios for over five hundred years. While physical mirrors provide real-time feedback, they do not inform dancers of their errors. Thus, technologies such as motion tracking have been used to augment what a physical mirror can provide. Current augmented mirrors, however, only implement one mode of communication, usually visual, and do not provide a holistic feedback to dancers that includes all the feedback elements commonly used in ballet classes.
Method
We conducted a mixed-method study with 16 novices and 16 expert dancers in which we compared two different modes of communication (visual and verbal), two different types of feedback (value and corrective) and two levels of guidance (mirror, or no mirror). Participants’ ballet technique scores were evaluated by a remote teacher on eight ballet combinations (tendue, adagio, pirouette, saute, plié, degage, frappe and battement tendue).
Results
We report quantitative and qualitative results that show how the level of guidance, mode of communication, and type of feedback, needs to be tuned in different ways for novices and experts.
2016
Remote Ballet Learning
Problem
Since its genesis in the 16th century, ballet training has largely been unchanged: it relies on the word of mouth expertise of ballet teachers passed down generation to generation. Top-tier training is only found in few locations around the world, and comes at an exceptional price. In this context, remote education might represent an affordable alternative: it would allow skilled dance teachers to connect with a wider audience. Although prior research has begun to develop technologies for dancers, there is no consensus on the type of feedback that should be provided to allow remote ballet learning.
Design
We prototyped four variations of a technologically simulated mirror to evaluate different combinations of feedback commonly used in traditional ballet settings.
Results
Our preliminary results suggest that systems for remote ballet learning should provide combinations of visual, verbal, value, and corrective feedback.
2015
Usability Evaluation of Kinect-Based System for Ballet
Aim
Since the 1800s, ballet education is influenced by the use of mirrors. Mirrors, however, cannot give feedback on movements. The aim of this study is to evaluate a Kinect-based system called Super Mirror, to discover if it has an impact on the usability in ballet instruction.
Method
Ballet students were evaluated on eight ballet movements (plié, élevé, grand plié, battement tendu (front, side and back), passé and développé) to measure the Super Mirror’s impact.
Results
The results show a potential usage in ballet education but improvements of Super Mirror are needed to comply with the standardized subject-matter expert’s criteria.
Milka Trajkova 