CLUE
An assistive Mixed Reality application that provides real-time captions and graphic illustrations in classrooms
PROJECT DESCRIPTION
Oral language instruction is the foundation for supporting a student's reading, writing, and communication in the classroom. While it has tremendous benefits, it falls short on addressing the learning requirements of 3 types of students: English language learners, students with hearing disabilities, and students with language-based learning disabilities.
Clue is an assistive Mixed Reality application that provides real-time captions and graphic illustrations in classrooms. It uses AI-powered speech-to-text technology and imports customized graphics from a database we created based on the curriculum of middle school subjects.
Project | Verizon 5G EdTech Challenge
Role | User Research, UX Design, Market Research, Concept Development
Team | Tuba Ozkan (Product/UX Design Lead), Nouf Aljowaysir (Developer, Researcher), Taeyeon Kim (Creative Technologist)
VERIZON CHALLENGE IDEA
The Verizon 5G EdTech Challenge was a nationwide open call to find innovative, cutting-edge, education technologies that will transform middle school education. They provided multiple design questions around this topic, and we chose to challenge ourselves with the following question: "How can we use 5G to improve the learning experience and reduce the resource gap for students with special and diverse learning needs in under-resourced middle schools?"
OUR CHALLENGE AREA
1 | Spoken language comprehension in the classroom is a barrier for many students with diverse learning needs.
2 | Public schools are underfunded and lack resources to improve teaching environments.
Design Question: How might we help classrooms become spaces where ALL students feel supported and engaged?
RESEARCH
Research shows that a simple classroom lecture can be a huge struggle for three main groups of learners: non-native English speakers, students with hearing disabilities and students with language-based learning disability.
ENGLISH LANGUAGE LEARNER
ELLs are often concentrated in low-performing schools with untrained or poorly trained teachers. They lag behind in academic achievement due to little or no access to quality instruction tailored to their needs.
DEAF & HARD OF HEARING
A mild hearing loss can cause a child to miss as much as 50 percent of classroom discussion. Providing specialized teachers and proper classroom settings is difficult and costly for many schools.
LANGUAGE-BASED LEARNING DISABILITIES (LBLD)
Students with LBLD experience the highest drop-out rate among all students with disabilities (19%). LBLD is often misunderstood by educators which leads to ineffective support services and classroom accommodations.
All these Students...
1 | Struggle with understanding oral language instruction.
2 | Need in-classroom tools and proper educational settings to fulfill their diverse learning requirements.
OUR SOLUTION
Our solution is an AI-powered assistive Mixed Reality application that provides real-time captions and graphic illustrations in classrooms.
OUR PRODUCT
1 | Adds Closed Captions & Emphasizes
Critical Concepts
2 | Provides Illustrated Animations
for Selected Words
3 | Transcribes Teacher Instruction
Digitally in an e-book Format
WHY THIS SOLUTION?
1 | Captions clarify dialogue for all listeners.
2 | The combination of both visual and textual instruction improves focus, comprehension and engagement in classrooms.
MARKET RESEARCH
BUSINESS MODEL
WHY WE NEED 5G?
1 | Our real-time speech-to-text technology is delayed due to current internet speeds.
2 | Our complex system of components (AI and MR) need high internet speeds, with almost zero latency, to ensure a smooth user experience.
Selected Works
