Oral language instruction is the foundation for supporting a student's reading, writing, and communication in the classroom. While it has tremendous benefits, it falls short on addressing the learning requirements of 3 types of students: English language learners, students with hearing disabilities, and students with language-based learning disabilities.
Clue is an assistive Mixed Reality application that provides real-time captions and graphic illustrations in classrooms. It uses AI-powered speech-to-text technology and imports customized graphics from a database we created based on the curriculum of middle school subjects.
Project | Verizon 5G EdTech Challenge
Role | User Research, UX Design, Market Research, Concept Development
Skills | Design Thinking, Storyboarding, Copywriting, Visual Design, Leadership
Team | Tuba Ozkan (Product/UX Design Lead), Nouf Aljowaysir (Developer, Researcher), Taeyeon Kim (Creative Technologist)
Verizon Challenge Area
The Verizon 5G EdTech Challenge was a nationwide open call to find innovative, cutting-edge, education technologies that will transform middle school education. They provided multiple design questions around this topic, and we chose to challenge ourselves with the following question: "How can we use 5G to improve the learning experience and reduce the resource gap for students with special and diverse learning needs in under-resourced middle schools?"
Our Challenge Area
1 | Spoken language comprehension in the classroom is a barrier for many students with diverse learning needs.
2 | Public schools are underfunded and lack resources to improve teaching environments.
Design Question: How might we help classrooms become spaces where ALL students feel supported and engaged?
Research shows that a simple classroom lecture can be a huge struggle for three main groups of learners:
non-native English speakers, students with hearing disabilities and students with language-based learning disability.
English Language Learner (ELL)
ELLs are often concentrated in low-performing schools with untrained or poorly trained teachers.
They lag behind in academic achievement due to little or no access to quality instruction tailored to their needs.
Deaf & Hard of Hearing (DHH)
A mild hearing loss can cause a child to miss as much as 50 percent of classroom discussion. Providing specialized teachers and proper classroom settings is difficult and costly for many schools.
Language-based Learning Disabilities (LBLD)
Students with LBLD experience the highest drop-out rate among all students with disabilities (19%). LBLD is often misunderstood by educators which leads to ineffective support services and classroom accommodations.
All these Students...
1 | Struggle with understanding oral language instruction.
2 | Need in-classroom tools and proper educational settings to fulfill their diverse learning requirements.
Our solution is an AI-powered assistive Mixed Reality application that provides real-time captions and graphic illustrations in classrooms.
1 | Adds Closed Captions & Emphasizes
2 | Provides Illustrated Animations
for Selected Words
3 | Transcribes Teacher Instruction
Digitally in an e-book Format
Why this solution?
1 | Captions clarify dialogue for all listeners.
2 | The combination of both visual and textual instruction improves focus, comprehension and engagement in classrooms.
Why we need 5G?
1 | Our real-time speech-to-text technology is delayed due to current internet speeds.
2 | Our complex system of components (AI and MR) need high internet speeds, with almost zero latency, to ensure a smooth user experience.