CLUE

An assistive Mixed Reality application that provides real-time captions and graphic illustrations in classrooms

PROJECT DESCRIPTION

Oral language instruction is the foundation for supporting a student's reading, writing, and communication in the classroom. While it has tremendous benefits, it falls short on addressing the learning requirements of 3 types of students: English language learners, students with hearing disabilities, and students with language-based learning disabilities. 

Clue is an assistive Mixed Reality application that provides real-time captions and graphic illustrations in classrooms. It uses AI-powered speech-to-text technology and imports customized graphics from a database we created based on the curriculum of middle school subjects.

Project | Verizon 5G EdTech Challenge
Role |  User Research, UX Design, Market Research, Concept Development
Team | Tuba Ozkan (Product/UX Design Lead), Nouf Aljowaysir (Developer, Researcher), Taeyeon Kim (Creative Technologist)

VERIZON CHALLENGE IDEA

The Verizon 5G EdTech Challenge was a nationwide open call to find innovative, cutting-edge, education technologies that will transform middle school education. They provided multiple design questions around this topic, and we chose to challenge ourselves with the following question: "How can we use 5G to improve the learning experience and reduce the resource gap for students with special and diverse learning needs in under-resourced middle schools?"

clueprocess

OUR CHALLENGE AREA 

1 | Spoken language comprehension in the classroom is a barrier for many students with diverse learning needs. 
2 | Public schools are underfunded and lack resources to improve teaching environments.

Design Question: How might we help classrooms become spaces where ALL students feel supported and engaged?

RESEARCH

Research shows that a simple classroom lecture can be a huge struggle for three main groups of learners: non-native English speakers, students with hearing disabilities and students with language-based learning disability.

y-70

ENGLISH LANGUAGE LEARNER

ELLs are often concentrated in low-performing schools with untrained or poorly trained teachers. They lag behind in academic achievement due to little or no access to quality instruction tailored to their needs.

a

DEAF & HARD OF HEARING

A mild hearing loss can cause a child to miss as much as 50 percent of classroom discussion. Providing specialized teachers and proper classroom settings is difficult and costly for many schools.

b

LANGUAGE-BASED LEARNING DISABILITIES (LBLD)

Students with LBLD experience the highest drop-out rate among all students with disabilities (19%). LBLD is often misunderstood by educators which leads to ineffective support services and classroom accommodations.

c

All these Students...

1 | Struggle with understanding oral language instruction.
2 | Need in-classroom tools and proper educational settings to fulfill their diverse learning requirements.

OUR SOLUTION

Our solution is an AI-powered assistive Mixed Reality application that provides real-time captions and graphic illustrations in classrooms.

d-69

OUR PRODUCT

aClue_Pitch Deck.002-01
aClue_Pitch Deck.002-02
aClue_Pitch Deck.002-03

1 | Adds Closed Captions & Emphasizes
Critical Concepts

2 | Provides Illustrated Animations
for Selected Words

3 | Transcribes Teacher Instruction
Digitally in an e-book Format

WHY THIS SOLUTION? 

1 | Captions clarify dialogue for all listeners.
2 | The combination of both visual and textual instruction improves focus, comprehension and engagement in classrooms.

Scenes_T-66

MARKET RESEARCH 

z-67

BUSINESS MODEL

d-68

WHY WE NEED 5G? 

1 | Our real-time speech-to-text technology is delayed due to current internet speeds.
2 | Our complex system of components (AI and MR) need high internet speeds, with almost zero latency, to ensure a smooth user experience.

 Selected Works