The Good Tech Companies - Why Speech Recognition Misses Human Context: Dr. Sunday David Ubur’s Affective Architecture

Episode Date: September 11, 2026

This story was originally published on HackerNoon at: https://hackernoon.com/why-speech-recognition-misses-human-context-dr-sunday-david-uburs-affective-architecture. Dr. Sunday Ubur's research explores emotion-aware AI captions that preserve tone, urgency, and context for deaf and hard-of-hearing users. Check more stories related to undefined at: https://hackernoon.com/c/undefined. You can also check exclusive content about #speech-recognition, #artificial-intelligence, #ai, #emotion-aware-captions, #accessible-captioning, #speech-recognition-ai, #speech-to-text-ai, #good-company, and more. This story was written by: @nicafurs. Learn more about this writer by checking @nicafurs's about page, and for more stories, please visit hackernoon.com. Dr. Sunday Ubur's research explores emotion-aware AI captions that preserve tone, urgency, and context for deaf and hard-of-hearing users.

This audio is presented by Hacker Noon, where anyone can learn anything about any technology. Why Speech Recognition misses human context, Dr. Sunday David Ubor's effective architecture, by Nika Furs. Building automated speech recognition has become one of the most resource-intensive arms races in modern computing. For years, the industry standard has centered on scaling foundational transformer models across hundreds of thousands of hours of speech data to drive word error rates closer to zero. Yet, as speech-to-text engines have become ubiquitously integrated into virtual meeting software, streaming platforms, and classroom tools, a fundamental imitation has become increasingly obvious to anyone relying on them for daily communication. While modern speech models

are remarkably adept to transcribing vocabulary, they remain largely oblivious to the emotional tone, cadence, and urgency that give spoken language its actual meaning. In high-stakes technical environments, such as engineering sprint retrospectives, architectural design reviews, or advanced university STEM lectures, spoken dialogue is rarely delivered as flat prose. A slight upward inflection can turn an apparent statement of fact into a skeptical question. A sudden drop in vocal pitch can signal a serious warning about a code vulnerability, and an urgent delivery can differentiate a critical design constraint from a casual suggestion. When standard automated speech recognition strips these acoustic cues away, leaving behind an uninflected block of text at the bottom of a display, it creates

what human computer interaction, HCI researchers call an intention gap. For deaf and hard of hearing professionals, the result is continuous cognitive strain, forcing them to guess speaker intent while rapidly darting their visual attention between slides, physical demonstrations, and disconnected caption windows. At the University of California, Irvine and Virginia Tech, HCI researcher Dr. Sunday David Ubor has taken an alternative engineering path to address this challenge. Rather than treating speech recognition solely as a sequence to sequence text translation problem, Dr. Ubor approaches the challenges in asynchronous distributed systems and spatial computing problem. Across several years of published laboratory work, his research has centered on constructing multimodal machine learning

pipelines and spatial augmented reality frameworks capable of detecting and rendering nonverbal vocal cues in real time, without creating computational latency or overwhelming the user's working memory. The core issue with modern transcription isn't simply that words get misheard, drive. Ubor explained in an interview, the deeper problem is that speech-to-text algorithms throw away almost all of the acoustic information that carries human intent. Our goal was never to pile more visual noise onto the user. It was to find an architectural method to preserve the critical vocal cues that conventional captions discard, while keeping the user's cognitive workload as low as possible. The first major

systems bottleneck Dr. Ubor tackled was low-latency multi-stream inference within his platform, interpretive caption. Capturing vocal nuance in live conversation requires running deep acoustic feature extraction alongside continuous speech recognition without introducing perceptible delay or thread blocking. To solve this, Dr. Ubor architected a parallel pipeline that bifurcates incoming audio streams into two asynchronous processing tracks. Onabranch roots the audio through speech recognition transformers to maintain lexical transcription accuracy, while the second branch feeds raw acoustic wave forms into a specialized deep acoustic neural network fine-tuned across standard benchmark vocal datasets. This dual pipeline setup extracts

prosodic features directly from the audio signal, mapping vocal delivery across distinct emotional and paralinguistic categories in real time while preserving synchronization, with the live speaker. However, extracting deep acoustic signals in real time solved only half the engineering equation. The more difficult challenge lay in interface design and user cognitive load. Early experimental systems across the accessibility sector often made the mistake of cluttering displays with bright banners, large animated avatars, or intrusive graphical tags that quickly overwhelmed USERS during demanding analytical tasks. Drawing upon established cognitive models of working memory, Dr. Ubor structured his system around an on-demand progressive disclosure interface.

By segmenting incoming captions into short, readable phrase units and transmitting them over asynchronous web sockets, the architecture presents clean, continuous text while embedding lightweight single-letter tokens adjacent to relevant phrases. These tokens remain visually subordinate to the primary transcript, expanding into plain language tooltips only when a user intentionally hovers over our focuses on them, ensuring that the interface never. obstructs the flow of technical information. Dr. Ubor applied a similar empirical rigor to immersive spatial computing within his easy caption framework. For years, teams developing consumer-mixed reality applications operated under the intuitive assumption that rendering expressive, animated digital

avatars alongside text captions would create a more engaging experience for deaf users. To evaluate whether this design choice actually supported comprehension, Dr. Ubor ran a study inside virtual reality head-mounted displays that combined electroencephalography, eG, brainwave monitoring with measures of user attention, engagement, and eye gaze. Their resulting data indicated that animated virtual avatars reduced sustained attention and engagement and slightly increased user stress compared with cleaner caption layouts presented without avatars. The findings gave interface designers concrete empirical evidence that assistive systems in virtual environments should prioritize clean geometry and customizable density over decorative visual

animations. To reduce the split attention penalty in physical environments such as lecture halls and collaborative laboratories, Dr. Ubor extended his captioning work into three-dimensional spatial augmented reality built with unity and mixed reality toolkits. Rather than requiring users to look away from a speaker down to amobile device or laptop screen, his augmented reality prototype anchors floating, head stabilized, emotionally annotated captions within the user's natural field of view, near the speaker. In preliminary evaluations, this AR-based approach improved comprehension and reduced cognitive effort compared with standard plain text captions. Dr. Ubor subsequently extended these spatial architectures into collaborative group settings through a speaker-aware captioning

interface presented at the ACMCHI conference on human factors in computing systems. In multi-speaker group discussions, failing to identify conversational turn taking often leads to mediate communicative breakdown. By integrating multi-speaker audio diarization with confidence-gated affect flags and an on-demand contextual repair module driven by language models, the platform allows users to follow fast-paced group discussions and recover missed conversational context without losing track of the live dialogue. The significance of Dr. Ubor's work has increasingly resonated beyond individual software prototypes, appearing at premier academic and technical venues including the ACM Conference on Human Factors in Computing Systems, Kai, the ACMSI-G-G-A-C-C-E-SS conference on computers and accessibility, assets,

and the IEEEEEEEEEEEEFR. The Rochester Institute of Technology's National Technical Institute for the Deaf, NTID, also invited him to present his research on AI-driven accessible captioning. His research path, progressing from competitive international fellowships to doctoral work at Virginia Tech Center for Human Computer Interaction, and on to Hiscrant postdoctoral research at the University of California, Irvine, where he leads independent research initiatives in accessible computing and classroom informatics, reflects a sustained focus on bridging the divide between theoretical artificial intelligence and practical accessibility engineering. Beyond system development, Dr. Ubor's scholarship examines how the field itself evaluates accessibility. In an auto-ethnographic account of his transition from secondary school to university in Nigeria, undertaken without formal disability accommodations or a deaf community, drive. Ubor and his co-authors introduced the concept of self-accommodation, the access work displaced onto

the disabled student when no institutional system exists to provide it, captured in the study's central image, my body became the access infrastructure. In a separate narrative review synthesizing dozens of peer-reviewed studies, he found that real-time speech to text tools still largely failed to convey effective nuance, and that emotion-aware captioning within immersive extended reality environments remains a persistent, under-addressed gap, an evidence-based case for building effect responsive, user-centered captioning rather than plain text transcription alone. Accessibility is frequently treated as an optional feature or a compliance check. box that gets tacked on at the end of a software cycle, Dr. Ubor noted.

In reality, when you engineer systems for edge case environments and users with distinct sensory constraints, you are forced to solve the hardest latency, cognitive load, and interface problems first. The resulting architectures end up being faster, more resilient, and far more usable for everyone. By approaching accessible communication through the rigorous lens of low latency systems engineering, neuro-organomics, and spatial computing, Dr. Uber research demonstrates that speech technology still has significant room to evolve. As intelligent interfaces and spatial headsets become standard tools across classrooms and workplaces, the systems that truly succeed will not just be those that transcribe raw

vocabulary, but those that capture and convey the full context of human dialogue. This article was published under Hackernoon's business blogging program. Thank you for listening to this Hackernoon story, read by artificial intelligence. Visit hackernoon.com to read, write, learn, publish.