The Good Tech Companies - Your AI Is Only as Smart as Its Data—And Humans Are Still the Best at Labeling It

Episode Date: March 24, 2025

This story was originally published on HackerNoon at: https://hackernoon.com/your-ai-is-only-as-smart-as-its-dataand-humans-are-still-the-best-at-labeling-it. The consensus method plays a key role in data annotation when it is necessary to ensure high accuracy and reduce subjectivity in labeling. Check more stories related to machine-learning at: https://hackernoon.com/c/machine-learning. You can also check exclusive content about #machine-learning, #data-annotation, #consensus, #data-labeling, #image-labeling, #consensus-in-data-annotation, #keymakr, #good-company, and more. This story was written by: @keymakr. Learn more about this writer by checking @keymakr's about page, and for more stories, please visit hackernoon.com. Consensus is achieved by gathering the opinions of multiple experts. Google, Tesla, Amazon, and Meta actively use consensus-based annotation to improve AI performance. Google Health applies consensus to enhance diagnostic accuracy. Tesla uses consensus to label data from autopilot cameras.

This audio is presented by Hacker Noon, where anyone can learn anything about any technology. Your AI is only as smart as its data, and humans are still the best at labeling it, by KeyMaker. The consensus method plays a key role in data annotation when it is necessary to ensure high accuracy and reduce subjectivity in labeling. Based in KeyMaker's experience, implementing a consensus approach with multiple expertene-specific cases can reduce annotation errors by 30 to 50 percent. Consensus minimizes mistakes, automates quality control, and helps create benchmark datasets, especially critical in high-responsibility areas such as medicine and autonomous driving. Tatiana Verbitskaya, a technical solution architect at Keymaker, talks about how this

method works and the projects in which it has been successfully applied. How it works. Consensus is achieved by gathering the opinions of multiple experts. When defining, ground truth, data, it is vital to establish an agreed upon standard of accuracy. Consensus is critical when training a model on subjective data, such as color and shape, or when high accuracy is required. This method IS actively used in the early stages when the model has not yet been trained

on sufficient data or when additional training is needed, particularly for specific cases, e.g. subjective judgments. Additionally, consensus is critical in large-scale projects, such as annotating data for self-driving cars or monitoring transportation, as it enhances precision while reducing errors. Key principles of consensus, odd number of experts. To avoid deadlocks, consensus relies on an odd number of annotators, ensuring a definitive outcome even in cases of disagreement. Disagreement analysis. This method doesn't just rely on the majority vote but also considers the frequency of disagreements.

If discrepancies are too significant, the data may be flagged for additional review or not even used for the model training. Error Detection Mechanisms. Even consensus-based data can contain errors if the cases are too subjective and not definitive. Global technology leaders like Google, Tesla, Amazon, and Meta actively use consensus-based annotation to improve AI model performance. Google Health, for instance, applies multiple radiologist annotations to X-rays to enhance diagnostic accuracy. Tesla uses consensus to label data from autopilot cameras, reducing training errors in autonomous driving. Amazon SageMaker Ground Truth incorporates consensus annotation in NLP, computer vision, and satellite imagery analysis, while Meta employs it for facial and object recognition

projects. Medical Consensus and Annotation Council One of the most critical applications of consensus is in medical image annotation for disease diagnosis. Experts say radiologists' diagnoses can vary be as much as 20 to 30 percent, directly impacting patient outcomes. When a consensus-based approach is employed, where multiple radiologists independently annotate images and their inputs are aggregated based on expertise-weighted scoring, annotation accuracy can be improved by up to 40%. KeyMaker actively applies this approach in complex medical projects. As a result, this helps to ensure precise image labeling for AI models trained to detect complex pathologies. Here, the process was built using the KeyLabs platform, where you can compare the opinions of

several experts, identify discrepancies, and form high-precision datasets. This approach significantly increases the reliability of algorithms used in automated diagnostics, minimizing the risk of wrong diagnosis. Consensus in copyright content usage monitoring. Currently, KeyMaker collaborates with SoundAware, a company that deploys automated music recognition technology to identify copyrighted music usage. The team reviews 10,000 URLs to assess the presence of copyrighted material. Video platforms are filled with content that can contain the author's material,

such as music, scenes from movies, or TV show fragments. Due to the vast amount of data and the subjective nature of copyright interpretation, manually analyzing each video is impractical. However, KeyMaker identifies cases where copyrighted content is used or modified in ways automated systems cannot detect reliably yet. These include parodies, fan art, and homages. To eliminate subjectivity, KeyMaker employs a consensus-based approach. Each video is evaluated by multiple independent experts who answer the following questions. Does the video contain copyrighted music? Does it feature scenes from a movie or TV show? Has the content been modified, such as through editing or remixing?

Based on the experts' responses, a final decision is made regarding potential copyright issues. Such projects are essential for enforcing copyright and ensuring rights holders receive fair compensation. Additionally, this process helps companies specializing in content monitoring refine their algorithms and accelerate the detection of copyrighted material. Consensus in vehicle and pedestrian tracking Consensus is also widely applied in AI training for autonomous vehicles, particularly in object recognition on roads, e AI training for autonomous vehicles, particularly in object recognition on roads, e.g. other vehicles, pedestrians, traffic signs. For instance,

a camera might capture a pedestrian in motion, and human annotators might disagree on whether the object is a person or a shadow. Consensus ensures precise labeling in such scenarios. Keymaker team recently worked with analysis of video recorded on cameras to track vehicles. It was necessary to track the vehicle's movement through several cameras at a crossroads and ensure that the system correctly identified the same vehicle in different frames. The cameras recorded one object, car, at several points. Several experts viewed the video from different cameras. They assessed whether this object is the same car because there could be differences in perception of appearance, for example, by color or brand. The information was used to train the model if five annotators confirmed the object's identity.

Otherwise, such data would have been excluded from the dataset. This has reduced the number of false alarms and increased the accuracy of car recognition systems, which is important for urban safety systems and automatic traffic control systems. The same approach can be applied to identify people in shopping malls or on the streets. Cameras capture movement by analyzing, for example, the color of clothes, height, or other characteristics. This method is used to, enhanced security monitoring, crime prevention, retail visitor behavior analysis, crowd flow assessment in public areas, the future of consensus in AI, the future of consensus based data annotation is promising, particularly as to reach $3.6 billion by 2027, and many companies are adopting multi-layered annotation verification to enhance data quality.

Studies show that models trained on datasets with consensus annotation demonstrate significantly higher accuracy than models trained on single-source labeling. Despite the development of automatic annotation and generative AI, the human factor remains key. Subjectivity and annotation disagreements necessitate multi-stage validation. Therefore, the consensus method will continue to be based, ensuring data reliability and reducing errors in critical areas such as autonomous systems, medicine, and financial analysis. Thank you for listening to this Hacker Noon story, read by and financial analysis.