Thumbnail Image
Publication

Subjective evaluation of operator responses for mobile defect identification in remanufacturing: application of NLP and disagreement tagging

Date
2025-12-04
Abstract
In the context of remanufacturing, particularly mobile device refurbishing, effective operator training is crucial for accurate defect identification and process inspection efficiency. This study examines the application of Natural Language Processing (NLP) techniques to evaluate operator expertise based on subjective textual responses gathered during a defect analysis task. Operators were asked to describe screen defects using open-ended questions, and their responses were compared with expert responses to evaluate their accuracy and consistency. We employed four NLP models, including finetuned Sentence-BERT (SBERT), pre-trained SBERT, Word2Vec, and Dice similarity, to determine their effectiveness in interpreting short, domain-specific text. A novel disagreement tagging framework was introduced to supplement traditional similarity metrics with explainable insights. This framework identifies the root causes of model–human misalignment across four categories: defect type, severity, terminology, and location. Results show that a finetuned SBERT model significantly outperforms other models by achieving Pearsons’s correlation of 0.93 with MAE and RMSE scores of 0.07 and 0.12, respectively, providing more accurate and context-aware evaluations. In contrast, other models exhibit limitations in semantic understanding and consistency. The results highlight the importance of finetuning NLP models for domain-specific applications and demonstrate how qualitative tagging methods can enhance interpretability and model debugging. This combined approach indicates a scalable and transparent methodology for the evaluation of operator responses, supporting the development of more effective training programmes in industrial settings where remanufacturing and sustainability generally are a key performance metric
Supervisor
Description
Publisher
MDPI
Citation
Big Data Cognitive Computing, 9, 312
Collections
Electronic & Computer Engineering
Show all metadata
Files
Thumbnail Image
Ahmed_2025_Subjective.pdf
Adobe PDF, 2.68 MB
Keywords
subjective assessment, disagreement analysis, natural language processing, deep learning, defect detection, remanufacturing
ULRR Identifiers
https://hdl.handle.net/10344/31496
 https://doi.org/10.34961/19135
Funding code
Funding Information
Sustainable Development Goals
External Link
License
Attribution-NonCommercial-ShareAlike 4.0 International
cbn
Embedded videos