{"@ID": "1039", "@Name": "Inadequate Detection or Handling of Adversarial Input Perturbations in Automated Recognition Mechanism", "@Abstraction": "Class", "@Structure": "Simple", "@Status": "Incomplete", "Description": "The product uses an automated mechanism such as machine learning to recognize complex data inputs (e.g. image or audio) as a particular concept or category, but it does not properly detect or handle inputs that have been modified or constructed in a way that causes the mechanism to detect a different, incorrect concept.", "Technical_Nuance": {"xhtml:p": "Targeted technologies include, but are not necessarily limited to:", "xhtml:ul": {"xhtml:li": ["automated speech recognition", "automated image recognition", "automated cyber defense", "chatbots, LLMs, generative AI"]}, "#text_blocks": ["Targeted technologies include, but are not necessarily limited to:", "automated speech recognition", "automated image recognition", "automated cyber defense", "chatbots, LLMs, generative AI"]}, "Related_Weaknesses": {"Related_Weakness": [{"@Nature": "ChildOf", "@CWE_ID": "693", "@View_ID": "1000", "@Ordinal": "Primary"}, {"@Nature": "ChildOf", "@CWE_ID": "697", "@View_ID": "1000"}]}, "Weakness_Ordinalities": {"Weakness_Ordinality": {"Ordinality": "Primary", "Description": "This weakness does not depend on other weaknesses and is the result of choices made during optimization."}}, "Applicable_Platforms": {"Language": {"@Class": "Not Language-Specific", "@Prevalence": "Undetermined"}, "Technology": {"@Name": "AI/ML", "@Prevalence": "Undetermined"}}, "Modes_Of_Introduction": {"Introduction": [{"Phase": "Architecture and Design", "Note": "This issue can be introduced into the automated algorithm itself due to inadequate or inappropriate training data, as well as lack of validation, verification, testing, and evaluation of the algorithm. These factors can affect the overall robustness of the algorithm when introduced into operational settings."}, {"Phase": "Implementation", "Note": "The developer might not apply external validation of inputs into the algorithm."}]}, "Common_Consequences": {"Consequence": [{"Scope": ["Availability", "Integrity"], "Impact": ["Bypass Protection Mechanism", "DoS: Resource Consumption (Other)", "DoS: Instability"], "Note": "When the automated mechanism is used in a\n               protection mechanism to automatically classify input\n               streams, and those classifications are used for\n               security-critical decisions, then any mistake in\n               classification can introduce a vulnerability that\n               allows attackers to cause the product to make the wrong\n               security decision or disrupt service of the automated\n               mechanism. An attacker may be able to craft inputs\n               that are misinterpreted in a way that grants excess\n               privileges."}, {"Scope": "Confidentiality", "Impact": "Read Application Data", "Note": "This weakness could lead to breaches of data privacy through exposing features of the training data, e.g., by using membership inference attacks or prompt injection attacks."}, {"Scope": "Other", "Impact": "Varies by Context", "Note": "The consequences depend on how the application applies or integrates the affected algorithm."}]}, "Detection_Methods": {"Detection_Method": [{"Method": "Dynamic Analysis with Manual Results Interpretation", "Phase": "Testing", "Description": "Use indicators from model performance deviations such as sudden drops in accuracy or unexpected outputs to verify the model."}, {"Method": "Dynamic Analysis with Manual Results Interpretation", "Phase": "Testing", "Custom_Phase": {"@Domain": "AI", "#text": "Data Collection"}, "Description": "Use indicators from input data collection mechanisms to verify that inputs are statistically within the distribution of the training and test data."}, {"Method": "Architecture or Design Review", "Description": "Use multiple models or model ensembling techniques to check for consistency of predictions/inferences."}]}, "Potential_Mitigations": {"Mitigation": [{"Phase": "Architecture and Design", "Description": "Algorithmic modifications such as model pruning or compression can help mitigate this weakness. Model pruning ensures that only weights that are most relevant to the task are used in the inference of incoming data and has shown resilience to adversarial perturbed data."}, {"Phase": "Architecture and Design", "Custom_Phase": {"@Domain": "AI", "#text": "Training"}, "Description": "Consider implementing adversarial training, a method that introduces adversarial examples into the training data to promote robustness of algorithm at inference time."}, {"Phase": "Architecture and Design", "Description": "Consider implementing model hardening to fortify the internal structure of the algorithm, including techniques such as regularization and optimization to desensitize algorithms to minor input perturbations and/or changes."}, {"Phase": "Implementation", "Description": "Consider implementing multiple models or using model ensembling techniques to improve robustness of individual model weaknesses against adversarial input perturbations."}, {"Phase": "Implementation", "Description": "Incorporate uncertainty estimations into the algorithm that trigger human intervention or secondary/fallback software when reached. This could be when inference predictions and confidence scores are abnormally high/low comparative to expected model performance."}, {"Phase": "Integration", "Description": "Reactive defenses such as input sanitization, defensive distillation, and input transformations can all be implemented before input data reaches the algorithm for inference."}, {"Phase": "Integration", "Description": "Consider reducing the output granularity of the inference/prediction such that attackers cannot gain additional information due to leakage in order to craft adversarially perturbed data."}]}, "Demonstrative_Examples": {"Demonstrative_Example": [{"Intro_Text": "Research into machine learning attacks [REF-1547] included an example in which an image of a Stop sign was transformed using minor changes, but it was still humanly recognizable as a Stop sign. However, due to the perturbations, the classifier determined that the transformed image was that of a Yield sign. Physical signs could be changed with small alterations using paint or stickers. For autonomous vehicles, this change could cause accidents if the vehicle does not stop when it is supposed to."}, {"Intro_Text": "Researchers were able to \"jailbreak\" Large Language Models (LLMs) and get answers to harmful questions by adding suffixes to chatbot questions [REF-1549]. A Reinforcement Learning (RL) technique was later discovered to \"optimize\" adversarial suffixes to increase the changes of success [REF-1548]."}]}, "Observed_Examples": {"Observed_Example": {"Reference": "CVE-2025-26644", "Description": "biometric authentication capability allows attackers to spoof biometric data and bypass authentication using adversarial input perturbations that cause the ML model to accepting the input", "Link": "https://www.cve.org/CVERecord?id=CVE-2025-26644"}}, "References": {"Reference": [{"@External_Reference_ID": "REF-16"}, {"@External_Reference_ID": "REF-17"}, {"@External_Reference_ID": "REF-15"}, {"@External_Reference_ID": "REF-13"}, {"@External_Reference_ID": "REF-14"}, {"@External_Reference_ID": "REF-1547"}, {"@External_Reference_ID": "REF-1548"}, {"@External_Reference_ID": "REF-1549"}]}, "Mapping_Notes": {"Usage": "Allowed-with-Review", "Rationale": "This CWE entry is a Class, but it does not have Base-level children.", "Comments": "This entry is classified in a part of CWE's hierarchy that does not have sufficiently low-level coverage, which might reflect a lack of classification-oriented weakness research in the software security community. Conduct careful root cause analysis to determine the original mistake that led to this weakness. If closer analysis reveals that this weakness is appropriate, then this might be the best available CWE to use for mapping. If no other option is available, then it is acceptable to map to this CWE.", "Reasons": {"Reason": [{"@Type": "Abstraction"}, {"@Type": "Other"}]}}, "Notes": {"Note": {"@Type": "Relationship", "#text": "Further investigation is needed to determine if better relationships exist or if additional organizational entries need to be created.  For example, this issue might be better related to \"recognition of input as an incorrect type,\" which might place it as a sibling of CWE-704 (incorrect type conversion)."}}, "Content_History": {"Submission": {"Submission_Name": "CWE Content Team", "Submission_Organization": "MITRE", "Submission_Date": "2018-03-12", "Submission_Version": "3.1", "Submission_ReleaseDate": "2018-03-29"}, "Modification": [{"Modification_Name": "CWE Content Team", "Modification_Organization": "MITRE", "Modification_Date": "2019-06-20", "Modification_Version": "3.3", "Modification_ReleaseDate": "2019-06-20", "Modification_Comment": "updated References"}, {"Modification_Name": "CWE Content Team", "Modification_Organization": "MITRE", "Modification_Date": "2020-02-24", "Modification_Version": "4.0", "Modification_ReleaseDate": "2020-02-24", "Modification_Comment": "updated Relationships"}, {"Modification_Name": "CWE Content Team", "Modification_Organization": "MITRE", "Modification_Date": "2023-04-27", "Modification_Version": "4.11", "Modification_ReleaseDate": "2023-04-27", "Modification_Comment": "updated References, Relationships"}, {"Modification_Name": "CWE Content Team", "Modification_Organization": "MITRE", "Modification_Date": "2023-06-29", "Modification_Version": "4.12", "Modification_ReleaseDate": "2023-06-29", "Modification_Comment": "updated Mapping_Notes"}, {"Modification_Name": "CWE Content Team", "Modification_Organization": "MITRE", "Modification_Date": "2024-07-16", "Modification_Version": "4.15", "Modification_ReleaseDate": "2024-07-16", "Modification_Comment": "updated Applicable_Platforms"}, {"Modification_Name": "CWE Content Team", "Modification_Organization": "MITRE", "Modification_Date": "2025-04-03", "Modification_Version": "4.17", "Modification_ReleaseDate": "2025-04-03", "Modification_Comment": "updated Common_Consequences, Description, Detection_Factors, Mapping_Notes, Modes_of_Introduction, Name, Potential_Mitigations, Time_of_Introduction"}, {"Modification_Name": "CWE Content Team", "Modification_Organization": "MITRE", "Modification_Date": "2025-09-09", "Modification_Version": "4.18", "Modification_ReleaseDate": "2025-09-09", "Modification_Comment": "updated References"}, {"Modification_Name": "CWE Content Team", "Modification_Organization": "MITRE", "Modification_Date": "2026-04-30", "Modification_Version": "4.20", "Modification_ReleaseDate": "2026-04-30", "Modification_Comment": "updated Observed_Examples, Relationships"}, {"Modification_Name": "CWE Content Team", "Modification_Organization": "MITRE", "Modification_Date": "2026-10-28", "Modification_Version": "5.0", "Modification_ReleaseDate": "2026-10-28", "Modification_Comment": "updated Common_Consequences, Demonstrative_Examples, Description, Modes_of_Introduction, References, Technical_Nuance"}], "Previous_Entry_Name": {"@Date": "2025-04-03", "#text": "Automated Recognition Mechanism with Inadequate Detection or Handling of Adversarial Input Perturbations"}}, "entry_type": "Weakness"}