When techniques such as machine learning are used to automatically classify input streams, and those classifications are used for security-critical decisions, then any mistake in classification can introduce a vulnerability that allows attackers to cause the product to make the wrong security decision or disrupt service of the automated mechanism. If the mechanism is not developed or "trained" with enough input data or has not adequately undergone test and evaluation, then attackers may be able to craft malicious inputs that intentionally trigger the incorrect classification.
Targeted technologies include, but are not necessarily limited to:
For example, an attacker might modify road signs or road surface markings to trick autonomous vehicles into misreading the sign/marking and performing a dangerous action. Another example includes an attacker that crafts highly specific and complex prompts to "jailbreak" a chatbot to bypass safety or privacy mechanisms, better known as prompt injection attacks.
| Impact | Details |
|---|---|
|
Bypass Protection Mechanism |
Scope: Integrity
When the automated recognition is used in a protection mechanism, an attacker may be able to craft inputs that are misinterpreted in a way that grants excess privileges.
|
|
DoS: Resource Consumption (Other); DoS: Instability |
Scope: Availability
There could be disruption to the service of the automated recognition system, which could cause further downstream failures of the software.
|
|
Read Application Data |
Scope: Confidentiality
This weakness could lead to breaches of data privacy through exposing features of the training data, e.g., by using membership inference attacks or prompt injection attacks.
|
|
Varies by Context |
Scope: Other
The consequences depend on how the application applies or integrates the affected algorithm.
|
| Phase(s) | Mitigation |
|---|---|
|
Architecture and Design |
Algorithmic modifications such as model pruning or compression can help mitigate this weakness. Model pruning ensures that only weights that are most relevant to the task are used in the inference of incoming data and has shown resilience to adversarial perturbed data.
|
|
Architecture and Design |
Consider implementing adversarial training, a method that introduces adversarial examples into the training data to promote robustness of algorithm at inference time.
|
|
Architecture and Design |
Consider implementing model hardening to fortify the internal structure of the algorithm, including techniques such as regularization and optimization to desensitize algorithms to minor input perturbations and/or changes.
|
|
Implementation |
Consider implementing multiple models or using model ensembling techniques to improve robustness of individual model weaknesses against adversarial input perturbations.
|
|
Implementation |
Incorporate uncertainty estimations into the algorithm that trigger human intervention or secondary/fallback software when reached. This could be when inference predictions and confidence scores are abnormally high/low comparative to expected model performance.
|
|
Integration |
Reactive defenses such as input sanitization, defensive distillation, and input transformations can all be implemented before input data reaches the algorithm for inference.
|
|
Integration |
Consider reducing the output granularity of the inference/prediction such that attackers cannot gain additional information due to leakage in order to craft adversarially perturbed data.
|
| Nature | Type | ID | Name |
|---|---|---|---|
| ChildOf | Pillar Pillar - a weakness that is the most abstract type of weakness and represents a theme for all class/base/variant weaknesses related to it. A Pillar is different from a Category as a Pillar is still technically a type of weakness that describes a mistake, while a Category represents a common characteristic used to group related things. | 693 | Protection Mechanism Failure |
| ChildOf | Pillar Pillar - a weakness that is the most abstract type of weakness and represents a theme for all class/base/variant weaknesses related to it. A Pillar is different from a Category as a Pillar is still technically a type of weakness that describes a mistake, while a Category represents a common characteristic used to group related things. | 697 | Incorrect Comparison |
| Phase | Note |
|---|---|
| Architecture and Design | This issue can be introduced into the automated algorithm itself due to inadequate training data used as well as lack of validation, verification, testing, and evaluation of the algorithm. These factors can affect the overall robustness of the algorithm when introduced into operational settings. |
| Implementation | The developer might not apply external validation of inputs into the algorithm. |
Class: Not Language-Specific (Undetermined Prevalence)
AI/ML (Undetermined Prevalence)
| Ordinality | Description |
|---|---|
|
Primary
|
(where the weakness exists independent of other weaknesses)
This weakness does not depend on other weaknesses and is the result of choices made during optimization.
|
| Method | Details |
|---|---|
|
Dynamic Analysis with Manual Results Interpretation |
Use indicators from model performance deviations such as sudden drops in accuracy or unexpected outputs to verify the model.
|
|
Dynamic Analysis with Manual Results Interpretation |
Use indicators from input data collection mechanisms to verify that inputs are statistically within the distribution of the training and test data.
|
|
Architecture or Design Review |
Use multiple models or model ensembling techniques to check for consistency of predictions/inferences.
|
| Nature | Type | ID | Name |
|---|---|---|---|
| MemberOf | CategoryCategory - a CWE entry that contains a set of other entries that share a common characteristic. | 1413 | Comprehensive Categorization: Protection Mechanism Failure |
Rationale
This CWE entry is a Class, but it does not have Base-level children.Comments
This entry is classified in a part of CWE's hierarchy that does not have sufficiently low-level coverage, which might reflect a lack of classification-oriented weakness research in the software security community. Conduct careful root cause analysis to determine the original mistake that led to this weakness. If closer analysis reveals that this weakness is appropriate, then this might be the best available CWE to use for mapping. If no other option is available, then it is acceptable to map to this CWE.Relationship
| Submissions | |||
|---|---|---|---|
| Submission Date | Submitter | Organization | |
|
2018年03月12日
(CWE 3.1, 2018年03月29日) |
CWE Content Team | MITRE | |
| Modifications | |||
| Modification Date | Modifier | Organization | |
|
2025年09月09日
(CWE 4.18, 2025年09月09日) |
CWE Content Team | MITRE | |
| updated References | |||
|
2025年04月03日
(CWE 4.17, 2025年04月03日) |
CWE Content Team | MITRE | |
| updated Common_Consequences, Description, Detection_Factors, Mapping_Notes, Modes_of_Introduction, Name, Potential_Mitigations, Time_of_Introduction | |||
|
2024年07月16日
(CWE 4.15, 2024年07月16日) |
CWE Content Team | MITRE | |
| updated Applicable_Platforms | |||
| 2023年06月29日 | CWE Content Team | MITRE | |
| updated Mapping_Notes | |||
| 2023年04月27日 | CWE Content Team | MITRE | |
| updated References, Relationships | |||
| 2020年02月24日 | CWE Content Team | MITRE | |
| updated Relationships | |||
| 2019年06月20日 | CWE Content Team | MITRE | |
| updated References | |||
| Previous Entry Names | |||
| Change Date | Previous Entry Name | ||
| 2025年04月03日 | Automated Recognition Mechanism with Inadequate Detection or Handling of Adversarial Input Perturbations | ||
Use of the Common Weakness Enumeration (CWE™) and the associated references from this website are subject to the Terms of Use. CWE is sponsored by the U.S. Department of Homeland Security (DHS) Cybersecurity and Infrastructure Security Agency (CISA) and managed by the Homeland Security Systems Engineering and Development Institute (HSSEDI) which is operated by The MITRE Corporation (MITRE). Copyright © 2006–2025, The MITRE Corporation. CWE, CWSS, CWRAF, and the CWE logo are trademarks of The MITRE Corporation.