Why Security Is Essential in Architectures with AI Components—and How to Effectively Implement It in Gen-AI

When designing software system architectures, we strive to make decisions that support the achievement of quality attributes essential to business success and to enhancing the end-user experience. These attributes are often aligned with established standards, such as ISO 25000. However, it’s critical to understand that when incorporating components under the umbrella of artificial intelligence, these quality attributes—while named similarly—carry different nuances.

Among the most vital attributes for both business performance and user experience is security, and that will be the focus of this blog post.

Imagine a system that appears highly functional, yet fails to uphold the CIA triad—confidentiality, integrity, and availability—three foundational aspects of security. In such a system, users might withhold their data due to inadequate confidentiality, only to have their information accessed by unauthorized parties. Picture entering your phone number to receive important alerts, but the system's poor data integrity alters that number, causing you to miss critical notifications. Or envision a scenario where the system simply isn’t available when you need it most. These examples underscore just how essential security is to a system’s success. Without the right security in place, users may abandon the system altogether jeopardizing the business objectives it was meant to serve.

When working with Gen AI, it becomes just another component within the broader system architecture, and thus its quality attributes must be safeguarded to ensure the overall success of the solution. As previously mentioned, AI components give these attributes a different weight and meaning—especially security, which demands more extensive consideration and a complementary approach.

To address system security effectively, we must consider multiple dimensions, beginning with the traditional CIA framework, and extending through DevSecOps practices—covering shared responsibility, static and dynamic security testing, and continuous monitoring. We must also factor in related concepts such as reliability and resilience, ideally operating under a zero-trust model.

However, introducing AI components adds even more complexity. These elements bring three additional attack surfaces:

• Training data
• The model itself
• The data used for inference or execution purposes

Below is a high-level overview of the types of attacks associated with each of these surfaces, along with tactics and strategies to mitigate them:

Poisoning Attacks: These target the training phase by injecting faulty data designed to subvert the intended model behavior. This threat can impact both training data and the model.

Privacy Attacks: These occur when an attacker extracts data the system was never meant to reveal. These attacks affect training data and the model.

Evasion Attacks: In these cases, the model is not directly compromised but generates incorrect responses. For example, altering just a few pixels in an image could cause a classification model to misinterpret it. This threat is linked to the data used for inference or execution purposes.

Abuse or Misuse Attacks: These involve the system producing responses it should not—such as content that promotes hate speech. This also correlates with the data used for inference or execution.

Bruce Schneier famously stated, “Security is a process, not a product.” In that spirit, here are key strategies and tactics for mitigating security threats—emphasizing that security is an ongoing, evolving journey:

• Zero Trust: Never trust, always verify.
• Least Privilege: Grant access only to the resources truly needed.
• Continuous Monitoring: Even after access is granted, monitor usage for suspicious activity.
• Encrypted Communications: Always encrypt information in transit.
• Network Segmentation.
• Traffic Restrictions: Allow only the traffic that is strictly necessary.
• Authorization: Enforce strict access controls for every service.
• Data Minimization: Use only the data essential for AI operations.
• Data Anonymization: When possible, anonymize sensitive data.
• Frequent Data Auditing: Monitor data access and usage for irregular activity.

We will now take a deeper dive into the use of Gen AI components—an integral part of the broader AI landscape—and revisit why security is crucial for both business impact and user satisfaction.

Prompt injection has become one of the primary entry points for attacks on Gen AI components.

In both legacy and modern information system architectures across various sectors, the adoption of Gen AI brings challenges in areas such as implementation, user experience, and data engineering. However, as stated before, our emphasis here is on security.

In this article, we will explore how to leverage established methodologies to identify threats within system architectures that integrate generative AI. We’ll also define controls to mitigate identified risks, taking into account the previously mentioned attack surfaces. This process will be broken down into 5 actionable steps.

Identify threats, prioritize threats, respond to threats, identify tactics, implement the appropriate technology

1. Identify Known Threats in the System

In a Gen-AI-enabled architecture, beyond traditional components like frontend interfaces, services, microservices, APIs, and data repositories, we also introduce specialized elements such as:

• Agents: Components responsible for orchestrating prompts, plugins, LLMs, and RAGs to define the tasks the solution will execute.
• Prompts: Natural language instructions that specify the tasks LLMs will perform.
• Plugins: Software components used to integrate Gen AI solutions with legacy systems or existing frameworks.
• LLMs (Large Language Models): High-capacity models designed for natural language processing. They handle content vectorization or input processing and can be used for general or specialized purposes using techniques such as distillation or machine learning.
• RAG (Retrieval-Augmented Generation): Components that allow the integration of specialized knowledge sources with LLMs to improve answer accuracy or provide access to exclusive information.
• Knowledge Bases: Vector databases that store contextual information used by RAG components.
• Data Flows: Pipelines that manage data ingestion into knowledge bases, APIs consumed by LLMs, and structured data sources.

Industry 5.0 refers to the combination of these components as the inference layer, and that’s how we’ll identify it.

The following diagram presents a contextual view of an agnostic Gen AI solution implementation:

To identify threats, we will use the OWASP Top 10 for LLMs as our reference:

Below is a table summarizing the vulnerabilities identified by OWASP for LLMs, along with descriptions and real-world attack examples:

Once these vulnerabilities are identified, we assess which components in our Gen AI architecture are exposed to specific threats.

The following diagram outlines the most common vulnerabilities across various Gen AI architecture components:

2. Prioritize Threats Based on Probability and Impact

Once threats have been identified, the next step is to prioritize them based on their potential impact. For this, we use a risk matrix—in this case, the ADM model.

Here’s an example of how to structure the prioritization:

3. Define How to Respond to Threats

To define how we’ll respond to the identified threats, we must first assess the level of security required for the implementation. Naturally, the higher the security level, the greater the effort and resource investment. However, this does not mean we should compromise security to make implementation feasible. The key is to find the right balance—a level of security that effectively meets both user needs and business objectives.

So how do we determine the appropriate level of security? While there’s no one-size-fits-all answer, a strong starting point is the LLM Security Verification Standard (LLMVS). Though still in draft form, it provides valuable baseline guidelines that help teams assess and align security requirements.

This is an overview of the LLMVS security levels:

The LLMVS outlines various security levels, each covering eight verification domains. Based on these levels, target controls must be implemented. With these controls and corresponding mitigation strategies in place, we define what type of control to apply to each prioritized threat. Once again, a risk matrix is used to establish both the selected control and the residual risk level.

4. Identify Tactics to Mitigate Threats

Once controls have been defined, the next step is to identify the specific techniques required to mitigate each threat. These techniques may already be established by the industry or the organization and provide clear guidance on how each threat should be addressed. Continuing with the methodology, we identify the technique within the risk matrix:

Select the Right Technologies and Components to Implement the Identified Tactics

To effectively implement the solution, it’s essential to identify the appropriate technological component for each of the tactics assigned to mitigate the threats. 

At this point, it’s crucial to specify a more detailed implementation plan—one that outlines each individual component of the solution and its corresponding security controls, as shown in the following diagram:

Finally, just as with any architecture design process, this stage requires alignment with architectural decisions, quality scenarios, and the definition of test suites capable of validating those scenarios. It also involves verifying that the system’s security measures are sufficient to meet its objectives—delivering value to users while supporting meaningful business outcomes.

Conclusion

Security is a fundamental attribute for meeting the needs of both end users and the business. When incorporating artificial intelligence components—especially generative AI—it’s essential to recognize that the traditional quality attributes take on different meanings. In the case of security, Gen-AI introduces three additional attack surfaces that must be considered when designing a system’s security architecture: The training data, the model itself, and the data used for inference or model execution.

Each of these surfaces is susceptible to distinct types of attacks, including poisoning, privacy breaches, evasion, and abuse or misuse. Fortunately, these threats can be mitigated through a combination of architectural approaches and proven tactical frameworks.

When designing a system architecture that includes Gen-AI components, organizations can apply well-established security methodologies—for example, threat modeling and secure design practices—leveraging tools like the LLM Security Verification Standard to identify vulnerabilities and define the target security level for the architecture.

By doing so, it’s possible to design and implement a system architecture that effectively addresses the needs of both end users and the business in terms of security. AWS plays a significant role in this effort by providing guardrails that support security implementation, including services such as AWS Identity and Access Management (IAM), AWS Certificate Manager (ACM), AWS WAF, AWS Key Management Service (KMS), AWS CloudWatch, AWS CloudTrail, AWS Secrets Manager, and Amazon Bedrock Guardrails