How to Secure LLM Apps Against Prompt Injection and Leakage

Current as of October 30, 2026. This guide draws on NIST AI 100-2 E2025 and NIST AI 600-1, the OWASP Top 10 for LLM Applications 2025, the New York Department of Financial Services (DFS) cybersecurity regulation (23 NYCRR Part 500) and two DFS industry letters, section 248.30 of the SEC's Regulation S-P, and FINRA's 2026 Annual Regulatory Oversight Report, as published on that date. It covers US texts only and is not legal advice.

Chatbots, document assistants and agents built on large language models take instructions and data through the same channel. NIST AI 100-2 puts it this way: "Because GenAI models combine the data and instruction channels, attackers can leverage the data channel to affect system operations by manipulating resources with which the system interacts." For a regulated institution, the question is how to keep that weakness from exposing customer information.

Unless a step quotes a rule, it is a recommendation, labeled (interpretation). Controls such as input and output filtering, least-privilege tool access, retrieval permissioning and red-teaming are named in the NIST and OWASP documents, which are voluntary; applying them to satisfy a binding rule is interpretation.

On scope: Part 500 binds any person "operating under or required to operate under a license, registration, charter, certificate, permit, accreditation or similar authorization under the Banking Law, the Insurance Law or the Financial Services Law, regardless of whether the covered entity is also regulated by other government agencies", subject to the exemptions in section 500.19 (smaller covered entities, for example, are exempt from sections 500.8 and 500.14(a)(1)). This guide used DFS's posted Second Amendment text, which DFS notes "is not an official version". Section 248.30 binds brokers, dealers, investment companies, and investment advisers and transfer agents registered with the SEC or another appropriate regulatory agency. Both NIST documents are voluntary; AI 100-2 says it "does not establish any legal standard or any other legal requirement or defense under any law, nor does it have the force or effect of law". The OWASP list comes from the OWASP Gen AI Security Project, which is not a regulator. FINRA's report is addressed to member firms and "does not create any new legal or regulatory requirements or new interpretations of existing requirements". Federal banking agencies' standards and other states' insurance and data security laws were not reviewed; firms outside Part 500 should map these steps to their own regulator's rules (interpretation).

Step 1: Inventory every LLM application and map its trust boundaries

Part 500 section 500.9(a) requires the risk assessment to be updated at least annually and "whenever a change in the business or technology causes a material change to the covered entity's cyber risk", and section 500.13(a) requires a documented asset inventory. DFS's 2024 letter says risk assessments should address the organization's own use of AI, AI used by its vendors, and vulnerabilities stemming from AI applications. Its September 10, 2026 risk-assessment letter, which "does not create new obligations", names AI adoption as an example of emerging risk.

For each application, record the model, provider, readable data, callable tools and every channel untrusted text can enter through, such as chat, files, retrieved documents and tool results (interpretation).

Step 2: Treat direct and indirect prompt injection as separate attack paths

NIST AI 100-2 describes direct prompt injection as attacks in which "the main user provides in-context instructions that are appended to higher-trust instructions like those provided by the application designer". Indirect prompt injection comes through resources the system ingests, and NIST notes that "it is the primary user of the model who is harmed by the compromise of the integrity, availability, or privacy of the GenAI system through an indirect prompt injection attack."

Threat-model both, including third-party content such as loan documents, claims attachments and client correspondence (interpretation).

Step 3: Enforce least privilege for tools, agents and retrieval outside the model

Part 500 section 500.7(a)(1) requires covered entities to "limit user access privileges to information systems that provide access to nonpublic information to only those necessary to perform the user's job". Applying it to an LLM application's service identity is interpretation.

OWASP's LLM06 entry advises: "Limit the extensions that LLM agents are allowed to call to only the minimum necessary." Its LLM07 entry states: "Critical controls such as privilege separation, authorization bounds checks, and similar must not be delegated to the LLM, either through the system prompt or otherwise." For retrieval, LLM08 recommends to "Implement fine-grained access controls and permission-aware vector and embedding stores."

Check entitlements at retrieval so the model sees only documents the requesting user could open, and require human approval for actions that move money, change records or send external messages (interpretation). FINRA's report, for member firms exploring AI agents, lists "how to establish guardrails or control mechanisms to limit or restrict agent behaviors, actions or decisions."

Step 4: Filter inputs, and validate outputs before anything acts on them

OWASP's LLM05 entry advises: "Treat the model as any other user, adopting a zero-trust approach, and apply proper input validation on responses coming from the model to backend functions." NIST AI 100-2 describes input filtering and designing systems "under the assumption that the AI model can and will produce malicious output if it is exposed to malicious actors."

OWASP's LLM07 entry says: "Avoid embedding any sensitive information (e.g. API keys, auth keys, database names, user roles, permission structure of the application) directly in the system prompts." Its LLM02 entry warns that system-prompt restrictions on what data the model returns "may not always be honored and could be bypassed via prompt injection or other methods." Part 500 section 500.8(a) requires secure development practices for in-house applications and procedures for testing the security of externally developed ones; covering LLM applications under it is interpretation.

Step 5: Limit what customer data the application can reach and keep

Regulation S-P section 248.30(a)(1): "Every covered institution must develop, implement, and maintain written policies and procedures that address administrative, technical, and physical safeguards for the protection of customer information." NIST AI 600-1 warns that "Models may leak, generate, or correctly infer sensitive information about individuals."

DFS's 2024 letter says covered entities that use AI "should also consider monitoring for unusual query behaviors that might indicate an attempt to extract NPI and blocking queries from personnel that might expose NPI to a public AI product or system." It also reads the Part 500 disposal duty in section 500.13(b) as reaching NPI used for AI purposes. Pass models only the fields a task needs, mask identifiers, and set retention for prompts, logs and vector stores (interpretation). For using customer data to train models, see our guide to training AI on customer data under GLBA and Regulation S-P.

Step 6: Red-team for prompt injection and leakage before and after release

NIST AI 600-1's suggested action MS-2.7-007 is to perform AI red-teaming against, among other things, GAI attacks such as prompt injection, and MS-2.10-001 suggests red-teaming for outputting of training data and revealing personal or sensitive information. OWASP's LLM01 entry recommends adversarial testing, "treating the model as an untrusted user to test the effectiveness of trust boundaries and access controls." For member firms, FINRA's report says "Testing areas to consider include areas such as privacy, integrity, reliability and accuracy."

Build direct and indirect injection tests from your own data channels, rerun them on every model, prompt or tool change, and record results for the risk assessment (interpretation).

Step 7: Log prompts and outputs and connect them to incident response

FINRA's report describes, for member firms, "Ongoing monitoring of prompts, responses and outputs to confirm the GenAI solution continues to perform as expected and results in compliant behavior." Part 500 section 500.14(a)(1) requires risk-based controls "designed to monitor the activity of authorized users and detect unauthorized access or use of, or tampering with, nonpublic information by such authorized users". Class A companies must also implement "a solution that centralizes logging and security event alerting" (section 500.14(b)), unless the CISO approves compensating controls.

Part 500 section 500.17(a)(1) requires notice to the superintendent "as promptly as possible but in no event later than 72 hours after determining that a cybersecurity incident has occurred at the covered entity, its affiliates, or a third-party service provider", where "cybersecurity incident" is defined in section 500.1(g). Regulation S-P section 248.30(a)(3) requires "a program reasonably designed to detect, respond to, and recover from unauthorized access to or use of customer information, including customer notification procedures", with notice to affected individuals whose sensitive customer information was involved, as soon as practicable and no later than 30 days after becoming aware that unauthorized access "has occurred or is reasonably likely to have occurred", subject to the rule's harm determination and Attorney General delay. Add model playbooks (disable tools, purge poisoned documents, rotate exposed credentials) to existing plans (interpretation).

Step 8: Hold model providers and AI vendors to the same controls

Part 500 section 500.11 requires third-party service provider policies addressing, to the extent applicable, due diligence, minimum cybersecurity practices, notice of cybersecurity events and representations and warranties. DFS's 2024 letter adds that "if TPSPs are using AI, Covered Entities should consider incorporating additional representations and warranties related to the secure use of Covered Entities' NPI". Regulation S-P section 248.30(a)(5) requires policies for oversight of service providers, reasonably designed to ensure they notify the covered institution "as soon as possible, but no later than 72 hours after becoming aware that a breach in security has occurred resulting in unauthorized access to a customer information system maintained by the service provider."

Ask each provider how it tests for prompt injection, isolates customers' data in shared retrieval stores and logs tool calls (interpretation). Our guides to AI vendor due diligence and AI contract terms cover the questions and clauses.

Checklist

Items restate the recommendations above (interpretation).

• Every LLM application is in the asset inventory and the risk assessment
• Each untrusted input channel is listed and threat-modeled for direct and indirect injection
• Tools and service identities are least-privilege; authorization is enforced outside the model
• Retrieval checks the requesting user's entitlements
• High-impact actions require human approval
• Model output is validated before any system acts on it; no secrets sit in system prompts
• Data passed to models is minimized, with retention set for prompts, logs and vector stores
• Injection and leakage tests rerun on every model, prompt or tool change
• Prompt and output logs feed monitoring and incident playbooks with the 72-hour and 30-day clocks
• Vendor contracts cover AI data use, testing and breach notice

How we sourced this guide: every factual statement in it is drawn from published rules, regulatory guidance and other public documents, and we keep a full set of sources and citations for it. If you would like them, or want to question a point, get in touch through our contact page and we will share them.