Тема безопасности искусственного интеллекта очень молодая и активно развивается. В первую очередь она делает это в англоязычном пространстве, из-за чего многие термины (те же LLM) приходят к нам в английском виде, а на русском или не существуют, или выглядят непривычно и иногда даже забавно (БЯМ). Это - пополняемый двуязычный словарь терминов с переводом на русский язык, в котором мы попытаемся дать русскоязычные замены для стандартных для сферы варваризмов.

Любой перевод дискуссионен.

Alignment - согласование.

Подразумевается, что результаты работы модели согласуются с целями, с которыми человек ее обучает и применяет, а методы достижения этих целей - с определенным набором ценностей. Перевод согласованность возможен, если под alignment подразумевается именно результат, а не процесс. Менее удачным кажется выравнивание (выравнивание с ценностями? относительно ценностей?). Вероятно, хорошим переводом может быть гармонизация (гармонизированная модель, гармонизация с ценностями, проблема гармонизации), но момент скорее всего быть упущен.

Capability uplift

Evaluation harness

Large language model (LLM) - большая языковая модель.

То, что в английском существует аббревиатура, не значит, что она всенепременно должна существовать на русском: скажите на встрече с клиентом, что ваш проект использует самые мощные бямы от ведущих бям-провайдеров, и посмотрите на результат.

Джейлбрейки

• Jailbroken: How Does LLM Safety Training Fail?, Wei et al., 2023
• Universal and Transferable Adversarial Attacks on Aligned Language Models, Zou et al., 2024
• AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models, Liu et al., 2024
• MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots, Deng et al., 2023
• Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study, Liu et al., 2023
• Jailbreaking Black Box Large Language Models in Twenty Queries, Chao et al., 2023
• Tree of Attacks: Jailbreaking Black-Box LLMs Automatically, Mehrotra et al., 2023
• Fundamental Limitations of Alignment in Large Language Models, Wolf et al., 2023
• Summon a Demon and Bind it: A Grounded Theory of LLM Red Teaming in the Wild, Inie et al., 2023
• ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs, Jiang et al., 2024
• Refusal in Language Models Is Mediated by a Single Direction, Arditi et al, 2024
• Does Refusal Training in LLMs Generalize to the Past Tense?, Andriushchenko and Flammarion, 2024
• Best-of-N Jailbreaking, John Hughes et al., 2024
• Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack, Mark Russinovich et al, Microsoft, 2023
• Removing RLHF Protections in GPT-4 via Fine-Tuning, Qiusi Zhan et al., 2023
• Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models, Xianjun Yang et al, 2023
• LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B, Simon Lermen et al, 2023
• Obfuscated Activations Bypass LLM Latent-Space Defenses, Bailey et al., 2024
• Fast Adversarial Attacks on Language Models In One GPU Minute, Sadasivan et al., University of Maryland, 2024
• Adversarial Poetry as a Universal Single-Turn Jailbreak Mechanism in Large Language Models, Bisconti et al., 2025
• In-Context Representation Hijacking, Yona et al., 2025

Prompt Injection

• Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection, Greshake at a.l, 2023
• ComPromptMized: Unleashing Zero-click Worms that Target GenAI-Powered Applications, Cohen et al., 2024
• A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systems, Wu et al., 2024
• The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions, Wallace et al., 2024
• Knowledge Return Oriented Prompting (KROP), Martin et al., 2024
• RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injection, Wen at al., 2025

Атаки на агентные системы

• Data Exfiltration from Slack AI via indirect prompt injection, PromptArmor, 2024
• Hacker Plants Computer 'Wiping' Commands in Amazon's AI Coding Agent
• Invitation Is All You Need! TARA for Targeted Promptware Attack Against Gemini-Powered Assistants, Nassi et al., 2025
• ForcedLeak: AI Agent risks exposed in Salesforce AgentForce, Sasi Levi, Noma Security, 2025
• Breaking down ‘EchoLeak’, the First Zero-Click AI Vulnerability Enabling Data Exfiltration from Microsoft 365 Copilot, Itay Ravia, Aim Labs, 2025
• CVE-2025-53773 - Visual Studio & Copilot – Wormable Command Execution via Prompt Injection, Persistent Security, 2025
• CamoLeak: Critical GitHub Copilot Vulnerability Leaks Private Source Code Legit Security, 2025
• Notion AI: Unpatched Data Exfiltration, PromptArmor, 2026

Inference Cost / Sponge-атаки

• OverThink: Slowdown Attacks on Reasoning LLMs, Kumar et al., University of Massachusetts Amherst, 2025
• Trapping misbehaving bots in an AI Labyrinth, Tatoris, Saxena and Miglietti, Cloudflare, 2025

LLM misuse

• An update on disrupting deceptive uses of AI, Nimmo & Flossman, OpenAI, 2024
• Adversarial Misuse of Generative AI, Google Threat Intelligence Group, 2025
• Disrupting malicious uses of AI: February 2025 update, Nimmo et al., OpenAI, 2025
• Unmasking EncryptHub: Help from ChatGPT & OPSEC blunders, Kraken Labs, Outpust24, 2025
• LLM Agent Honeypot: Monitoring AI Hacking Agents in the Wild, Reworr and Dmitrii Volkov, Palisade Research, 2024
• Disrupting malicious uses of AI: June 2025, OpenAI, 2025
• Threat Intelligence Report: August 2025, Anthropic, 2025
• Disrupting malicious uses of our models: an update, October 2025, OpenAI, 2025
• GTIG AI Threat Tracker: Advances in Threat Actor Usage of AI Tools, Google Threat Intelligence Group, 2025
• Disrupting the first reported AI-orchestrated cyber espionage campaign
  Anthropic, 2025

LLM в offensive security

• Evaluating Large Language Models' Capability to Launch Fully Automated Spear Phishing Campaigns: Validated on Human Subjects, Heiding et al., 2024
• Catastrophic Cyber Capabilities Benchmark (3CB): Robustly Evaluating LLM Agent Cyber Offense Capabilities, Anurin et al., Apart Research, 2024
• Evaluating AI cyber capabilities with crowdsourced elicitation, Petrov and Volkov, Palisade Research, 2025
• XBOW Unleashes GPT-5’s Hidden Hacking Power, Doubling Performance, De Moor, Ziegler, XBOW, 2025

LLM в киберзащите

• BinMetric: A Comprehensive Binary Analysis Benchmark for Large Language Models, Shang et al., Hefei University of Science and Technology, 2025
• LLM4Decompile: Decompiling Binary Code with Large Language Models, Tan et al., 2025
• DecompileBench: A Comprehensive Benchmark for Evaluating Decompilers in Real-World Scenarios, Gao et al., 2025
• CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning, Deason et al., 2025

Защита LLM-систем

• Baseline Defenses for Adversarial Attacks Against Aligned Language Models, Jain et al., 2023
• Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes, Hu et al., 2024
• Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations, Inan et al., 2023
• ShieldGemma: Generative AI Content Moderation Based on Gemma, ShieldGemma Team, Google LLC, 2024
• Rapid Response: Mitigating LLM Jailbreaks with a Few Examples, Peng et al., 2024
• Defending Against Indirect Prompt Injection Attacks With Spotlighting, Keegan Hines et al, Microsoft, 2024
• Are you still on track!? Catching LLM Task Drift with Activations, Abdelnabi et al., 2024
• Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming, Mrinank Sharma et al., Anthropic. 2025
• The Dual LLM pattern for building AI assistants that can resist prompt injection, Simon Willison, 2023
• LlamaFirewall: An open source guardrail system for building secure AI agents, Chennabasappa et al, Meta, 2025
• Introducing gpt-oss-safeguard, OpenAI & ROOST, 2025
• Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreaks, Cunningham et al., Anthropic, 2026
• Qwen3 Guard, Qwen Team, 2025

Бенчмарки

• Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models, Bhatt et al., 2023
• CYBERSECEVAL 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Models, Bhatt et al., 2024
• CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models, Wan et al., 2024
• AIR-BENCH 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policies, Zeng et al., 2024
• AgentDojo: A Dynamic Environment to Evaluate Attacks and Defenses for LLM Agents, Edoardo Debenedetti et al., 2024
• A StrongREJECT for Empty Jailbreaks, Souly et al., 2024
• Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models, Andy K. Zhang et al, Stanford, 2024
• The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning, Li et al, 2024

Policy

• The Coming Wave, Mustafa Suleyman, 2024
• AI existential risk probabilities are too unreliable to inform policy, Narayanan and Kapoor, 2024

Safety & Reliability

• Towards Understanding Sycophancy in Language Models, Sharma et al, 2023
• Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models, Denison et al, 2024
• AI Risk Categorization Decoded (AIR 2024): From Government Regulations to Corporate Policies, Zeng et al., 2024
• Constitutional AI: Harmlessness from AI Feedback, Bai et al., Anthropic, 2022
• Frontier Models are Capable of In-context Scheming, Alexander Meinke et al., Apollo Research, 2024
• Demonstrating specification gaming in reasoning models, Alexander Bondarenko et al., Palisade Research, 2025
• Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs, Jan Betley et al., 2025
• Reasoning models don't always say what they think, Chen et al., Anthropic, 2025
• Claude Sonnet 3.7 (often) knows when it’s in alignment evaluations, Apollo Research, 2025
• SHADE-Arena: Evaluating sabotage and monitoring in LLM agents, Kutasov et al., 2025
• Spiral-Bench, Samuel Paech, 2025

AI Alignment Course

• Week 1: AI and the Years Ahead
• Week 2: What is AI alignment

Model Stealing & Inversion

• LLMmap: Fingerprinting For Large Language Models, Pasquini et al., 2024
• Stealing Part of a Production Language Model, Carlini et al., 2024

Гайдлайны

• Google's Secure AI Framework: A practitioner’s guide to navigating AI security

Misc

• What Was Your Prompt? A Remote Keylogging Attack on AI Assistants, Weiss et al., 2024
• Smuggling arbitrary data through an emoji, Paul Butler, 2025
• LLM Backdoors at the Inference Level: The Threat of Poisoned Templates, Ariel Fogel, 2025, Pillar Security

Полезные каналы

• https://t.me/addlist/40D9BRf6rDoxNzg6 - большой список каналов на тему AI + Security
• https://t.me/pwnai
• https://t.me/rybolos_channel
• https://t.me/aisecnews
• https://t.me/kokuykin

Теги: AI Safety, AI Security, LLM Security, LLM Safety, Adversarial ML, AI in Cybersecurity, атаки на LLM, атаки на большие языковые модели, защита больших языковых моделей, разборы на русском языке