Skip to content
Guardrails and security
- Guardrails AI - Runs input/output validators around a model call and enforces structured output.
- NeMo Guardrails - NVIDIA toolkit for input, output, dialogue, and tool rails defined in Colang and Python.
- OpenAI Guardrails - Drop-in OpenAI client wrapper that runs configurable input and output checks (keywords, secret keys, PII, URLs, moderation, jailbreak and prompt-injection detection) and raises on a tripwire; preview release.
- garak - LLM vulnerability scanner probing for jailbreaks, prompt injection, and data leakage.
- PyRIT - Microsoft AI Red Team’s toolkit that sends single-turn or adaptive multi-turn attacks, optionally transformed by converters such as Base64, to a model, an HTTP endpoint, or a browser-driven app, and scores the responses with deterministic or LLM-judge scorers, from Python, a CLI scanner, or a web GUI; MIT.
- DeepTeam - Red-teaming framework that simulates jailbreaks, prompt injection, and related attacks against LLM and agent apps, built on DeepEval.
- PurpleLlama - Meta’s safety repository: Llama Guard and Prompt Guard classifier models, LlamaFirewall, which runs Prompt Guard, alignment-check, regex, and CodeShield scanners over agent inputs and outputs and returns allow, block, or human review, CodeShield’s Semgrep and regex rules for insecure generated code, and the CyberSecEval benchmarks; code is MIT, the models use Llama community licenses.
- Presidio - PII detection, redaction, and anonymisation for text and images; useful for scrubbing traces before export.
- Snyk agent-scan - Security scanner for MCP servers, agents, and skills, formerly Invariant Labs’
mcp-scan; inspect lists what a config exposes locally, while scan needs a Snyk token and calls Snyk’s hosted API.
- MCP Scanner - Scans MCP servers’ tools, prompts, resources, and instructions for poisoning and exfiltration patterns using YARA rules, optional LLM and behavioural analyzers, and a Cisco API analyzer; Apache-2.0.