AI Safety, Alignment & Interpretability

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and evaluation. Integrates with CrewAI, Agno, OpenAI Agents SDK, LangChain, Autogen, AG2, and CamelAI. MIT licensed.

Open-source agentic framework for automated behavioral evaluations of frontier AI models. Generates targeted evaluation suites to probe LLMs for specific behaviors (sycophancy, self-preservation, political bias, etc.) with quantitative elicitation rates. From Anthropic's safety research team. MIT licensed.

Apache-2.0 corpus of 537 test cases and a TypeScript runner for benchmarking AI agent guardrail products, maintained by a vendor whose own product it scores.

Complete recipes for full-stack alignment.

Fit interpretable models and explain blackbox machine learning with state-of-the-art explainability techniques including Explainable Boosting Machines and SHAP-based explanations.

Gold-standard for mechanistic interpretability.

Sparse autoencoders for interpretable features.

Library for inspecting, tracing, and intervening on neural network internals at scale.

PyTorch's official interpretability library.

Easy-to-use knowledge editing framework for LLMs. Enables precise modification of model knowledge and behavior to correct hallucinations or outdated information. ACL 2024. MIT licensed.

Comprehensive AI explainability toolkit with interpretability algorithms for data and machine learning models. Includes TED, BRCG, and ProtoNN methods for diverse explanation needs. Apache 2.0 licensed.

User-friendly explainability library for transparent ML models. Beautiful visualizations with explicit labels that everyone can understand. Generates web reports and integrates with SHAP/LIME. Apache 2.0 licensed.

Comprehensive toolkit for detecting, understanding, and mitigating unwanted algorithmic bias in datasets and ML models.

Python package to assess and improve fairness of machine learning models. Provides metrics for disparity assessment and algorithms for unfairness mitigation with scikit-learn integration. MIT licensed.

Python Risk Identification Tool for generative AI. Microsoft's open-source framework for automated red teaming with multi-modal attack support, crescendo strategies, and 100+ operations experience. MIT licensed.

Open-source system for automatic censorship and robustness suppression removal in language-model outputs.

Agentic LLM vulnerability scanner and AI red teaming kit with multi-step attack simulation and automated security probing. Apache 2.0 licensed.

Programmable guardrails toolkit for LLM-based conversational systems. Uses Colang DSL to define safety rules, dialog flows, and content boundaries. Integrates with LangChain, LangGraph, and LlamaIndex for production deployments. Apache 2.0 licensed.

Input/output validation framework for building reliable AI applications. Detects and mitigates risks through composable validators for PII, toxicity, prompt injection, and structured output validation. Features Guardrails Hub with 50+ pre-built validators. Apache 2.0 licensed.

Trained models and code to predict toxic comments on all 3 Jigsaw Toxic Comment Challenges. Built using PyTorch Lightning and Transformers for toxicity, severe toxicity, obscene, threat, insult, identity attack, and sexual explicit content detection. Apache 2.0 licensed.

AI-powered agentic red team framework that automates offensive security operations from reconnaissance to exploitation to post-exploitation with zero human intervention. Integrates multiple security tools for comprehensive penetration testing. MIT licensed.

Cybersecurity AI framework for semi- and fully-automating offensive and defensive security tasks. Purpose-built for cybersecurity use cases with agent-based architecture for vulnerability assessment and security operations. MIT licensed.

Full-stack AI Red Teaming platform securing AI ecosystems via OpenClaw Security Scan, Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak evaluation. Apache 2.0 licensed.

AI agent framework for black-box security testing, supporting bug bounty, red-team, and penetration testing workflows. MIT licensed.