How Good Are the LLM Guardrails on the Market? A Comparative Study on the Effectiveness of LLM Content Filtering Across Major GenAI Platforms

This report evaluates the built-in guardrails of three anonymized cloud LLM platforms using 1,123 prompts (1,000 benign; 123 malicious/jailbreak) to measure input and output filtering efficacy at strict settings, focusing on false positives/negatives. Platform 3 blocked the most malicious prompts at input (~92%) but had high benign false positives (13.1%); Platform 2 achieved similar malicious blocking (~91%) with far fewer FPs (0.6%); Platform 1 was most permissive (~53% malicious blocked; 0.1% FPs). Output filters rarely triggered because model alignment refused most harmful requests (109/123), though some harmful outputs still bypassed output guardrails. The study concludes guardrail tuning requires balancing strictness and usability, with model alignment serving as the primary defense and guardrails providing complementary protection.