Gemini 2.0 Flash Security Report

February 2025 • Model Security & Safety Evaluation

Released
February 05, 2025

Compare Model Performance

Want to see how Gemini 2.0 Flash stacks up against other models? Use our comparison tool to analyze security metrics side by side.

Compare Models →

Google's Gemini 2.0 Flash launched in February 2025, introducing significant improvements in processing speed and context handling. Following its general availability announcement across Google AI Studio and Vertex AI, we conducted a comprehensive security evaluation to understand its safety measures and potential vulnerabilities.

As enterprise adoption increases, this analysis examines both the model's security features and areas for potential improvement.

Company Logo

"2.0 Flash delivers significant performance improvements across key benchmarks, with expanded capabilities including image generation and text-to-speech in development."Google Announcement, February 2025

About This Evaluation

Red Teaming

Open-source AI red teaming covering 50+ vulnerability types.

Learn more

Quick Start

Run your own security evaluation with our step-by-step guide.

View guide

Documentation

Learn more about our AI red teaming framework.

Read docs
About the Model
Let's take a closer look at the model before diving into our security findings.

Core Capabilities

Input Support

Text, Image, Video, Audio processing, Multimodal capabilities

Output Type

Text generation

Token Limits

1M input tokens input • 8k output tokens output

Use Cases

Primary Focus

Low latency scenarios

Key Applications

• Search as a tool • Code execution • Automating tasks

Availability

Access Points

• Google AI Studio • Gemini API • Vertex AI • Gemini App

Knowledge Cutoff

June 2024

Security Test Results

Comprehensive analysis across 39 test categories, sorted by severity and pass rate:

Performance Overview

Our security testing suite demonstrates varying levels of compliance across different categories. The Gemini 2.0 Flash model's overall pass rate distribution shows 3 critical, 5 high, 15 medium, and 16 low severity findings.

Top performing areas include ASCII Smuggling (100%), Unauthorized Commitments (82.22%), Divergent Repetition (77.78%). These categories show satisfactory security practices.

Areas requiring attention include Pliny Prompt Injections (0%), Disinformation Campaigns (17.78%), Overreliance (20%). These categories present opportunities for improvement in security measures.

Showing 39 of 39 tests
Type
Severity
Description
Pass Rate
Low
Tests vulnerability to Unicode tag-based instruction smuggling attacks
100%
Low
Tests handling of malicious prompts from the BeaverTails dataset
40%
Medium
Tests for unauthorized contractual commitments and legal exposure
82.22%
Medium
Tests prompt injection attacks from Meta's CyberSecEval dataset
44.44%
Medium
Tests for training data leaks through repetitive pattern exploitation that causes model divergence
77.78%
Medium
Tests for unauthorized actions beyond defined system boundaries
68.89%
Medium
Tests for fabrication of false or misleading information
62.22%
High
Tests handling of WMD-related content
42.22%
Critical
Tests handling of child exploitation content
57.78%
Low
Tests handling of intellectual property violations
33.33%
Low
Tests handling of cybercrime-related content
40%
Low
Tests creation of malicious code
37.78%
Medium
Tests handling of graphic or violent content
57.78%
Low
Tests handling of harassment and bullying content
44.44%
Critical
Tests handling of hate speech and discriminatory content
62.22%
Medium
Tests handling of general illegal activities
33.33%
Medium
Tests handling of illegal drug-related content
42.22%
Low
Tests handling of methamphetamine-related content
42.22%
Medium
Tests handling of weapons-related content
55.56%
Low
Tests handling of personal attacks and insults
42.22%
Medium
Tests handling of IP theft and violations
44.44%
Medium
Tests handling of false information campaigns
17.78%
Medium
Tests handling of non-violent criminal content
35.56%
Low
Tests handling of profane or inappropriate language
51.11%
Low
Tests handling of extremist content and recruitment
46.67%
Critical
Tests handling of self-harm and suicide content
62.22%
High
Tests handling of sexual crime content
51.11%
Medium
Tests handling of explicit sexual content
42.22%
Medium
Tests handling of unauthorized professional advice
44.44%
Low
Tests handling of dangerous activity instructions
35.56%
High
Tests handling of violent crime content
55.56%
Low
Tests handling of Improvised Explosive Devices
53.33%
High
Tests for unauthorized resource usage and purpose deviation
60%
Low
Tests for unauthorized impersonation of entities
31.11%
Low
Tests for overreliance on system assumptions
20%
High
Tests for direct PII exposure vulnerabilities
60%
Medium
Tests handling of Pliny prompt injections
0%
Low
Tests handling of political content and bias
55.56%
Low
Tests handling of religious content and bias
51.11%
Total Tests: 39
Critical: 3High: 5Medium: 15Low: 16

Our evaluation included over 1,600 test probes across four security categories. The analysis revealed both strengths and areas requiring additional safety measures:

Security & Access Control

68% Pass Rate

Protection against unauthorized access, data exposure, and system vulnerabilities

46 failed probes

Compliance & Legal

46% Pass Rate

Assessment of compliance with legal requirements and prevention of illegal content

392 failed probes

Trust & Safety

50% Pass Rate

Prevention of harmful content and protection of user safety

252 failed probes

Brand

46% Pass Rate

Protection of brand integrity and prevention of misuse

195 failed probes
Standards
Security analysis of Gemini 2.0 Flash against industry-standard frameworks for Large Language Models

OWASP Top 10 for LLMs (2025) and MITRE ATLAS represent the primary security assessment frameworks for large language models. OWASP focuses on critical deployment risks, while ATLAS maps adversarial techniques against AI systems, similar to the MITRE ATT&CK framework for traditional systems.

The Gemini 2.0 Flash model's evaluation against these frameworks reveals several areas requiring attention across 6 identified findings. While Gemini 2.0 Flash has no critical vulnerabilities, high-severity issues should be prioritized for improvement.

High-severity concerns include LLM02: Sensitive Information Disclosure.

MITRE ATLAS identified Prompt Injections and Jailbreak as high-severity concerns.

OWASP Top 10 for LLMs 2025 - Gemini 2.0 Flash Assessment

High Risk Areas

LLM02: Sensitive Information Disclosure

Moderate Risk Areas

LLM06: Excessive Agency
LLM09: Misinformation

MITRE ATLAS Findings for Gemini 2.0 Flash

High Severity Findings

Prompt Injections
Jailbreak

Moderate Severity Findings

Erode ML Model Integrity
Statistical Analysis
Comprehensive breakdown of test results across 39 security test categories

Total Test Categories

39

Average Pass Rate

48.3%

Critical Issues

3

Severity Distribution

critical3 tests
high5 tests
medium15 tests
low16 tests

Performance Summary

Security & Access Control

Pass Rate68.0%

Compliance & Legal

Pass Rate46.0%

Trust & Safety

Pass Rate50.0%

Brand

Pass Rate46.0%

Worst Performing Tests

Pliny Prompt Injections

Tests handling of Pliny prompt injections

0%

Disinformation Campaigns

Tests handling of false information campaigns

17.78%

Overreliance

Tests for overreliance on system assumptions

20%

Entity Impersonation

Tests for unauthorized impersonation of entities

31.11%

Illegal Activity

Tests handling of general illegal activities

33.33%

Best Performing Tests

ASCII Smuggling

Tests vulnerability to Unicode tag-based instruction smuggling attacks

100%

Unauthorized Commitments

Tests for unauthorized contractual commitments and legal exposure

82.22%

Divergent Repetition

Tests for training data leaks through repetitive pattern exploitation that causes model divergence

77.78%

Excessive Agency

Tests for unauthorized actions beyond defined system boundaries

68.89%

Self-Harm

Tests handling of self-harm and suicide content

62.22%

Key Observations

Strengths

  • Strong performance in handling ASCII Smuggling (100% pass rate)
  • Effective prevention in Unauthorized Commitments (82.22%)
  • Consistent performance across critical security tests

Areas for Improvement

  • Low pass rate (0%) for Pliny Prompt Injections
  • 3 critical severity issues identified
  • Average pass rate of 48.3% indicates room for improvement