PettingZoo : Multi-Agent Reinforcement Learning

Last Updated : 18 Jul, 2025

PettingZoo is a Python library designed specifically for multi-agent reinforcement learning (MARL) research. Its aim is to provide a standardized, scalable and user-friendly interface analogous to what Gymnasium (formerly OpenAI Gym) provides for single-agent RL, but adapted for complex multi-agent scenarios. 

2
Playground Domain

In a game like playground domain, the Petting Zoo library can be used to simulate and manage multi-agent environments where multiple players (agents) interact, compete or cooperate, mimicking playground scenarios. It provides standardized APIs and environment setups that make it easy to develop, test and benchmark multi agent reinforcement learning algorithms within customizable playground inspired settings.

Installation and Usage

Base Installation:

pip install pettingzoo

With specific environments:

pip install 'pettingzoo[atari]'

Full dependency installation:

pip install 'pettingzoo[all]'

Environment Versioning

PettingZoo follows strict environment versioning (e.g _v3, _v4) for reproducibility and benchmarking, so researchers can confidently compare algorithms across standard versions.

Core Architecture and API

PettingZoo models environments as Agent Environment Cycles (AEC), which cleanly support both turn-based and simultaneous action multi-agent systems. This abstraction allows the library to represent a wide range of classic, cooperative and competitive MARL tasks in a unified manner.

Main Components

AECEnv (Agent Environment Cycle API):

  • Each agent acts in turn.
  • Agents actions, observations and rewards are processed one at a time in a cycle.
  • Example use:
Python 
from pettingzoo.butterfly import pistonball_v6
env = pistonball_v6.env()
env.reset()
for agent in env.agent_iter():
    obs, reward, terminated, truncated, info = env.last()
    action = get_action(obs) if not terminated and not truncated else None
    env.step(action)

Parallel API:

  • Used for simultaneous-action environments where all agents step together.
  • Facilitates faster batch processing and compatibility with vectorized RL frameworks.
  • Example use:
Python 
env = pistonball_v6.parallel_env()
observations = env.reset()
while True:
    actions = {agent: env.action_space(agent).sample() for agent in env.agents}
    obs, rewards, terminations, truncations, infos = env.step(actions)
    if all(terminations.values()) or all(truncations.values()):
        break

Environment Families

PettingZoo provides various environment suites, including:

  • Atari: Multiplayer Atari 2600 games
  • Butterfly: High-coordination cooperative games
  • Classic: Board and card games (Chess, TicTacToe, etc.)
  • MPE: Simple particle-based communication tasks
  • SISL: Cooperative environments focused on navigation and control tasks

Integration with RL Libraries

PettingZoo's popularity and broad adoption have resulted in widespread integration with major RL libraries:

  • Ray RLlib: Native support to train multi-agent policies using PettingZoo environments, with specialized tutorials available for Ray users.
  • Sample Factory: Supports PettingZoo’s Parallel API by wrapping environments in the PettingZooParallelEnv wrapper.
  • TorchRL: Includes a dedicated PettingZooEnv wrapper, enabling seamless use with PyTorch-based RL pipelines.
  • Various other libraries (CleanRL, Tianshou, AgileRL) use PettingZoo as their MARL benchmark suite.

Wrappers and Utilities

Since PettingZoo intentionally leaves out environment preprocessing logic, the companion library SuperSuit provides common wrappers for:

1. Frame stacking

  • Purpose: Provides agents with a temporal context by stacking the last N frames of observations as a single input.
  • How it helps: Many environments, especially those with visual inputs or partial observability, benefit because agents can "see" recent dynamics, helping them infer velocity, movement trends or remember prior states.

2. Observation normalization

  • Purpose: Scales or centers environmental observations (e.g., pixel values, continuous states) to a standard range, such as [0, with zero mean and unit variance.
  • How it helps: Prevents learning instabilities caused by widely varying input scales. Algorithms tend to converge faster and are less sensitive to initialization, hyperparameters or outliers.

3.Action discretization

  • Purpose: Converts continuous action spaces into discrete bins or applies simple mappings (useful for RL algorithms that only handle discrete actions).
  • How it helps: Expands compatibility allows algorithms designed for discrete action spaces to work with a broader variety of environments.

4. Environment vectorization

  • Purpose: Runs multiple instances of the environment in parallel (vectorized), allowing agents to collect experience more efficiently.
  • How it helps: Greatly accelerates training by increasing sample throughput, improving exploration and supporting data parallelism. This is especially useful for scalable RL research and benchmarking.

Example - Usage Pattern

A typical workflow might be:

Step 1: Install PettingZoo and SuperSuit

Python 
!pip install pettingzoo
!pip install 'pettingzoo[butterfly]'
!pip install supersuit

SuperSuit is a useful library for preprocessing, but is optional for basic demos.

Step 2: Import and Initialize Environment

  • Import the appropriate environment (here, pistonball_v6).
  • render_mode can be set to "human" to show a visualization window (in local Jupyter setups) or "rgb_array" to produce numpy arrays as images.
  • Calling env.reset(seed=42) sets the random seed so runs are reproducible.
Python 
from pettingzoo.butterfly import pistonball_v6

# Initialize the environment 
env = pistonball_v6.env(render_mode="human") 
env.reset(seed=42)

Step 3: Interact with the Environment (Agent Environment Cycle API)

1. Agent Environment Cycle (AEC) API: In this API, the environment alternates control between agents each agent acts sequentially.

2. Loop Details:

  • env.agent_iter() gives the correct agent order.
  • env.last() returns the agent’s observation, reward and flags for termination/truncation.
  • If the agent is done (terminated or truncated), you pass None to env.step(); otherwise, you typically sample a random action or call your policy.
Python 
# Loop over the agents in order
for agent in env.agent_iter():
    observation, reward, termination, truncation, info = env.last()
    if termination or truncation:
        action = None  # No action if the episode is over for that agent
    else:
        # Replace with your own policy:
        action = env.action_space(agent).sample()
    env.step(action)
env.close()

Step 4: Alternative Interact Using Parallel API

The Parallel API steps all agents at once:

Python 
env = pistonball_v6.parallel_env(render_mode=None)
observations = env.reset(seed=42)

terminated = {agent: False for agent in env.agents}
truncated = {agent: False for agent in env.agents}

while not all(terminated.values()):
    actions = {agent: env.action_space(agent).sample() for agent in env.agents}
    observations, rewards, terminated, truncated, infos = env.step(actions)

Advantages of Peetingzoo

  • Standardized, easy-to-use API for multi-agent reinforcement learning.
  • Wide variety of built-in environments (Atari, board games, coordination tasks, etc.).
  • Supports both turn-based and simultaneous agent interactions.
  • Compatible with major RL libraries like Stable Baselines3 and RLlib.
  • Clean interface allows flexible integration and custom experimentation.
  • Accelerates research by enabling reproducible and comparable benchmarking
Comment

Explore