Rate this Page
★ ★ ★ ★ ★

OpenApp Environment#

OpenApps Environment

A web application simulation environment for OpenEnv that wraps the OpenApps framework and BrowserGym.

Overview#

The OpenApp environment provides a simulated web application ecosystem where agents can interact with various apps (calendar, todo, messenger, maps) using browser-based actions.

OpenApps Demo

This environment is ideal for:

  • Training and evaluating UI agents

  • Testing web automation strategies

  • Researching human-computer interaction

  • Developing multimodal agents

Features#

  • Multiple Apps: Interact with calendar, todo list, messenger, and map applications

  • Browser-Based Actions: Click, fill forms, navigate, scroll, and more

  • Task-Based Evaluation: Optional task goals with automatic reward calculation

  • Configurable: Customize app configurations and behavior

  • BrowserGym Integration: Built on top of BrowserGym for robust browser interaction

Directory Structure#

openapp_env/
├── __init__.py                   # Package exports
├── client.py                     # HTTP client for connecting to OpenApp
├── models.py                     # Data models for actions and observations
├── pyproject.toml                # Package dependencies and configuration
├── openenv.yaml                  # OpenEnv environment configuration
├── test_openapp_env.py           # Unit tests for environment structure
├── README.md                     # This file
├── IMPLEMENTATION.md             # Implementation details and design decisions
├── example_usage.py              # Basic usage example (legacy)
├── assets/                       # Images and media
│   ├── OpenApps_OpenEnv_RL.png   # Environment overview diagram
│   └── openapps-demo.gif         # Demo animation
└── server/                       # Server-side environment implementation
    ├── __init__.py
    ├── app.py                    # FastAPI server application
    ├── openapp_environment.py    # Core environment logic (BrowserGym + OpenApps)
    ├── Dockerfile                # Docker image definition
    └── start.sh                  # Container startup script (runs both servers)

Key Components:

  • client.py: OpenAppEnv class that extends HTTPEnvClient for remote environment interaction

  • models.py: OpenAppAction and OpenAppObservation dataclasses with validation

  • server/openapp_environment.py: OpenAppEnvironment class that wraps BrowserGym and OpenApps

  • server/app.py: FastAPI server that exposes the environment via HTTP endpoints

  • server/Dockerfile: Self-contained Docker image with OpenApps server and FastAPI server

  • server/start.sh: Startup script that launches both OpenApps (port 5001) and FastAPI (port 8000)

Installation#

There are two ways to use the OpenApp environment: Docker mode (recommended, fully self-contained) or Local mode (requires manual server setup).

Option 2: Local Mode#

Local mode requires manual setup of the OpenApps server. This mode is useful for development or when you need to customize the OpenApps configuration.

Prerequisites:

  • Python 3.11+ installed

  • UV package manager (recommended) or pip

Step 1: Install openapp_env

cd envs/openapp_env
pip install -e .

This installs the environment package along with dependencies (BrowserGym, Playwright, etc.).

Step 2: Install Playwright browsers

playwright install chromium

Step 3: Clone and set up OpenApps (for running the server)

# Clone OpenApps repository
git clone https://github.com/facebookresearch/OpenApps.git
cd OpenApps

# Install dependencies
uv sync  # or: pip install -e .

Why do I need the OpenApps repository?

The OpenApps Python package (installed via pip in Step 1) provides the library code, but the repository contains:

  • launch.py - The server startup script

  • config/ - Hydra configuration files

  • Application templates and assets

In Docker mode, all of this is included in the container, so you don’t need to clone anything.

Quick Start#

Running Locally#

For local usage, you need the OpenApps repository to run the server:

Step 1: Clone OpenApps (if you haven’t already)

git clone https://github.com/facebookresearch/OpenApps.git
cd OpenApps
uv sync

Step 2: Start OpenApps Server (in terminal 1)

To run the server in headless mode (no browser window):

cd OpenApps  # or wherever you cloned it
uv run launch.py

# or instead of the uv run you can use the Python command:
python OpenApps/launch.py

To run the server with visible browser for visualization:

cd OpenApps
python OpenApps/launch.py browsergym_env_args.headless=False

Wait for the server to start (you’ll see “Port 5001 is available” or similar).

Step 3: Run your code (in terminal 2)

export OPENAPPS_URL=http://localhost:5001
python examples/openapp_example.py --mode local

Note: The OpenApps Python package (installed via pip) provides the modules, but you need the full repository to run launch.py with its config files.

Example Script#

# Run with Docker (recommended)
python examples/openapp_example.py --mode docker

# Run locally (requires OpenApps server running)
export OPENAPPS_URL=http://localhost:5001
python examples/openapp_example.py --mode local

# Show browser window to visualize agent actions
python examples/openapp_example.py --mode local --show-browser

# Run with custom number of steps
python examples/openapp_example.py --mode docker --num-steps 20

# See all options
python examples/openapp_example.py --help

Visualizing Agent Interactions#

There are multiple ways to see what the agent is doing:

Option 1: Show Browser Window (Local Mode)

The key is to start the OpenApps server with visualization enabled:

# Terminal 1: Start OpenApps server with visible browser
cd OpenApps
python OpenApps/launch.py browsergym_env_args.headless=False

# Terminal 2: Run your agent code
export OPENAPPS_URL=http://localhost:5001
python examples/openapp_example.py --mode local

Important: The browser visualization is controlled by the OpenApps server, not the client. You must launch the server with browsergym_env_args.headless=False to see the browser window.

Option 2: Access Web Interface Directly

While the OpenApps server is running, open your browser to:

  • Main page: http://localhost:5001

  • Calendar: http://localhost:5001/calendar

  • Todo: http://localhost:5001/todo

  • Messenger: http://localhost:5001/messages

  • Maps: http://localhost:5001/maps

Option 3: Docker Web Interface

When running in Docker mode, you can also access a web interface for manual testing:

# Start a container and keep it running
docker run -d -p 8000:8000 openapp-env:latest

# Access the web interface
# - Interactive UI: http://localhost:8000/web
# - API docs: http://localhost:8000/docs
# - OpenApps (internal): http://localhost:5001 (inside container)

Note: In Docker mode, the OpenApps server runs inside the container and is not directly accessible from your host machine. The FastAPI server at port 8000 acts as a proxy to interact with OpenApps.

Basic Usage#

from envs.openapp_env import OpenAppAction, OpenAppEnv

# Create environment from Docker image
client = OpenAppEnv.from_docker_image("openapp-env:latest")

# Reset to initial state
result = client.reset()
print(f"Starting URL: {result.observation.url}")

# Navigate to calendar app
result = client.step(OpenAppAction(
    action_type="goto",
    url="http://localhost:5001/calendar"
))

# Click on a button (example bid)
result = client.step(OpenAppAction(
    action_type="click",
    bid="add-event-btn"
))

# Fill in a form field
result = client.step(OpenAppAction(
    action_type="fill",
    bid="event-title-input",
    text="Team Meeting"
))

print(f"Reward: {result.reward}")
print(f"Done: {result.done}")

# Cleanup
client.close()

Action Types#

The environment supports the following action types:

  • click: Click on an element

    • Required: bid (BrowserGym element ID)

  • fill: Fill a text input field

    • Required: bid, text

  • select_option: Select from dropdown

    • Required: bid, value

  • goto: Navigate to a URL

    • Required: url

  • scroll: Scroll the page

    • Required: direction (“up” or “down”)

  • send_keys: Send keyboard input

    • Required: text

  • noop: No operation

Observations#

Each observation includes:

  • html: Current page HTML content

  • url: Current page URL

  • open_pages_urls: List of all open page URLs

  • active_page_index: Index of currently active page

  • screenshot: Base64-encoded screenshot (optional)

  • axtree_txt: Accessibility tree for element interaction

  • app_state: Current state of all apps (calendar events, todos, messages, etc.)

  • task_info: Information about current task (if using tasks)

  • last_action_error: Error message if last action failed

Configuration#

Environment Parameters#

from envs.openapp_env.server.openapp_environment import OpenAppEnvironment

env = OpenAppEnvironment(
    web_app_port=5001,                   # Port for OpenApps server
    headless=True,                       # Run browser in headless mode
    task_name="add_meeting",             # Optional task name
    apps_config={},                      # App-specific configuration
    max_steps=50,                        # Maximum steps per episode
)

Note: OpenApps is automatically detected from the installed Python package. You can optionally override with openapps_path parameter or OPENAPPS_PATH environment variable if needed.

Tasks and Rewards#

The environment can be configured with specific tasks from OpenApps. Tasks define:

  • Goal state (e.g., “Add a meeting with Dennis to the calendar”)

  • Reward function based on app state changes

  • Success criteria

See OpenApps documentation for available tasks.

Example: Task-Based Training#

from envs.openapp_env import OpenAppAction, OpenAppEnv

# Create environment with a specific task
client = OpenAppEnv.from_docker_image("openapp-env:latest")

# The task will guide the agent toward a specific goal
# Rewards will be based on progress toward completing the task
result = client.reset()

# Agent interacts to complete the task
# ... agent logic here ...

client.close()

Development#

Running Server Locally (without Docker)#

cd envs/openapp_env
uv run server

The server will start at http://localhost:8000

Testing#

from openapp_env.server.openapp_environment import OpenAppEnvironment
from openapp_env.models import OpenAppAction

def test_environment():
    env = OpenAppEnvironment()

    # Test reset
    obs = env.reset()
    assert obs.url != ""

    # Test step
    action = OpenAppAction(action_type="noop")
    obs = env.step(action)
    assert env.state.step_count == 1

    # Cleanup
    env.close()

test_environment()

Attribution#

This environment integrates:

  • OpenApps - Web application simulation framework

  • BrowserGym - Browser automation environment

Troubleshooting#

Docker Build Issues#

Error: Container did not become ready

If you’re behind a corporate proxy (Meta/Facebook networks), set NO_PROXY:

export NO_PROXY=localhost,127.0.0.1
docker build -t openapp-env:latest -f envs/openapp_env/server/Dockerfile .

Error: Environment variable 'USER' not found

This is automatically handled in the Dockerfile with ENV USER=root. If you see this, rebuild the image.

Container exits immediately

Check the logs to see which server failed:

docker logs <container-id>

Common causes:

  • OpenApps server failed to start (check for port conflicts)

  • Missing dependencies (rebuild with --no-cache)

Local Mode Issues#

Error: OPENAPPS_URL not set

Set the environment variable before running:

export OPENAPPS_URL=http://localhost:5001
python examples/openapp_example.py --mode local

Error: Connection refused to localhost:5001

Make sure the OpenApps server is running:

cd OpenApps
uv run launch.py

Browser visualization not working

The visualization is controlled by the server, not the client:

# Start server with visible browser
cd OpenApps
python launch.py browsergym_env_args.headless=False

Performance Issues#

Docker container is slow

The container runs both a full Chromium browser and web applications. For faster performance:

  • Increase Docker memory allocation (6GB+ recommended)

  • Use headless mode (default)

  • Reduce max_steps in environment configuration

Large Docker image size

The image is ~5.7GB due to:

  • Chromium browser (~1.5GB)

  • OpenApps dependencies (~2GB)

  • BrowserGym and ML libraries (~2GB)

This is expected for a full browser automation environment.

License#

BSD 3-Clause License (see LICENSE file in OpenEnv root directory)

Citation#

If you use this environment in your research, please cite both OpenEnv and OpenApps:

@article{ullrich2025openapps0,
  title   = {OpenApps: Simulating Environment Variations to Measure UI-Agent Reliability},
  author  = {Karen Ullrich and Jingtong Su and Claudia Shi and Arjun Subramonian and Amir Bar and Ivan Evtimov and Nikolaos Tsilivis and Randall Balestriero and Julia Kempe and Mark Ibrahim},
  year    = {2025},
  journal = {arXiv preprint arXiv: 2511.20766}
}
On this page
Edit on GitHub
Show Source
PyTorch Libraries