FinArena — Banking Fraud Agent Benchmark

How to Participate

What is FinArena?

FinArena benchmarks LLM agents on multi-turn banking fraud investigation scenarios. Your agent plays the role of a professional fraud analyst: it receives an alert about a customer, must call the right banking tools in the right order, reach the correct decision, and respond to the customer — all within a live OBP (Open Bank Project) environment.

Quick Start

1. Download model.py from the Files tab under solution/
2. Set your API credentials — edit api_url, api_key, and model_name in model.py
3. Zip and submit — upload to the platform
4. View results — leaderboard and a standalone HTML report are generated automatically

Required Files

Your submission must contain:

submission.zip
├── model.py          # Your agent implementation (required)
└── requirements.txt  # Extra Python dependencies (optional)

Agent Interface

Implement one class with one method:

class FinArenaAgent:
    def __init__(self):
        self.api_url   = "https://your-litellm-endpoint/v1/chat/completions"
        self.api_key   = "YOUR_API_KEY"
        self.model_name = "your-model-name"

    def chat(self, messages: list[dict], tools: list[dict]) -> dict:
        """
        Receive the current conversation + available tool schemas.
        Return an OpenAI-compatible message dict.

        Args:
            messages  — conversation history in OpenAI format
            tools     — list of tool schemas the model may call

        Returns:
            An assistant message dict, e.g.:
            {
              "role": "assistant",
              "content": null,
              "tool_calls": [{"id": "...", "type": "function",
                              "function": {"name": "freeze_card",
                                           "arguments": "{\"card_id\": \"abc123\"}"}}]
            }
        """
        ...

The benchmark's ingestion program manages the full conversation loop, tool execution against the live OBP bank, reward scoring, and result logging. You only implement chat().

Available Tools

Your agent has access to these banking tools:

Evaluation Environment

• Each submission is run inside an isolated container with a live PostgreSQL + OBP API instance
• Bank data (customers, accounts, transactions, cards) is seeded fresh for each run
• Your agent interacts with the real OBP REST API via the tool wrappers above
• The benchmark runs 1 RL step with 1–4 rollouts per scenario by default