New: support for Claude Code and Codex CLI available

NIKA Break the Network. Put Your AI Agent On Call.

A unified platform for benchmarking AI agents on realistic network troubleshooting tasks.

View on GitHub Read the Paper Dataset
Scroll to explore
Overview

What is NIKA?

Think about SWE-bench, but for network troubleshooting. Agents interact with real CLIs and telemetry across hundreds of reproducible network incidents, all running on your laptop, cluster, or cloud.

NIKA Architecture Diagram

Benchmark Suite

NIKA contains 685 curated network incidents from data center networks to ISPs. Powered by state-of-the-art network emulators, the benchmark challenges AI agents against real-world protocols (e.g., BGP, VXLAN, DNS) and network stacks.

# pick a network, a fault, its target
scenario: dc-clos-bgp
problem: bgp-asn-misconfig
inject: { host_name: leaf0 }

Orchestration Platform

A modular platform that orchestrates incident replay, connects AI agents to the network environment via MCP servers for live diagnostic interactions, and provides human-facing evaluations of agent performance.

# deploy, run an agent, evaluate
nika env run dc-clos-bgp
nika agent run --agent claude
nika eval judge
Features

Built for AI + Networking Research

Kathará-based Environment

Standardised network emulation built on Kathará, enabling reproducible and realistic network topologies with real routing daemons.

Pre-built Scenarios & Faults

54 realistic faults spanning 6 categories, ready to inject into any scenario at any topology scale with a single CLI command.

MCP Tool Integration

Model Context Protocol servers expose network operations (ping, iperf, BGP/OSPF config, flow rules, telemetry) as callable tools for any MCP-compatible agent.

Reproducible Evaluation

Automatic evaluation via numeric metrics, LLM-as-judge scoring, and CSV publishing, ensuring fair and consistent comparisons across agents and models.

Network Scenarios

Live Networks and Workloads

Each network issue can be instantiated on multiple topologies, and tested with realistic workloads. Need more? Use the NIKA SDK to create your own network scenarios.

Scalable

Data Center (CLOS)

Multi-tier leaf–spine fabric with edge servers. Models modern hyper-scale data centre networking.

BGPECMPLeaf-Spine
Scalable

Campus Network (3-Tier)

Enterprise core–distribution–access topology. Represents typical university or corporate networks.

OSPFVLANsAccess
Scalable

ISP Backbone (Meshed)

Provider-style backbone with core and access nodes. Models Internet Service Provider infrastructure.

BGPMPLSMesh
Scalable

SDN Cloud POP (CLOS/Star)

SDN fabric with centralised controller and edge switches. Models software-defined cloud edge deployments.

OpenFlowSDNStar
In-Progress

llm-d AI Inference

Reproduce an LLM inference cluster and test resilience to network disruption: P/D disaggregation, load balancing, and disaggregated KV Cache storage.

VXLANFlannelOverlay
Beta

Multi-Tenant Microservices

Attach a k3s cluster running concurrent microservice applications, exposed via MetalLB in BGP mode and NGINX ingress.

k3sEBGPMetalLB
Define a Task

Define a New Task in Pure YAML

A benchmark test is just a short, declarative yaml definition. Choose a network, the size to run it at, the fault to introduce, and where it happens, NIKA takes care of the rest.

my_tests.yaml 
cases:
  - scenario: dc_clos_bgp
    topo_size: s
    problem: bgp_asn_misconfig
    inject:
      host_name: leaf0
  - scenario: ospf_enterprise_dhcp
    topo_size: m
    problem: dns_record_error
    inject:
      host_name: dns_server
      target_website: web0

Anatomy of a test

  • scenario

    Which network to run the test on.

  • topo_size

    How large to make it.

  • problem

    The fault to introduce.

  • inject

    Where the fault happens.

Want a brand-new network or fault of your own? The authoring guide walks through adding both.

Agents & Harnesses

Bring Any Agent, Any Model

Frontier models, coding assistants, and your own custom agents: swap them in and out of the NIKA benchmark with a single command.

Bring Your Own (BYO)

Already built an agent on a popular framework? Plug it in and benchmark it against everything else.

Coding Assistants

Put general-purpose coding assistants to the test on realistic network incidents, out of the box.

Native Agents

Want to focus on the creation of new tasks, not agent plumbing? Use default agents and go ahead.

Community

Explore agents and skills shared by the community, and contribute your own. Leaderboard is coming soon.

Quick Start

Up and Running in Minutes

Requires Kathará (Docker-based) and Python ≥ 3.12.

1

Install

Clone the repository, sync dependencies with uv, and activate the virtual environment.

bash 
# Clone the repository
git clone https://github.com/sands-lab/nika
cd nika

# Sync dependencies with uv and activate the venv
uv sync
source .venv/bin/activate
2

Configure

Copy .env.example to .env and fill in your API keys. NIKA ships no hard-coded defaults: every run needs a configured .env or explicit CLI flags.

.env 
# Copy the template, then fill in values
cp .env.example .env

# LLM API keys (in .env)
OPENAI_API_KEY=<your-key>
DEEPSEEK_API_KEY=<your-key>
ANTHROPIC_API_KEY=<your-key>

# Optional: observability (LangSmith / Langfuse)
LANGSMITH_TRACING=true
LANGSMITH_API_KEY=<your-key>
3

Deploy & Inject

List scenarios, deploy one (each run creates a session), then inject a fault with its parameters.

bash 
# List available scenarios
nika env list

# Deploy a scenario
nika env run dc-clos-bgp-link-flapping
4

Run an Agent & Evaluate

Run any agent against the session, then close it and evaluate with numeric metrics, an LLM judge, and a CSV summary.

bash 
# Run an agent against the session
nika agent run -a byo.langgraph -p openai -m gpt-5-mini -n 20

# Close the session, then evaluate
nika session close -y
nika eval metrics
nika eval judge -p openai -m gpt-5-mini

# Publish a summary CSV
nika eval summary
MCP Servers & Tools

Network Operations as Agent Tools

NIKA exposes network operations through MCP servers, letting any MCP-compatible agent (Claude, GPT-4, Gemini, open-source LLMs…) directly interact with the network environment.

Base Kathará Server

  • get_reachability: Ping all host pairs
  • iperf_test: Bandwidth test between two hosts
  • systemctl_ops: Start, stop, or restart services
  • get_host_net_config: Retrieve host network config
  • nft_list_ruleset: Dump nftables ruleset

BMv2 P4 Server

  • bmv2_get_log: Retrieve BMv2 switch logs
  • bmv2_get_counter_arrays: Read counter arrays

FRRouting Server

  • frr_get_bgp_conf: Retrieve BGP configuration
  • frr_get_ospf_conf: Retrieve OSPF configuration

INT / InfluxDB Server

  • influx_list_buckets: List InfluxDB buckets
  • influx_get_measurements: Retrieve measurements
  • influx_query_measurement: Query telemetry data

Generic Server

  • google_search: Web search for context

Task Management Server

  • list_avail_problems: List available problems
  • get_submission_template: Get submission template
  • submit: Submit a solution

More tools coming soon. You can also plug in any custom MCP server; browse mcp.so for the community catalogue.

Citation

Cite NIKA

If you use NIKA in your research, please cite the following papers.

NIKA Pre-print

ArXiv →
BibTeX 
@misc{nika,
  title   = {A Network Arena for Benchmarking AI Agents
             on Network Troubleshooting},
  author  = {Zhihao Wang and Alessandro Cornacchia and
             Alessio Sacco and Franco Galante and
             Marco Canini and Dingde Jiang},
  year    = {2025},
  eprint  = {2512.16381},
  archivePrefix = {arXiv},
  primaryClass  = {cs.NI},
  url     = {https://arxiv.org/abs/2512.16381},
}

NGNO Workshop at ACM SIGCOMM '25

DOI →
BibTeX 
@inproceedings{llm4netlab,
  author    = {Wang, Zhihao and Cornacchia, Alessandro
               and Galante, Franco and Centofanti, Carlo
               and Sacco, Alessio and Jiang, Dingde},
  title     = {Towards a Playground to Democratize
               Experimentation and Benchmarking of AI
               Agents for Network Troubleshooting},
  year      = {2025},
  doi       = {10.1145/3748496.3748990},
  booktitle = {Proceedings of the 1st Workshop on
               Next-Generation Network Observability},
  series    = {NGNO '25},
}