Skip to main content

🧪 Harmony AI Evaluation Metrics

Harmony AI provides tools for simulating real-world and adversarial scenarios to evaluate AI agents' robustness and reliability. It enables robust testing through persona generation, prompt variations, hostile prompt injection, and metric-based evaluations.

📈 Evaluation Metrics​

Use the following endpoints to score your AI agent's responses to generated prompts:

MetricEndpointExample Use Case
🟢 Stability/metrics/v3/stabilityResponse consistency for similar prompts
✅ Completion/metrics/v3/completionAccuracy and thoroughness of standard replies
❌ Refusal/metrics/v3/refusalRejection of inappropriate content
📌 Adherence/metrics/v3/adherencePolicy and rules compliance

🔹 Stability Evaluation /metrics/v3/stability​

Purpose: Ensure agents respond consistently to reworded prompts.

Inputs:

  • responses: List of responses (minimum 2)

Example Use Cases:

  • Testing paraphrase or persona-based variation responses.

🔹 Completion Evaluation /metrics/v3/completion​

Purpose: Evaluate whether the agent completes the task as requested.

Inputs:

  • query: User prompt
  • response: Agent reply

Example Use Cases:

  • Recipe assistant giving full ingredients and instructions.

🔹 Refusal Evaluation /metrics/v3/refusal​

Purpose: Check how clearly and appropriately the agent refuses harmful prompts.

Inputs:

  • query: Prompt
  • response: Agent reply

Example Use Cases:

  • Agent refusing to answer questions about illegal activity.

🔹 Adherence Evaluation /metrics/v3/adherence​

Purpose: Check if the agent strictly follows defined policies.

Inputs:

  • policy: Policy text
  • response: Agent reply

Example Use Cases:

  • Enforcing disclaimers in medical/legal responses.

🔐 Authentication & API Access​

All endpoints are secured via Bearer Token Authentication.

Headers Required:

Authorization: Bearer YOUR_API_KEY
Content-Type: application/json

Response Format: Standardized JSON responses with status, payload, and metadata.