Contribute to HarnessEval

Everyone is welcome to contribute to HarnessEval. Contact us with case or skill contributions, evaluation model additions, or other concrete suggestions.

Select a contribution type

Case / Taxonomy

Contribute an evaluation case.

Define the initial world, action, observation, and taxonomy.

Skill

Contribute an evaluator skill.

Define one evaluation question, its contract, implementation, and tests.

Submission template

submission.json

{
  "schema_version": "harnesseval.case_submission.v1",
  "kind": "case",
  "case_id": "replace_with_case_id",
  "title": "Concise case title",
  "evaluation_question": "What observable behavior should this case evaluate?",
  "taxonomy": {},
  "world": {
    "initial_observation": "assets/initial_observation.png"
  },
  "action": {
    "type": "physical_parameter_condition",
    "text": "Describe the requested action and constraints."
  },
  "provenance": {
    "author": "github-user",
    "license": "Apache-2.0",
    "redistribution_allowed": true
  }
}

Acknowledging Contributions

We welcome contributions and concrete suggestions. Contributions accepted following human review will be acknowledged, and substantial contributions may be considered for authorship in the next version. We invite everyone to help build and improve this benchmark together.