# Evaluation and scoring

> How each solution is checked and how the scores add up.

Source: <https://frontieror-challenge.com/docs/self-hosted/scoring>

## Which instances count

Every submission covers all of the open round's private Self-hosted instances. They are separate from the Main track's instances and are never published. Public instances are for development and don't count.

## What can happen to an instance

After the submission closes, the checker verifies each uploaded solution and recomputes its objective.

| Result                                  | Score  | Message                                                       |
| --------------------------------------- | ------ | ------------------------------------------------------------- |
| Feasible                                | Scored | –                                                             |
| No solution uploaded                    | 0      | No solution submitted                                         |
| Infeasible                              | 0      | Infeasible: followed by the constraints the solution violates |
| The checker could not read the solution | 0      | Checker exited with code 1                                    |
| A problem on our side, 3 times          | 0      | Evaluation failed on our side                                 |

The checker recomputes the objective from your decisions, so a claimed `objective_value` that does not match them is rejected. A problem on our side, such as a checker that fails to start, is retried a minute later; only after 3 failed tries does the instance score 0.

## Instance score

A feasible solution scores the ratio of its objective to the instance's reference objective, a frozen best-known value. Matching the reference scores 1, a worse solution less, and a better one more, up to 2.

| Objective and reference                       | Minimize                | Maximize                |
| --------------------------------------------- | ----------------------- | ----------------------- |
| Both positive                                 | reference ÷ objective   | objective ÷ reference   |
| Both negative                                 | objective ÷ reference   | reference ÷ objective   |
| On opposite sides of 0, or the reference is 0 | 2 if better, 0 if worse | 2 if better, 0 if worse |

Equal values score 1, including both 0. The objective is the one the checker recomputes from your decisions. Infeasible solutions and solutions the checker rejects score 0.

## From instances to a submission score

Instance scores are averaged within each problem, then across problems, so every problem weighs the same however many instances it has.

## Leaderboard

- The leaderboard is per stage and per track. Self-hosted and Main are ranked separately and never compared.
- A team's stage score comes from one **Completed** submission per round: the one with the highest score, then the least time used, then the earliest. Its problem scores are averaged over every problem released so far. A round the team skipped counts 0 on its problems, and so does the open round until the team submits to it.
- Teams are ranked by that score, highest first. Equal scores are ranked by time used, lowest first: each private instance's time from download to upload as a share of its time limit plus the 60-second grace, or 1 without a feasible solution, averaged like the score. A skipped round's problems count 1, so leaving work undone never helps.
- Next to the score, **Avg passed** is the mean over only the problems your round-best submissions passed (a feasible solution for every private instance, none failed), and **Avg submitted** the mean over those they uploaded at least one solution for. Under each, how many of the released problems that is ("5 of 6"). Neither is used for ranking.
- The menu and the sub-track cards rank teams on one category or one formulation instead: the average of its released problems' scores in each team's round-best submissions, with ties broken by time used on the same problems.

## What you see

For every instance: its score, or the message it scored 0 with.