Basic implementation of partial scoring. #3047

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Draft

meisterT wants to merge 14 commits into DOMjudge:main

from meisterT:partial_scoring

Draft

Basic implementation of partial scoring. #3047

meisterT wants to merge 14 commits into DOMjudge:main from meisterT:partial_scoring

Conversation

meisterT

Copy link

Member

@meisterT meisterT commented Aug 3, 2025

This follows the legacy spec in:
https://icpc.io/problem-package-format/spec/legacy.html#graders

We might decided at a later point whether we want to support this or just the new (currently draft) spec, but basic constructs of the change should be the same.

Not yet implemented:

any form of testing
anything on the team interface
shadowing

Part of #2518

@meisterT


 Basic implementation of partial scoring.

5c9d385

This follows the legacy spec in:
https://icpc.io/problem-package-format/spec/legacy.html#graders
We might decided at a later point whether we want to support this or
just the new (currently draft) spec, but basic constructs of the change
should be the same.
Not yet implemented:
- any form of testing
- anything on the team interface
- shadowing
Part of DOMjudge#2518

@meisterT meisterT force-pushed the partial_scoring branch from 3821281 to 5c9d385 Compare

August 3, 2025 17:34

meisterT added 3 commits

August 4, 2025 21:51

@meisterT


 Basic import of score while shadowing. No display or comparison yet.

bebe929

@meisterT


 Base scoring scoreboard in shadow mode on external scores.

414b06e

@meisterT


 Display more external results.

4ba5683

@meisterT

Copy link

Member Author

meisterT commented Aug 5, 2025

Some notes:

there's a to-do already, but we need to talk about internal precision and precision in presentation
we should add to the jury submission page some info that allows following the result calculation with test case groups (should be easy to do with the existing function)
the number of tries on a scoring scoreboard should really be the number of total tries (not until the first correct attempt like for pass-fail)
it seems we are not handling the accept_if_any_accepted case always correctly, needs more testing
while we can ingest results from a primary CCS now, we don't compare the scores yet for shadow differences
the layout of the shadow differences table needs to be fixed
optimization problems are by their very nature less deterministic as they incentivize teams to do stuff like try this approach for that amount of time, then this other approach for that amount of time and generally the use of RNG is much more prevalent

nickygerritsen

nickygerritsen reviewed

Aug 17, 2025

View reviewed changes

Copy link

Member

@nickygerritsen nickygerritsen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initial round of review done. Of course there are the todos and comments from PHPStan / PHPCS

webapp/src/Entity/Problem.php

@@ -211,6 +211,11 @@ class Problem extends BaseApiEntity implements

#[Serializer\Exclude]

private Collection $languages;

#[ORM\ManyToOne(inversedBy: 'problems')]

Copy link

Member

@nickygerritsen nickygerritsen Aug 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not seeing this reverse relation? And should it be manytoone? When does a testcasegroup have many problems?

Copy link

Member

@nickygerritsen nickygerritsen Aug 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And for me calling it "parent testcase group" sounds weird, maybe "root testcase group"?

Copy link

Member

@nickygerritsen nickygerritsen Aug 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did push the change for the reverse relation.

webapp/src/Service/ExternalContestSourceService.php

@@ -1855,6 +1859,10 @@ protected function importRun(Event $event, EventData $data): void

->setRuntime($runTime)

->setResult($judgementTypeId === null ? null : $verdictsFlipped[$judgementTypeId]);

if (isset($data->score)) {

$run->setScore($data->score);

Copy link

Member

@nickygerritsen nickygerritsen Aug 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment from PHPStan is not lying 😛

webapp/src/Service/ScoreboardService.php

@@ -259,6 +260,8 @@ public function calculateScoreRow(

$contestStartTime = $contest->getStarttime();

$pointsJury = "0";

Copy link

Member

@nickygerritsen nickygerritsen Aug 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not $scoreJury? We have points already for multipoint problems, so this might be confusing?

webapp/src/Service/SubmissionService.php

Comment on lines +62 to +67

/**

* Returns a two-element array:

* - The score for the testcase group, or null if not all results are ready.

* - The result for the testcase group, or null if not all results are ready.

* @return array{string|null, string|null}

Copy link

Member

@nickygerritsen nickygerritsen Aug 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would return a string indexed array to make this more clear:

Suggested change

/**

* Returns a two-element array:

* - The score for the testcase group, or null if not all results are ready.

* - The result for the testcase group, or null if not all results are ready.

* @return array{string|null, string|null}

/**

* Returns a two-element array:

* - The score for the testcase group, or null if not all results are ready.

* - The result for the testcase group, or null if not all results are ready.

* @return array{score: string|null, result: string|null}

nickygerritsen added 10 commits

August 17, 2025 14:08

@nickygerritsen


 Add reverse relation for problem parent testcase group

b7c1310

@nickygerritsen


 Import/export contest scoreboard type from/to YAML

be6b593

@nickygerritsen


 Fix score on single team scoreboard

899583b

@nickygerritsen


 Show all tries for scoring scoreboards, not only the first correct one

dcf9cb9

@nickygerritsen


 Add score to team submission list and detail page

7ac3835

@nickygerritsen


 It can be the case that we don't have a judging yet when calculating ...

6c794ee

...the scoreboard cache

@nickygerritsen


 Check scoreboard type when shadowing

35c7e53

@nickygerritsen


 Do not assume we have one event if we have exactly 0

62f28bd

@nickygerritsen


 Do not import directories when importing shadow submissions

8f1b0e9

@nickygerritsen


 Fix shadow differences submissions table for scoring contests

97ae23a

@nickygerritsen

Copy link

Member

nickygerritsen commented Aug 17, 2025

Some notes:

the number of tries on a scoring scoreboard should really be the number of total tries (not until the first correct attempt like for pass-fail)

I changed this to show the total number of submissions. I wonder if this is what we want? Say you have 5 submissions:

AC, score = 50
AC, score = 75
AC, score = 100
WA
WA

Do we want 3 or 5 tries? And what if the fourth one is AC, score = 100?

Also please check my logic. Basically I do not stop processing submissions after the first correct one for scoring problems (and have changed the FTS logic since this was dependent on this assumption).

the layout of the shadow differences table needs to be fixed

Fixed

I also fixed some other things:

I added the reverse relation for problem->testcaseGroup.
We now correctly import (and export) the scoreboard type from contest.yaml/json (but for this I have renamed the SCORING enum case to SCORE as that is what the spec says).
I fixed the single team scoreboard row, it didn't show the scores.
I added the score to the team submission list and detail modal.
I fixed an issue where if we calculated the scorecache for a submission without judgement, it would crash.
I now check the scoreboard type when starting up the shadow script.
I fixed some logic for importing an event with no items, as that is what we had in the replay for the NAC challenge.
I also fixed importing the ZIPs from that challenge that contained folders.

Labels

None yet

2 participants

@meisterT @nickygerritsen

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Basic implementation of partial scoring. #3047

Are you sure you want to change the base?

Basic implementation of partial scoring. #3047

Uh oh!

Conversation

@meisterT meisterT commented Aug 3, 2025

Uh oh!

meisterT commented Aug 5, 2025

Uh oh!

@nickygerritsen nickygerritsen left a comment

Choose a reason for hiding this comment

Uh oh!

@nickygerritsen nickygerritsen Aug 17, 2025

Choose a reason for hiding this comment

Uh oh!

@nickygerritsen nickygerritsen Aug 17, 2025

Choose a reason for hiding this comment

Uh oh!

@nickygerritsen nickygerritsen Aug 17, 2025

Choose a reason for hiding this comment

Uh oh!

@nickygerritsen nickygerritsen Aug 17, 2025

Choose a reason for hiding this comment

Uh oh!

@nickygerritsen nickygerritsen Aug 17, 2025

Choose a reason for hiding this comment

Uh oh!

@nickygerritsen nickygerritsen Aug 17, 2025

Choose a reason for hiding this comment

Uh oh!

nickygerritsen commented Aug 17, 2025

Uh oh!

Uh oh!