Duckietown Challenges Home Challenges Submissions

Evaluator 31

ID31
evaluatordavincistage-05
owneruser1000001
machinedavincistage
processdavincistage-05
version6.2.40
first heard
last heard
statusinactive
# evaluating
# success6 7539
# timeout3 7494
# failed3 7492
# error
# aborted
# host-error1 7531
arm0
x86_640
Mac0
gpu available1
Number of processors64
Processor frequency (MHz)2.8 GHz
Free % of processors84%
RAM total (MB)251.6 GB
RAM free (MB)235.9 GB
Disk (MB)491.1 GB
Disk available (MB)331.3 GB
Docker Hub
P11
P2
Cloud simulations1
PI Camera0
# Duckiebots0
Map 3x3 avaiable
Number of duckies
gpu cores
AIDO 2 Map LF public
AIDO 2 Map LF private
AIDO 2 Map LFV public
AIDO 2 Map LFV private
AIDO 2 Map LFVI public
AIDO 2 Map LFVI private
AIDO 3 Map LF public
AIDO 3 Map LF private
AIDO 3 Map LFV public
AIDO 3 Map LFV private
AIDO 3 Map LFVI public
AIDO 3 Map LFVI private
AIDO 5 Map large loop
ETU track
for 2021, ETH_small_intersect
IPFS mountpoint /ipfs available
IPNS mountpoint /ipns available

Evaluator jobs

Job IDsubmissionuseruser labelchallengestepstatusup to dateevaluatordate starteddate completeddurationmessage
7551672Bea Baselines 🐤template-tensorflowaido-LF-sim-testingsim-2of4timeoutnodavincistage-05----No reset possible
7550672Bea Baselines 🐤template-tensorflowaido-LF-sim-testingsim-3of4failednodavincistage-050:01:25
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 298, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 156, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 328, in read_reply
    raise RemoteNodeAborted(msg) from None
zuper_nodes.structures.RemoteNodeAborted: Remote node closed communication (None)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges/cie_concrete.py", line 691, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/experiment_manager.py", line 60, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/experiment_manager.py", line 33, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 307, in main
    raise InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: Getting agent protocol
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
7549715Bea Baselines 🐤template-randomaido-LF-sim-testingsim-3of4successnodavincistage-050:02:12
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.022815306577956607
survival_time_median3.899999999999994
deviation-center-line_median0.026662490367679023
in-drivable-lane_median2.399999999999994


other stats
deviation-center-line_max0.026662490367679023
deviation-center-line_mean0.026662490367679023
deviation-center-line_min0.026662490367679023
deviation-heading_max1.125353902266408
deviation-heading_mean1.125353902266408
deviation-heading_median1.125353902266408
deviation-heading_min1.125353902266408
distance-from-start_max0.4891723059495436
distance-from-start_mean0.4891723059495436
distance-from-start_median0.4891723059495436
distance-from-start_min0.4891723059495436
driven_any_max0.5134230938905437
driven_any_mean0.5134230938905437
driven_any_median0.5134230938905437
driven_any_min0.5134230938905437
driven_lanedir_consec_max0.022815306577956607
driven_lanedir_consec_mean0.022815306577956607
driven_lanedir_consec_min0.022815306577956607
driven_lanedir_max0.022815306577956607
driven_lanedir_mean0.022815306577956607
driven_lanedir_median0.022815306577956607
driven_lanedir_min0.022815306577956607
in-drivable-lane_max2.399999999999994
in-drivable-lane_mean2.399999999999994
in-drivable-lane_min2.399999999999994
per-episodes
details{"zigzag-000-ego0": {"driven_any": 0.5134230938905437, "survival_time": 3.899999999999994, "driven_lanedir": 0.022815306577956607, "in-drivable-lane": 2.399999999999994, "deviation-heading": 1.125353902266408, "distance-from-start": 0.4891723059495436, "deviation-center-line": 0.026662490367679023, "driven_lanedir_consec": 0.022815306577956607}}
simulation-passed1
survival_time_max3.899999999999994
survival_time_mean3.899999999999994
survival_time_min3.899999999999994
No reset possible
7547715Bea Baselines 🐤template-randomaido-LF-sim-testingsim-0of4successnodavincistage-050:04:19
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.05434655099627461
survival_time_median27.600000000000257
deviation-center-line_median0.634658591949439
in-drivable-lane_median20.050000000000185


other stats
deviation-center-line_max0.634658591949439
deviation-center-line_mean0.634658591949439
deviation-center-line_min0.634658591949439
deviation-heading_max5.904235246936352
deviation-heading_mean5.904235246936352
deviation-heading_median5.904235246936352
deviation-heading_min5.904235246936352
distance-from-start_max0.7960980578537318
distance-from-start_mean0.7960980578537318
distance-from-start_median0.7960980578537318
distance-from-start_min0.7960980578537318
driven_any_max6.576498432956264
driven_any_mean6.576498432956264
driven_any_median6.576498432956264
driven_any_min6.576498432956264
driven_lanedir_consec_max0.05434655099627461
driven_lanedir_consec_mean0.05434655099627461
driven_lanedir_consec_min0.05434655099627461
driven_lanedir_max0.05769325072273146
driven_lanedir_mean0.05769325072273146
driven_lanedir_median0.05769325072273146
driven_lanedir_min0.05769325072273146
in-drivable-lane_max20.050000000000185
in-drivable-lane_mean20.050000000000185
in-drivable-lane_min20.050000000000185
per-episodes
details{"autolab-000-ego0": {"driven_any": 6.576498432956264, "survival_time": 27.600000000000257, "driven_lanedir": 0.05769325072273146, "in-drivable-lane": 20.050000000000185, "deviation-heading": 5.904235246936352, "distance-from-start": 0.7960980578537318, "deviation-center-line": 0.634658591949439, "driven_lanedir_consec": 0.05434655099627461}}
simulation-passed1
survival_time_max27.600000000000257
survival_time_mean27.600000000000257
survival_time_min27.600000000000257
No reset possible
7546715Bea Baselines 🐤template-randomaido-LF-sim-testingsim-1of4successnodavincistage-050:02:41
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.02606899309120614
survival_time_median11.950000000000037
deviation-center-line_median0.3267562024231576
in-drivable-lane_median8.500000000000023


other stats
deviation-center-line_max0.3267562024231576
deviation-center-line_mean0.3267562024231576
deviation-center-line_min0.3267562024231576
deviation-heading_max2.556935476404166
deviation-heading_mean2.556935476404166
deviation-heading_median2.556935476404166
deviation-heading_min2.556935476404166
distance-from-start_max0.6291689351083347
distance-from-start_mean0.6291689351083347
distance-from-start_median0.6291689351083347
distance-from-start_min0.6291689351083347
driven_any_max2.617379035765366
driven_any_mean2.617379035765366
driven_any_median2.617379035765366
driven_any_min2.617379035765366
driven_lanedir_consec_max0.02606899309120614
driven_lanedir_consec_mean0.02606899309120614
driven_lanedir_consec_min0.02606899309120614
driven_lanedir_max0.029415931236242088
driven_lanedir_mean0.029415931236242088
driven_lanedir_median0.029415931236242088
driven_lanedir_min0.029415931236242088
in-drivable-lane_max8.500000000000023
in-drivable-lane_mean8.500000000000023
in-drivable-lane_min8.500000000000023
per-episodes
details{"loop-000-ego0": {"driven_any": 2.617379035765366, "survival_time": 11.950000000000037, "driven_lanedir": 0.029415931236242088, "in-drivable-lane": 8.500000000000023, "deviation-heading": 2.556935476404166, "distance-from-start": 0.6291689351083347, "deviation-center-line": 0.3267562024231576, "driven_lanedir_consec": 0.02606899309120614}}
simulation-passed1
survival_time_max11.950000000000037
survival_time_mean11.950000000000037
survival_time_min11.950000000000037
No reset possible
7544783Andrea Danieletemplate-randomaido-LF-sim-validationsim-2of4successnodavincistage-050:03:35
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.26764947507969805
survival_time_median25.75000000000023
deviation-center-line_median0.7555292890833561
in-drivable-lane_median17.20000000000021


other stats
deviation-center-line_max0.7555292890833561
deviation-center-line_mean0.7555292890833561
deviation-center-line_min0.7555292890833561
deviation-heading_max5.523299123594233
deviation-heading_mean5.523299123594233
deviation-heading_median5.523299123594233
deviation-heading_min5.523299123594233
distance-from-start_max0.7960980578537241
distance-from-start_mean0.7960980578537241
distance-from-start_median0.7960980578537241
distance-from-start_min0.7960980578537241
driven_any_max6.239826156091173
driven_any_mean6.239826156091173
driven_any_median6.239826156091173
driven_any_min6.239826156091173
driven_lanedir_consec_max0.26764947507969805
driven_lanedir_consec_mean0.26764947507969805
driven_lanedir_consec_min0.26764947507969805
driven_lanedir_max0.2496387990426845
driven_lanedir_mean0.2496387990426845
driven_lanedir_median0.2496387990426845
driven_lanedir_min0.2496387990426845
in-drivable-lane_max17.20000000000021
in-drivable-lane_mean17.20000000000021
in-drivable-lane_min17.20000000000021
per-episodes
details{"loop2-000-ego0": {"driven_any": 6.239826156091173, "survival_time": 25.75000000000023, "driven_lanedir": 0.2496387990426845, "in-drivable-lane": 17.20000000000021, "deviation-heading": 5.523299123594233, "distance-from-start": 0.7960980578537241, "deviation-center-line": 0.7555292890833561, "driven_lanedir_consec": 0.26764947507969805}}
simulation-passed1
survival_time_max25.75000000000023
survival_time_mean25.75000000000023
survival_time_min25.75000000000023
No reset possible
7540789Andrea Danieletemplate-pytorchaido-LF-sim-validationsim-1of4successnodavincistage-050:08:20
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median1.9238763308972584
survival_time_median59.99999999999873
deviation-center-line_median2.3619014386467865
in-drivable-lane_median34.79999999999905


other stats
deviation-center-line_max2.3619014386467865
deviation-center-line_mean2.3619014386467865
deviation-center-line_min2.3619014386467865
deviation-heading_max20.677903565041166
deviation-heading_mean20.677903565041166
deviation-heading_median20.677903565041166
deviation-heading_min20.677903565041166
distance-from-start_max0.39109185572637784
distance-from-start_mean0.39109185572637784
distance-from-start_median0.39109185572637784
distance-from-start_min0.39109185572637784
driven_any_max7.154554195734862
driven_any_mean7.154554195734862
driven_any_median7.154554195734862
driven_any_min7.154554195734862
driven_lanedir_consec_max1.9238763308972584
driven_lanedir_consec_mean1.9238763308972584
driven_lanedir_consec_min1.9238763308972584
driven_lanedir_max1.9238763308972584
driven_lanedir_mean1.9238763308972584
driven_lanedir_median1.9238763308972584
driven_lanedir_min1.9238763308972584
in-drivable-lane_max34.79999999999905
in-drivable-lane_mean34.79999999999905
in-drivable-lane_min34.79999999999905
per-episodes
details{"loop-000-ego0": {"driven_any": 7.154554195734862, "survival_time": 59.99999999999873, "driven_lanedir": 1.9238763308972584, "in-drivable-lane": 34.79999999999905, "deviation-heading": 20.677903565041166, "distance-from-start": 0.39109185572637784, "deviation-center-line": 2.3619014386467865, "driven_lanedir_consec": 1.9238763308972584}}
simulation-passed1
survival_time_max59.99999999999873
survival_time_mean59.99999999999873
survival_time_min59.99999999999873
No reset possible
7539785Andrea Danieletemplate-rosaido-LF-sim-validationsim-2of4successnodavincistage-050:02:49
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.6053731617161837
survival_time_median7.299999999999982
deviation-center-line_median0.2763461308160487
in-drivable-lane_median3.8499999999999863


other stats
deviation-center-line_max0.2763461308160487
deviation-center-line_mean0.2763461308160487
deviation-center-line_min0.2763461308160487
deviation-heading_max2.118644646411014
deviation-heading_mean2.118644646411014
deviation-heading_median2.118644646411014
deviation-heading_min2.118644646411014
distance-from-start_max1.7345836618190156
distance-from-start_mean1.7345836618190156
distance-from-start_median1.7345836618190156
distance-from-start_min1.7345836618190156
driven_any_max1.842952450176782
driven_any_mean1.842952450176782
driven_any_median1.842952450176782
driven_any_min1.842952450176782
driven_lanedir_consec_max0.6053731617161837
driven_lanedir_consec_mean0.6053731617161837
driven_lanedir_consec_min0.6053731617161837
driven_lanedir_max0.6322581510531685
driven_lanedir_mean0.6322581510531685
driven_lanedir_median0.6322581510531685
driven_lanedir_min0.6322581510531685
in-drivable-lane_max3.8499999999999863
in-drivable-lane_mean3.8499999999999863
in-drivable-lane_min3.8499999999999863
per-episodes
details{"loop2-000-ego0": {"driven_any": 1.842952450176782, "survival_time": 7.299999999999982, "driven_lanedir": 0.6322581510531685, "in-drivable-lane": 3.8499999999999863, "deviation-heading": 2.118644646411014, "distance-from-start": 1.7345836618190156, "deviation-center-line": 0.2763461308160487, "driven_lanedir_consec": 0.6053731617161837}}
simulation-passed1
survival_time_max7.299999999999982
survival_time_mean7.299999999999982
survival_time_min7.299999999999982
No reset possible
7531789Andrea Danieletemplate-pytorchaido-LF-sim-validationsim-2of4host-errornodavincistage-050:00:23
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/auth

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 856, in get_cr
    cr = run_single(
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 1058, in run_single
    docker_pull_retry(client, image, ntimes=4, wait=5, credentials=credentials, quiet=False)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 47, in docker_pull_retry
    return docker_pull(client, image_name, quiet=quiet, credentials=credentials)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 84, in docker_pull
    do_login_for_registry(client, credentials, br.registry)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_logging_in.py", line 78, in do_login_for_registry
    docker_login(client, registry, credentials[registry]["username"], credentials[registry]["secret"])
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_logging_in.py", line 35, in docker_login
    res = client.login(username=docker_username, password=docker_password, reauth=True, registry=registry)
  File "/usr/local/lib/python3.8/dist-packages/docker/client.py", line 199, in login
    return self.api.login(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/daemon.py", line 152, in login
    return self._result(response, json=True)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 274, in _result
    self._raise_for_status(response)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e) from e
  File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation) from e
docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.41/auth: Internal Server Error ("Get https://registry-stage2.duckietown.org/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)")
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
7515785Andrea Danieletemplate-rosaido-LF-sim-validationsim-0of4timeoutnodavincistage-05----No reset possible
7509789Andrea Danieletemplate-pytorchaido-LF-sim-validationsim-3of4failednodavincistage-050:01:50
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 293, in main
    sim_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 156, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 328, in read_reply
    raise RemoteNodeAborted(msg) from None
zuper_nodes.structures.RemoteNodeAborted: Remote node closed communication (None)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges/cie_concrete.py", line 691, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/experiment_manager.py", line 60, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/experiment_manager.py", line 33, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 423, in main
    raise dc.InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: external protocol violation
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
7494785Andrea Danieletemplate-rosaido-LF-sim-validationsim-0of4timeoutnodavincistage-05----No reset possible
7492789Andrea Danieletemplate-pytorchaido-LF-sim-validationsim-3of4failednodavincistage-050:05:53
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 293, in main
    sim_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 156, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 328, in read_reply
    raise RemoteNodeAborted(msg) from None
zuper_nodes.structures.RemoteNodeAborted: Remote node closed communication (None)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges/cie_concrete.py", line 691, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/experiment_manager.py", line 60, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/experiment_manager.py", line 33, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 423, in main
    raise dc.InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: external protocol violation
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible