Duckietown Challenges Home Challenges Submissions

Evaluator 66

ID66
evaluatordavinci-stage-04
owneruser1000001
machinedavinci-stage
processdavinci-stage-04
version6.2.43
first heard
last heard
statusinactive
# evaluating
# success4 8102
# timeout1 8229
# failed2 8092
# error
# aborted
# host-error1 8185
arm0
x86_641
Mac0
gpu available1
Number of processors64
Processor frequency (MHz)2.8 GHz
Free % of processors96%
RAM total (MB)251.6 GB
RAM free (MB)234.8 GB
Disk (MB)491.1 GB
Disk available (MB)270.2 GB
Docker Hub
P11
P2
Cloud simulations1
PI Camera0
# Duckiebots0
Map 3x3 avaiable
Number of duckies
gpu cores
AIDO 2 Map LF public
AIDO 2 Map LF private
AIDO 2 Map LFV public
AIDO 2 Map LFV private
AIDO 2 Map LFVI public
AIDO 2 Map LFVI private
AIDO 3 Map LF public
AIDO 3 Map LF private
AIDO 3 Map LFV public
AIDO 3 Map LFV private
AIDO 3 Map LFVI public
AIDO 3 Map LFVI private
AIDO 5 Map large loop
ETU track
for 2021, ETH_small_intersect
IPFS mountpoint /ipfs available
IPNS mountpoint /ipns available

Evaluator jobs

Job IDsubmissionuseruser labelchallengestepstatusup to dateevaluatordate starteddate completeddurationmessage
8229875Bea Baselines 🐤template-pytorchaido-LF-sim-testingsim-2of4timeoutnodavinci-stage-04----No reset possible
8192876Bea Baselines 🐤template-pytorchaido-LFVI_multi-sim-testingsim-0of4successnodavinci-stage-040:26:38
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median56.9999999999989
in-drivable-lane_median48.87499999999912
driven_lanedir_consec_median0.9039075203662736
deviation-center-line_median0.7199465049687266


other stats
deviation-center-line_max1.6606280263960318
deviation-center-line_mean0.7861554527977063
deviation-center-line_min0.04410077485734022
deviation-heading_max22.3628794974069
deviation-heading_mean7.859463446659802
deviation-heading_median4.229356531373676
deviation-heading_min0.6162612264849561
distance-from-start_max0.862642227355799
distance-from-start_mean0.46728263787989766
distance-from-start_median0.46892241995223793
distance-from-start_min0.06864348425931588
driven_any_max8.762066198961993
driven_any_mean3.296445106258416
driven_any_median2.1677356061789603
driven_any_min0.08824301371375172
driven_lanedir_consec_max2.06026946356039
driven_lanedir_consec_mean0.972276999376903
driven_lanedir_consec_min0.021023493214674893
driven_lanedir_max2.06026946356039
driven_lanedir_mean0.972276999376903
driven_lanedir_median0.9039075203662736
driven_lanedir_min0.021023493214674893
in-drivable-lane_max56.2999999999989
in-drivable-lane_mean45.47499999999914
in-drivable-lane_min27.84999999999943
per-episodes
details{"udem-000-ego0": {"driven_any": 8.762066198961993, "survival_time": 56.9999999999989, "driven_lanedir": 2.06026946356039, "in-drivable-lane": 43.99999999999933, "deviation-heading": 6.207875743533875, "distance-from-start": 0.7293670163168379, "deviation-center-line": 1.258324252571141, "driven_lanedir_consec": 2.06026946356039}, "udem-000-ego1": {"driven_any": 3.5501440546283263, "survival_time": 56.9999999999989, "driven_lanedir": 1.5020649932068166, "in-drivable-lane": 27.84999999999943, "deviation-heading": 22.3628794974069, "distance-from-start": 0.862642227355799, "deviation-center-line": 1.6606280263960318, "driven_lanedir_consec": 1.5020649932068166}, "udem-000-ego2": {"driven_any": 0.785327157729594, "survival_time": 56.9999999999989, "driven_lanedir": 0.30575004752573065, "in-drivable-lane": 53.749999999998906, "deviation-heading": 2.2508373192134767, "distance-from-start": 0.20847782358763803, "deviation-center-line": 0.18156875736631173, "driven_lanedir_consec": 0.30575004752573065}, "udem-000-ego3": {"driven_any": 0.08824301371375172, "survival_time": 56.9999999999989, "driven_lanedir": 0.021023493214674893, "in-drivable-lane": 56.2999999999989, "deviation-heading": 0.6162612264849561, "distance-from-start": 0.06864348425931588, "deviation-center-line": 0.04410077485734022, "driven_lanedir_consec": 0.021023493214674893}}
simulation-passed1
survival_time_max56.9999999999989
survival_time_mean56.9999999999989
survival_time_min56.9999999999989
No reset possible
8185876Bea Baselines 🐤template-pytorchaido-LFVI_multi-sim-testingsim-1of4host-errornodavinci-stage-040:01:43
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 298, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 156, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 338, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego2" aborted with the following error:

error in ego2 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 388, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 37, in init
              ||     self.model = DDPG(state_dim=self.preprocessor.shape, action_dim=2, max_action=1, net_type="cnn")
              ||   File "/submission/model.py", line 153, in __init__
              ||     self.actor = ActorCNN(action_dim, max_action).to(device)
              ||   File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 612, in to
              ||     return self._apply(convert)
              ||   File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
              ||     module._apply(fn)
              ||   File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 381, in _apply
              ||     param_applied = fn(param)
              ||   File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 610, in convert
              ||     return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
              || RuntimeError: CUDA error: out of memory
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges/cie_concrete.py", line 691, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/experiment_manager.py", line 60, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/experiment_manager.py", line 33, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 306, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 298, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 156, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 338, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego2" aborted with the following error:

error in ego2 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 388, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 37, in init
              ||     self.model = DDPG(state_dim=self.preprocessor.shape, action_dim=2, max_action=1, net_type="cnn")
              ||   File "/submission/model.py", line 153, in __init__
              ||     self.actor = ActorCNN(action_dim, max_action).to(device)
              ||   File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 612, in to
              ||     return self._apply(convert)
              ||   File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
              ||     module._apply(fn)
              ||   File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 381, in _apply
              ||     param_applied = fn(param)
              ||   File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 610, in convert
              ||     return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
              || RuntimeError: CUDA error: out of memory
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
8123877Bea Baselines 🐤template-pytorchaido-LFV_multi-sim-testingsim-0of4successnodavinci-stage-040:28:51
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median59.99999999999873
in-drivable-lane_median43.62499999999888
driven_lanedir_consec_median0.8415371570467847
deviation-center-line_median1.1760200230144895


other stats
deviation-center-line_max3.0536084945481936
deviation-center-line_mean1.4408115959874614
deviation-center-line_min0.35759784337267253
deviation-heading_max20.780768512928834
deviation-heading_mean12.089442545981584
deviation-heading_median12.49855514894086
deviation-heading_min2.5798913731157875
distance-from-start_max1.0381404289666083
distance-from-start_mean0.4150916716625676
distance-from-start_median0.2289180294191429
distance-from-start_min0.16439019884537617
driven_any_max5.751472255952354
driven_any_mean3.271833085706188
driven_any_median2.9530270841935766
driven_any_min1.4298059184852423
driven_lanedir_consec_max1.2458541367040588
driven_lanedir_consec_mean0.8034077211506871
driven_lanedir_consec_min0.2847024338051203
driven_lanedir_max1.7440149206152895
driven_lanedir_mean0.9279479171284948
driven_lanedir_median0.8415371570467847
driven_lanedir_min0.2847024338051203
in-drivable-lane_max56.59999999999874
in-drivable-lane_mean44.137499999998965
in-drivable-lane_min32.69999999999936
per-episodes
details{"autolab-000-ego0": {"driven_any": 5.751472255952354, "survival_time": 59.99999999999873, "driven_lanedir": 1.7440149206152895, "in-drivable-lane": 35.49999999999906, "deviation-heading": 18.84424205387151, "distance-from-start": 1.0381404289666083, "deviation-center-line": 1.7290428169170196, "driven_lanedir_consec": 1.2458541367040588}, "autolab-000-ego1": {"driven_any": 4.177258947789016, "survival_time": 59.99999999999873, "driven_lanedir": 1.162000625224923, "in-drivable-lane": 32.69999999999936, "deviation-heading": 20.780768512928834, "distance-from-start": 0.16439019884537617, "deviation-center-line": 3.0536084945481936, "driven_lanedir_consec": 1.162000625224923}, "autolab-000-ego2": {"driven_any": 1.7287952205981372, "survival_time": 59.99999999999873, "driven_lanedir": 0.521073688868646, "in-drivable-lane": 51.74999999999869, "deviation-heading": 6.152868244010208, "distance-from-start": 0.22097606829681735, "deviation-center-line": 0.6229972291119599, "driven_lanedir_consec": 0.521073688868646}, "autolab-000-ego3": {"driven_any": 1.4298059184852423, "survival_time": 59.99999999999873, "driven_lanedir": 0.2847024338051203, "in-drivable-lane": 56.59999999999874, "deviation-heading": 2.5798913731157875, "distance-from-start": 0.2368599905414684, "deviation-center-line": 0.35759784337267253, "driven_lanedir_consec": 0.2847024338051203}}
simulation-passed1
survival_time_max59.99999999999873
survival_time_mean59.99999999999873
survival_time_min59.99999999999873
No reset possible
8110878Bea Baselines 🐤template-pytorchaido-LFP-sim-testingsim-2of4successnodavinci-stage-040:05:53
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median4.849999999999991
in-drivable-lane_median2.6499999999999906
driven_lanedir_consec_median0.300560331728813
deviation-center-line_median0.11302606553186674


other stats
deviation-center-line_max0.11302606553186674
deviation-center-line_mean0.11302606553186674
deviation-center-line_min0.11302606553186674
deviation-heading_max1.1867901365912077
deviation-heading_mean1.1867901365912077
deviation-heading_median1.1867901365912077
deviation-heading_min1.1867901365912077
distance-from-start_max0.4162332061863731
distance-from-start_mean0.4162332061863731
distance-from-start_median0.4162332061863731
distance-from-start_min0.4162332061863731
driven_any_max0.7420773117311594
driven_any_mean0.7420773117311594
driven_any_median0.7420773117311594
driven_any_min0.7420773117311594
driven_lanedir_consec_max0.300560331728813
driven_lanedir_consec_mean0.300560331728813
driven_lanedir_consec_min0.300560331728813
driven_lanedir_max0.300560331728813
driven_lanedir_mean0.300560331728813
driven_lanedir_median0.300560331728813
driven_lanedir_min0.300560331728813
in-drivable-lane_max2.6499999999999906
in-drivable-lane_mean2.6499999999999906
in-drivable-lane_min2.6499999999999906
per-episodes
details{"loop2-000-ego0": {"driven_any": 0.7420773117311594, "survival_time": 4.849999999999991, "driven_lanedir": 0.300560331728813, "in-drivable-lane": 2.6499999999999906, "deviation-heading": 1.1867901365912077, "distance-from-start": 0.4162332061863731, "deviation-center-line": 0.11302606553186674, "driven_lanedir_consec": 0.300560331728813}}
simulation-passed1
survival_time_max4.849999999999991
survival_time_mean4.849999999999991
survival_time_min4.849999999999991
No reset possible
8102880Bea Baselines 🐤template-rosaido-LF-sim-validationsim-3of4successnodavinci-stage-040:02:22
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.6840969183737273
survival_time_median10.550000000000017
deviation-center-line_median0.283090308451433
in-drivable-lane_median7.300000000000011


other stats
deviation-center-line_max0.283090308451433
deviation-center-line_mean0.283090308451433
deviation-center-line_min0.283090308451433
deviation-heading_max2.1745400958541197
deviation-heading_mean2.1745400958541197
deviation-heading_median2.1745400958541197
deviation-heading_min2.1745400958541197
distance-from-start_max2.6322694392362207
distance-from-start_mean2.6322694392362207
distance-from-start_median2.6322694392362207
distance-from-start_min2.6322694392362207
driven_any_max2.8066728501164544
driven_any_mean2.8066728501164544
driven_any_median2.8066728501164544
driven_any_min2.8066728501164544
driven_lanedir_consec_max0.6840969183737273
driven_lanedir_consec_mean0.6840969183737273
driven_lanedir_consec_min0.6840969183737273
driven_lanedir_max0.6840969183737273
driven_lanedir_mean0.6840969183737273
driven_lanedir_median0.6840969183737273
driven_lanedir_min0.6840969183737273
in-drivable-lane_max7.300000000000011
in-drivable-lane_mean7.300000000000011
in-drivable-lane_min7.300000000000011
per-episodes
details{"zigzag-000-ego0": {"driven_any": 2.8066728501164544, "survival_time": 10.550000000000017, "driven_lanedir": 0.6840969183737273, "in-drivable-lane": 7.300000000000011, "deviation-heading": 2.1745400958541197, "distance-from-start": 2.6322694392362207, "deviation-center-line": 0.283090308451433, "driven_lanedir_consec": 0.6840969183737273}}
simulation-passed1
survival_time_max10.550000000000017
survival_time_mean10.550000000000017
survival_time_min10.550000000000017
No reset possible
8099881Bea Baselines 🐤template-tensorflowaido-LFV-sim-validationsim-3of4failednodavinci-stage-040:01:32
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 298, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 156, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 328, in read_reply
    raise RemoteNodeAborted(msg) from None
zuper_nodes.structures.RemoteNodeAborted: Remote node closed communication (None)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges/cie_concrete.py", line 691, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/experiment_manager.py", line 60, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/experiment_manager.py", line 33, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 307, in main
    raise InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: Getting agent protocol
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
8092881Bea Baselines 🐤template-tensorflowaido-LFV-sim-validationsim-1of4failednodavinci-stage-040:02:06
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 298, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 156, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 328, in read_reply
    raise RemoteNodeAborted(msg) from None
zuper_nodes.structures.RemoteNodeAborted: Remote node closed communication (None)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges/cie_concrete.py", line 691, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/experiment_manager.py", line 60, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/experiment_manager.py", line 33, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 307, in main
    raise InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: Getting agent protocol
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible