Duckietown Challenges Home Challenges Submissions

Submission 118

Submission118
Competinguser retired
Challengeaido-LFV_multi-sim-testing
UserBea Baselines 🐤
Date submitted
Last status update
Completecomplete
DetailsEvaluation is complete.
Sisters
Result💚
Jobssim-0of4: 2992 sim-1of4: 3011 sim-2of4: 3004 sim-3of4: 2991
Next
User labeltemplate-ros
Admin priority50
Blessingn/a
User priority50

The highlights are available only to the owner and the admins.

The highlights are available only to the owner and the admins.

The highlights are available only to the owner and the admins.

The highlights are available only to the owner and the admins.

Evaluation jobs for this submission

Show only up-to-date jobs
Job IDstepstatusup to datedate starteddate completeddurationmessage
3011sim-1of4successno0:04:18
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median5.399999999999989
in-drivable-lane_median2.7999999999999914
driven_lanedir_consec_median0.3203547834263183
deviation-center-line_median0.1669452440510627


other stats
deviation-center-line_max0.33600303350245964
deviation-center-line_mean0.1816298217174575
deviation-center-line_min0.056625765265245
deviation-heading_max3.7004216538613743
deviation-heading_mean1.5626809351971132
deviation-heading_median0.9863363711914852
deviation-heading_min0.5776293445441084
distance-from-start_max1.3477428954819668
distance-from-start_mean0.8530664029154171
distance-from-start_median1.0242076166386966
distance-from-start_min0.01610748290230824
driven_any_max1.372674345984347
driven_any_mean0.8658439056700842
driven_any_median1.0372962687375546
driven_any_min0.016108739220880432
driven_lanedir_consec_max0.6647000713545803
driven_lanedir_consec_mean0.3295086277974497
driven_lanedir_consec_min0.01262487298258197
driven_lanedir_max0.6647000713545803
driven_lanedir_mean0.3295086277974497
driven_lanedir_median0.3203547834263183
driven_lanedir_min0.01262487298258197
in-drivable-lane_max4.549999999999988
in-drivable-lane_mean2.5374999999999925
in-drivable-lane_min0.0
per-episodes
details{"zigzag-000-ego0": {"driven_any": 1.372674345984347, "survival_time": 5.399999999999989, "driven_lanedir": 0.1377120239032874, "in-drivable-lane": 4.549999999999988, "deviation-heading": 0.5776293445441084, "distance-from-start": 1.3477428954819668, "deviation-center-line": 0.056625765265245, "driven_lanedir_consec": 0.1377120239032874}, "zigzag-000-ego1": {"driven_any": 1.1518473244795526, "survival_time": 5.399999999999989, "driven_lanedir": 0.6647000713545803, "in-drivable-lane": 2.6499999999999906, "deviation-heading": 0.7047171510885128, "distance-from-start": 1.1358919442223092, "deviation-center-line": 0.1405297214950147, "driven_lanedir_consec": 0.6647000713545803}, "zigzag-000-ego2": {"driven_any": 0.9227452129955566, "survival_time": 5.399999999999989, "driven_lanedir": 0.5029975429493492, "in-drivable-lane": 2.949999999999992, "deviation-heading": 1.2679555912944571, "distance-from-start": 0.912523289055084, "deviation-center-line": 0.19336076660711068, "driven_lanedir_consec": 0.5029975429493492}, "zigzag-000-ego3": {"driven_any": 0.016108739220880432, "survival_time": 5.399999999999989, "driven_lanedir": 0.01262487298258197, "in-drivable-lane": 0.0, "deviation-heading": 3.7004216538613743, "distance-from-start": 0.01610748290230824, "deviation-center-line": 0.33600303350245964, "driven_lanedir_consec": 0.01262487298258197}}
simulation-passed1
survival_time_max5.399999999999989
survival_time_mean5.399999999999989
survival_time_min5.399999999999989
No reset possible
3004sim-2of4successno0:07:46
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median11.65000000000003
in-drivable-lane_median10.52500000000003
driven_lanedir_consec_median0.21506180698901867
deviation-center-line_median0.10967823584205684


other stats
deviation-center-line_max0.3742973283056312
deviation-center-line_mean0.16293538184557607
deviation-center-line_min0.05808772739255946
deviation-heading_max1.491506023507465
deviation-heading_mean0.7646413489093553
deviation-heading_median0.6006243965770725
deviation-heading_min0.3658105789758113
distance-from-start_max2.99559325401033
distance-from-start_mean1.301352760841175
distance-from-start_median0.9907902817827896
distance-from-start_min0.22823722578879013
driven_any_max3.1823421060704407
driven_any_mean1.36832271002256
driven_any_median1.031292129254593
driven_any_min0.22836447551061304
driven_lanedir_consec_max1.7699309601533475
driven_lanedir_consec_mean0.5778477364746453
driven_lanedir_consec_min0.11133637176719646
driven_lanedir_max1.7699309601533475
driven_lanedir_mean0.5778477364746453
driven_lanedir_median0.21506180698901867
driven_lanedir_min0.11133637176719646
in-drivable-lane_max10.75000000000003
in-drivable-lane_mean9.237500000000034
in-drivable-lane_min5.150000000000046
per-episodes
details{"autolab-000-ego0": {"driven_any": 0.7600066497414949, "survival_time": 11.65000000000003, "driven_lanedir": 0.13927918700191588, "in-drivable-lane": 10.75000000000003, "deviation-heading": 0.6026973972604321, "distance-from-start": 0.7182020376696778, "deviation-center-line": 0.05808772739255946, "driven_lanedir_consec": 0.13927918700191588}, "autolab-000-ego1": {"driven_any": 1.302577608767691, "survival_time": 11.65000000000003, "driven_lanedir": 0.29084442697612145, "in-drivable-lane": 10.30000000000003, "deviation-heading": 0.3658105789758113, "distance-from-start": 1.2633785258959016, "deviation-center-line": 0.13709742558407031, "driven_lanedir_consec": 0.29084442697612145}, "autolab-000-ego2": {"driven_any": 3.1823421060704407, "survival_time": 11.65000000000003, "driven_lanedir": 1.7699309601533475, "in-drivable-lane": 5.150000000000046, "deviation-heading": 1.491506023507465, "distance-from-start": 2.99559325401033, "deviation-center-line": 0.3742973283056312, "driven_lanedir_consec": 1.7699309601533475}, "autolab-000-ego3": {"driven_any": 0.22836447551061304, "survival_time": 11.65000000000003, "driven_lanedir": 0.11133637176719646, "in-drivable-lane": 10.75000000000003, "deviation-heading": 0.5985513958937128, "distance-from-start": 0.22823722578879013, "deviation-center-line": 0.08225904610004332, "driven_lanedir_consec": 0.11133637176719646}}
simulation-passed1
survival_time_max11.65000000000003
survival_time_mean11.65000000000003
survival_time_min11.65000000000003
No reset possible
3003sim-1of4host-errorno0:05:16
The container "simul [...]
The container "simulator" exited with code 137.


Error code 137 means an out-of-memory error. The container exceeded the memory available to it. 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
2994sim-2of4host-errorno0:06:45
Uncaught exception w [...]
Uncaught exception while running Docker Compose:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1344, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/dist-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.8/dist-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 447, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 336, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 1247, in run_one
    stats = container.stats(stream=False)
  File "/usr/local/lib/python3.8/dist-packages/docker/models/containers.py", line 422, in stats
    return self.client.api.stats(self.id, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/container.py", line 1139, in stats
    return self._result(self._get(url, params={'stream': False}),
  File "/usr/local/lib/python3.8/dist-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
2993sim-1of4host-errorno0:06:55
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 862, in get_cr
    uploaded = upload_files(wd, aws_config, copy_to_machine_cache=copy_to_machine_cache)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/uploading.py", line 48, in upload_files
    uploaded = upload(aws_config, toupload, copy_to_machine_cache=copy_to_machine_cache)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/uploading.py", line 275, in upload
    shutil.copy(realfile0, realfile)
  File "/usr/lib/python3.8/shutil.py", line 418, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib/python3.8/shutil.py", line 264, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/duckietown/aido-LFV_multi-sim-testing/submission118/sim-1of4-nogpu-staging-spot-0-04-job2993-a-wd/challenge-evaluation-output/episodes/zigzag-000/ego1/camera.mp4.firstpass.mp4'
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
2992sim-0of4successno0:02:30
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median2.799999999999998
in-drivable-lane_median1.7749999999999977
driven_lanedir_consec_median0.15220107198756783
deviation-center-line_median0.0861080815796236


other stats
deviation-center-line_max0.10570276129254724
deviation-center-line_mean0.0861080815796236
deviation-center-line_min0.06651340186669996
deviation-heading_max0.6775616356694971
deviation-heading_mean0.5446829221518549
deviation-heading_median0.5446829221518549
deviation-heading_min0.4118042086342126
distance-from-start_max0.5092552010086382
distance-from-start_mean0.4757980815461358
distance-from-start_median0.4757980815461358
distance-from-start_min0.44234096208363344
driven_any_max0.5107762144942676
driven_any_mean0.4804114262349891
driven_any_median0.4804114262349891
driven_any_min0.4500466379757107
driven_lanedir_consec_max0.1765866027819183
driven_lanedir_consec_mean0.15220107198756783
driven_lanedir_consec_min0.12781554119321736
driven_lanedir_max0.1765866027819183
driven_lanedir_mean0.15220107198756783
driven_lanedir_median0.15220107198756783
driven_lanedir_min0.12781554119321736
in-drivable-lane_max1.849999999999998
in-drivable-lane_mean1.7749999999999977
in-drivable-lane_min1.6999999999999975
per-episodes
details{"loop2-000-ego0": {"driven_any": 0.5107762144942676, "survival_time": 2.799999999999998, "driven_lanedir": 0.12781554119321736, "in-drivable-lane": 1.849999999999998, "deviation-heading": 0.6775616356694971, "distance-from-start": 0.5092552010086382, "deviation-center-line": 0.06651340186669996, "driven_lanedir_consec": 0.12781554119321736}, "loop2-000-ego1": {"driven_any": 0.4500466379757107, "survival_time": 2.799999999999998, "driven_lanedir": 0.1765866027819183, "in-drivable-lane": 1.6999999999999975, "deviation-heading": 0.4118042086342126, "distance-from-start": 0.44234096208363344, "deviation-center-line": 0.10570276129254724, "driven_lanedir_consec": 0.1765866027819183}}
simulation-passed1
survival_time_max2.799999999999998
survival_time_mean2.799999999999998
survival_time_min2.799999999999998
No reset possible
2991sim-3of4successno0:05:20
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median9.55
in-drivable-lane_median7.150000000000002
driven_lanedir_consec_median0.21833518725577916
deviation-center-line_median0.1412306583626028


other stats
deviation-center-line_max0.4396919671328122
deviation-center-line_mean0.2020654079199312
deviation-center-line_min0.08610834782170707
deviation-heading_max6.020150074556843
deviation-heading_mean2.1034402436750663
deviation-heading_median1.009515340177968
deviation-heading_min0.3745802197874866
distance-from-start_max2.2448008645522264
distance-from-start_mean1.2429650652003963
distance-from-start_median1.3635296981246796
distance-from-start_min2.220446049250313e-16
driven_any_max2.6003164615436836
driven_any_mean1.3821267790340894
driven_any_median1.4640953272963158
driven_any_min4.2410519540680986e-14
driven_lanedir_consec_max0.7708406450102978
driven_lanedir_consec_mean0.30187775488046403
driven_lanedir_consec_min0.0
driven_lanedir_max0.7708406450102978
driven_lanedir_mean0.30187775488046403
driven_lanedir_median0.21833518725577916
driven_lanedir_min0.0
in-drivable-lane_max8.65
in-drivable-lane_mean5.737500000000001
in-drivable-lane_min0.0
per-episodes
details{"loop-000-ego0": {"driven_any": 2.6003164615436836, "survival_time": 9.55, "driven_lanedir": 0.28770955197369874, "in-drivable-lane": 7.9, "deviation-heading": 1.338440239726293, "distance-from-start": 2.18897409705177, "deviation-center-line": 0.12805386621251968, "driven_lanedir_consec": 0.28770955197369874}, "loop-000-ego1": {"driven_any": 0.5514149192230159, "survival_time": 9.55, "driven_lanedir": 0.14896082253785958, "in-drivable-lane": 8.65, "deviation-heading": 0.3745802197874866, "distance-from-start": 0.5380852991975891, "deviation-center-line": 0.08610834782170707, "driven_lanedir_consec": 0.14896082253785958}, "loop-000-ego2": {"driven_any": 2.3767757353696157, "survival_time": 9.55, "driven_lanedir": 0.7708406450102978, "in-drivable-lane": 6.400000000000004, "deviation-heading": 0.6805904406296432, "distance-from-start": 2.2448008645522264, "deviation-center-line": 0.15440745051268592, "driven_lanedir_consec": 0.7708406450102978}, "loop-000-ego3": {"driven_any": 4.2410519540680986e-14, "survival_time": 9.55, "driven_lanedir": 0.0, "in-drivable-lane": 0.0, "deviation-heading": 6.020150074556843, "distance-from-start": 2.220446049250313e-16, "deviation-center-line": 0.4396919671328122, "driven_lanedir_consec": 0.0}}
simulation-passed1
survival_time_max9.55
survival_time_mean9.55
survival_time_min9.55
No reset possible