Duckietown Challenges Home Challenges Submissions

Submission 185

Submission185
Competinguser retired
Challengeaido-LFV_multi-state-sim-validation
UserBea Baselines 🐤
Date submitted
Last status update
Completecomplete
DetailsEvaluation is complete.
Sisters
Result💚
Jobssim-0of4: 1424 sim-1of4: 1440 sim-2of4: 1470 sim-3of4: 1422
Next
User labelminimal-agent-state
Admin priority50
Blessingn/a
User priority50

1470

Click the images to see detailed statistics about the episode.

loop-000

1440

Click the images to see detailed statistics about the episode.

autolab-000

1424

Click the images to see detailed statistics about the episode.

loop2-000

1422

Click the images to see detailed statistics about the episode.

zigzag-000

Evaluation jobs for this submission

Show only up-to-date jobs
Job IDstepstatusup to datedate starteddate completeddurationmessage
1470258successno0:09:20
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median59.99999999999873
in-drivable-lane_median3.2749999999998938
driven_lanedir_consec_median2.7120049795825034
deviation-center-line_median6.8302336697805295


other stats
deviation-center-line_max7.154226688493771
deviation-center-line_mean6.804680160035498
deviation-center-line_min6.404026612087161
deviation-heading_max8.100880506476408
deviation-heading_mean5.973168085654567
deviation-heading_median5.501308310203183
deviation-heading_min4.789175215735496
distance-from-start_max2.4160153979932883
distance-from-start_mean1.609314369969162
distance-from-start_median1.566033493936358
distance-from-start_min0.8891750940106445
driven_any_max3.519587859799123
driven_any_mean3.48283846738281
driven_any_median3.481341572069241
driven_any_min3.449082865593636
driven_lanedir_consec_max3.3303389224329396
driven_lanedir_consec_mean2.6711178042987727
driven_lanedir_consec_min1.930122335597144
driven_lanedir_max3.3876167841121254
driven_lanedir_mean3.3082911061481024
driven_lanedir_median3.3180661038703403
driven_lanedir_min3.2094154327396036
in-drivable-lane_max4.349999999999898
in-drivable-lane_mean3.2249999999999286
in-drivable-lane_min2.0000000000000284
per-episodes
details{"loop-000-ego0": {"driven_any": 3.4760910958005855, "survival_time": 59.99999999999873, "driven_lanedir": 3.305793285307741, "in-drivable-lane": 3.449999999999882, "deviation-heading": 5.348097201635604, "distance-from-start": 1.6303675543068192, "deviation-center-line": 7.154226688493771, "driven_lanedir_consec": 3.305793285307741}, "loop-000-ego1": {"driven_any": 3.4865920483378963, "survival_time": 59.99999999999873, "driven_lanedir": 3.3303389224329396, "in-drivable-lane": 3.0999999999999055, "deviation-heading": 4.789175215735496, "distance-from-start": 2.4160153979932883, "deviation-center-line": 6.91888991254649, "driven_lanedir_consec": 3.3303389224329396}, "loop-000-ego2": {"driven_any": 3.449082865593636, "survival_time": 59.99999999999873, "driven_lanedir": 3.2094154327396036, "in-drivable-lane": 4.349999999999898, "deviation-heading": 8.100880506476408, "distance-from-start": 0.8891750940106445, "deviation-center-line": 6.404026612087161, "driven_lanedir_consec": 1.930122335597144}, "loop-000-ego3": {"driven_any": 3.519587859799123, "survival_time": 59.99999999999873, "driven_lanedir": 3.3876167841121254, "in-drivable-lane": 2.0000000000000284, "deviation-heading": 5.654519418770762, "distance-from-start": 1.5016994335658966, "deviation-center-line": 6.741577427014568, "driven_lanedir_consec": 2.118216673857266}}
simulation-passed1
survival_time_max59.99999999999873
survival_time_mean59.99999999999873
survival_time_min59.99999999999873
No reset possible
1448258host-errorno0:16:05
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 392, in main
    stats = future.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 689, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 60, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 33, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 426, in main
    raise dc.InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: broken process pool
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
1440257successno0:09:17
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median59.99999999999873
in-drivable-lane_median2.024999999999956
driven_lanedir_consec_median2.354349999155144
deviation-center-line_median6.190490094517614


other stats
deviation-center-line_max7.342897137212392
deviation-center-line_mean5.815429155063692
deviation-center-line_min3.537839294007148
deviation-heading_max8.643847237176693
deviation-heading_mean5.769308033135152
deviation-heading_median5.462417882322537
deviation-heading_min3.508549130718843
distance-from-start_max2.1932800589165744
distance-from-start_mean1.7770214146438683
distance-from-start_median1.7822126331905102
distance-from-start_min1.3503803332778788
driven_any_max3.519587859801346
driven_any_mean3.018964710108239
driven_any_median3.3875880363682387
driven_any_min1.781094907895131
driven_lanedir_consec_max3.3210418145715233
driven_lanedir_consec_mean2.428232830048234
driven_lanedir_consec_min1.683189507311125
driven_lanedir_max3.387629081865671
driven_lanedir_mean2.8833297718361015
driven_lanedir_median3.2312502490838044
driven_lanedir_min1.683189507311125
in-drivable-lane_max2.749999999999872
in-drivable-lane_mean2.162499999999953
in-drivable-lane_min1.8500000000000263
per-episodes
details{"autolab-000-ego0": {"driven_any": 3.278087216378781, "survival_time": 59.99999999999873, "driven_lanedir": 3.141458683596086, "in-drivable-lane": 2.0499999999998835, "deviation-heading": 5.270318376228317, "distance-from-start": 2.1932800589165744, "deviation-center-line": 3.537839294007148, "driven_lanedir_consec": 2.5904401579936893}, "autolab-000-ego1": {"driven_any": 1.781094907895131, "survival_time": 59.99999999999873, "driven_lanedir": 1.683189507311125, "in-drivable-lane": 1.8500000000000263, "deviation-heading": 3.508549130718843, "distance-from-start": 1.3503803332778788, "deviation-center-line": 7.342897137212392, "driven_lanedir_consec": 1.683189507311125}, "autolab-000-ego2": {"driven_any": 3.4970888563576965, "survival_time": 59.99999999999873, "driven_lanedir": 3.3210418145715233, "in-drivable-lane": 2.749999999999872, "deviation-heading": 8.643847237176693, "distance-from-start": 2.0627256300706325, "deviation-center-line": 5.639409158878009, "driven_lanedir_consec": 3.3210418145715233}, "autolab-000-ego3": {"driven_any": 3.519587859801346, "survival_time": 59.99999999999873, "driven_lanedir": 3.387629081865671, "in-drivable-lane": 2.0000000000000284, "deviation-heading": 5.654517388416758, "distance-from-start": 1.5016996363103878, "deviation-center-line": 6.741571030157219, "driven_lanedir_consec": 2.118259840316598}}
simulation-passed1
survival_time_max59.99999999999873
survival_time_mean59.99999999999873
survival_time_min59.99999999999873
No reset possible
1433258host-errorno0:09:43
Uncaught exception w [...]
Uncaught exception while running Docker Compose:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1344, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/dist-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.8/dist-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 447, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 336, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 1247, in run_one
    stats = container.stats(stream=False)
  File "/usr/local/lib/python3.8/dist-packages/docker/models/containers.py", line 422, in stats
    return self.client.api.stats(self.id, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/container.py", line 1139, in stats
    return self._result(self._get(url, params={'stream': False}),
  File "/usr/local/lib/python3.8/dist-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
1431257host-errorno0:02:17
The container "simul [...]
The container "simulator" exited with code 137.


Error code 137 means an out-of-memory error. The container exceeded the memory available to it.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
1424256successno0:04:38
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median59.99999999999873
in-drivable-lane_median3.6749999999999297
driven_lanedir_consec_median2.3321113241668914
deviation-center-line_median6.1369806805591525


other stats
deviation-center-line_max7.370362503106452
deviation-center-line_mean6.1369806805591525
deviation-center-line_min4.903598858011853
deviation-heading_max7.744574522427148
deviation-heading_mean7.106635260692304
deviation-heading_median7.106635260692304
deviation-heading_min6.468695998957461
distance-from-start_max1.648096422950854
distance-from-start_mean1.4270486060438758
distance-from-start_median1.4270486060438758
distance-from-start_min1.2060007891368978
driven_any_max3.4850840176975995
driven_any_mean3.469336713167126
driven_any_median3.469336713167126
driven_any_min3.4535894086366525
driven_lanedir_consec_max3.243830729974445
driven_lanedir_consec_mean2.3321113241668914
driven_lanedir_consec_min1.420391918359338
driven_lanedir_max3.290537400716066
driven_lanedir_mean3.2671840653452557
driven_lanedir_median3.2671840653452557
driven_lanedir_min3.243830729974445
in-drivable-lane_max4.199999999999893
in-drivable-lane_mean3.6749999999999297
in-drivable-lane_min3.1499999999999666
per-episodes
details{"loop2-000-ego0": {"driven_any": 3.4850840176975995, "survival_time": 59.99999999999873, "driven_lanedir": 3.290537400716066, "in-drivable-lane": 3.1499999999999666, "deviation-heading": 7.744574522427148, "distance-from-start": 1.2060007891368978, "deviation-center-line": 4.903598858011853, "driven_lanedir_consec": 1.420391918359338}, "loop2-000-ego1": {"driven_any": 3.4535894086366525, "survival_time": 59.99999999999873, "driven_lanedir": 3.243830729974445, "in-drivable-lane": 4.199999999999893, "deviation-heading": 6.468695998957461, "distance-from-start": 1.648096422950854, "deviation-center-line": 7.370362503106452, "driven_lanedir_consec": 3.243830729974445}}
simulation-passed1
survival_time_max59.99999999999873
survival_time_mean59.99999999999873
survival_time_min59.99999999999873
No reset possible
1423258host-errorno0:04:23
Uncaught exception w [...]
Uncaught exception while running Docker Compose:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/92f3aa2c8d3e4f3f3a9fef6102cca888c4b99f90e95c6808407a6167bcde10c4/json

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 1245, in run_one
    container = client.containers.get(container_id)
  File "/usr/local/lib/python3.8/dist-packages/docker/models/containers.py", line 897, in get
    resp = self.client.api.inspect_container(container_id)
  File "/usr/local/lib/python3.8/dist-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/container.py", line 771, in inspect_container
    return self._result(
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 274, in _result
    self._raise_for_status(response)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.41/containers/92f3aa2c8d3e4f3f3a9fef6102cca888c4b99f90e95c6808407a6167bcde10c4/json: Not Found ("No such container: 92f3aa2c8d3e4f3f3a9fef6102cca888c4b99f90e95c6808407a6167bcde10c4")
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
1422259successno0:09:46
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median59.99999999999873
in-drivable-lane_median4.3499999999998895
driven_lanedir_consec_median2.267584946510809
deviation-center-line_median5.560918040395352


other stats
deviation-center-line_max5.665543832826052
deviation-center-line_mean5.372714923116391
deviation-center-line_min4.703479778848808
deviation-heading_max10.454471794519389
deviation-heading_mean9.605748195757917
deviation-heading_median9.67062083759416
deviation-heading_min8.627279313323946
distance-from-start_max3.040685334907007
distance-from-start_mean2.0347241292792138
distance-from-start_median2.024381858977471
distance-from-start_min1.049447464254907
driven_any_max3.5795904164549035
driven_any_mean3.481407758763022
driven_any_median3.4491559187079783
driven_any_min3.4477287811812265
driven_lanedir_consec_max3.5003518913422345
driven_lanedir_consec_mean2.3551360398422125
driven_lanedir_consec_min1.3850223750049966
driven_lanedir_max3.5003518913422345
driven_lanedir_mean3.2651578642468557
driven_lanedir_median3.1933043173233298
driven_lanedir_min3.173670930998528
in-drivable-lane_max4.399999999999963
in-drivable-lane_mean3.2749999999999355
in-drivable-lane_min0.0
per-episodes
details{"zigzag-000-ego0": {"driven_any": 3.4490819319910093, "survival_time": 59.99999999999873, "driven_lanedir": 3.2020366385783157, "in-drivable-lane": 4.349999999999895, "deviation-heading": 8.627279313323946, "distance-from-start": 1.049447464254907, "deviation-center-line": 5.586766629990746, "driven_lanedir_consec": 1.3850223750049966}, "zigzag-000-ego1": {"driven_any": 3.4477287811812265, "survival_time": 59.99999999999873, "driven_lanedir": 3.173670930998528, "in-drivable-lane": 4.399999999999963, "deviation-heading": 10.454471794519389, "distance-from-start": 1.5345351360434747, "deviation-center-line": 5.665543832826052, "driven_lanedir_consec": 1.633024912418889}, "zigzag-000-ego2": {"driven_any": 3.5795904164549035, "survival_time": 59.99999999999873, "driven_lanedir": 3.5003518913422345, "in-drivable-lane": 0.0, "deviation-heading": 9.518767273646697, "distance-from-start": 3.040685334907007, "deviation-center-line": 4.703479778848808, "driven_lanedir_consec": 3.5003518913422345}, "zigzag-000-ego3": {"driven_any": 3.449229905424948, "survival_time": 59.99999999999873, "driven_lanedir": 3.184571996068344, "in-drivable-lane": 4.349999999999884, "deviation-heading": 9.822474401541626, "distance-from-start": 2.514228581911467, "deviation-center-line": 5.535069450799957, "driven_lanedir_consec": 2.9021449806027295}}
simulation-passed1
survival_time_max59.99999999999873
survival_time_mean59.99999999999873
survival_time_min59.99999999999873
No reset possible
1421257host-errorno0:04:14
Uncaught exception w [...]
Uncaught exception while running Docker Compose:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1344, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/dist-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.8/dist-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 447, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 336, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 1247, in run_one
    stats = container.stats(stream=False)
  File "/usr/local/lib/python3.8/dist-packages/docker/models/containers.py", line 422, in stats
    return self.client.api.stats(self.id, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/container.py", line 1139, in stats
    return self._result(self._get(url, params={'stream': False}),
  File "/usr/local/lib/python3.8/dist-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
1417256host-errorno0:01:35
The container "simul [...]
The container "simulator" exited with code 137.


Error code 137 means an out-of-memory error. The container exceeded the memory available to it.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible