Submission 1647

Submission

1647

Competing

yes

Challenge

aido-LFVI-sim-testing

User

Bea Baselines 🐤

Date submitted

2022-11-27 20:18:46+00:00

Last status update

2022-11-29 12:16:21+00:00

Complete

complete

Details

Evaluation is complete.

Sisters

Result

💚

Jobs

sim-0of4: 16900 sim-1of4: 16890 sim-2of4: 16866 sim-3of4: 16868

User label

baseline-duckietown-ml

Admin priority

Blessing

n/a

User priority

The highlights are available only to the owner and the admins.

Evaluation jobs for this submission

See previous jobs for previous versions of challenges

Job ID

step

status

up to date

date started

date completed

duration

message

16900

sim-0of4

success

yes

2022-11-29 12:04:45+00:00

2022-11-29 12:16:04+00:00

0:11:19

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

survival_time_median	59.99999999999873
in-drivable-lane_median	0.0
driven_lanedir_consec_median	0.0001430511474609375
deviation-center-line_median	0.40653839040804857

other stats

deviation-center-line_max	0.40653839040804857
deviation-center-line_mean	0.40653839040804857
deviation-center-line_min	0.40653839040804857
deviation-heading_max	40.967332753486254
deviation-heading_mean	40.967332753486254
deviation-heading_median	40.967332753486254
deviation-heading_min	40.967332753486254
distance-from-start_max	2.220446049250313e-16
distance-from-start_mean	2.220446049250313e-16
distance-from-start_median	2.220446049250313e-16
distance-from-start_min	2.220446049250313e-16
driven_any_max	2.6645352591003757e-13
driven_any_mean	2.6645352591003757e-13
driven_any_median	2.6645352591003757e-13
driven_any_min	2.6645352591003757e-13
driven_lanedir_consec_max	0.0001430511474609375
driven_lanedir_consec_mean	0.0001430511474609375
driven_lanedir_consec_min	0.0001430511474609375
driven_lanedir_max	0.0001430511474609375
driven_lanedir_mean	0.0001430511474609375
driven_lanedir_median	0.0001430511474609375
driven_lanedir_min	0.0001430511474609375
in-drivable-lane_max	0.0
in-drivable-lane_mean	0.0
in-drivable-lane_min	0.0
per-episodes	details {"udem-000-ego0": {"driven_any": 2.6645352591003757e-13, "survival_time": 59.99999999999873, "driven_lanedir": 0.0001430511474609375, "in-drivable-lane": 0.0, "deviation-heading": 40.967332753486254, "distance-from-start": 2.220446049250313e-16, "deviation-center-line": 0.40653839040804857, "driven_lanedir_consec": 0.0001430511474609375}}
simulation-passed	1
survival_time_max	59.99999999999873
survival_time_mean	59.99999999999873
survival_time_min	59.99999999999873

No reset possible

16890

sim-1of4

success

yes

2022-11-29 11:57:22+00:00

2022-11-29 12:11:06+00:00

0:13:44

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

survival_time_median	59.99999999999873
in-drivable-lane_median	0.0
driven_lanedir_consec_median	0.0
deviation-center-line_median	0.4065243130134252

other stats

deviation-center-line_max	0.4065243130134252
deviation-center-line_mean	0.4065243130134252
deviation-center-line_min	0.4065243130134252
deviation-heading_max	40.967332753486254
deviation-heading_mean	40.967332753486254
deviation-heading_median	40.967332753486254
deviation-heading_min	40.967332753486254
distance-from-start_max	5.551115123125783e-17
distance-from-start_mean	5.551115123125783e-17
distance-from-start_median	5.551115123125783e-17
distance-from-start_min	5.551115123125783e-17
driven_any_max	6.661338147750939e-14
driven_any_mean	6.661338147750939e-14
driven_any_median	6.661338147750939e-14
driven_any_min	6.661338147750939e-14
driven_lanedir_consec_max	0.0
driven_lanedir_consec_mean	0.0
driven_lanedir_consec_min	0.0
driven_lanedir_max	0.0
driven_lanedir_mean	0.0
driven_lanedir_median	0.0
driven_lanedir_min	0.0
in-drivable-lane_max	0.0
in-drivable-lane_mean	0.0
in-drivable-lane_min	0.0
per-episodes	details {"4way-000-ego0": {"driven_any": 6.661338147750939e-14, "survival_time": 59.99999999999873, "driven_lanedir": 0.0, "in-drivable-lane": 0.0, "deviation-heading": 40.967332753486254, "distance-from-start": 5.551115123125783e-17, "deviation-center-line": 0.4065243130134252, "driven_lanedir_consec": 0.0}}
simulation-passed	1
survival_time_max	59.99999999999873
survival_time_mean	59.99999999999873
survival_time_min	59.99999999999873

No reset possible

16884

sim-1of4

timeout

yes

2022-11-28 21:40:47+00:00

No reset possible

16883

sim-0of4

host-error

yes

2022-11-28 21:40:42+00:00

2022-11-28 21:45:14+00:00

0:04:32

Uncaught exception: [...]

Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/uploading.py", line 139, in download_artefacts
    get_file_from_cache(fn, sha256hex)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner_cache.py", line 64, in get_file_from_cache
    raise KeyError(msg)
KeyError: 'Hash not in cache'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 901, in get_cr
    cr = run_single(
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 1032, in run_single
    prepare_dir(wd, aws_config, steps2artefacts=steps2artefacts, steps2scores=steps2scores, use_ipfs=use_ipfs)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 1465, in prepare_dir
    download_artefacts(aws_config, steps2artefacts, previous_steps_dir, use_ipfs=use_ipfs)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/uploading.py", line 155, in download_artefacts
    get_object(aws_config, bucket_name, object_key, fn)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/uploading.py", line 204, in get_object
    aws_object.download_file(fn)
  File "/usr/local/lib/python3.8/dist-packages/boto3/s3/inject.py", line 359, in object_download_file
    return self.meta.client.download_file(
  File "/usr/local/lib/python3.8/dist-packages/boto3/s3/inject.py", line 190, in download_file
    return transfer.download_file(
  File "/usr/local/lib/python3.8/dist-packages/boto3/s3/transfer.py", line 320, in download_file
    future.result()
  File "/usr/local/lib/python3.8/dist-packages/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
  File "/usr/local/lib/python3.8/dist-packages/s3transfer/futures.py", line 266, in result
    raise self._exception
  File "/usr/local/lib/python3.8/dist-packages/s3transfer/tasks.py", line 269, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/s3transfer/download.py", line 354, in _submit
    response = client.head_object(
  File "/usr/local/lib/python3.8/dist-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.8/dist-packages/botocore/client.py", line 943, in _make_api_call
    http, parsed_response = self._make_request(
  File "/usr/local/lib/python3.8/dist-packages/botocore/client.py", line 966, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/lib/python3.8/dist-packages/botocore/endpoint.py", line 119, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/lib/python3.8/dist-packages/botocore/endpoint.py", line 202, in _send_request
    while self._needs_retry(
  File "/usr/local/lib/python3.8/dist-packages/botocore/endpoint.py", line 354, in _needs_retry
    responses = self._event_emitter.emit(
  File "/usr/local/lib/python3.8/dist-packages/botocore/hooks.py", line 412, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/botocore/hooks.py", line 256, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python3.8/dist-packages/botocore/hooks.py", line 239, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.8/dist-packages/botocore/utils.py", line 1595, in redirect_from_error
    new_region = self.get_bucket_region(bucket, response)
  File "/usr/local/lib/python3.8/dist-packages/botocore/utils.py", line 1665, in get_bucket_region
    response = self._client.head_bucket(Bucket=bucket)
  File "/usr/local/lib/python3.8/dist-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.8/dist-packages/botocore/client.py", line 943, in _make_api_call
    http, parsed_response = self._make_request(
  File "/usr/local/lib/python3.8/dist-packages/botocore/client.py", line 966, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/lib/python3.8/dist-packages/botocore/endpoint.py", line 119, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/lib/python3.8/dist-packages/botocore/endpoint.py", line 202, in _send_request
    while self._needs_retry(
  File "/usr/local/lib/python3.8/dist-packages/botocore/endpoint.py", line 354, in _needs_retry
    responses = self._event_emitter.emit(
  File "/usr/local/lib/python3.8/dist-packages/botocore/hooks.py", line 412, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/botocore/hooks.py", line 256, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python3.8/dist-packages/botocore/hooks.py", line 239, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.8/dist-packages/botocore/retryhandler.py", line 207, in __call__
    if self._checker(**checker_kwargs):
  File "/usr/local/lib/python3.8/dist-packages/botocore/retryhandler.py", line 284, in __call__
    should_retry = self._should_retry(
  File "/usr/local/lib/python3.8/dist-packages/botocore/retryhandler.py", line 320, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/usr/local/lib/python3.8/dist-packages/botocore/retryhandler.py", line 363, in __call__
    checker_response = checker(
  File "/usr/local/lib/python3.8/dist-packages/botocore/retryhandler.py", line 247, in __call__
    return self._check_caught_exception(
  File "/usr/local/lib/python3.8/dist-packages/botocore/retryhandler.py", line 416, in _check_caught_exception
    raise caught_exception
  File "/usr/local/lib/python3.8/dist-packages/botocore/endpoint.py", line 281, in _do_get_response
    http_response = self._send(request)
  File "/usr/local/lib/python3.8/dist-packages/botocore/endpoint.py", line 377, in _send
    return self.http_session.send(request)
  File "/usr/local/lib/python3.8/dist-packages/botocore/httpsession.py", line 484, in send
    raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://duckietown-ai-driving-olympics-1.s3.us-east-2.amazonaws.com/"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 927, in get_cr
    uploaded = upload_files(wd, aws_config, copy_to_machine_cache=copy_to_machine_cache)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/uploading.py", line 50, in upload_files
    uploaded = upload(aws_config, toupload, copy_to_machine_cache=copy_to_machine_cache, quiet=True)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/uploading.py", line 293, in upload
    aws_object.load()
  File "/usr/local/lib/python3.8/dist-packages/boto3/resources/factory.py", line 564, in do_action
    response = action(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/boto3/resources/action.py", line 88, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "/usr/local/lib/python3.8/dist-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.8/dist-packages/botocore/client.py", line 943, in _make_api_call
    http, parsed_response = self._make_request(
  File "/usr/local/lib/python3.8/dist-packages/botocore/client.py", line 966, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/lib/python3.8/dist-packages/botocore/endpoint.py", line 119, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/lib/python3.8/dist-packages/botocore/endpoint.py", line 202, in _send_request
    while self._needs_retry(
  File "/usr/local/lib/python3.8/dist-packages/botocore/endpoint.py", line 354, in _needs_retry
    responses = self._event_emitter.emit(
  File "/usr/local/lib/python3.8/dist-packages/botocore/hooks.py", line 412, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/botocore/hooks.py", line 256, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python3.8/dist-packages/botocore/hooks.py", line 239, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.8/dist-packages/botocore/retryhandler.py", line 207, in __call__
    if self._checker(**checker_kwargs):
  File "/usr/local/lib/python3.8/dist-packages/botocore/retryhandler.py", line 284, in __call__
    should_retry = self._should_retry(
  File "/usr/local/lib/python3.8/dist-packages/botocore/retryhandler.py", line 320, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/usr/local/lib/python3.8/dist-packages/botocore/retryhandler.py", line 363, in __call__
    checker_response = checker(
  File "/usr/local/lib/python3.8/dist-packages/botocore/retryhandler.py", line 247, in __call__
    return self._check_caught_exception(
  File "/usr/local/lib/python3.8/dist-packages/botocore/retryhandler.py", line 416, in _check_caught_exception
    raise caught_exception
  File "/usr/local/lib/python3.8/dist-packages/botocore/endpoint.py", line 281, in _do_get_response
    http_response = self._send(request)
  File "/usr/local/lib/python3.8/dist-packages/botocore/endpoint.py", line 377, in _send
    return self.http_session.send(request)
  File "/usr/local/lib/python3.8/dist-packages/botocore/httpsession.py", line 484, in send
    raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://duckietown-ai-driving-olympics-1.s3.amazonaws.com/v3/staging/by-value/sha256/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

16870

sim-1of4

timeout

yes

2022-11-28 19:38:10+00:00

No reset possible

16868

sim-3of4

success

yes

2022-11-28 19:36:43+00:00

2022-11-28 19:38:24+00:00

0:01:41

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

other stats

simulation-passed	1
skipped	1

No reset possible

16867

sim-0of4

timeout

yes

2022-11-28 19:36:43+00:00

No reset possible

16866

sim-2of4

success

yes

2022-11-28 19:36:27+00:00

2022-11-28 19:38:13+00:00

0:01:46

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

other stats

simulation-passed	1
skipped	1

No reset possible