Benchmarking Performance

We provide a benchmarking script to enable users to easily analyze the performance of CARLA in their own environment. The script can be configured to run a number of scenarios that combine different maps, sensors and weather conditions. It reports the average and standard deviation of FPS under the requested scenarios.

In this section we detail the requirements to run the benchmark, where to find the script, the flags available to customize the scenarios that are run and examples on how to run the commands.

We have also included our results of a separate benchmark which measures performance in CARLA in a specific environment when using different combinations of number of vehicles, enabling physics and/or enabling Traffic Manager. The results are presented alongside the CARLA version used and the environment the test was performed in.

The benchmark script

The benchmark script can be found in PythonAPI/util. It has several flags available to customize the scenarios to be tested which are detailed in the synopsis below.

Before you begin

The benchmarking script requires some dependencies to be installed before you can run it:

python -m pip install -U py-cpuinfo==5.0.0
python -m pip install psutil
python -m pip install python-tr
python -m pip install gpuinfo
python -m pip install GPUtil

Synopsis

python3 performance_benchmark.py [--host HOST] [--port PORT] [--file FILE] [--tm] [--ticks TICKS] [--sync] [--async]) [--fixed_dt FIXED_DT] [--render_mode]) [--no_render_mode] [--show_scenarios]) [--sensors SENSORS [SENSORS ...]]) [--maps MAPS [MAPS ...]]) [--weather WEATHER [WEATHER ...]]

Flags

--host: IP_ADDRESS

Default: Localhost.

Configures the host of the server.

--port: PORT

Default: 2000

Configures the TCP port to listen to.

--file: filename.md

Default: benchmark.md

Writes results in markdown table format to a file.

--tm

Switch to Traffic Manager benchmark

--ticks

Default: 100

Sets the number of ticks to use for each scenario.

--sync

Default mode.

Runs benchmark in synchronous mode.

--async

Runs benchmark in asynchronous mode.

--fixed_dt

Default: 0.05

For use with synchronous mode if you would like to set the delta timestep.

--render_mode

Runs benchmark in rendering mode.

--no_render_mode

Default mode.

Runs benchmark in non-rendering mode.

--show_scenarios

When the script is run with only this flag you will see a list of all the scenario parameters available.

When combined with other flags you will see a preview of the scenarios that will be run without actually executing them.

--sensors: INTEGER

Default: All

Sensors to be used in the benchmark. Chose between LIDAR and RGB camera:

  • 0: cam-300x200
  • 1: cam-800x600
  • 2: cam-1900x1080
  • 3: cam-300x200 cam-300x200 (two cameras)
  • 4: LIDAR: 100k
  • 5: LIDAR: 500k
  • 6: LIDAR: 1M
--maps: TownName

Default: All maps

All CARLA maps, both layered and sub-layered, are available.

--weather: INTEGER

Default: All weather conditions

Change the weather conditions:

  • 0: ClearNoon
  • 1: CloudyNoon
  • 2: SoftRainSunset

How to run the benchmark

  1. Start CARLA:

    # Linux:
./CarlaUE4.sh
# Windows:
CarlaUE4.exe
# Source:
make launch

  2. In a separate terminal navigate to PythonAPI/util to find the performance_benchmark.py script:

  • Show all possible scenarios without running them:
python3 performance_benchmark.py --show_scenarios
  • Show what scenarios will run when configurations are applied without actually executing them:
python3 performance_benchmark.py --sensors 2 5 --maps Town03 Town05 --weather 0 1 --show_scenarios`
  • Execute the performance benchmark for those scenarios:
python3 performance_benchmark.py --sensors 2 5 --maps Town03 Town05 --weather 0 1
  • Perform the benchmark for asynchronous mode and rendering mode:
python3 performance_benchmark.py --async --render_mode

CARLA performance report

The following table details the performance effect on average FPS when running CARLA with increasing numbers of vehicles and different combinations of enabling and/or disabling physics and Traffic Manager.

  • CARLA Version: Dev branch on 29/01/21 (commit 198fa38c9b1317c114ac15dff130766253c02832)
  • Environment Specs: Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz / 32 GB / NVIDIA GeForce GTX 1080 Ti
Num Vehicles Phy: Off TM: Off Phy: On TM: Off Phy: Off TM: On Phy: On TM: On
0 1220 1102 702 729
1 805 579 564 422
10 473 223 119 98
50 179 64 37 26
100 92 34 22 15
150 62 21 17 10
200 47 15 14 7
250 37 11 12 6

If you have any questions regarding the performance benchmarks then don't hesitate to post in the forum.

CARLA forum