Measuring performance with parsl-perf
parsl-perf is tool for making basic performance measurements of Parsl
configurations.
It runs repeated batches of no-op apps, giving a measurement of tasks per
second. By default, parsl-perf will attempt to estimate a batch size
that will take 2 minutes to execute.
This can give a basic measurement of some of the overheads in task execution.
parsl-perf must be invoked with a configuration file, which is a Python
file containing a variable config which contains a Config object, or
a function fresh_config which returns a Config object. The
fresh_config format is the same as used with the pytest test suite.
To specify a parsl_resource_specification for tasks, add a --resources
argument.
To change the target runtime from the default of 120 seconds, add a
--time parameter.
To change the iteration algorithm used to pick the next batch size and to
decide when to stop, use the --iterate parameter:
--iterate=estimate(the default) will run a small batch, then repeatedly re-estimate a new batch size until it finds one that takes at least 75% of the target run time. This will rapidly find a batch size of around the target size.
--iterate=exponentialwill start with a small batch, then double the size in each subsequent batch until reaching the target run time. This is useful for understanding how a configuration scales down as batch sizes increase, which is sometimes sub-linear, and to get more consistent batch sizes across runs ofparsl-perf.
For example:
$ python -m parsl.benchmark.perf --config parsl/tests/configs/workqueue_ex.py --resources '{"cores":1, "memory":0, "disk":0}'
==== Iteration 1 ====
Will run 10 tasks to target 120 seconds runtime
Submitting tasks / invoking apps
warning: using plain-text when communicating with workers.
warning: use encryption with a key and cert when creating the manager.
All 10 tasks submitted ... waiting for completion
Submission took 0.008 seconds = 1248.676 tasks/second
Runtime: actual 3.668s vs target 120s
Tasks per second: 2.726
[...]
==== Iteration 4 ====
Will run 57640 tasks to target 120 seconds runtime
Submitting tasks / invoking apps
All 57640 tasks submitted ... waiting for completion
Submission took 34.839 seconds = 1654.487 tasks/second
Runtime: actual 364.387s vs target 120s
Tasks per second: 158.184
Cleaning up DFK
The end