Benchmark Builder
Abstract
Novel human-robot interaction (HRI) approaches need careful validation. This is often achieved with user studies in laboratory environments, which mostly rely on highly individual, complex prototype setups in early design phases. Lately, the lack of replicability imposed by such experiments has vitally been discussed. Sharing common benchmark tasks among researchers would be a valuable basis to overcome this issue. We therefore offer a benchmark toolkit to compose scalable, synthetic standard tasks as a unified basis for future efforts towards more replicable HRI research.
Contact
Cite as
A publication introducing the benchmark toolkit concepts is accepted for presentation at the 31st IEEE International Conference on Robot & Human Interactive Communication
Accompanying Video
Dominik, Riedelbauch; Jonathan, Hümmer;
A Benchmark Toolkit for Collaborative Human-Robot Interaction
2022, March
Source Code of the Benchmark Builder
https://resy-gitlab.inf.uni-bayreuth.de/hrc-benchmarks/benchmark_builder
Object Detection Source Code and Reproducible Experiment
https://resy-gitlab.inf.uni-bayreuth.de/hrc-benchmarks/benchmark_builder_experiments
How-To Use the Benchmark Builder
Press the button on the bottom of this page to launch the Benchmark Builder Application. Then, follow the three steps outlined below to generate a benchmark problem including parts for 3D printing and a precedence graph task model:
Step 1
Push one of the buttons to the left (A) to add a part to the scene. Inserted parts will appear at the (+) location. Use drag-and-drop to transfer the part to its initial location (B). The next part can be inserted once you have moved away the last one from the (+) location. Turning the mouse wheel while dragging a part will rotate it. If you want to delete a part, drag-and-drop it to the waste bin (C). Proceed to Step 2 once you have added all necessary parts.
Step 2
In step 2, you will construct the goal state of your benchmark task (D). Drag-and-drop each part to its goal location. Stacking parts may cause "earlier-later"-relations. These will be displayed as a precedence graph to the right (E). Note that parts underneath other parts can no longer be moved - you will have to disassemble the structure top down to re-arrange them! Once you are done with arranging the parts, proceed to Step 3.
Step 3
You have finished the configuration of your individualized benchmark task! The Benchmark Builder will now offer you a .zip file for download. This file will contain the precedence graph task model, a set of .stl CAD files for 3D printing all necessary parts, and a .pdf file with further instructions on printing and assembling the benchmark parts.