STTT
International Journal on Software Tools for Technology Transfer
Competitions and Challenges (CoCha)
Research Competitions and Challenges can be a driving force in transferring theoretical results into working software tools that demonstrate the state of the art. Competitions provide a controlled comparison and independent evaluation of tool capabilities on subject matter selected by experts within the field. Challenges pose community-oriented research problems and tool capability objectives and may provide exemplar systems on which many teams of researchers may apply their techniques to comparatively assess capabilities, identify community technology gaps, and discover and illustrate synergies between technologies developed by different groups.
This STTT theme area provides a forum for overview articles about new competitions and challenges, progress reports of established efforts, and articles that provide insights about research challenges as a driver of progress within the scientific community.
Competitions: Typical characteristics of tool competitions include (a) a common collection of examples/benchmarks to which tools will be applied, where examples have been curated by experts within the field to exercise different dimensions of tool capabilities, (b) controlled application of tools to examples in a manner that removes ambiguities and uncertainties about tool configuration and enables results to be repeated, (c) independent evaluation of results by expert, and (d) a wide call and participation that helps ensure that competition winners are reflective of the state-of-the-art. Some of the most impactful competitions for formal methods tools have been the SAT competitions (first held in 1992) for satisfiability solvers and the CASC competitions (first held in 1996) for automated theorem provers. Since 2000, the number of dedicated verification competitions has been steadily increasing, also in areas different from verification. Competition victories provide motivation and “bragging rights” to individual research groups. Yet, strong benefits for the entire community include the ability to identify both successful strategies and algorithms as well as weaknesses and capability gaps that the community can address. Yearly competitions enable experts within the community as well as potential consumers of the technology to track the advance of tool capabilities across time.
This STTT theme area seeks Competition-related papers on (but not limited to) the following topics:
- reports about competitions that describe the progress of technology;
- articles that describe benchmark sets sets that are used in research competitions;
- system descriptions that indicate which portions of system development and behavior were effectively covered by tools in a competition, along with relative results and successes on different aspects of the system;
- analysis articles and surveys on the general the topic of competitions and their value in our community; and
- articles that focus on evaluation techniques that aid reproducibility of results and advance benchmarking technology.
Challenge Problems: Challenge problems come in several forms. They may be narratives describing research goals that require broad community action to be met. A well-known example is the Verifying Compiler Grand Challenge of Tony Hoare. They may also be system artifact exemplars with challenge goals for verification or some other aspect of development. An example is the Pacemaker Challenge which presented an industry requirements document for a pacemaker medical device and called on researchers to demonstrate the effectiveness of rigorous development methods for behavior specification, architecture descriptions, implementation, testing/verification, and certification using the real-world description provided by the requirements. The intent of both types of challenges is to spur the community to action around a common, concrete and challenging system – where clear successes in building and assuring the system correspond to significant improvements in the state-of-the-art that are meaningful to a variety of stakeholders.
This STTT theme area seeks Challenge-related papers on (but not limited to) the following topics:
- statements of narrative challenge problems with clearly defined goals and benefits to the community;
- descriptions of system exemplars as challenge problems, with descriptions of potential technologies to be brought to bear, expected benefits to developers, challenge issues in the context of development processes and tool chains;
- reports on technologies, tools, and methodologies that address challenge problems;
- assessments and surveys of challenge problem solutions and the interrelationships.
Dirk Beyer (LMU Munich, Germany)
Marieke Huisman (Twente University, Netherlands)
Theme Editors-in-Chief of CoCha