Cyber security exercises: A comparison of participant evaluation metrics and scoring systems

Abstract

Cyber security exercises became an important tool for the information security domain for various reasons. Raising awareness, improving readiness to cyber-attacks, and testing capabilities of experts before a real incident are some. However, measuring the success of exercises and the participants participating in the game is a vague area, which no scientifically elaborated approach has been followed, according to the current literature. In this research, some of the wellknown cyber security exercises are compared and contrasted by evaluation metrics they use and scoring systems they have implemented to their game. Other than capture the flag type events, which collecting points depend on how many challenges participants solve, this paper elaborates on red team vs. blue team exercises. The ultimate goal is to observe strengths and weaknesses of their approach and identify which metrics are commonly used. According to the findings, after a detailed comparison of these exercises, it was realized that current evaluation techniques in cyber security exercises mostly focus on metrics representing the defensive and offensive success of participants. Keeping the systems up and running, successfully defending systems, or attacking other players are some of the key elements. Furthermore, it was realized that exercises provide a useful way to raise awareness, improve technical competence, and enhance the readiness of cyber security experts in the field. Thus, it is important to extract meaningful outcomes from those exercises, such as understanding which participant has enough capacity to deal with eminent cyber security attacks. There are a number of exercises that aim to achieve this goal; however, it is not clearly described how to evaluate the success metrics. Other than that, evaluating players with a solid, reusable, and meaningful approach is still missing. Non-technical, but important, issues like reporting and media relations lack in many of them. It was seen after this research that it should not be just about who comes first in these cyber security exercises; rather, it should tell who is ready for the real combat.