Carl Westin

116 Consistency and agreement in conflict resolution adaptive cruise control systems) 241–243 and intelligent agents (e.g., unmanned vehi- cles and robots). 197 In order for a system to personalize its support, the system must know some- thing about who it is interacting with and how that person thinks. For example, in the context of ATC conflict resolution the system must know how the controller reasons when determining a solution. As a first step, however, it is necessary to de- termine the degree of agreement between controllers (i.e., the inter-rater reliability) since there is little benefit of personalizing a system if all think the same or agree on the same solution. Second, it is necessary to determine how consistent controllers make decisions and solve conflicts over time (i.e., the intra-rater reliability). Pre- vious research provides an inconclusive picture in regards to controller consistency and agreement in conflict resolution. More generally, research by Shanteau and col- leagues 236 indicates that experts often disagree and that there appears to be higher consistency than agreement. Consistency and agreement may be especially low in fields that have no “gold standard” (i.e., a benchmark, correct, or optimal solution) such as ATC. 131, 244 This paper aims to determine the degree of consistency and agreement in oper- ator decision-making, using ATC conflict resolution as the example domain. How- ever, previous research investigating controllers’ decision-making in conflict resolu- tion has been limited in terms of subjective data collection methods and use of static traffic scenarios. For this purpose, a novel experimental design was developed for investigating controllers’ conflict resolution performance in a dynamic environment. Furthermore, a novel conflict solution framework was developed against which con- sistency and agreement could be measured objectively. A series of human-in-the-loop simulations were conducted in two separate stud- ies. These investigated controller’s conflict solving patterns (i.e., actions taken to solve a conflict) across repeated conflicts (each conflict was repeated four times) and identified their individual and consistent problem-solving style (i.e., solving the same conflict using the same pattern in at least 75% of all repetitions). Participants’ problem-solving styles were then compared to determine the agreement between participants. Since conflict solving is highly situation dependent, different scenarios and conflicts were used in the two studies. In total five different conflicts with vary- ing geometries (e.g., convergence angles) and parameters (e.g., relative distances and speeds) were investigated. The empirical findings can be helpful for determin- ing the benefit of developing decision-support systems sensitive to the individual.