This paper presents a study involving focus group interviews with pilots. Ten pilots listened to performances of test takers taking a variety of aviation English tests. The pilots were asked to rate the acceptability of the pilot’s language for (a) communicating with other pilots and (b) radiotelephony communications with air traffic control. The focus groups had two aims: (1) to establish the ‘indigenous’ assessment criteria pilots use when assessing the language ability of peers and (2) to establish what level is sufficient as the operational level. The results showed that the pilots focused on some but not all of the criteria on the ICAO scale. Whilst listening to the performances, they also often focused on the speakers’ technical knowledge. The paper proposes a model of how industry professionals can be involved in the validation of an LSP rating scales.