Achieving reliable grading across trainees is precursor to effective competency development in industry, such as the aviation industry where continuous improvement in pilot’s professional knowledge is crucial. Grading data is an important digital asset to a company in terms of gaining understanding of the competence level of its knowledge workers, such as pilots. The direct output of quality grading data is the design of training programs. Achieve grading reliability that varies across assessors is an enduring quest for excellence due to its nature of subjective and situational judgement. Here, we study assessors’ grading practices in pilot training sessions of SIA. Specifically, we seek to understand:
1. What are the salient situational factors affecting the trainee grades?
2. If so, can non-invasive intervention such as refreshing workshop articulating the grading expectation and criteria, change the   grading behaviour of the assessors?

By answering the two questions, we seek to provide instrumental suggestions to the partnering airline company on salient aspects to be mindful of when deriving training programs and grading assessment sessions. Based on a set of grading data obtained, we seek to identify salient factors that could have contributed to the grades that the trainees received. We observed assessor, training session timing, and nature of the grading dimensions could explain for variation in the grading scores. Furthermore, we observed that the bulk of the grades is hovering around the mid-point of 3 (scoring range of 1 to 5). This makes the discrimination of pilots who are supposedly vary in their competencies more challenging. To address this issue, a refreshing workshop on grading for assessors was conducted in reiterating the grading expectations and criteria. Interestingly, we observed assessors graded more variably, resulting in more informative grading. Overall, the quest for reliable grading is ongoing and this paper reports the preliminary findings we made.