Interrater Agreement and Accuracy in Judging Character Strengths from a Single Facial Photograph

ParticipantsA total of 99 university students participated in this study. Familiar Group 1 consisted of 33 undergraduates from a class (26 females, Mean Age = 20.4, SD = 0.7), and Familiar Group 2 also included 33 undergraduates from another class (19 females, mean age = 19.3, SD = 0.5). The participants from the same class had known each other for more than one and a half years and were intimately acquainted due to their participation in the same courses and activities. The Unfamiliar Group included 33 undergraduates (17 females, mean age = 19.5, SD = 1.9), who had no contact experience with Group 1 and Group 2. All participants were right-handed and had normal or corrected-to-normal vision. They all volunteered for the experiments and signed an informed consent form before the experiments. Participants were compensated upon completion of the study. Design and ProcedureThe Evaluation Process of Character Strengths and AttractivenessTo help participants understand the experimental process, we conducted two practice trials before the formal experiment. During these practice trials, participants evaluated the character strengths and attractiveness of one male face and one female face. The faces used in the practice phase will not be included in the formal experiment. Figure 2 depicts the process for a single trial. Initially, a fixation point appeared in the center of the screen for 500 ms. This was followed by the display of a face for 2000 ms, followed by a blank screen for 200 ms. After this short pause, the target face reappeared, prompting participants to rate its attractiveness on a scale from 1 to 6, where "1" represents very unattractive and "6" denotes very attractive. Participants also assessed the character strengths using the same scale, with "1" indicating not fitting and "6" signifying fitting.The order of character strengths evaluations was randomized, with a 200-ms blank screen displayed between each evaluation. Participants were required to evaluate all ten traits for one face before proceeding to the next face. This process continued until all trials were completed.Each participant made assessments of female and male faces in two different blocks, and the order of blocks was random. Familiar Group 1 and Group 2 evaluated the faces of SET 1 and 2, respectively. The unfamiliar group evaluated all the faces of SET 1 and 2. Statistical MethodsAnalysis Method for Interrater Agreement:First, we calculated the interrater agreement for the familiar and unfamiliar groups regarding their assessments of each face's relationship, vitality, conscientiousness, and attractiveness. The index of interrater agreement was rwg, which was defined as the proportional reduction in error variance (James et al., 1993), and the calculation formula was as follows: rwg=1-(Sx2/tE2);Sx2represents the error variation, that is, the observed variation of each face in each group; tE2 represents the expected variation. tE2 = (p*p-1)/12, and p is the number of response alternatives (in the current study, p = 6). We recommended the 0.70 for rwg as indicating a moderate level of interrater agreement (Brown & Hauenstein, 2005). Next, the rwg values for relationship, vitality, conscientiousness, and attractiveness in both the unfamiliar and familiar groups were converted into Z-scores using the Fisher formula. Following this, LSD paired samples t-tests were conducted. If familiarity has an impact on the assessments of relationship, vitality, conscientiousness, and attractiveness, we would expect to see a significant difference between the interrater agreements of the unfamiliar and familiar groups in these dimensions.Analysis Method for Accuracy :Using targets' self-ratings as the criterion measure, we evaluated accuracy by correlating these self-ratings with the ratings from observers, including both the Familiar and Unfamiliar Groups (Naumann et al., 2009). To compare the strength of these correlations (Familiar-Self vs. Unfamiliar-Self), we analyzed the size of these correlations using William's T, a statistical test that evaluates the difference between two correlations sharing a dependent variable- specifically, the traits of the targets (Diedenhofen & Musch, 2015; Satchell, 2019). Significant correlations between observers’ ratings and self-ratings would indicate that the observers accurately assessed the traits (affiliation, vitality, willpower, and attractiveness). Furthermore, if familiarity enhances judgment accuracy, we should observe significantly stronger correlations in the Familiar-Self comparisons than in the Unfamiliar-Self comparisons.