SaikoCTF, In-Person, Hack-In-The-Box (HITB) Security Conference Study – ASCEND Project

IARPA’s Reimagining Security with Cyberpsychology-Informed Network Defenses (ReSCIND) program aims to enhance cybersecurity by leveraging attackers’ human limitations. It focuses on developing novel defenses that exploit decision-making biases and cognitive vulnerabilities. These defenses help rebalance the asymmetry of cyber defense, imposing penalties on attackers and thwarting their efforts.

IARPA’s ReSCIND program funds the ASCEND (Adaptive Security through Cognitive Exploitation for Defense) project. Led by SRI, the multidisciplinary team behind ASCEND includes researchers and experts from the Florida Institute for Human and Machine Cognition, George Mason University, RAD Science Solution, SimSpace, Two Six Technologies, the University of Florida, and Virtual Reality Medical Center, as well as independent consultants, will create a game-changing, cyberpsychology-informed cyber defense system (see http://ascend.sri.com for more information).

Experiment Objectives

The overall objective of this study is to determine how cyber attackers change strategy, behavior, and physiologic response when presented with different cyber-attack countermeasures. ASCEND defines Cognitive Vulnerabilities (CogVulns) as decision-making and cognitive biases plus attacker’s culture, cognitive-emotional state, personality traits, and cyber-psychological characteristics. 

This study targets Loss Aversion (LA) Bias, Representativeness Bias (RB), and two aspects of Socio-Cultural Bias (SCB), namely Age Bias (SCB-AB) and Gender Bias (SCB-GB).  

We conducted experiments using targeted challenges in a capture-the-flag (CTF) event to simulate real-world adversarial behavior and attendees of the Hack In The Box Security Conference (HITB) in Bangkok, Thailand, as proxies for hackers.  

Experiment Description

The study begins with consenting, online individual differences measures (IDM) (e.g., demographics, personality) and an online skill-screener provisioned through pwn.college. At the beginning and end of the study, participants answer a questionnaire about their mental state.  

Participants can opt into wearing sensors that detect their brainwaves, heart rate, sweat, and respiration while they sit at a table using a laptop to participate in SaikoCTF. Before a participant who opted for physiological sensors starts the CTF cyber-attack challenges, they complete a physio-sensor calibration session to determine their individual baseline values.  

Participants are pseudo-randomly assigned to be in one of two groups (1 and 2). SaikoCTF uses a within-subjects design. Each challenge has a control (no CogVuln trigger present) and a treatment (CogVuln trigger present) version. There are two CTF challenges (A/B versions) for each CogVuln, for a total of four challenges per CogVuln (version A control, version A treatment, version B control, version B treatment). The A/B pairs have similar objectives and target the same CogVuln but have enough differences to control for human learning. The order in which control and treatment versions of each CTF challenge is presented is counter-balanced between groups 1 and 2 to control for order of conditions. After each CTF challenge, participants answer additional IDM and CogVuln measures (questionnaires and surveys) to assess their biases, personality traits, cultural values, and cognitive-emotional and cyber-psychological attributes. CTF challenges are time limited.  

CTF challenges are implemented in the SimSpace Cyber Range Platform (simspace.com/platform). For the three CogVulns tested in this study there are six targeted CTF challenges, each particularly designed to elicit the effectiveness of one CogVuln trigger deployed in the treatment version of the challenge. Furthermore, cyber behavior data is collected to evaluate hypothesized CogVuln sensors in relation to the established methods (IDMs and Bias measures) during analysis.  

The CTF challenges for LA target the Loss/Gain Framing Effect facet of LA by informing participants that after three failed login attempts, they will be locked out for 30 seconds and their activities will be logged. The goal is to test whether participants susceptible to loss/gain framing effect take less risky actions when faced with temporary suspended access.  

The CTF challenges for RB target the Sample Size Insensitivity facet of RB by providing logs with proportionally more alerts for a web-server endpoint or service that is not compromised than for an endpoint or service that is compromised. The goal is to test whether participants who see a lot of mentions of a compromised service are more likely to target that service.  

The CTF challenges for SCB-AG present participants with a grid of four AI-generated photos and usernames of males in their 30’s and 40’s and two photos and usernames of males in their 20’s and two photos and usernames of males in their 60s. The goal is to test whether participants pick a young or old profile over other profiles or positional bias.  

The CTF challenges for SCB-GB present participants with a grid of ten AI-generated photos and usernames common for white persons ages 30-44 (to factor out race and age bias) and two female photos and usernames. The goal is to test whether participants pick a female over male bias or positional bias.  

Experimental Results 

For CogVuln Sensor analysis, we used two approaches: Clustering and Least Absolute Shrinkage and Selection Operator (LASSO).  

LASSO identified for all CogVulns linear combinations of cyber behavior topics that predict average CogVulns nearly as well as the established methods (small SD difference, <0.20). Cyber behavior topics explain more variation for Base Rate Neglect and Sunk Cost than for other CogVulns. However, for these CogVulns, the model identified complex interactions of cyber behavior topics, and therefore results are more difficult to interpret. For the Reflection Effect and Sunk Cost facets of LA, and for the Nonrandom Sequence facet of RB, one or two cyber behavior topics were identified as predictive and had statistically significant associations with those CogVulns.  

Clustering identified low-level cyber measures that predict CogVulns well and are also ecologically valid, so they can be turned into CogVuln sensors. Clustering behaviors works well for predicting LA Framing Effect, but we found no predictive clusters for RB or SCB. 

Physio Analysis: The physiological data analysis showed that multi-sensor biometric measurements can reliably detect differences in participant states under cognitive bias triggers. In the Loss Aversion experiments, we observed that roughly half of the standard psychophysiological measures exceeded the Phase 1 effect size threshold (Cohen’s d > 0.3) between treatment and control groups. Many of these measures, particularly those related to heart rate variability, respiratory patterns, and skin conductance, also yielded statistically significant differences (p < 0.05), confirming that these signals carry meaningful information about the cognitive and emotional responses of participants during CTF challenges.  

These findings validate the operational value of integrating physiological measures into cognitive vulnerability sensing. Physiological signals provide an independent, high-temporal-resolution channel of information that complements behavioral and system log data. In scenarios where overt behavioral indicators may be subtle or delayed, physio data offers the potential for earlier detection of state changes, increasing the timeliness and confidence of ASCEND’s assessments. The convergence of significant physio patterns with our known bias-trigger points suggests that this modality can enhance ASCEND’s multi-sensor fusion capability, ultimately improving its resilience against adversary deception and its ability to adapt detection thresholds to individual profiles. 

LA CogVuln Trigger analysis: Within the paired modeling of participant behavior in continuous and binary measures, we saw only one metric with significance related to the trigger, namely Mean Time Between Login Tool Invocation (MTBLTI).  

RB CogVuln Trigger analysis: Within the paired modeling of participant behavior in continuous and binary measures, we consistently saw significance (p < 0.05) of the time on vulnerable path for event, challenge, and near significance for condition (p < 0.08). For the near significance in the trigger effect, we saw an effect size of approximately –0.19 for a pseudo-Cohen’s d across 170 participants. 

SCB-A and SCB-G CogVuln Trigger analysis: Within the paired modeling of participant behavior in continuous and binary measures, we saw significance (p < 0.05) in the trigger effect, inducing choosing a non-biased position (p < 0.018, odds ratio = 0.39). The treatment group (those exposed to a trigger of biased age or gender) were 2.5 times less likely to choose the biased position relative to not having an induced behavior. This was not expected behavior, and it seems counterintuitive to a second result, namely: We also saw significance in the trigger effect predicting the Manhattan distance of their choice from a biased position (p < 0.001, pseudo-Cohen’s d of 1.7).