SaikoCTF, In-Person, EkoParty Security Conference Study – ASCEND Project

IARPA’s Reimagining Security with Cyberpsychology-Informed Network Defenses (ReSCIND) program aims to enhance cybersecurity by leveraging attackers’ human limitations. It focuses on developing novel defenses that exploit decision-making biases and cognitive vulnerabilities. These defenses help rebalance the asymmetry of cyber defense, imposing penalties on attackers and thwarting their efforts.

IARPA’s ReSCIND program funds the ASCEND (Adaptive Security through Cognitive Exploitation for Defense) project. Led by SRI, the multidisciplinary team behind ASCEND includes researchers and experts from the Florida Institute for Human and Machine Cognition, George Mason University, RAD Science Solution, SimSpace, Two Six Technologies, the University of Florida, and Virtual Reality Medical Center, as well as independent consultants, will create a game-changing, cyberpsychology-informed cyber defense system (see http://ascend.sri.com for more information).

Experiment Objectives

The overall objective of this study is to determine how cyber attackers change strategy, behavior and physiologic response when presented with different cyber-attack countermeasures. ASCEND defines Cognitive Vulnerabilities (CogVulns) as decision-making and cognitive biases plus attacker’s culture, cognitive-emotional state, personality traits, and cyber-psychological characteristics.

This study targets Anchoring Bias (AB) Bias, Confirmation Bias (CB), and two aspects of Socio-Cultural Bias (SCB), namely Age Bias (SCB-AB) and Gender Bias (SCB-GB).

We conducted experiments using targeted challenges in a capture-the-flag (CTF) event to simulate real-world adversarial behavior and attendees of the Ekoparty Security Conference (EkoParty) in Buenos Aires, Argentina, as proxies for hackers.

Experiment Description

The study begins with consenting, online individual differences measures (IDM) (e.g., demographics, personality) and an online skill-screener provisioned through pwn.college. At the beginning and end of the study, participants answer a questionnaire about their mental state.

Participants can opt into wearing sensors that detect their brainwaves, heart rate, sweat, and respiration while they sit at a table using a laptop to participate in SaikoCTF. Before a participant who opted for physiological sensors starts the CTF cyber-attack challenges, they complete a physio-sensor calibration session to determine their individual baseline values.

Participants are pseudo-randomly assigned to be in one of two groups (1 and 2). SaikoCTF uses a within-subjects design. Each challenge has a control (no CogVuln trigger present) and a treatment (CogVuln trigger present) version. There are two CTF challenges (A/B versions) for each CogVuln, for a total of four challenges per CogVuln (version A control, version A treatment, version B control, version B treatment). The A/B pairs have similar objectives and target the same CogVuln but have enough differences to control for human learning. The order in which control and treatment versions of each CTF challenge is presented is counter-balanced between groups 1 and 2 to control for order of conditions. After each CTF challenge, participants answer additional IDM and CogVuln measures (questionnaires and surveys) to assess their biases, personality traits, cultural values, cognitive-emotional and cyber-psychological attributes. CTF challenges are time limited.

CTF challenges are implemented in the SimSpace Cyber Range Platform (simspace.com/platform). For the three CogVulns tested in this study there are six, targeted CTF challenges, each particularly designed to elicit the effectiveness of one CogVuln trigger deployed in the treatment version of the challenge. Furthermore, cyber behavior data is collected to evaluate hypothesized CogVuln sensors in relation to the established methods (IDMs and Bias measures) during analysis.

The CTF challenges for AB target the numeric priming facet of AB. Participants are told to find the target server and port on the network. In the A version of the challenge, participants are given access to an administrator workstation with an admin password that ends in “44,” and there is only one port that contains the number 4. In the B version of the challenge, the participant in the treatment groups are given access to an administrator’s workstation that has the number “9” in its password, and there is only one IP address that contains the number 9.

The CTF challenges for CB are testing whether susceptible participants who are initially shown evidence of a network vulnerability and script will continuously attempt to exploit that vector even if a simpler and easier path exists out of sight. In the A version of the CB challenge, participants are given network access and login credentials to the target machine, which has a directory with potential attack scripts to try. The target machine has the root login credentials stored in a hidden location that linpeas can find. Participants must escalate privileges to get the flag that is only readable by root. In the treatment version, the participants are given a linpeas output that indicates dirtycow vulnerability, but linpeas is disrupted halfway through and is incomplete. The goal is to test whether susceptible participants will assume the output of linpeas to be correct and attempt multiple dirtycow exploits, rather than re-running linpeas to confirm the results. In the B version of the CB challenge, participants are given a file that contains the output of an nmap scan showing port 80/http open and port 445/smb open, with port 80 being vulnerable to a number of Apache 2.4.49 exploits. The goal is to test whether susceptible participants will continue to scan and attack the web server using the provided vulnerability scripts rather than re-scanning the box to see that SMB is enabled and allows anonymous login.

The CTF challenge for SCB-AG presents participants with a grid of four AI-generated photos and usernames of males in their 30s and 40s and two photos and usernames of males in their 60s. The goal is to test whether participants pick a young or old profile over other profiles or positional bias.

The CTF challenge for SCB-GB presents participants with a grid of ten AI-generated photos and usernames common for white persons ages 30-44 (to factor out race and age bias) and two female photos and usernames. The goal is to test whether participants pick a female over male bias or positional bias.

Experimental Results 

For CogVuln Sensor analysis, we used two approaches: Clustering and Least Absolute Shrinkage and Selection Operator (LASSO).

LASSO identified for all CogVulns linear combinations of cyber behavior topics that predict average CogVulns nearly as well as the established methods (small SD difference, <0.20). Cyber behavior topics explain more variation for CB than for AB or SCB. However, for these CogVulns, the model identified complex interactions of cyber behavior topics, and therefore results are more difficult to interpret.

Clustering identified low-level cyber measures that predict CogVulns well and are also ecologically valid, so they can be turned into CogVuln sensors. Clustering behaviors works well for predicting CB and AB, but results were not strong for the SCBs we tested.

Physio Analysis: The physiological data analysis showed that multi-sensor biometric measurements can reliably detect differences in participant states under cognitive bias triggers. In the Loss Aversion experiments, we observed that roughly half of the standard psychophysiological measures exceeded the Phase 1 effect size threshold (Cohen’s d > 0.3) between treatment and control groups. Many of these measures, particularly those related to heart rate variability, respiratory patterns, and skin conductance, also yielded statistically significant differences (p < 0.05), confirming that these signals carry meaningful information about the cognitive and emotional responses of participants during CTF challenges.

These findings validate the operational value of integrating physiological measures into cognitive vulnerability sensing. Physiological signals provide an independent, high-temporal-resolution channel of information that complements behavioral and system log data. In scenarios where overt behavioral indicators may be subtle or delayed, physio data offers the potential for earlier detection of state changes, increasing the timeliness and confidence of ASCEND’s assessments. The convergence of significant physio patterns with our known bias-trigger points suggests that this modality can enhance ASCEND’s multi-sensor fusion capability, ultimately improving its resilience against adversary deception and its ability to adapt detection thresholds to individual profiles.

CB CogVuln Trigger analysis: Within the paired modeling of participant behavior in continuous and binary measures we saw significance (p < 0.05) in the trigger effect (condition going from control to treatment) in the Levenshtein distance (p < 0.00, pseudo-Cohen’s d of 0.72) for syntax and the binary measure of biased behavior (p < 0.0, odds ratio = 13), as well as significance from challenge and established measures (but at a lower effect size compared to trigger). For the continuous Levenshtein distance metric, we saw significance for the syntax used between the control and treatment groups. The distance to the correct command syntax (the command was mentioned in the instructions for both control and treatment group) was greater in the treatment group. We interpret this to mean that the participant is so caught up in the planted data/information that how they run the command does not matter to them, even though it is given.

AB CogVuln Trigger analysis: We found no significant effect between control and treatment for all paired t-test groupings. Performing logistic regression model analyses to predict the effect on future target choices by looking at participants’ first/second/third selected target also showed no significant results. The CogVuln Trigger for AB as designed did not have the intended effect. We posit that the graphical order of ports/IP addresses was more strongly triggering a Position Bias than an Anchoring Bias.

SCB-A and SCB-G CogVuln Trigger analysis: Within the paired modeling of participant behavior in continuous and binary measures, we saw significance (p < 0.05) in the trigger effect, inducing choosing a non-biased position (p < 0.018, odds ratio = 0.39). The treatment group (those exposed to a trigger of biased age or gender) were 2.5 times less likely to choose the biased position relative to not having an induced behavior. This was not expected behavior, and it seems counterintuitive to a second result, namely: We also saw significance in the trigger effect predicting the Manhattan distance of their choice from a biased position (p < 0.001, pseudo-Cohen’s d of 1.7).