One-sentence Summary

In a preregistered cross-sectional study with tertiary-level students ($N = 100$N=100), researchers at $ETH~Z\"u rich$ETH Zu¨rich found that both written-communication proficiency and computer-science achievement significantly predict vibe-coding performance, with CS achievement remaining a significant predictor after controlling for domain-general cognitive skills, thereby informing tool and curriculum design for LLM-driven programming.

Key Contributions

• Reports a preregistered cross-sectional study with 100 tertiary-level students measuring computer-science achievement, domain-general cognitive skills, written-communication proficiency, and vibe-coding performance, using tasks curated by an eight-expert consensus process and a purpose-built environment that mirrors commercial tools while enabling controlled evaluation.
• Introduces a controlled vibe-coding assessment environment that hides source code from participants, allowing iterative GUI-oriented programming with an LLM in a stateless, single-turn setup, which differs from typical LLM workflows but provides tighter experimental control.
• Finds that both written-communication skills (r = .29) and computer-science achievement (r = .39) significantly predict vibe-coding performance, with computer-science achievement remaining a significant predictor after controlling for domain-general cognitive skills, and both constructs contributing independently to GUI-oriented vibe-coding outcomes.

Introduction

LLM integrated development platforms now let users build software by describing it in plain language, a workflow known as vibe coding. This shift raises questions about which skills matter most for future developers, especially as written communication skills appear to be declining among students and adults. Prior work has examined how prompt quality or programming experience affects LLM guided coding, but few studies directly measure writing proficiency alongside computer science achievement in a controlled setting, and none isolate the pure, no-code variant of vibe coding where users never see the generated source.

The authors address this gap with a preregistered, cross-sectional study of 100 university students. They measure CS achievement, written communication skills, domain-general reasoning, and vibe coding performance using a custom platform that mimics commercial AI programming tools. Participants solved GUI-oriented tasks by iteratively prompting an LLM without viewing or editing code, and the task suite was validated by eight experts through a structured consensus process.

The authors find that both CS achievement and writing skills significantly predict vibe coding performance, with CS achievement contributing roughly twice the unique variance of writing skills in a joint model. They also contribute a vetted assessment suite for GUI-oriented vibe coding proficiency and a research platform designed for controlled experiments on LLM native software creation, supporting future empirical work on the construct.

Dataset

The authors construct their dataset through a combination of expert elicitation, standardized instruments, and locally developed assessments. The data is used to evaluate how well participants can perform programming tasks with the assistance of an LLM-based agent, alongside measures of their written communication, computer science knowledge, and general reasoning skills.

Task Design and Dataset Composition

• The core dataset consists of three programming tasks that participants are asked to solve using an LLM-based agent.
• Two of the tasks were acquired through a Delphi-style expert elicitation process. Eight experts, diverse in geography and occupation, were recruited to propose and rate tasks based on criteria including interdisciplinarity, real-world relevance, GUI-first implementation, non-trivial complexity, time feasibility, clear specification, and culture-fairness.
• After two rating rounds, three tasks met the consensus threshold (median rating ≥ 4 on all scales, IQR ≤ 1 for at least 80% of scales). Two of these tasks were similar in description and were combined into one.
• The third task was intentionally decontextualized and non-authentic, designed as a small GUI toy application with opaque labeling to avoid strong priors that might bias the LLM.
• Each task is paired with a rubric for evaluating behavioral fidelity, defined before the study. Participants are told that imitating the style or design of the base applications is not necessary.

Survey Instruments and Additional Data Subsets

• Written Communication Skills: A task-based assessment and analytic rubric were designed by two instructors in written communication. The prompt asks participants to write a 300-to-450-word description of a technical concept for a college-educated, non-expert adult. The rubric has five categories, each scored from 1 to 5, with a maximum total of 100. Essays are graded blindly by two independent raters, with a third blinded rater used when scores differ by more than ten points.
• Computer Science Achievement: A 12-item subset of the validated SCS1 assessment is used, which is language-independent and pseudocode-based. The subset includes four items each on definitional knowledge, code tracing, and code completion, selected based on item-response-theory difficulty and discrimination values. A 25-minute time limit is imposed, and validity evidence was gathered in a pilot study.
• General Reasoning Skills: The ICAR16, a 16-question subset of the International Cognitive Ability Resource, is used to measure domain-general cognitive skills. It includes four questions each on verbal reasoning, matrix reasoning, letter series, and three-dimensional rotation. A 12-minute time limit was chosen after piloting, and the order of items was randomized.

Data Usage and Processing

• The dataset is used to measure the construct of vibe coding, defined as iterative code generation from human task specifications.
• Participants are required to formulate the target behavior themselves, rather than copying a textual specification, to evaluate their ability to articulate a system design for an LLM agent.
• The written communication rubric is used to index a domain-general proficiency, with individual criteria treated as reflective indicators. The authors test whether the underlying ability to structure and convey technical information predicts performance on LLM-mediated programming tasks.
• The SCS1 subset and ICAR16 are used as covariates or control measures to account for prior CS knowledge and general cognitive ability.
• The time limits for the SCS1 subset and ICAR16 were determined through pilot studies to avoid construct-irrelevant speed while minimizing testing fatigue.

Method

The authors designed the study tasks using a Delphi-style expert elicitation process to ensure authenticity and appropriate complexity. An expert panel generated tasks meeting specific criteria, including interdisciplinarity, real-world relevance, GUI-first implementation, and time feasibility. The final task set comprised two context-rich tasks (replication and feature extension) and one decontextualized task to evaluate expressive specification without relying on strong prior labels.

To guarantee reproducible access and strict data privacy, the authors developed a dedicated vibe-coding platform. The interface deliberately conceals the source code to align with the pure vibe coding paradigm.

As shown in the figure above, the platform features a chat window on the left for interacting with the AI agent and a live application preview on the right. The left sidebar also displays the task description and a timer. The system integrates convenience features such as iteration rollback and a deliberately blurred live stream of the model's output to convey progress without allowing code copying. For the underlying AI, the authors selected Anthropic Claude Sonnet 4 based on performance evaluations. To optimize throughput, the platform utilizes a diff-based update mechanism where the model returns only the necessary code fragments rather than rewriting the entire file.

To ensure participants fully explored the sample applications before attempting replication or extension, the authors implemented a feature-tracking mechanism. The sample applications were instrumented to monitor feature usage and provide explicit feedback to the user.

As illustrated in the figure above, a banner notifies participants if they have missed interacting with relevant features in the sample application, preserving construct validity by focusing on coding ability rather than application comprehension.

The experimental procedure followed a counterbalanced design to mitigate order effects.

As depicted in the figure above, participants were randomly assigned to two groups. Both groups completed a demographic survey, followed by a test battery (including ICAR16, an essay task, and SCS1) and the vibe coding tasks (replication, feature addition, and decontextualized tasks) in alternating order. This structured flow ensured a rigorous evaluation of the relationship between written communication skills and vibe coding performance.

Experiment

The experiments included a pilot study to calibrate time limits and refine a vibe coding platform, followed by a main study with 100 students measuring writing skills, CS achievement, cognitive ability, and vibe coding performance. The pilot validated the SCS1 time limit, reduced ICAR16 from 16 to 12 minutes, and extended the essay task to 300-450 words, while also adding platform notifications to preserve construct validity. In the main study, writing skills correlated positively with vibe coding performance, but this correlation became non-significant after controlling for domain-general cognitive skills, whereas CS achievement remained a significant predictor and contributed roughly twice the unique variance in performance. Exploratory analyses found that prompt quality, as rated by humans and measured via lexical diversity, mediated the relationship between writing skill and vibe coding outcomes, with higher-quality prompts linking stronger writing to better task performance.

Expert panelists rated proposed vibe coding tasks on five criteria, with selected tasks showing strong overall ratings. The exam scheduler, meal planner, and course registration platform were chosen, generally receiving higher or more consistent scores across criteria. Some tasks, like the eye tracker app, showed notably lower clarity and wider disagreement. Selected tasks (exam scheduler, meal planner, course registration) tended to have higher mean ratings, especially for challenge and purpose clarity. The eye tracker app received the lowest purpose clarity rating and showed the widest interquartile ranges across several criteria, indicating less consensus. The isometric room builder and celebrity ranker scored lower on authenticity and challenge compared to the selected tasks. Culture fairness ratings were generally high across all tasks, with most medians near or above 4.4.

Scores across the study instruments varied widely, with writing skills showing the highest average and vibe coding tasks the lowest, while replication and decontextualized tasks exhibited the broadest score ranges. Reliability checks showed acceptable consistency for essay grading and the self-report scale, but lower consistency for the cognitive test, which was treated as a covariate. Correlations among primary constructs were mostly positive and significant, with writing skills and CS achievement both relating to vibe coding performance, though only the CS achievement link held after controlling for the cognitive test. Writing skills had the highest mean score, while vibe coding tasks had the lowest mean score among the instruments. Replication and decontextualized tasks showed the widest score ranges, spanning nearly the full normalized scale. Essay grading reliability was acceptable after recalibration, and the self-report scale met the reliability threshold, but the cognitive test fell short and was used only as a covariate. All primary construct correlations were positive and significant except between CS achievement and writing skills, which was small and non-significant. After adjusting for the cognitive test, the link between CS achievement and vibe coding remained, but the link between writing skills and vibe coding did not.

Vibe coding performance correlates positively with CS achievement, cognitive ability, and writing skills, with the strongest associations observed for CS achievement and cognitive ability. When controlling for cognitive ability, the correlation between writing and vibe coding remains positive but is reduced, suggesting a partial mediation by cognitive factors. The pattern is consistent with prompt quality acting as a linking mechanism between writing proficiency and vibe coding success. Vibe coding performance shows a moderate positive correlation with CS achievement and cognitive ability, both statistically significant. Writing skills correlate positively with vibe coding performance, but the association weakens after controlling for cognitive ability. The relationship between writing and vibe coding appears to be mediated by prompt quality, as the direct effect becomes non-significant when prompt quality is included.

Writing skills show a small but significant positive correlation with overall vibe coding performance, though this relationship weakens after controlling for general cognitive ability. Among the three coding tasks, the decontextualized task and feature addition task correlate similarly with writing, while the replication task shows no significant correlation. Prompt quality appears to mediate much of the association between writing proficiency and task performance. The overall correlation between writing skills and vibe coding performance is significant but small, and becomes non-significant when controlling for general cognitive skills. The decontextualized and feature addition tasks both correlate significantly with writing skills, whereas the replication task does not. Prompt quality accounts for about half of the total association between writing and vibe coding performance, suggesting it serves as an intermediate link.

Lexical diversity in prompts, measured by HD-D and MTLD, is positively and significantly correlated with essay lexical diversity, essay grades, and vibe coding performance. Human-graded prompt quality also correlates positively with essay grades and vibe coding performance, and mediation analysis suggests prompt quality may link writing skill to vibe coding outcomes. Higher prompt lexical diversity is associated with higher essay lexical diversity, essay grades, and vibe coding performance. Human-graded prompt quality correlates positively with both essay grades and vibe coding performance. Prompt quality appears to mediate the relationship between writing skill and vibe coding performance, accounting for about half of the total association.

Expert panelists selected three vibe coding tasks (exam scheduler, meal planner, course registration) based on higher and more consistent ratings across criteria like challenge and purpose clarity, while tasks such as the eye tracker app showed lower clarity and wider disagreement. Instrument scores varied widely, with writing skills averaging highest and vibe coding lowest, and reliability was acceptable for essay grading and self-report but not the cognitive test, which was treated as a covariate. Vibe coding performance correlated positively with CS achievement, cognitive ability, and writing skills, though the writing link weakened after controlling for cognitive ability and was largely mediated by prompt quality. Prompt lexical diversity and human-graded prompt quality also correlated with essay grades and vibe coding performance, supporting prompt quality as an intermediate mechanism between writing proficiency and task success.