Large Language Models in Experimental Research

Summary of the Previous Session

So far, we’ve been discussing various experimental designs intended to study human behaviors and decision-making directly. However, with the rapid advancements and widespread adoption of Large Language Models (LLMs) and Generative AI, we’re now at an exciting turning point. In this session, we will explore how these cutting-edge AI tools can significantly enhance the way we design, implement, and analyze experiments, opening new avenues for more efficient, scalable, and insightful experimental research.

Why LLMs Matter for Experiments

• LLMs can process economic scenarios, simulate participant responses, and generate experimental materials (e.g., instructions, code) with minimal human effort.

• Their scalability allows researchers to run large-scale simulations at low cost, unlike human-based experiments, which are resource-intensive.

• They address challenges in online experiments, such as participant comprehension and compliance, by providing tailored interactions.

Homo Silicus as a Computational Model

Definition: John Horton introduces Homo Silicus as an analogy to the classical Homo Economicus. While Homo Economicus assumes perfect rationality to model human behavior, Homo Silicus represents Large Language Models (LLMs) acting as computational models of humans. These models simulate behavior by drawing from vast, real-world training data—capturing not only rational choices but also heuristic-driven and sometimes irrational decisions.

Significance: Homo Silicus bridges theoretical and empirical research. Like Homo Economicus, it helps economists explore behavior in abstracted scenarios—but with a critical advantage: LLMs simulate responses that are dynamic, data-driven, and more reflective of the messy realities of human decision-making.

Relevance to Business: In business contexts, Homo Silicus enables rapid and low-cost piloting of strategies—such as pricing or incentive structures—by simulating consumer and employee responses.

Mechanism: Endowing LLMs for Simulation

How It Works: LLMs can be “endowed” with specific characteristics—like preferences or beliefs—via prompt engineering, allowing them to act as agents in economic scenarios. Unlike static models, LLMs use computational simulation, grounded in their exposure to vast textual data.

Technical Details:

• Prompt Engineering: Prompts serve as experimental treatments. For instance, “You are a low-income consumer” frames the model’s role, much like assigning participants in a behavioral lab .

• Stochastic Responses: LLMs produce varied outputs using parameters like temperature (e.g., 0.8), enabling simulations of heterogeneous agents. Repeated API calls generate a sample of responses.

• Training Data: LLMs are trained on discussions of economic behavior (e.g., bargaining, consumption), allowing them to simulate realistic decision patterns based on embedded social norms and knowledge.

Here is an example of how we can use these tools:
https://colab.research.google.com/drive/13S7eM4Gh94paBgHsQCubA_dudLoo3UcV?usp=sharing

Advantages Over Homo Economicus

• Realism: Homo Silicus simulates behaviors like fairness, loss aversion, and status quo bias—features absent in perfectly rational models.

• Flexibility: LLMs can instantly switch roles (e.g., manager, worker, customer) through prompts, enabling diverse scenario testing.

• Scalability and Cost-Efficiency: Experiments using LLMs are inexpensive and fast. Horton ran experiments costing $\sim\$50$ in minutes. A ride-sharing simulation with 5 API calls cost less than $0.10.

• Business Applications: Firms can simulate consumer reactions to dynamic pricing or test employee incentives before real-world implementation, aligning with the tech industry’s shift toward data-driven experimentation.

Limitations and Considerations

• Performativity: LLMs might echo textbook answers if prompted with familiar scenarios.

• Training Biases: Models trained on Western-centric data may not generalize globally.

• Hallucinations: LLMs can generate convincing yet inaccurate outputs, requiring human oversight .

• Prompt Dependency: Outcomes vary with prompt quality, necessitating thoughtful design .

Mitigation: Design clear prompts, compare against real data, and log interactions to ensure transparency and replicability. Also read the paer by Yang et al. (2024) [Are large language models (llms) good social predictors? arXiv preprint arXiv:2402.12620, 2024] as it seems to pour cold water on many of these expectations.

How LLMs Improve the Experiments

Enhancing Experiment Design

Research Question Formulation

LLMs assist researchers in identifying promising hypotheses by analyzing vast bodies of academic literature and economic trends. They detect gaps, suggest variables of interest, and recommend experiment types such as economic games or randomized controlled trials.

• Prompt Engineering: Prompts like “Based on recent literature on consumer pricing, suggest a hypothesis for a field experiment” elicit hypotheses grounded in prior studies and contextual knowledge.

• Trend Extraction: LLMs infer emerging topics and behavioral patterns from training data such as journal articles, online discussions, and working papers.

Treatment and Setting Selection

LLMs guide the selection of experimental settings (e.g., lab, field, online) and help ensure appropriate randomization strategies to satisfy key assumptions like SUTVA and exclusion restrictions. They also recommend sample sizes, stratification methods, and ethical considerations.

• Algorithmic Suggestions: LLMs propose sample sizes, assess statistical power and effect sizes, and optimize randomization strategies under given resource constraints.

• Simulation: Researchers can simulate treatment effects using LLMs to test robustness and fine-tune design parameters.

Example: A researcher studying employee incentives might prompt, “Suggest an experimental setting and randomization strategy for testing bonus structures,” and receive detailed, feasible recommendations.

Instruction Wording

LLMs generate clear, concise, and accessible instructions tailored to diverse participant demographics. By accounting for variations in education and language proficiency, they reduce measurement error and improve participant compliance.

• Natural Language Processing: LLMs adjust tone and complexity to match target populations, using techniques from tools like QualiGPT.

• Iterative Refinement: LLMs generate multiple instruction drafts, integrating feedback from pilot runs to improve clarity and engagement.

Example: In a ride-sharing simulation, GPT-3.5-turbo was prompted: “You are a low-income commuter. Rate a 50% ride-sharing price increase as Completely Fair, Acceptable, Unfair, or Very Unfair, and explain why.” The model produced clear instructions and comprehension checks that helped detect ambiguity.

Code and Documentation

LLMs accelerate the implementation of experiments by generating clean, usable code and standardized documentation. They support multiple platforms like Python, Qualtrics, and oTree, and can create IRB documentation and replication packages for full transparency.

• Code Generation: LLMs convert plain-language task descriptions into executable code across different platforms.

• Documentation: They produce structured replication packages, including pre-registration templates and IRB documents.

Example: A researcher prompts, “Generate Python code for a Qualtrics survey on consumer pricing preferences.” GPT-4 generates both the logic and documentation, reducing human error and promoting reproducibility.

Enhancing the Implementation of Experiments

Conversational Support

AI chatbots enhance online experiments by providing real-time, personalized support. They clarify instructions and reduce confusion, improving participant comprehension and compliance—addressing common challenges such as measurement errors and dropout.

• Conversational AI: LLMs use natural language processing to simulate human-like interactions and tailor responses to individual participant profiles (e.g., language proficiency).

• API Integration: Real-time API calls allow chatbots to handle large volumes of simultaneous queries, making them scalable for large experiments.

Example: In an online survey on employee incentives, a participant asks, “What does ‘performance-based bonus’ mean?” The AI chatbot responds: “It’s a bonus tied to your task performance, like meeting sales targets.” This ensures clarity and reduces attrition.

Simulation of Social Interactions

LLMs can simulate realistic social dynamics—such as bargaining, group decision-making, or negotiations—thereby enhancing the ecological validity of experimental designs.

• Role-Playing: LLMs adopt specific roles (e.g., buyer, seller) via prompts and generate contextually grounded dialogue based on behavioral patterns in their training data.

• Multi-Agent Simulation: Multiple API calls simulate interactions among distinct agents, enabling the modeling of strategic behavior and social preferences.

Example: In a dictator game simulation, GPT-3 agents endowed with different preferences (e.g., fairness, efficiency) produced allocation choices that mirrored real human behavior, enabling a controlled study of social norms.

Engagement and Monitoring

AI tools monitor participant engagement in real-time, reduce attrition, and detect non-compliant behaviors (e.g., cheating). Interactive prompts keep participants focused and ensure high-quality data.

• Behavioral Tracking: LLMs integrate with browser-based plugins to detect tab-switching or inactivity and flag potential disengagement.

• Engagement Prompts: Dynamic reminders such as “Please focus on the task” help re-engage participants and maintain data quality.

Example: In an online study, the system detects repeated tab-switching. The AI prompts: “Please stay on the survey page to ensure accurate responses.” This intervention helps reduce noise and preserve data integrity.

Here is an example of how we can use AI tools to simulate a negotiation:
https://colab.research.google.com/drive/13S7eM4Gh94paBgHsQCubA_dudLoo3UcV?usp=sharing

Required readings