Agent-Guided Formula Construction Documentation

suggest_formula_for_design(
    pseudobulk_modality="my_data_pseudobulk",
    analysis_goal="Compare treatment vs control",  # Optional
    show_metadata_summary=True
)

construct_de_formula_interactive(
    pseudobulk_modality="my_data_pseudobulk",
    main_variable="condition",
    covariates=["batch", "age"],  # Optional
    include_interactions=False,
    validate_design=True
)

run_differential_expression_with_formula(
    pseudobulk_modality="my_data_pseudobulk",
    formula=None,  # Uses stored formula if None
    contrast=["condition", "treatment", "control"],
    alpha=0.05,
    lfc_threshold=0.0
)

iterate_de_analysis(
    pseudobulk_modality="my_data_pseudobulk",
    new_formula="~condition + batch + gender",  # Try adding gender
    compare_to_previous=True,
    iteration_name="with_gender_covariate"
)

compare_de_iterations(
    pseudobulk_modality="my_data_pseudobulk",
    iteration_1="simple_model",
    iteration_2="batch_corrected",
    show_overlap=True,
    show_unique=True
)

User: "I need to run differential expression on my pseudobulk data"

Agent: [Uses suggest_formula_for_design automatically]

Agent Response:
"📊 **Formula Design Analysis for 'pbmc_pseudobulk'**

**Metadata Summary:**
• Samples: 24
• Variables analyzed: 6
• Categorical variables: 3
• Continuous variables: 2

**Key Variables:**
• **condition**: categorical, 2 levels (treatment, control)
• **batch**: categorical, 3 levels (batch1, batch2, batch3)
• **gender**: categorical, 2 levels (M, F)
• **age**: continuous variable

📝 **Recommended Formula Options:**

**1. Simple Model** *(recommended for Initial exploratory analysis)*
   Formula: `~condition`
   Description: Compare condition groups directly
   ✅ Pros: Maximum statistical power, Straightforward interpretation
   ⚠️ Cons: Ignores potential confounders, May miss batch effects
   Min samples needed: 6

**2. Batch-corrected Model** *(recommended for Multi-batch experiments)*
   Formula: `~condition + batch`
   Description: Compare condition while accounting for batch effects
   ✅ Pros: Controls for technical/batch variation, More reliable effect estimates
   ⚠️ Cons: Reduces degrees of freedom, Requires balanced design
   Min samples needed: 8

**3. Multi-factor Model** *(recommended for Final analysis with adequate samples)*
   Formula: `~condition + batch + gender + age`
   Description: Comprehensive model accounting for condition and 3 covariates
   ✅ Pros: Controls for multiple confounders, Publication-ready analysis
   ⚠️ Cons: Requires larger sample size, More complex interpretation
   Min samples needed: 15

💡 **Recommendation**: Start with the simple model for exploration, then use the batch-corrected model if you see batch effects.

**Next step**: Use `construct_de_formula_interactive` to build and validate your chosen formula."

User: "Let's start with the batch-corrected model"

Agent: [Uses construct_de_formula_interactive]

User: "The results look good, but I want to see what happens if I add gender as a covariate"

Agent: [Uses iterate_de_analysis]

Agent Response:
"🔄 **DE Analysis Iteration 'with_gender' Complete**

**Formula**: `~condition + batch + gender`
**Contrast**: treatment vs control (in condition)

📊 **Current Results**:
• Significant genes: 387
• Upregulated: 198
• Downregulated: 189

🔄 **Comparison with Previous Iteration**:
• Overlapping significant genes: 298
• New in current: 89
• Lost from previous: 44
• Fold change correlation: 0.924

📈 **Iteration Summary**:
• Total iterations: 2
• Current iteration: 2

💡 **Interpretation**: High correlation (0.924) suggests similar biological effects, but adding gender reveals 89 additional significant genes.

**Next steps**: Use `compare_de_iterations` to compare all iterations or continue iterating with different parameters."

User: "Show me a detailed comparison between the two models"

Agent: [Uses compare_de_iterations]

SingleCell Expert Agent
├── Formula Construction Tools (5 new tools)
├── Enhanced Formula Service (3 new methods)
├── Workflow Tracker (lightweight iteration tracking)
└── Integration with existing pseudobulk pipeline

# Formula construction data
adata.uns['constructed_formula'] = {
    'formula': '~condition + batch',
    'main_variable': 'condition',
    'covariates': ['batch'],
    'formula_components': {...},
    'design_result': {...}
}

# DE analysis results
adata.uns['de_results_formula_condition_treatment_vs_control'] = {
    'results_df': results_dataframe,
    'analysis_stats': {...},
    'parameters': {...}
}

# Iteration tracking
adata.uns['de_iterations'] = {
    'iterations': [...],
    'current_iteration': 2
}

User: "I have pseudobulk data and need to find differentially expressed genes"

Agent: "I'll help you set up a proper statistical analysis. Let me first examine your pseudobulk data structure."

[Agent uses suggest_formula_for_design]

Agent: "I've analyzed your data and found you have 36 samples with these key variables:
• condition: disease (n=18) vs healthy (n=18) - Your main comparison
• batch: 4 different batches - Important for controlling technical variation
• age: continuous variable (22-67 years) - Potential biological confounder

I recommend starting with a batch-corrected model: `~condition + batch`
This will compare disease vs healthy while accounting for batch effects.

Should I construct and validate this formula for you?"

User: "Yes, that sounds good"

[Agent uses construct_de_formula_interactive]

Agent: "Formula constructed successfully! The design looks statistically sound with 36 samples and 5 coefficients, giving you good power to detect medium to large effects.

Ready to run the analysis?"

User: "The analysis found 342 significant genes. Can we try adding age as a covariate?"

Agent: "Great idea! Adding age can help control for age-related expression changes. Let me run this as iteration 2 and compare with your previous results."

[Agent uses iterate_de_analysis]

Agent: "Iteration 2 complete with formula: `~condition + batch + age`

Results: 387 significant genes (vs 342 previously)
• 298 genes overlap with previous analysis (87% concordance)
• 89 new genes became significant
• 44 genes lost significance

The high overlap suggests consistent core biology, while the new genes likely represent age-sensitive disease effects that were previously masked.

Age appears to explain meaningful variance - I recommend keeping this model."

User: "I want to compare the simple model vs the full model to see which is better"

Agent: [Uses compare_de_iterations automatically]

Agent: "📊 **Model Comparison: Simple vs Full**

**Simple Model** (~condition):
• 456 significant genes
• Higher statistical power
• Clear biological interpretation

**Full Model** (~condition + batch + age + gender):
• 298 significant genes
• More conservative results
• Controls for multiple confounders

**Overlap Analysis:**
• 267 genes significant in both (89% of full model genes)
• 31 genes unique to full model (high-confidence hits)
• 189 genes only in simple model (may include false positives)

**Recommendation**: The full model is more conservative and likely more accurate for publication. The 31 genes unique to the full model represent high-confidence discoveries after controlling for confounders."

Agent: "❌ **Formula Construction Failed**

**Formula**: `~condition + nonexistent_variable`
**Error**: Variables not found in metadata: ['nonexistent_variable']

💡 **Suggestions**:
• Check variable names are spelled correctly
• Available variables: condition, batch, age, gender, cell_type
• Try: `construct_de_formula_interactive('my_data', 'condition', ['batch'])`"

# 1. Get formula suggestions
agent.suggest_formula_for_design("pbmc_pseudobulk")

# 2. Build chosen formula interactively
agent.construct_de_formula_interactive(
    pseudobulk_modality="pbmc_pseudobulk",
    main_variable="condition",
    covariates=["batch"]
)

# 3. Run analysis
agent.run_differential_expression_with_formula("pbmc_pseudobulk")

# 4. Try alternative model
agent.iterate_de_analysis(
    pseudobulk_modality="pbmc_pseudobulk",
    new_formula="~condition + batch + age",
    iteration_name="with_age_correction"
)

# 5. Compare models
agent.compare_de_iterations("pbmc_pseudobulk", "iteration_1", "with_age_correction")

# Custom reference levels
agent.construct_de_formula_interactive(
    pseudobulk_modality="pbmc_pseudobulk",
    main_variable="condition",
    covariates=["batch", "age"],
    include_interactions=True  # Include condition*batch interaction
)

# Run with specific parameters
agent.run_differential_expression_with_formula(
    pseudobulk_modality="pbmc_pseudobulk",
    reference_levels={"condition": "healthy"},  # Set healthy as reference
    alpha=0.01,  # More stringent significance
    lfc_threshold=0.5  # Require |LFC| >= 0.5
)

"📈 **Design Matrix Preview**:
This shows how your samples are encoded for statistical analysis:
• Each row = one sample
• Each column = one statistical parameter
• Values of 1 mean 'this parameter applies to this sample'
• Values of 0 mean 'this parameter doesn't apply'

For example, 'condition[T.treatment]' = 1 for treatment samples, 0 for controls."

lobster/
├── agents/
│   └── transcriptomics_expert.py     # Enhanced with 5 new tools
├── tools/
│   ├── differential_formula_service.py  # Enhanced with 3 helper methods
│   └── workflow_tracker.py             # New lightweight tracker
└── tests/
    └── integration/
        └── test_agent_guided_formula_construction.py  # Integration tests