47. Microbiome Metadata Harmonization Workflow

# In Zotero:
# 1. Select your collection of 10 publications
# 2. File → Export Library → Format: RIS
# 3. Save as: ibd_microbiome_studies.ris

TY  - JOUR
TI  - Gut microbiome alterations in colorectal cancer patients
AU  - Smith, John
AU  - Jones, Alice
PY  - 2023
JO  - Nature Microbiome
AB  - We performed 16S rRNA sequencing on fecal samples...
PMID- 37123456
KW  - microbiome
KW  - 16S rRNA
KW  - colorectal cancer
ER  -

TY  - JOUR
TI  - Fecal microbiota composition in ulcerative colitis
AU  - Brown, Charlie
PY  - 2022
JO  - Cell Host & Microbe
AB  - Illumina MiSeq 16S sequencing revealed...
PMID- 36789012
KW  - microbiome
KW  - ulcerative colitis
ER  -

lobster chat
> /queue load ibd_microbiome_studies.ris

✅ Imported 10 publications to queue
   - 7 microbiome studies detected
   - 2 proteomics studies detected
   - 1 general study detected

> Process all microbiome publications from the queue and extract SRA identifiers

✅ Processed 7 microbiome publications
   - Total SRA runs identified: 142
   - Metadata files cached: 142
   - workspace_metadata_keys populated

> Filter all publications for human fecal 16S samples and standardize disease terms

📊 Filtering Results:
   - Original samples: 142
   - After 16S validation: 138 (97.2% retention)
   - After host validation: 136 (95.8% retention)
   - After sample type filter: 89 (62.7% retention - fecal only)
   - After disease standardization: 89 (100% mapping success)

   Disease Distribution:
   - CRC: 34 samples (38.2%)
   - UC: 22 samples (24.7%)
   - CD: 18 samples (20.2%)
   - Healthy: 15 samples (16.9%)

> Export harmonized metadata to CSV

✅ Harmonized dataset exported:
   - File: workspace/metadata/exports/harmonized_ibd_microbiome_2024-11-19.csv
   - Total samples: 89
   - Columns: 34 (28 SRA metadata + 6 harmonized fields - schema-driven)
   - Provenance: Full W3C-PROV tracking available via /pipeline export

from lobster.core.schemas.publication_queue import PublicationQueueEntry

entry = PublicationQueueEntry(
    entry_id="pub_queue_37123456_abc123",
    pmid="37123456",
    title="Gut microbiome alterations in colorectal cancer patients",
    schema_type="microbiome",  # Auto-inferred from keywords

    # Multi-agent handoff fields
    workspace_metadata_keys=[
        "SRR12345678_metadata.json",
        "SRR12345679_metadata.json",
        "SRR12345680_metadata.json"
    ],
    harmonization_metadata={
        "samples": [
            {"run_accession": "SRR12345678", "organism": "Homo sapiens", "sample_type": "fecal", "disease": "crc"},
            {"run_accession": "SRR12345679", "organism": "Homo sapiens", "sample_type": "fecal", "disease": "uc"}
        ],
        "filtering_stats": {
            "original_samples": 142,
            "filtered_samples": 89,
            "retention_rate": 62.7
        },
        "validation_status": "passed"
    },

    extracted_identifiers={
        "sra": ["SRR12345678", "SRR12345679", "SRR12345680"],
        "bioproject": ["PRJNA720345"],
        "biosample": ["SAMN12345678", "SAMN12345679", "SAMN12345680"]
    }
)

# Get full paths to metadata files
paths = entry.get_workspace_metadata_paths(workspace_dir="/path/to/workspace")
# Returns: [
#   '/path/to/workspace/metadata/SRR12345678_metadata.json',
#   '/path/to/workspace/metadata/SRR12345679_metadata.json',
#   '/path/to/workspace/metadata/SRR12345680_metadata.json'
# ]

def validate_16s_amplicon(
    metadata: Dict[str, Any],
    strict: bool = False
) -> Tuple[Dict[str, Any], Dict[str, Any], AnalysisStep]:
    """
    Validate if metadata represents a 16S amplicon study.

    Args:
        metadata: Metadata dictionary to validate
        strict: If True, require exact keyword matches (default: False)

    Returns:
        Tuple of (filtered_metadata, stats, ir)
    """

from lobster_metadata.services.metadata.microbiome_filtering_service import MicrobiomeFilteringService

service = MicrobiomeFilteringService()

# Non-strict matching (default)
metadata = {
    "platform": "Illumina MiSeq",
    "library_strategy": "AMPLICON",
    "organism": "Homo sapiens"
}

filtered, stats, ir = service.validate_16s_amplicon(metadata, strict=False)

print(stats)
# {
#   'is_valid': True,
#   'reason': '16S amplicon detected in library_strategy',
#   'matched_field': 'library_strategy',
#   'matched_value': 'AMPLICON',
#   'strict_mode': False,
#   'fields_checked': ['platform', 'library_strategy', 'assay_type']
# }

def validate_host_organism(
    metadata: Dict[str, Any],
    allowed_hosts: Optional[List[str]] = None,
    fuzzy_threshold: float = 85.0
) -> Tuple[Dict[str, Any], Dict[str, Any], AnalysisStep]:
    """
    Validate host organism using fuzzy string matching.

    Args:
        metadata: Metadata dictionary containing host organism info
        allowed_hosts: List of allowed host names (default: ["Human", "Mouse"])
        fuzzy_threshold: Minimum fuzzy match score (0-100, default: 85.0)

    Returns:
        Tuple of (filtered_metadata, stats, ir)
    """

# Example 1: Standard host validation
metadata = {"organism": "Homo sapiens"}
filtered, stats, ir = service.validate_host_organism(metadata)

print(stats)
# {
#   'is_valid': True,
#   'reason': 'Host organism matched: Human',
#   'matched_host': 'Human',
#   'confidence_score': 100.0,
#   'allowed_hosts': ['Human', 'Mouse'],
#   'fuzzy_threshold': 85.0
# }

# Example 2: Fuzzy matching with typo
metadata = {"organism": "homo sapeins"}  # Typo: 'sapeins'
filtered, stats, ir = service.validate_host_organism(metadata, fuzzy_threshold=80.0)

print(stats)
# {
#   'is_valid': True,
#   'reason': 'Host organism matched: Human',
#   'matched_host': 'Human',
#   'confidence_score': 92.3,  # High confidence despite typo
#   'allowed_hosts': ['Human', 'Mouse'],
#   'fuzzy_threshold': 80.0
# }

# Example 3: Rejection due to low threshold
metadata = {"organism": "Rattus norvegicus"}  # Rat (not in allowed_hosts)
filtered, stats, ir = service.validate_host_organism(metadata)

print(stats)
# {
#   'is_valid': False,
#   'reason': 'Host organism not in allowed list: [\'Human\', \'Mouse\']',
#   'matched_host': None,
#   'confidence_score': None,
#   'allowed_hosts': ['Human', 'Mouse'],
#   'fuzzy_threshold': 85.0
# }

def standardize_disease_terms(
    metadata: pd.DataFrame,
    disease_column: str = "disease"
) -> Tuple[pd.DataFrame, Dict[str, Any], AnalysisStep]:
    """
    Standardize disease terminology in metadata.

    Args:
        metadata: DataFrame with disease information
        disease_column: Column name containing disease labels

    Returns:
        Tuple of (standardized_metadata, statistics, provenance_ir)
    """

# Level 1: Exact match
"colorectal cancer" → "crc" (exact match in variant list)

# Level 2: Value contains variant
"stage III colorectal cancer" → "crc" (contains "colorectal cancer")

# Level 3: Variant contains value (reverse)
"tumor" → "crc" (variant "tumor" matched)

# Level 4: Token-based matching
"crc adenocarcinoma patient" → "crc" (tokens "crc", "adenocarcinoma" match)

# Level 5: No match found
"diabetes" → "diabetes" (unmapped, returns original value)

from lobster_metadata.services.metadata.disease_standardization_service import DiseaseStandardizationService
import pandas as pd

service = DiseaseStandardizationService()

# Original metadata with messy disease labels
metadata = pd.DataFrame({
    'sample_id': ['S1', 'S2', 'S3', 'S4', 'S5'],
    'disease': ['colorectal cancer', 'UC', 'Crohn disease', 'healthy control', 'adenocarcinoma']
})

standardized, stats, ir = service.standardize_disease_terms(metadata, disease_column='disease')

print(standardized[['sample_id', 'disease', 'disease_original']])
#   sample_id  disease         disease_original
# 0        S1      crc  colorectal cancer
# 1        S2       uc                 UC
# 2        S3       cd       Crohn disease
# 3        S4  healthy      healthy control
# 4        S5      crc       adenocarcinoma

print(stats)
# {
#   'total_samples': 5,
#   'standardization_rate': 100.0,  # All samples mapped
#   'mapping_stats': {
#       'exact_matches': 1,          # "UC" → "uc"
#       'contains_matches': 2,       # "colorectal cancer", "Crohn disease"
#       'reverse_contains_matches': 0,
#       'token_matches': 2,          # "healthy control", "adenocarcinoma"
#       'unmapped': 0
#   },
#   'unique_mappings': {
#       'colorectal cancer': 'crc',
#       'UC': 'uc',
#       'Crohn disease': 'cd',
#       'healthy control': 'healthy',
#       'adenocarcinoma': 'crc'
#   },
#   'disease_distribution': {
#       'crc': 2,
#       'uc': 1,
#       'cd': 1,
#       'healthy': 1
#   },
#   'original_column': 'disease_original'
# }

process_metadata_queue(filter_criteria="16S human fecal_stool CRC")
→ ❌ Insufficient disease data: 15% coverage < 50% threshold
→ User stuck with no clear solution

process_metadata_queue(...) → Validation fails
→ metadata_assistant asks supervisor for permission
→ enrich_samples_with_disease (4-phase hierarchy)
→ 15% → 90% coverage improvement
→ Re-validation passes → workflow continues

@tool
def enrich_samples_with_disease(
    workspace_key: str,
    enrichment_mode: str = "hybrid",
    manual_mappings: Optional[str] = None,
    confidence_threshold: float = 0.8,
    auto_retry_validation: bool = True,
    dry_run: bool = False
) -> str:
    """
    Enrich samples with missing disease annotation using 4-phase hierarchy.

    Args:
        workspace_key: Metadata key (e.g., "aggregated_filtered_samples")
        enrichment_mode: "column_scan", "llm_auto", "manual", "hybrid" (default)
        manual_mappings: JSON string {"pub_id": "disease"}
        confidence_threshold: Min LLM confidence (0.0-1.0), default 0.8
        auto_retry_validation: Re-validate after enrichment (default True)
        dry_run: Preview without saving (default False)
    """

# After validation fails
enrich_samples_with_disease(
    workspace_key="aggregated_filtered_samples",
    enrichment_mode="hybrid",
    confidence_threshold=0.8
)

# Output:
## Disease Enrichment Report
**Total Samples**: 89
**Initial Coverage**: 15.0% (13/89)
**Final Coverage**: 95.5% (85/89)
**Improvement**: +80.5% (+72 samples)

### Phase Results
Phase 1 (Column): +5 samples
Phase 2 (Abstract): +67 samples
Phase 3 (Methods): Skipped (coverage ≥50%)
Phase 4 (Manual): 0 samples

### Re-validation
✅ PASSED: 95.5% ≥ 50% threshold

enrich_samples_with_disease(
    workspace_key="aggregated_filtered_samples",
    enrichment_mode="manual",
    manual_mappings='{"pub_queue_pmid_12345": "crc", "pub_queue_doi_10_5678": "uc"}'
)

enrich_samples_with_disease(
    workspace_key="aggregated_filtered_samples",
    enrichment_mode="hybrid",
    dry_run=True  # Preview only, no changes saved
)

1. process_metadata_queue() fails validation (<50% coverage)
2. metadata_assistant → supervisor: "Can I enrich disease data?"
3. Supervisor → user: "metadata_assistant suggests enrichment. Proceed?"
4. User: "Yes"
5. Supervisor → metadata_assistant: "Proceed with enrichment"
6. enrich_samples_with_disease(enrichment_mode="hybrid")
7. Re-validation → ≥50% → workflow continues

❌ Disease validation failed: 15.0% coverage < 50% threshold

I can attempt automatic enrichment.

**Estimated improvement**: +30-60% coverage (10 cached publications)
**Cost**: ~$0.03 (LLM calls)
**Time**: ~20s

Should I proceed?

def filter_by_sample_type(
    metadata: pd.DataFrame,
    sample_types: List[str],
    sample_columns: Optional[List[str]] = None
) -> Tuple[pd.DataFrame, Dict[str, Any], AnalysisStep]:
    """
    Filter samples by sample type (fecal vs gut tissue).

    Args:
        metadata: DataFrame with sample information
        sample_types: List of sample types to keep (e.g., ["fecal"])
        sample_columns: Columns to check for sample type info
                       (if None, checks common columns)

    Returns:
        Tuple of (filtered_metadata, statistics, provenance_ir)
    """

# Original metadata with mixed sample types
metadata = pd.DataFrame({
    'sample_id': ['S1', 'S2', 'S3', 'S4', 'S5', 'S6'],
    'disease': ['crc', 'uc', 'cd', 'healthy', 'crc', 'uc'],
    'sample_type': ['fecal', 'fecal', 'biopsy', 'fecal', 'tissue', 'stool']
})

# Filter for fecal samples only
filtered, stats, ir = service.filter_by_sample_type(
    metadata,
    sample_types=["fecal"]
)

print(filtered[['sample_id', 'disease', 'sample_type']])
#   sample_id disease sample_type
# 0        S1     crc       fecal
# 1        S2      uc       fecal
# 3        S4 healthy       fecal
# 5        S6      uc       stool

print(stats)
# {
#   'original_samples': 6,
#   'filtered_samples': 4,
#   'retention_rate': 66.67,
#   'sample_types_requested': ['fecal'],
#   'columns_checked': ['sample_type'],
#   'matches_by_column': {'sample_type': 4}
# }

protocol_extraction/
├── __init__.py              # Factory: get_protocol_service(domain)
├── base.py                  # IProtocolExtractionService ABC, BaseProtocolDetails
├── amplicon/                # 16S/ITS microbiome (FULLY IMPLEMENTED)
│   ├── __init__.py
│   ├── details.py           # AmpliconProtocolDetails dataclass
│   ├── service.py           # AmpliconProtocolService
│   └── resources/
│       └── primers.json     # 15 primers with sequences, sources
├── mass_spec/               # Proteomics (STUB - NotImplementedError)
│   └── ...
└── rnaseq/                  # Transcriptomics (STUB - NotImplementedError)
    └── ...

from lobster.services.metadata.protocol_extraction import get_protocol_service

# Get amplicon service (supports domain aliases)
service = get_protocol_service("amplicon")  # Primary
service = get_protocol_service("16s")       # Alias
service = get_protocol_service("its")       # Alias
service = get_protocol_service("metagenomics")  # Alias

# Extract protocol from methods text
text = """
The V3-V4 region was amplified using primers 515F and 806R.
PCR for 30 cycles at 55°C. Sequenced on MiSeq (2×250 bp).
Processed with QIIME2/DADA2 using SILVA v138.
"""
details, result = service.extract_protocol(text, source="pmc")

print(details.v_region)           # "V3-V4"
print(details.forward_primer)     # "515F"
print(details.reverse_primer)     # "806R"
print(details.pcr_cycles)         # 30
print(details.platform)           # "Illumina MiSeq"
print(details.pipeline)           # "DADA2"
print(details.reference_database) # "SILVA"
print(details.confidence)         # 1.0 (100%)

schema_dict = details.to_schema_dict()
# {
#   'forward_primer_name': '515F',
#   'forward_primer_seq': 'GTGCCAGCMGCCGCGGTAA',
#   'reverse_primer_name': '806R',
#   'amplicon_region': 'V3-V4',
#   'sequencing_platform': 'Illumina MiSeq',
#   'primer_set': '515F-806R'
# }

uns_dict = details.to_uns_dict()
# {
#   'forward_primer': '515F',
#   'forward_primer_sequence': 'GTGCCAGCMGCCGCGGTAA',
#   'target_region': 'V3-V4',
#   'amplicon_target': '16S rRNA V3-V4',
#   'taxonomy_database': 'SILVA',
#   'pipeline': 'DADA2',
#   'pcr_conditions': {
#       'annealing_temperature_celsius': 55.0,
#       'pcr_cycles': 30
#   },
#   'extraction_confidence': 0.83
# }

import anndata as ad
import numpy as np

# Create sample AnnData
adata = ad.AnnData(np.random.rand(10, 5))

# Store protocol details
adata = service.store_in_adata(adata, details, store_in_obs=True)

# Access stored data
print(adata.uns["protocol_details"]["forward_primer"])  # '515F'
print(adata.obs["amplicon_region"].iloc[0])             # 'V3-V4'

from lobster.tools.providers.pmc_provider import PMCProvider

provider = PMCProvider()
full_text = provider.extract_full_text("PMID:35042229")

# Parameters auto-extracted from methods section
print(full_text.parameters)
# {
#   'v_region': 'V3-V4',
#   'forward_primer': '515F',
#   'forward_primer_sequence': 'GTGCCAGCMGCCGCGGTAA',
#   'sequencing_platform': 'Illumina MiSeq',
#   'bioinformatics_pipeline': 'DADA2',
#   'reference_database': 'SILVA v138'
# }

# In protocol_extraction/__init__.py
if domain_lower in ("new_domain", "alias1", "alias2"):
    from .new_domain.service import NewDomainProtocolService
    service = NewDomainProtocolService()
    _service_cache[domain_lower] = service
    return service

@tool
def filter_samples_by(
    workspace_key: str,
    filter_criteria: str,
    strict: bool = True
) -> str:
    """
    Filter samples by multi-modal criteria (16S amplicon + host organism +
    sample type + disease).

    Args:
        workspace_key: Key for workspace metadata (e.g., "geo_gse123456")
        filter_criteria: Natural language criteria (e.g., "16S human fecal CRC")
        strict: Use strict matching for 16S detection (default: True)

    Returns:
        Formatted markdown report with filtering results
    """

# In lobster chat:
> Filter the workspace metadata 'geo_gse123456' for human fecal 16S samples with CRC, UC, CD, or healthy controls

filter_samples_by(
    workspace_key="geo_gse123456",
    filter_criteria="16S human fecal CRC UC CD healthy",
    strict=True
)

# 📊 Microbiome Filtering Results

## Criteria Parsed:
- ✅ 16S Amplicon: Enabled (strict mode)
- ✅ Host Organism: ["Human"]
- ✅ Sample Type: ["fecal"]
- ✅ Disease Standardization: Enabled

## Sequential Filtering:

### Step 1: 16S Amplicon Validation
- Original samples: 142
- Valid 16S samples: 138
- Retention: 97.2%
- Matched fields: library_strategy (135), platform (3)

### Step 2: Host Organism Validation
- Input samples: 138
- Valid host samples: 136
- Retention: 98.6%
- Matched host: Human (avg confidence: 94.8%)

### Step 3: Sample Type Filter
- Input samples: 136
- Matching sample types: 89
- Retention: 65.4%
- Matched columns: sample_type (72), body_site (17)

### Step 4: Disease Standardization
- Input samples: 89
- Standardized samples: 89
- Standardization rate: 100.0%
- Disease distribution:
  - crc: 34 (38.2%)
  - uc: 22 (24.7%)
  - cd: 18 (20.2%)
  - healthy: 15 (16.9%)

## Final Results:
- **Original Samples**: 142
- **Filtered Samples**: 89
- **Overall Retention Rate**: 62.7%

## Next Steps:
Use harmonization_metadata field to export filtered dataset to CSV.

# Step 1: Load publications
lobster chat
> /load ibd_studies.ris
# ✅ Imported 5 publications (5 microbiome studies)

# Step 2: Process publications (research_agent)
> Process all microbiome publications and extract SRA metadata
# ✅ Processed 5 publications, 87 SRA runs identified

# Step 3: Filter samples (metadata_assistant)
> Filter all publications for human fecal 16S samples
# ✅ Filtered 87 → 56 samples (64.4% retention)

# Step 4: Standardize diseases (metadata_assistant)
> Standardize disease terms for CRC, UC, CD, and healthy controls
# ✅ Standardized 56 samples (100% mapping rate)

# Step 5: Export CSV
> Export harmonized metadata to CSV
# ✅ Exported to workspace/metadata/exports/harmonized_ibd_2024-11-19.csv

# Step 1: Load publications
> /load mouse_microbiome.ris
# ✅ Imported 3 publications

# Step 2: Process and filter for mouse fecal
> Process publications and filter for mouse fecal 16S samples
# ✅ Filtered 45 → 28 fecal samples

# Step 3: Process and filter for mouse gut tissue
> Filter for mouse gut tissue samples instead
# ✅ Filtered 45 → 17 gut tissue samples

# Step 4: Export both datasets for comparison
> Export fecal dataset to CSV as mouse_fecal (auto-timestamp appended)
> Export gut tissue dataset to CSV as mouse_gut_tissue (auto-timestamp appended)

# Step 1: Load large RIS file
> /load crc_meta_analysis.ris
# ✅ Imported 10 publications (10 microbiome studies)

# Step 2: Process all publications
> Process all publications, extract SRA metadata, and cache to workspace
# ✅ Processed 10 publications, 234 SRA runs identified

# Step 3: Apply strict filters for consistency
> Filter for human fecal 16S samples only, require CRC or healthy controls
# ✅ Filtered 234 → 156 samples (66.7% retention)
#    - CRC: 89 samples
#    - Healthy: 67 samples

# Step 4: Validate harmonization quality
> Check metadata completeness for age, sex, BMI, sequencing depth
# ✅ Validation passed:
#    - age: 94% complete
#    - sex: 98% complete
#    - bmi: 67% complete
#    - sequencing_depth: 100% complete

# Step 5: Export for statistical analysis
> Export harmonized dataset with provenance tracking
# ✅ Exported: workspace/metadata/exports/crc_meta_analysis_2024-11-19.csv
#    Provenance: workspace/metadata/exports/crc_meta_analysis_2024-11-19_provenance.json

from lobster.core.data_manager_v2 import DataManagerV2
from lobster.core.schemas.publication_queue import PublicationStatus

data_manager = DataManagerV2(workspace_path="./workspace")
queue = data_manager.publication_queue

# Get entry by ID
entry = queue.get_entry("pub_queue_37123456_abc123")

# Access workspace metadata keys
print(entry.workspace_metadata_keys)
# ['SRR12345678_metadata.json', 'SRR12345679_metadata.json']

# Get full paths to metadata files
paths = entry.get_workspace_metadata_paths(workspace_dir="./workspace")
# ['/workspace/metadata/SRR12345678_metadata.json', '/workspace/metadata/SRR12345679_metadata.json']

# Update harmonization metadata (metadata_assistant)
queue.update_status(
    entry_id="pub_queue_37123456_abc123",
    status=PublicationStatus.COMPLETED,
    harmonization_metadata={
        "samples": [...],
        "filtering_stats": {...}
    }
)

# Read harmonization metadata (research_agent)
entry = queue.get_entry("pub_queue_37123456_abc123")
harmonized_data = entry.harmonization_metadata

from lobster_metadata.services.metadata.microbiome_filtering_service import MicrobiomeFilteringService

service = MicrobiomeFilteringService()

# 16S amplicon validation
metadata = {"platform": "Illumina MiSeq", "library_strategy": "AMPLICON"}
filtered, stats, ir = service.validate_16s_amplicon(metadata, strict=False)

# Host organism validation
metadata = {"organism": "Homo sapiens"}
filtered, stats, ir = service.validate_host_organism(
    metadata,
    allowed_hosts=["Human", "Mouse"],
    fuzzy_threshold=85.0
)

from lobster_metadata.services.metadata.disease_standardization_service import DiseaseStandardizationService
import pandas as pd

service = DiseaseStandardizationService()

# Disease standardization
metadata = pd.DataFrame({
    'sample_id': ['S1', 'S2'],
    'disease': ['colorectal cancer', 'UC']
})
standardized, stats, ir = service.standardize_disease_terms(metadata)

# Sample type filtering
filtered, stats, ir = service.filter_by_sample_type(
    metadata,
    sample_types=["fecal"],
    sample_columns=["sample_type", "body_site"]
)

from lobster.core.schemas.sra import validate_sra_sample, validate_sra_samples_batch
from lobster.core.interfaces.validator import ValidationResult

# Validate single sample
sample = {"run_accession": "SRR001", ...}
result = validate_sra_sample(sample)

if result.is_valid:
    print(f"Valid: {result.summary()}")
else:
    print(f"Errors: {result.format_messages()}")

# Batch validation (aggregates results)
samples = [sample1, sample2, sample3]
batch_result = validate_sra_samples_batch(samples)

print(f"Validation rate: {batch_result.metadata['validation_rate']:.1f}%")
print(f"Valid samples: {batch_result.metadata['valid_samples']}/{batch_result.metadata['total_samples']}")

run_accession: str          # SRR/ERR/DRR accession
experiment_accession: str   # SRX/ERX accession
sample_accession: str       # SRS/ERS accession
study_accession: str        # SRP/ERP accession
bioproject: str             # PRJNA/PRJEB/PRJDB accession
biosample: str              # SAMN accession
library_strategy: str       # AMPLICON, RNA-Seq, WGS, etc.
library_source: str         # METAGENOMIC, TRANSCRIPTOMIC, etc.
library_selection: str      # PCR, cDNA, RANDOM, etc.
library_layout: str         # SINGLE or PAIRED
organism_name: str          # Scientific name
organism_taxid: str         # NCBI Taxonomy ID
instrument: str             # Sequencing instrument
instrument_model: str       # Instrument model

public_url: Optional[str]   # NCBI public URL
ncbi_url: Optional[str]     # NCBI direct URL
aws_url: Optional[str]      # AWS S3 URL
gcp_url: Optional[str]      # GCP URL

# Valid sample
sample = {
    "run_accession": "SRR21960766",
    "experiment_accession": "SRX17944370",
    "sample_accession": "SRS15461891",
    "study_accession": "SRP403291",
    "bioproject": "PRJNA891765",
    "biosample": "SAMN31357800",
    "library_strategy": "AMPLICON",
    "library_source": "METAGENOMIC",
    "library_selection": "PCR",
    "library_layout": "PAIRED",
    "organism_name": "human metagenome",
    "organism_taxid": "646099",
    "instrument": "Illumina MiSeq",
    "instrument_model": "Illumina MiSeq",
    "public_url": "https://sra-downloadb.be-md.ncbi.nlm.nih.gov/...",
    "env_medium": "Stool"
}

result = validate_sra_sample(sample)
# result.is_valid == True
# result.warnings == []  # Has env_medium
# result.info == ["Sample SRR21960766 validated successfully..."]

# In metadata_assistant tools
samples, validation_result = _extract_samples_from_workspace(ws_data)

# Only valid samples returned
for sample in samples:
    assert sample["run_accession"]  # Guaranteed present

# Validation stats available
print(f"Extracted: {validation_result.metadata['total_samples']}")
print(f"Valid: {validation_result.metadata['valid_samples']}")
print(f"Rate: {validation_result.metadata['validation_rate']:.1f}%")

## Entry Processed: pub_queue_37249910_test
**Samples Extracted**: 247
**Samples Valid**: 247
**Samples After Filter**: 104
**Retention**: 42.1%
**Filter**: 16S human fecal
**Validation**: 0 errors, 0 warnings

# Check for failed entries
failed_entries = queue.list_entries(handoff_status=HandoffStatus.METADATA_FAILED)

for entry in failed_entries:
    print(f"Failed: {entry.entry_id}")
    print(f"Error: {entry.error_log[-1]}")  # Most recent error

    # Option 1: Retry with different parameters
    # Fix underlying data issue first, then retry

    # Option 2: Skip and continue with other entries
    # Batch processing continues despite failures

❌ All 142 samples failed validation (425 errors).
   Check workspace files for data quality issues.

First validation errors for entry pub_queue_12345:
  - Field 'run_accession': Field required
  - Field 'library_strategy': Field required
  - Sample SRR001: No download URLs available

[INFO] Processing entry pub_queue_12345: 'Microbiome Study' (4 workspace keys)
[INFO] Processing SRA sample file: sra_PRJNA891765_samples
[INFO] Sample extraction complete: 247/247 valid (100.0%)
[INFO] Extracted 247 valid samples from sra_PRJNA891765_samples (validation_rate: 100.0%)
[INFO] Entry pub_queue_12345 validation summary: 247/247 samples valid (100.0%), 0 errors, 0 warnings

# Manual identifier extraction
entry = queue.get_entry("pub_queue_12345678")

# Check extracted metadata for methods section
print(entry.extracted_metadata.get("methods", "No methods extracted"))

# Add identifiers manually if found in supplementary materials
entry.extracted_identifiers = {
    "sra": ["SRR12345678", "SRR12345679"],
    "bioproject": ["PRJNA720345"]
}
queue.update_status(
    entry_id=entry.entry_id,
    status=PublicationStatus.COMPLETED,
    extracted_identifiers=entry.extracted_identifiers
)

# Option 1: Use non-strict 16S validation
filter_samples_by(
    workspace_key="geo_gse123456",
    filter_criteria="16S human fecal",
    strict=False  # More permissive
)

# Option 2: Adjust fuzzy threshold for host validation
# (Requires direct service call, not via natural language tool)
service = MicrobiomeFilteringService()
filtered, stats, ir = service.validate_host_organism(
    metadata,
    allowed_hosts=["Human"],
    fuzzy_threshold=75.0  # Lower threshold (default: 85.0)
)

# Option 3: Check original metadata for sample type values
# Use read_sample_metadata tool to inspect column names

# Inspect unmapped values
stats = {...}  # From standardize_disease_terms
unmapped_count = stats["mapping_stats"]["unmapped"]
unique_mappings = stats["unique_mappings"]

# Check which values weren't mapped
original_values = [k for k, v in unique_mappings.items() if v == k]
print(f"Unmapped values: {original_values}")

# Manual mapping for custom categories
metadata["disease"] = metadata["disease"].replace({
    "Stage III CRC": "crc",
    "baseline": "healthy",
    "follow-up": "healthy"
})

# Re-run standardization
standardized, stats, ir = service.standardize_disease_terms(metadata)

# Check publication queue entry status
entry = queue.get_entry("pub_queue_12345678")
print(f"Status: {entry.status}")
print(f"Identifiers: {entry.extracted_identifiers}")
print(f"Workspace keys: {entry.workspace_metadata_keys}")

# If identifiers exist but workspace_metadata_keys empty:
# Re-fetch SRA metadata manually
# (research_agent should handle this automatically)

# 1. Check entry status
entry = queue.get_entry("pub_queue_12345")
print(f"Handoff status: {entry.handoff_status}")
print(f"Error: {entry.error_log[-1]}")

# 2. Inspect workspace file
workspace_key = entry.workspace_metadata_keys[0]
ws_path = data_manager.workspace_path / "metadata" / f"{workspace_key}.json"

with open(ws_path) as f:
    data = json.load(f)
    sample = data["data"]["samples"][0]
    print(f"Sample fields: {sample.keys()}")

    # Check for missing required fields
    required = ["run_accession", "experiment_accession", "library_strategy",
                "organism_name", "bioproject", "biosample"]
    missing = [f for f in required if f not in sample]
    print(f"Missing required fields: {missing}")

# 3. Re-fetch SRA metadata if corrupted
# Delete workspace file and re-run research_agent.fetch_sra_metadata()
ws_path.unlink()

# Option 1: Accept warnings and continue (most common)
# env_medium is optional, samples are still valid

# Option 2: Manually add env_medium from publication methods
entry = queue.get_entry("pub_queue_12345")
harmonization = entry.harmonization_metadata
for sample in harmonization["samples"]:
    if not sample.get("env_medium"):
        # Infer from publication or sample_type
        if "fecal" in sample.get("sample_type", "").lower():
            sample["env_medium"] = "Stool"
        elif "tissue" in sample.get("sample_type", "").lower():
            sample["env_medium"] = "Gut tissue"

# Update harmonization_metadata
queue.update_status(
    entry_id=entry.entry_id,
    status=entry.status,
    harmonization_metadata=harmonization
)

# Good: Ensure RIS file keywords match schema type
# In RIS file: KW - microbiome, KW - 16S rRNA
# Result: schema_type="microbiome" (auto-inferred)

# Bad: Missing keywords
# In RIS file: KW - gut bacteria
# Result: schema_type="general" (default, not optimal)

# Good: Check sample counts before filtering
> Query workspace metadata 'geo_gse123456' and show sample count
# Output: 142 samples

> Filter for human fecal 16S samples
# Output: 89 samples (62.7% retention) ✅ Reasonable

# Bad: Blind filtering without validation
> Filter for human fecal 16S samples
# Output: 5 samples (3.5% retention) ⚠️ Too restrictive

# Target retention rates by filter type:
# - 16S validation: >90% (if dataset is primarily 16S)
# - Host validation: >95% (if host is consistent)
# - Sample type filter: 50-80% (expected reduction for fecal-only)
# - Disease standardization: 95-100% (mapping rate)

# Alert if retention rate drops below expected:
if retention_rate < 50:
    print(f"⚠️ Warning: Low retention rate ({retention_rate:.1f}%)")
    print("Consider relaxing filter criteria or checking metadata quality")

# Good: Process publications in batches across sessions
# Session 1: Load and extract identifiers
> /load batch1.ris
> Process all publications
# Exit session

# Session 2: Resume filtering (metadata persists)
> Filter all publications for human fecal 16S
# ✅ Metadata loaded from workspace cache (fast)

# Bad: Re-process everything in single session
# (Memory pressure, long session times)

# Use harmonization_metadata to document decisions
entry.harmonization_metadata = {
    "samples": filtered_samples,
    "filtering_criteria": "16S human fecal CRC UC CD healthy",
    "filtering_rationale": "Focus on human gut microbiome with IBD diseases",
    "filtering_date": "2024-11-19",
    "analyst": "researcher@databiomix.com",
    "validation_status": "passed",
    "quality_checks": {
        "16s_validation": "strict mode, 97.2% retention",
        "host_validation": "fuzzy threshold=85.0, 98.6% retention",
        "sample_type_filter": "fecal only, 65.4% retention",
        "disease_standardization": "100% mapping rate"
    }
}