— name: lamindb description: This skill should be used when working with LaminDB, an open-source data framework for biology that makes data queryable, traceable, reproducible, and FAIR. Use when managing biological datasets (scRNA-seq, spatial, flow cytometry, etc.), tracking computational workflows, curating and validating data with biological ontologies, building data lakehouses, or ensuring data lineage and reproducibility in biological research. Covers data management, annotation, ontologies (genes, cell types, diseases, tissues), schema validation, integrations with workflow managers (Nextflow, Snakemake) and MLOps platforms (W&B, MLflow), and deployment strategies. — # LaminDB ## Overview LaminDB is an open-source data framework for biology designed to make data queryable, traceable, reproducible, and FAIR (Findable, Accessible, Interoperable, Reusable). It provides a unified platform that combines lakehouse architecture, lineage tracking, feature stores, biological ontologies, LIMS (Laboratory Information Management System), and ELN (Electronic Lab Notebook) capabilities through a single Python API. **Core Value Proposition:** – **Queryability**: Search and filter datasets by metadata, features, and ontology terms – **Traceability**: Automatic lineage tracking from raw data through analysis to results – **Reproducibility**: Version control for data, code, and environment – **FAIR Compliance**: Standardized annotations using biological ontologies ## When to Use This Skill Use this skill when: – **Managing biological datasets**: scRNA-seq, bulk RNA-seq, spatial transcriptomics, flow cytometry, multi-modal data, EHR data – **Tracking computational workflows**: Notebooks, scripts, pipeline execution (Nextflow, Snakemake, Redun) – **Curating and validating data**: Schema validation, standardization, ontology-based annotation – **Working with biological ontologies**: Genes, proteins, cell types, tissues, diseases, pathways (via Bionty) – **Building data lakehouses**: Unified query interface across multiple datasets – **Ensuring reproducibility**: Automatic versioning, lineage tracking, environment capture – **Integrating ML pipelines**: Connecting with Weights & Biases, MLflow, HuggingFace, scVI-tools – **Deploying data infrastructure**: Setting up local or cloud-based data management systems – **Collaborating on datasets**: Sharing curated, annotated data with standardized metadata ## Core Capabilities LaminDB provides six interconnected capability areas, each documented in detail in the references folder. ### 1. Core Concepts and Data Lineage **Core entities:** – **Artifacts**: Versioned datasets (DataFrame, AnnData, Parquet, Zarr, etc.) – **Records**: Experimental entities (samples, perturbations, instruments) – **Runs & Transforms**: Computational lineage tracking (what code produced what data) – **Features**: Typed metadata fields for annotation and querying **Key workflows:** – Create and version artifacts from files or Python objects – Track notebook/script execution with `ln.track()` and `ln.finish()` – Annotate artifacts with typed features – Visualize data lineage graphs with `artifact.view_lineage()` – Query by provenance (find all outputs from specific code/inputs) **Reference:** `references/core-concepts.md` – Read this for detailed information on artifacts, records, runs, transforms, features, versioning, and lineage tracking. ### 2. Data Management and Querying **Query capabilities:** – Registry exploration and lookup with auto-complete – Single record retrieval with `get()`, `one()`, `one_or_none()` – Filtering with comparison operators (`__gt`, `__lte`, `__contains`, `__startswith`) – Feature-based queries (query by annotated metadata) – Cross-registry traversal with double-underscore syntax – Full-text search across registries – Advanced logical queries with Q objects (AND, OR, NOT) – Streaming large datasets without loading into memory **Key workflows:** – Browse artifacts with filters and ordering – Query by features, creation date, creator, size, etc. – Stream large files in chunks or with array slicing – Organize data with hierarchical keys – Group artifacts into collections **Reference:** `references/data-management.md` – Read this for comprehensive query patterns, filtering examples, streaming strategies, and data organization best practices. ### 3. Annotation and Validation **Curation process:** 1. **Validation**: Confirm datasets match desired schemas 2. **Standardization**: Fix typos, map synonyms to canonical terms 3. **Annotation**: Link datasets to metadata entities for queryability **Schema types:** – **Flexible schemas**: Validate only known columns, allow additional metadata – **Minimal required schemas**: Specify essential columns, permit extras – **Strict schemas**: Complete control over structure and values **Supported data types:** – DataFrames (Parquet, CSV) – AnnData (single-cell genomics) – MuData (multi-modal) – SpatialData (spatial transcriptomics) – TileDB-SOMA (scalable arrays) **Key workflows:** – Define features and schemas for data validation – Use `DataFrameCurator` or `AnnDataCurator` for validation – Standardize values with `.cat.standardize()` – Map to ontologies with `.cat.add_ontology()` – Save curated artifacts with schema linkage – Query validated datasets by features **Reference:** `references/annotation-validation.md` – Read this for detailed curation workflows, schema design patterns, handling validation errors, and best practices. ### 4. Biological Ontologies **Available ontologies (via Bionty):** – Genes (Ensembl), Proteins (UniProt) – Cell types (CL), Cell lines (CLO) – Tissues (Uberon), Diseases (Mondo, DOID) – Phenotypes (HPO), Pathways (GO) – Experimental factors (EFO), Developmental stages – Organisms (NCBItaxon), Drugs (DrugBank) **Key workflows:** – Import public ontologies with `bt.CellType.import_source()` – Search ontologies with keyword or exact matching – Standardize terms using synonym mapping – Explore hierarchical relationships (parents, children, ancestors) – Validate data against ontology terms – Annotate datasets with ontology records – Create custom terms and hierarchies – Handle multi-organism contexts (human, mouse, etc.) **Reference:** `references/ontologies.md` – Read this for comprehensive ontology operations, standardization strategies, hierarchy navigation, and annotation workflows. ### 5. Integrations **Workflow managers:** – Nextflow: Track pipeline processes and outputs – Snakemake: Integrate into Snakemake rules – Redun: Combine with Redun task tracking **MLOps platforms:** – Weights & Biases: Link experiments with data artifacts – MLflow: Track models and experiments – HuggingFace: Track model fine-tuning – scVI-tools: Single-cell analysis workflows **Storage systems:** – Local filesystem, AWS S3, Google Cloud Storage – S3-compatible (MinIO, Cloudflare R2) – HTTP/HTTPS endpoints (read-only) – HuggingFace datasets **Array stores:** – TileDB-SOMA (with cellxgene support) – DuckDB for SQL queries on Parquet files **Visualization:** – Vitessce for interactive spatial/single-cell visualization **Version control:** – Git integration for source code tracking **Reference:** `references/integrations.md` – Read this for integration patterns, code examples, and troubleshooting for third-party systems. ### 6. Setup and Deployment **Installation:** – Basic: `uv pip install lamindb` – With extras: `uv pip install 'lamindb[gcp,zarr,fcs]'` – Modules: bionty, wetlab, clinical **Instance types:** – Local SQLite (development) – Cloud storage + SQLite (small teams) – Cloud storage + PostgreSQL (production) **Storage options:** – Local filesystem – AWS S3 with configurable regions and permissions – Google Cloud Storage – S3-compatible endpoints (MinIO, Cloudflare R2) **Configuration:** – Cache management for cloud files – Multi-user system configurations – Git repository sync – Environment variables **Deployment patterns:** – Local dev → Cloud production migration – Multi-region deployments – Shared storage with personal instances **Reference:** `references/setup-deployment.md` – Read this for detailed installation, configuration, storage setup, database management, security best practices, and troubleshooting. ## Common Use Case Workflows ### Use Case 1: Single-Cell RNA-seq Analysis with Ontology Validation “`python import lamindb as ln import bionty as bt import anndata as ad # Start tracking ln.track(params={"analysis": "scRNA-seq QC and annotation"}) # Import cell type ontology bt.CellType.import_source() # Load data adata = ad.read_h5ad("raw_counts.h5ad") # Validate and standardize cell types adata.obs["cell_type"] = bt.CellType.standardize(adata.obs["cell_type
f:T2ff4,# LaminDB Annotation & Validation This document covers data curation, validation, schema management, and annotation best practices in LaminDB. ## Overview LaminDB's curation process ensures datasets are both validated and queryable through three essential steps: 1. **Validation**: Confirming datasets match desired schemas 2. **Standardization**: Fixing inconsistencies like typos and mapping synonyms 3. **Annotation**: Linking datasets to metadata entities for queryability ## Schema Design Schemas define expected data structure, types, and validation rules. LaminDB supports three main schema approaches: ### 1. Flexible Schema Validates only columns matching Feature registry names, allowing additional metadata: “`python import lamindb as ln # Create flexible schema schema = ln.Schema( name="valid_features", itype=ln.Feature # Validates against Feature registry ).save() # Any column matching a Feature name will be validated # Additional columns are permitted but not validated “` ### 2. Minimal Required Schema Specifies essential columns while permitting extra metadata: “`python # Define required features required_features = [ ln.Feature.get(name="cell_type"), ln.Feature.get(name="tissue"), ln.Feature.get(name="donor_id") ] # Create schema with required features schema = ln.Schema( name="minimal_immune_schema", features=required_features, flexible=True # Allows additional columns ).save() “` ### 3. Strict Schema Enforces complete control over data structure: “`python # Define all allowed features all_features = [ ln.Feature.get(name="cell_type"), ln.Feature.get(name="tissue"), ln.Feature.get(name="donor_id"), ln.Feature.get(name="disease") ] # Create strict schema schema = ln.Schema( name="strict_immune_schema", features=all_features, flexible=False # No additional columns allowed ).save() “` ## DataFrame Curation Workflow The typical curation process involves six key steps: ### Step 1-2: Load Data and Establish Registries “`python import pandas as pd import lamindb as ln # Load data df = pd.read_csv("experiment.csv") # Define and save features ln.Feature(name="cell_type", dtype=str).save() ln.Feature(name="tissue", dtype=str).save() ln.Feature(name="gene_count", dtype=int).save() ln.Feature(name="experiment_date", dtype="date").save() # Populate valid values (if using controlled vocabulary) import bionty as bt bt.CellType.import_source() bt.Tissue.import_source() “` ### Step 3: Create Schema “`python # Link features to schema features = [ ln.Feature.get(name="cell_type"), ln.Feature.get(name="tissue"), ln.Feature.get(name="gene_count"), ln.Feature.get(name="experiment_date") ] schema = ln.Schema( name="experiment_schema", features=features, flexible=True ).save() “` ### Step 4: Initialize Curator and Validate “`python # Initialize curator curator = ln.curators.DataFrameCurator(df, schema) # Validate dataset validation = curator.validate() # Check validation results if validation: print("✓ Validation passed") else: print("✗ Validation failed") curator.non_validated # See problematic fields “` ### Step 5: Fix Validation Issues #### Standardize Values “`python # Fix typos and synonyms in categorical columns curator.cat.standardize("cell_type") curator.cat.standardize("tissue") # View standardization mapping curator.cat.inspect_standardize("cell_type") “` #### Map to Ontologies “`python # Map values to ontology terms curator.cat.add_ontology("cell_type", bt.CellType) curator.cat.add_ontology("tissue", bt.Tissue) # Look up public ontologies for unmapped terms curator.cat.lookup(public=True).cell_type # Interactive lookup “` #### Add New Terms “`python # Add new valid terms to registry curator.cat.add_new_from("cell_type") # Or manually create records new_cell_type = bt.CellType(name="my_novel_cell_type").save() “` #### Rename Columns “`python # Rename columns to match feature names df = df.rename(columns={"celltype": "cell_type"}) # Re-initialize curator with fixed DataFrame curator = ln.curators.DataFrameCurator(df, schema) “` ### Step 6: Save Curated Artifact “`python # Save with schema linkage artifact = curator.save_artifact( key="experiments/curated_data.parquet", description="Validated and annotated experimental data" ) # Verify artifact has schema artifact.schema # Returns the schema object artifact.describe() # Shows validation status “` ## AnnData Curation For composite structures like AnnData, use "slots" to validate different components: ### Defining AnnData Schemas “`python # Create schemas for different slots obs_schema = ln.Schema( name="cell_metadata", features=[ ln.Feature.get(name="cell_type"), ln.Feature.get(name="tissue"), ln.Feature.get(name="donor_id") ] ).save() var_schema = ln.Schema( name="gene_ids", features=[ln.Feature.get(name="ensembl_gene_id")] ).save() # Create composite AnnData schema anndata_schema = ln.Schema( name="scrna_schema", otype="AnnData", slots={ "obs": obs_schema, "var.T": var_schema # .T indicates transposition } ).save() “` ### Curating AnnData Objects “`python import anndata as ad # Load AnnData adata = ad.read_h5ad("data.h5ad") # Initialize curator curator = ln.curators.AnnDataCurator(adata, anndata_schema) # Validate all slots validation = curator.validate() # Fix issues by slot curator.cat.standardize("obs", "cell