— name: ai-multimodal description: Process and generate multimedia content using Google Gemini API. Capabilities include analyze audio files (transcription with timestamps, summarization, speech understanding, music/sound analysis up to 9.5 hours), understand images (captioning, object detection, OCR, visual Q&A, segmentation), process videos (scene detection, Q&A, temporal analysis, YouTube URLs, up to 6 hours), extract from documents (PDF tables, forms, charts, diagrams, multi-page), generate images (text-to-image, editing, composition, refinement). Use when working with audio/video files, analyzing images or screenshots, processing PDF documents, extracting structured data from media, creating images from text prompts, or implementing multimodal AI features. Supports multiple models (Gemini 2.5/2.0) with context windows up to 2M tokens. license: MIT allowed-tools: – Bash – Read – Write – Edit — # AI Multimodal Processing Skill Process audio, images, videos, documents, and generate images using Google Gemini's multimodal API. Unified interface for all multimedia content understanding and generation. ## Core Capabilities ### Audio Processing – Transcription with timestamps (up to 9.5 hours) – Audio summarization and analysis – Speech understanding and speaker identification – Music and environmental sound analysis – Text-to-speech generation with controllable voice ### Image Understanding – Image captioning and description – Object detection with bounding boxes (2.0+) – Pixel-level segmentation (2.5+) – Visual question answering – Multi-image comparison (up to 3,600 images) – OCR and text extraction ### Video Analysis – Scene detection and summarization – Video Q&A with temporal understanding – Transcription with visual descriptions – YouTube URL support – Long video processing (up to 6 hours) – Frame-level analysis ### Document Extraction – Native PDF vision processing (up to 1,000 pages) – Table and form extraction – Chart and diagram analysis – Multi-page document understanding – Structured data output (JSON schema) – Format conversion (PDF to HTML/JSON) ### Image Generation – Text-to-image generation – Image editing and modification – Multi-image composition (up to 3 images) – Iterative refinement – Multiple aspect ratios (1:1, 16:9, 9:16, 4:3, 3:4) – Controllable style and quality ## Capability Matrix | Task | Audio | Image | Video | Document | Generation | |——|:—–:|:—–:|:—–:|:——–:|:———-:| | Transcription | ✓ | – | ✓ | – | – | | Summarization | ✓ | ✓ | ✓ | ✓ | – | | Q&A | ✓ | ✓ | ✓ | ✓ | – | | Object Detection | – | ✓ | ✓ | – | – | | Text Extraction | – | ✓ | – | ✓ | – | | Structured Output | ✓ | ✓ | ✓ | ✓ | – | | Creation | TTS | – | – | – | ✓ | | Timestamps | ✓ | – | ✓ | – | – | | Segmentation | – | ✓ | – | – | – | ## Model Selection Guide ### Gemini 2.5 Series (Recommended) – **gemini-2.5-pro**: Highest quality, all features, 1M-2M context – **gemini-2.5-flash**: Best balance, all features, 1M-2M context – **gemini-2.5-flash-lite**: Lightweight, segmentation support – **gemini-2.5-flash-image**: Image generation only ### Gemini 2.0 Series – **gemini-2.0-flash**: Fast processing, object detection – **gemini-2.0-flash-lite**: Lightweight option ### Feature Requirements – **Segmentation**: Requires 2.5+ models – **Object Detection**: Requires 2.0+ models – **Multi-video**: Requires 2.5+ models – **Image Generation**: Requires flash-image model ### Context Windows – **2M tokens**: ~6 hours video (low-res) or ~2 hours (default) – **1M tokens**: ~3 hours video (low-res) or ~1 hour (default) – **Audio**: 32 tokens/second (1 min = 1,920 tokens) – **PDF**: 258 tokens/page (fixed) – **Image**: 258-1,548 tokens based on size ## Quick Start ### Prerequisites **API Key Setup**: Supports both Google AI Studio and Vertex AI. The skill checks for `GEMINI_API_KEY` in this order: 1. Process environment: `export GEMINI_API_KEY="your-key"` 2. Project root: `.env` 3. `.claude/.env` 4. `.claude/skills/.env` 5. `.claude/skills/ai-multimodal/.env` **Get API key**: https://aistudio.google.com/apikey **For Vertex AI**: “`bash export GEMINI_USE_VERTEX=true export VERTEX_PROJECT_ID=your-gcp-project-id export VERTEX_LOCATION=us-central1 # Optional “` **Install SDK**: “`bash pip install google-genai python-dotenv pillow “` ### Common Patterns **Transcribe Audio**: “`bash python scripts/gemini_batch_process.py \ –files audio.mp3 \ –task transcribe \ –model gemini-2.5-flash “` **Analyze Image**: “`bash python scripts/gemini_batch_process.py \ –files image.jpg \ –task analyze \ –prompt "Describe this image" \ –output docs/assets/<output-name>.md \ –model gemini-2.5-flash “` **Process Video**: “`bash python scripts/gemini_batch_process.py \ –files video.mp4 \ –task analyze \ –prompt "Summarize key points with timestamps" \ –output docs/assets/<output-name>.md \ –model gemini-2.5-flash “` **Extract from PDF**: “`bash python scripts/gemini_batch_process.py \ –files document.pdf \ –task extract \ –prompt "Extract table data as JSON" \ –output docs/assets/<output-name>.md \ –format json “` **Generate Image**: “`bash python scripts/gemini_batch_process.py \ –task generate \ –prompt "A futuristic city at sunset" \ –output docs/assets/<output-file-name> \ –model gemini-2.5-flash-image \ –aspect-ratio 16:9 “` **Optimize Media**: “`bash # Prepare large video for processing python scripts/media_optimizer.py \ –input large-video.mp4 \ –output docs/assets/<output-file-name> \ –target-size 100MB # Batch optimize multiple files python scripts/media_optimizer.py \ –input-dir ./videos \ –output-dir docs/assets/optimized \ –quality 85 “` **Convert Documents to Markdown**: “`bash # Convert to PDF python scripts/document_converter.py \ –input document.docx \ –output docs/assets/document.md # Extract pages python scripts/document_converter.py \ –input large.pdf \ –output docs/assets/chapter1.md \ –pages 1-20 “` ## Supported Formats ### Audio – WAV, MP3, AAC, FLAC, OGG Vorbis, AIFF – Max 9.5 hours per request – Auto-downsampled to 16 Kbps mono ### Images – PNG, JPEG, WEBP, HEIC, HEIF – Max 3,600 images per request – Resolution: ≤384px = 258 tokens, larger = tiled ### Video – MP4, MPEG, MOV, AVI, FLV, MPG, WebM, WMV, 3GPP – Max 6 hours (low-res) or 2 hours (default) – YouTube URLs supported (public only) ### Documents – PDF only for vision processing – Max 1,000 pages – TXT, HTML, Markdown supported (text-only) ### Size Limits – **Inline**: <20MB total request – **File API**: 2GB per file, 20GB project quota – **Retention**: 48 hours auto-delete ## Reference Navigation For detailed implementation guidance, see: ### Audio Processing – `references/audio-processing.md` – Transcription, analysis, TTS – Timestamp handling and segment analysis – Multi-speaker identification – Non-speech audio analysis – Text-to-speech generation ### Image Understanding – `references/vision-understanding.md` – Captioning, detection, OCR – Object detection and localization – Pixel-level segmentation – Visual question answering – Multi-image comparison ### Video Analysis – `references/video-analysis.md` – Scene detection, temporal understanding – YouTube URL processing – Timestamp-based queries – Video clipping and FPS control – Long video optimization ### Document Extraction – `references/document-extraction.md` – PDF processing, structured output – Table and form extraction – Chart and diagram analysis – JSON schema validation – Multi-page handling ### Image Generation – `references/image-generation.md` – Text-to-image, editing – Prompt engineering strategies – Image editing and composition – Aspect ratio selection – Safety settings ## Cost Optimization ### Token Costs **Input Pricing**: – Gemini 2.5 Flash: $1.00/1M input, $0.10/1M output – Gemini 2.5 Pro: $3.00/1M input, $12.00/1M output – Gemini 1.5 Flash: $0.70/1M input, $0.175/1M output **Token Rates**: – Audio: 32 tokens/second (1 min = 1,920 tokens) – Video: ~300 tokens/second (default) or ~100 (low-res) – PDF: 258 tokens/page (fixed) – Image: 258-1,548 tokens based on size **TTS Pricing**: – Flash TTS: $10/1M tokens – Pro TTS: $20/1M tokens ### Best Practices 1. Use `gemini-2.5-flash` for most tasks (best price/performance) 2. Use File API for files >20MB or repeated queries 3. Optimize media before upload (see `media_optimizer.py`) 4. Process specific segments instead of full videos 5. Use lower FPS for static content 6. Implement context caching for repeated queries 7. Batch process multiple files in parallel ## Rate Limits **Free Tier**: – 10-15 RPM (requests per minute) – 1M-4M TPM (tokens per minute) – 1,500 RPD (requests per day) **YouTube Limits**: – Free tier: 8 hours/day – Paid tier: No length limits – Public videos only **Storage Limits**: – 20GB per project – 2GB per file – 48-hour retention ## Error Handling Common errors and solutions: – **400**: Invalid format/size – validate before upload – **401**: Invalid API key – check configuration – **403**: Permission denied – verify API key restrictions – **404**: File not found – ensure file uploaded and active – **429**: Rate limit exceeded – implement exponential backoff – **500**: Server error – retry with backoff ## Scripts Overview All scripts support unified API key detection and error handling: **gemini_batch_process.py**: Batch process multiple media files – Supports all modalities (audio, image, video, PDF) – Progress tracking and error recovery – Output formats: JSON, Markdown, CSV – Rate limiting and retry logic – Dry-run mode **media_optimizer.py**: Prepare media for Gemini API – Compress videos/audio for size limits – Resize images appropriately – Split long videos into chunks – Format conversion – Quality vs size optimization **document_converter.py**: Convert documents to PDF – Convert DOCX, XLSX, PPTX to PDF – Extract page ranges – Optimize PDFs for Gemini – Extract images from PDFs – Batch conversion support Run any script with `–help` for detailed usage. ## Resources – [Audio API Docs](https://ai.google.dev/gemini-api/docs/audio) – [Image API Docs](https://ai.google.dev/gemini-api/docs/image-understanding) – [Video API Docs](https://ai.google.dev/gemini-api/docs/video-understanding) – [Document API Docs](https://ai.google.dev/gemini-api/docs/document-processing) – [Image Gen Docs](https://ai.google.dev/gemini-api/docs/image-generation) – [Get API Key](https://aistudio.google.com/apikey) – [Pricing](https://ai.google.dev/pricing)