— name: peer-review description: "Systematic peer review toolkit. Evaluate methodology, statistics, design, reproducibility, ethics, figure integrity, reporting standards, for manuscript and grant review across disciplines." allowed-tools: [Read, Write, Edit, Bash] — # Scientific Critical Evaluation and Peer Review ## Overview Peer review is a systematic process for evaluating scientific manuscripts. Assess methodology, statistics, design, reproducibility, ethics, and reporting standards. Apply this skill for manuscript and grant review across disciplines with constructive, rigorous evaluation. ## When to Use This Skill This skill should be used when: – Conducting peer review of scientific manuscripts for journals – Evaluating grant proposals and research applications – Assessing methodology and experimental design rigor – Reviewing statistical analyses and reporting standards – Evaluating reproducibility and data availability – Checking compliance with reporting guidelines (CONSORT, STROBE, PRISMA) – Providing constructive feedback on scientific writing ## Visual Enhancement with Scientific Schematics **When creating documents with this skill, always consider adding scientific diagrams and schematics to enhance visual communication.** If your document does not already contain schematics or diagrams: – Use the **scientific-schematics** skill to generate AI-powered publication-quality diagrams – Simply describe your desired diagram in natural language – Nano Banana Pro will automatically generate, review, and refine the schematic **For new documents:** Scientific schematics should be generated by default to visually represent key concepts, workflows, architectures, or relationships described in the text. **How to generate schematics:** “`bash python scripts/generate_schematic.py "your diagram description" -o figures/output.png “` The AI will automatically: – Create publication-quality images with proper formatting – Review and refine through multiple iterations – Ensure accessibility (colorblind-friendly, high contrast) – Save outputs in the figures/ directory **When to add schematics:** – Peer review workflow diagrams – Evaluation criteria decision trees – Review process flowcharts – Methodology assessment frameworks – Quality assessment visualizations – Reporting guidelines compliance diagrams – Any complex concept that benefits from visualization For detailed guidance on creating schematics, refer to the scientific-schematics skill documentation. — ## Peer Review Workflow Conduct peer review systematically through the following stages, adapting depth and focus based on the manuscript type and discipline. ### Stage 1: Initial Assessment Begin with a high-level evaluation to determine the manuscript's scope, novelty, and overall quality. **Key Questions:** – What is the central research question or hypothesis? – What are the main findings and conclusions? – Is the work scientifically sound and significant? – Is the work appropriate for the intended venue? – Are there any immediate major flaws that would preclude publication? **Output:** Brief summary (2-3 sentences) capturing the manuscript's essence and initial impression. ### Stage 2: Detailed Section-by-Section Review Conduct a thorough evaluation of each manuscript section, documenting specific concerns and strengths. #### Abstract and Title – **Accuracy:** Does the abstract accurately reflect the study's content and conclusions? – **Clarity:** Is the title specific, accurate, and informative? – **Completeness:** Are key findings and methods summarized appropriately? – **Accessibility:** Is the abstract comprehensible to a broad scientific audience? #### Introduction – **Context:** Is the background information adequate and current? – **Rationale:** Is the research question clearly motivated and justified? – **Novelty:** Is the work's originality and significance clearly articulated? – **Literature:** Are relevant prior studies appropriately cited? – **Objectives:** Are research aims/hypotheses clearly stated? #### Methods – **Reproducibility:** Can another researcher replicate the study from the description provided? – **Rigor:** Are the methods appropriate for addressing the research questions? – **Detail:** Are protocols, reagents, equipment, and parameters sufficiently described? – **Ethics:** Are ethical approvals, consent, and data handling properly documented? – **Statistics:** Are statistical methods appropriate, clearly described, and justified? – **Validation:** Are controls, replicates, and validation approaches adequate? **Critical elements to verify:** – Sample sizes and power calculations – Randomization and blinding procedures – Inclusion/exclusion criteria – Data collection protocols – Computational methods and software versions – Statistical tests and correction for multiple comparisons #### Results – **Presentation:** Are results presented logically and clearly? – **Figures/Tables:** Are visualizations appropriate, clear, and properly labeled? – **Statistics:** Are statistical results properly reported (effect sizes, confidence intervals, p-values)? – **Objectivity:** Are results presented without over-interpretation? – **Completeness:** Are all relevant results included, including negative results? – **Reproducibility:** Are raw data or summary statistics provided? **Common issues to identify:** – Selective reporting of results – Inappropriate statistical tests – Missing error bars or measures of variability – Over-fitting or circular analysis – Batch effects or confounding variables – Missing controls or validation experiments #### Discussion – **Interpretation:** Are conclusions supported by the data? – **Limitations:** Are study limitations acknowledged and discussed? – **Context:** Are findings placed appropriately within existing literature? – **Speculation:** Is speculation clearly distinguished from data-supported conclusions? – **Significance:** Are implications and importance clearly articulated? – **Future directions:** Are next steps or unanswered questions discussed? **Red flags:** – Overstated conclusions – Ignoring contradictory evidence – Causal claims from correlational data – Inadequate discussion of limitations – Mechanistic claims without mechanistic evidence #### References – **Completeness:** Are key relevant papers cited? – **Currency:** Are recent important studies included? – **Balance:** Are contrary viewpoints appropriately cited? – **Accuracy:** Are citations accurate and appropriate? – **Self-citation:** Is there excessive or inappropriate self-citation? ### Stage 3: Methodological and Statistical Rigor Evaluate the technical quality and rigor of the research with particular attention to common pitfalls. **Statistical Assessment:** – Are statistical assumptions met (normality, independence, homoscedasticity)? – Are effect sizes reported alongside p-values? – Is multiple testing correction applied appropriately? – Are confidence intervals provided? – Is sample size justified with power analysis? – Are parametric vs. non-parametric tests chosen appropriately? – Are missing data handled properly? – Are exploratory vs. confirmatory analyses distinguished? **Experimental Design:** – Are controls appropriate and adequate? – Is replication sufficient (biological and technical)? – Are potential confounders identified and controlled? – Is randomization properly implemented? – Are blinding procedures adequate? – Is the experimental design optimal for the research question? **Computational/Bioinformatics:** – Are computational methods clearly described and justified? – Are software versions and parameters documented? – Is code made available for reproducibility? – Are algorithms and models validated appropriately? – Are assumptions of computational methods met? – Is batch correction applied appropriately? ### Stage 4: Reproducibility and Transparency Assess whether the research meets modern standards for reproducibility and open science. **Data Availability:** – Are raw data deposited in appropriate repositories? – Are accession numbers provided for public databases? – Are data sharing restrictions justified (e.g., patient privacy)? – Are data formats standard and accessible? **Code and Materials:** – Is analysis code made available (GitHub, Zenodo, etc.)? – Are unique materials available or described sufficiently for recreation? – Are protocols detailed in sufficient depth? **Reporting Standards:** – Does the manuscript follow discipline-specific reporting guidelines (CONSORT, PRISMA, ARRIVE, MIAME, MINSEQE, etc.)? – See `references/reporting_standards.md` for common guidelines – Are all elements of the appropriate checklist addressed? ### Stage 5: Figure and Data Presentation Evaluate the quality, clarity, and integrity of data visualization. **Quality Checks:** – Are figures high resolution and clearly labeled? – Are axes properly labeled with units? – Are error bars defined (SD, SEM, CI)? – Are statistical significance indicators explained? – Are color schemes appropriate and accessible (colorblind-friendly)? – Are scale bars included for images? – Is data visualization appropriate for the data type? **Integrity Checks:** – Are there signs of image manipulation (duplications, splicing)? – Are Western blots and gels appropriately presented? – Are representative images truly representative? – Are all conditions shown (no selective presentation)? **Clarity:** – Can figures stand alone with their legends? – Is the message of each figure immediately clear? – Are there redundant figures or panels? – Would data be better presented as tables or figures? ### Stage 6: Ethical Considerations Verify that the research meets ethical standards and guidelines. **Human Subjects:** – Is IRB/ethics approval documented? – Is informed consent described? – Are vulnerable populations appropriately protected? – Is patient privacy adequately protected? – Are potential conflicts of interest disclosed? **Animal Research:** – Is IACUC or equivalent approval documented? – Are procedures humane and justified? – Are the 3Rs (replacement, reduction, refinement) considered? – Are euthanasia methods appropriate? **Research Integrity:** – Are there concerns about data fabrication or falsification? – Is authorship appropriate and justified? – Are competing interests disclosed? – Is funding source disclosed? – Are there concerns about plagiarism or duplicate publication? ### Stage 7: Writing Quality and Clarity Assess the manuscript's clarity, organization, and accessibility. **Structure and Organization:** – Is the manuscript logically organized? – Do sections flow coherently? – Are transitions between ideas clear? – Is the narrative compelling and clear? **Writing Quality:** – Is the language clear, precise, and concise? – Are jargon and acronyms minimized and defined? – Is grammar and spelling correct? – Are sentences unnecessarily complex? – Is the passive voice overused? **Accessibility:** – Can a non-specialist understand the main findings? – Are technical terms explained? – Is the significance clear to a broad audience? ## Structuring Peer Review Reports Organize feedback in a hierarchical structure that prioritizes issues and provides actionable guidance. ### Summary Statement Provide a concise overall assessment (1-2 paragraphs): – Brief synopsis of the research – Overall recommendation (accept, minor revisions, major revisions, reject) – Key strengths (2-3 bullet points) – Key weaknesses (2-3 bullet points) – Bottom-line assessment of significance and soundness ### Major Comments List critical issues that significantly impact the manuscript's validity, interpretability, or significance. Number these sequentially for easy reference. **Major comments typically include:** – Fundamental methodological flaws – Inappropriate statistical analyses – Unsupported or overstated conclusions – Missing critical controls or experiments – Serious reproducibility concerns – Major gaps in literature coverage – Ethical concerns **For each major comment:** 1. Clearly state the issue 2. Explain why it's problematic 3. Suggest specific solutions or additional experiments 4. Indicate if addressing it is essential for publication ### Minor Comments List less critical issues that would improve clarity, completeness, or presentation. Number these sequentially. **Minor comments typically include:** – Unclear figure labels or legends – Missing methodological details – Typographical or grammatical errors – Suggestions for improved data presentation – Minor statistical reporting issues – Supplementary analyses that would strengthen conclusions – Requests for clarification **For each minor comment:** 1. Identify the specific location (section, paragraph, figure) 2. State the issue clearly 3. Suggest how to address it ### Specific Line-by-Line Comments (Optional) For manuscripts requiring detailed feedback, provide section-specific or line-by-line comments: – Reference specific page/line numbers or sections – Note factual errors, unclear statements, or missing citations – Suggest specific edits for clarity ### Questions for Authors List specific questions that need clarification: – Methodological details that are unclear – Seemingly contradictory results – Missing information needed to evaluate the work – Requests for additional data or analyses ## Tone and Approach Maintain a constructive, professional, and collegial tone throughout the review. **Best Practices:** – **Be constructive:** Frame criticism as opportunities for improvement – **Be specific:** Provide concrete examples and actionable suggestions – **Be balanced:** Acknowledge strengths as well as weaknesses – **Be respectful:** Remember that authors have invested significant effort – **Be objective:** Focus on the science, not the scientists – **Be thorough:** Don't overlook issues, but prioritize appropriately – **Be clear:** Avoid ambiguous or vague criticism **Avoid:** – Personal attacks or dismissive language – Sarcasm or condescension – Vague criticism without specific examples – Requesting unnecessary experiments beyond the scope – Demanding adherence to personal preferences vs. best practices – Revealing your identity if reviewing is double-blind ## Special Considerations by Manuscript Type ### Original Research Articles – Emphasize rigor, reproducibility, and novelty – Assess significance and impact – Verify that conclusions are data-driven – Check for complete methods and appropriate controls ### Reviews and Meta-Analyses – Evaluate comprehensiveness of literature coverage – Assess search strategy and inclusion/exclusion criteria – Verify systematic approach and lack of bias – Check for critical analysis vs. mere summarization – For meta-analyses, evaluate statistical approach and heterogeneity ### Methods Papers – Emphasize validation and comparison to existing methods – Assess reproducibility and availability of protocols/code – Evaluate improvements over existing approaches – Check for sufficient detail for implementation ### Short Reports/Letters – Adapt expectations for brevity – Ensure core findings are still rigorous and significant – Verify that format is appropriate for findings ### Preprints – Recognize that these have not undergone formal peer review – May be less polished than journal submissions – Still apply rigorous standards for scientific validity – Consider providing constructive feedback to help authors improve before journal submission ### Presentations and Slide Decks **⚠️ CRITICAL: For presentations, NEVER read the PDF directly. ALWAYS convert to images first.** When reviewing scientific presentations (PowerPoint, Beamer, slide decks): #### Mandatory Image-Based Review Workflow **NEVER attempt to read presentation PDFs directly** – this causes buffer overflow errors and doesn't show visual formatting issues. **Required Process:** 1. Convert PDF to images using Python: “`bash python skills/scientific-slides/scripts/pdf_to_images.py presentation.pdf review/slide –dpi 150 # Creates: review/slide-001.jpg, review/slide-002.jpg, etc. “` 2. Read and inspect EACH slide image file sequentially 3. Document issues with specific slide numbers 4. Provide feedback on visual formatting and content **Print when starting review:** “` [HH:MM:SS] PEER REVIEW: Presentation detected – converting to images for review [HH:MM:SS] PDF REVIEW: NEVER reading PDF directly – using image-based inspection “` #### Presentation-Specific Evaluation Criteria **Visual Design and Readability:** – [ ] Text is large enough (minimum 18pt, ideally 24pt+ for body text) – [ ] High contrast between text and background (4.5:1 minimum, 7:1 preferred) – [ ] Color scheme is professional and colorblind-accessible – [ ] Consistent visual design across all slides – [ ] White space is adequate (not cramped) – [ ] Fonts are clear and professional **Layout and Formatting (Check EVERY Slide Image):** – [ ] No text overflow or truncation at slide edges – [ ] No element overlaps (text over images, overlapping shapes) – [ ] Titles are consistently positioned – [ ] Content is properly aligned – [ ] Bullets and text are not cut off – [ ] Figures fit within slide boundaries – [ ] Captions and labels are visible and readable **Content Quality:** – [ ] One main idea per slide (not overloaded) – [ ] Minimal text (3-6 bullets per slide maximum) – [ ] Bullet points are concise (5-7 words each) – [ ] Figures are simplified and clear (not copy-pasted from papers) – [ ] Data visualizations have large, readable labels – [ ] Citations are present and properly formatted – [ ] Results/data slides dominate the presentation (40-50% of content) **Structure and Flow:** – [ ] Clear narrative arc (introduction → methods → results → discussion) – [ ] Logical progression between slides – [ ] Slide count appropriate for talk duration (~1 slide per minute) – [ ] Title slide includes authors, affiliation, date – [ ] Introduction cites relevant background literature (3-5 papers) – [ ] Discussion cites comparison papers (3-5 papers) – [ ] Conclusions slide summarizes key findings – [ ] Acknowledgments/funding slide at end **Scientific Content:** – [ ] Research question clearly stated – [ ] Methods adequately summarized (not excessive detail) – [ ] Results presented logically with clear visualizations – [ ] Statistical significance indicated appropriately – [ ] Conclusions supported by data shown – [ ] Limitations acknowledged where appropriate – [ ] Future directions or broader impact discussed **Common Presentation Issues to Flag:** **Critical Issues (Must Fix):** – Text overflow making content unreadable – Font sizes too small (<18pt) – Element overlaps obscuring data – Insufficient contrast (text hard to read) – Figures too complex or illegible – No citations (completely unsupported claims) – Slide count drastically mismatched to duration **Major Issues (Should Fix):** – Inconsistent design across slides – Too much text (walls of text, not bullets) – Poorly simplified figures (axis labels too small) – Cramped layout with insufficient white space – Missing key structural elements (no conclusion slide) – Poor color choices (not colorblind-safe) – Minimal results content (<30% of slides) **Minor Issues (Suggestions for Improvement):** – Could use more visuals/diagrams – Some slides slightly text-heavy – Minor alignment inconsistencies – Could benefit from more white space – Additional citations would strengthen claims – Color scheme could be more modern #### Review Report Format for Presentations **Summary Statement:** – Overall impression of presentation quality – Appropriateness for target audience and duration – Key strengths (visual design, content, clarity) – Key weaknesses (formatting issues, content gaps) – Recommendation (ready to present, minor revisions, major revisions) **Layout and Formatting Issues (By Slide Number):** “` Slide 3: Text overflow – bullet point 4 extends beyond right margin Slide 7: Element overlap – figure overlaps with caption text Slide 12: Font size – axis labels too small to read from distance Slide 18: Alignment – title not centered “` **Content and Structure Feedback:** – Adequacy of background context and citations – Clarity of research question and objectives – Quality of methods summary – Effectiveness of results presentation – Strength of conclusions and implications **Design and Accessibility:** – Overall visual appeal and professionalism – Color contrast and readability – Colorblind accessibility – Consistency across slides **Timing and Scope:** – Whether slide count matches intended duration – Appropriate level of detail for talk type – Balance between sections #### Example Image-Based Review Process “` [14:30:00] PEER REVIEW: Starting review of presentation [14:30:05] PEER REVIEW: Presentation detected – converting to images [14:30:10] PDF REVIEW: Running pdf_to_images.py on presentation.pdf [14:30:15] PDF REVIEW: Converted 25 slides to images in review/ directory [14:30:20] PDF REVIEW: Inspecting slide 1/25 – title slide [14:30:25] PDF REVIEW: Inspecting slide 2/25 – introduction … [14:35:40] PDF REVIEW: Inspecting slide 25/25 – acknowledgments [14:35:45] PDF REVIEW: Completed image-based review [14:35:50] PEER REVIEW: Found 8 layout issues, 3 content issues [14:35:55] PEER REVIEW: Generating structured feedback by slide number “` **Remember:** For presentations, the visual inspection via images is MANDATORY. Never attempt to read presentation PDFs as text – it will fail and miss all visual formatting issues. ## Resources This skill includes reference materials to support comprehensive peer review: ### references/reporting_standards.md Guidelines for major reporting standards across disciplines (CONSORT, PRISMA, ARRIVE, MIAME, STROBE, etc.) to evaluate completeness of methods and results reporting. ### references/common_issues.md Catalog of frequent methodological and statistical issues encountered in peer review, with guidance on identifying and addressing them. ## Final Checklist Before finalizing the review, verify: – [ ] Summary statement clearly conveys overall assessment – [ ] Major concerns are clearly identified and justified – [ ] Suggested revisions are specific and actionable – [ ] Minor issues are noted but properly categorized – [ ] Statistical methods have been evaluated – [ ] Reproducibility and data availability assessed – [ ] Ethical considerations verified – [ ] Figures and tables evaluated for quality and integrity – [ ] Writing quality assessed – [ ] Tone is constructive and professional throughout – [ ] Review is thorough but proportionate to manuscript scope – [ ] Recommendation is consistent with identified issues