Add skills tools and enhance model integration
- Introduced new skills tools: `skills_categories`, `skills_list`, and `skill_view` in `model_tools.py`, allowing for better organization and access to skill-related functionalities. - Updated `toolsets.py` to include a new `skills` toolset, providing a dedicated space for skill tools. - Enhanced `batch_runner.py` to recognize and validate skills tools during batch processing. - Added comprehensive tool definitions for skills tools, ensuring compatibility with OpenAI's expected format. - Created new shell script `test_skills_kimi.sh` for testing skills tool functionality with Kimi K2.5. - Added example skill files demonstrating the structure and usage of skills within the Hermes-Agent framework, including `SKILL.md` for example and audiocraft skills. - Improved documentation for skills tools and their integration into the existing tool framework, ensuring clarity for future development and usage.
This commit is contained in:
teknium
parent
8e8b6be690
commit
f172f7d4aa
189 changed files with 116214 additions and 2 deletions
361
skills/mlops/ml-paper-writing/references/checklists.md
Normal file
361
skills/mlops/ml-paper-writing/references/checklists.md
Normal file
|
|
@ -0,0 +1,361 @@
|
|||
# Conference Paper Checklists
|
||||
|
||||
This reference documents the mandatory checklist requirements for major ML/AI conferences. All major venues now require paper checklists—missing them results in desk rejection.
|
||||
|
||||
---
|
||||
|
||||
## Contents
|
||||
|
||||
- [NeurIPS Paper Checklist](#neurips-paper-checklist)
|
||||
- [ICML Paper Checklist](#icml-paper-checklist)
|
||||
- [ICLR Requirements](#iclr-requirements)
|
||||
- [ACL Requirements](#acl-requirements)
|
||||
- [Universal Pre-Submission Checklist](#universal-pre-submission-checklist)
|
||||
|
||||
---
|
||||
|
||||
## NeurIPS Paper Checklist
|
||||
|
||||
### Mandatory Components
|
||||
|
||||
All NeurIPS submissions must include a completed paper checklist. Papers lacking this element face **automatic desk rejection**. The checklist appears after references and supplemental material, outside the page limit.
|
||||
|
||||
### 16 Required Checklist Items
|
||||
|
||||
#### 1. Claims Alignment
|
||||
Authors must verify that abstract and introduction claims match theoretical and experimental results, with clearly stated contributions, assumptions, and limitations.
|
||||
|
||||
**What to check:**
|
||||
- [ ] Abstract claims match actual results
|
||||
- [ ] Introduction doesn't overclaim
|
||||
- [ ] Contributions are specific and falsifiable
|
||||
|
||||
#### 2. Limitations Discussion
|
||||
Papers should include a dedicated "Limitations" section addressing strong assumptions, robustness to violations, scope constraints, and performance-influencing factors.
|
||||
|
||||
**What to include:**
|
||||
- [ ] Dedicated Limitations section
|
||||
- [ ] Honest assessment of scope
|
||||
- [ ] Conditions where method may fail
|
||||
|
||||
#### 3. Theory & Proofs
|
||||
Theoretical contributions require full assumption statements and complete proofs (main paper or appendix with proof sketches for intuition).
|
||||
|
||||
**What to check:**
|
||||
- [ ] All assumptions stated formally
|
||||
- [ ] Complete proofs provided (main text or appendix)
|
||||
- [ ] Proof sketches for intuition in main text
|
||||
|
||||
#### 4. Reproducibility
|
||||
Authors must describe steps ensuring results verification through code release, detailed instructions, model access, or checkpoints appropriate to their contribution type.
|
||||
|
||||
**What to provide:**
|
||||
- [ ] Clear reproducibility statement
|
||||
- [ ] Code availability information
|
||||
- [ ] Model checkpoints if applicable
|
||||
|
||||
#### 5. Data & Code Access
|
||||
Instructions for reproducing main experimental results should be provided (supplemental material or URLs), including exact commands and environment specifications.
|
||||
|
||||
**What to include:**
|
||||
- [ ] Exact commands to run experiments
|
||||
- [ ] Environment specifications (requirements.txt, conda env)
|
||||
- [ ] Data access instructions
|
||||
|
||||
#### 6. Experimental Details
|
||||
Papers must specify training details: data splits, hyperparameters, and selection methods in the main paper or supplementary materials.
|
||||
|
||||
**What to document:**
|
||||
- [ ] Train/val/test split details
|
||||
- [ ] All hyperparameters used
|
||||
- [ ] Hyperparameter selection method
|
||||
|
||||
#### 7. Statistical Significance
|
||||
Results require error bars, confidence intervals, or statistical tests with clearly stated calculation methods and underlying assumptions.
|
||||
|
||||
**What to include:**
|
||||
- [ ] Error bars or confidence intervals
|
||||
- [ ] Number of runs/seeds
|
||||
- [ ] Calculation method (std dev vs std error)
|
||||
|
||||
#### 8. Compute Resources
|
||||
Specifications needed: compute worker types (CPU/GPU), memory, storage, execution time per run, and total project compute requirements.
|
||||
|
||||
**What to document:**
|
||||
- [ ] GPU type and count
|
||||
- [ ] Training time per run
|
||||
- [ ] Total compute used
|
||||
|
||||
#### 9. Ethics Code Compliance
|
||||
Authors confirm adherence to the NeurIPS Code of Ethics, noting any necessary deviations.
|
||||
|
||||
**What to verify:**
|
||||
- [ ] Read NeurIPS Code of Ethics
|
||||
- [ ] Confirm compliance
|
||||
- [ ] Note any deviations with justification
|
||||
|
||||
#### 10. Broader Impacts
|
||||
Discussion of potential negative societal applications, fairness concerns, privacy risks, and possible mitigation strategies when applicable.
|
||||
|
||||
**What to address:**
|
||||
- [ ] Potential negative applications
|
||||
- [ ] Fairness considerations
|
||||
- [ ] Privacy implications
|
||||
- [ ] Mitigation strategies
|
||||
|
||||
#### 11. Safeguards
|
||||
High-risk models (language models, internet-scraped datasets) require controlled release mechanisms and usage guidelines.
|
||||
|
||||
**What to consider:**
|
||||
- [ ] Release strategy for sensitive models
|
||||
- [ ] Usage guidelines if needed
|
||||
- [ ] Access controls if appropriate
|
||||
|
||||
#### 12. License Respect
|
||||
All existing assets require creator citations, license names, URLs, version numbers, and terms-of-service acknowledgment.
|
||||
|
||||
**What to document:**
|
||||
- [ ] Dataset licenses cited
|
||||
- [ ] Code licenses respected
|
||||
- [ ] Version numbers included
|
||||
|
||||
#### 13. Asset Documentation
|
||||
New releases need structured templates documenting training details, limitations, consent procedures, and licensing information.
|
||||
|
||||
**For new datasets/models:**
|
||||
- [ ] Datasheet or model card
|
||||
- [ ] Training data documentation
|
||||
- [ ] Known limitations
|
||||
|
||||
#### 14. Human Subjects
|
||||
Crowdsourcing studies must include participant instructions, screenshots, compensation details, and comply with minimum wage requirements.
|
||||
|
||||
**What to include:**
|
||||
- [ ] Task instructions
|
||||
- [ ] Compensation details
|
||||
- [ ] Time estimates
|
||||
|
||||
#### 15. IRB Approvals
|
||||
Human subjects research requires documented institutional review board approval or equivalent, with risk descriptions disclosed (maintaining anonymity at submission).
|
||||
|
||||
**What to verify:**
|
||||
- [ ] IRB approval obtained
|
||||
- [ ] Risk assessment completed
|
||||
- [ ] Anonymized at submission
|
||||
|
||||
#### 16. LLM Declaration
|
||||
Usage of large language models as core methodology components requires disclosure; writing/editing use doesn't require declaration.
|
||||
|
||||
**What to disclose:**
|
||||
- [ ] LLM used as core methodology component
|
||||
- [ ] How LLM was used
|
||||
- [ ] (Writing assistance doesn't require disclosure)
|
||||
|
||||
### Response Format
|
||||
|
||||
Authors select "yes," "no," or "N/A" per question, with optional 1-2 sentence justifications.
|
||||
|
||||
**Important:** Reviewers are explicitly instructed not to penalize honest limitation acknowledgment.
|
||||
|
||||
---
|
||||
|
||||
## ICML Paper Checklist
|
||||
|
||||
### Broader Impact Statement
|
||||
|
||||
ICML requires a Broader Impact Statement at the end of the paper, before references. This does NOT count toward the page limit.
|
||||
|
||||
**Required elements:**
|
||||
- Potential positive impacts
|
||||
- Potential negative impacts
|
||||
- Mitigation strategies
|
||||
- Who may be affected
|
||||
|
||||
### ICML Specific Requirements
|
||||
|
||||
#### Reproducibility Checklist
|
||||
|
||||
- [ ] Data splits clearly specified
|
||||
- [ ] Hyperparameters listed
|
||||
- [ ] Search ranges documented
|
||||
- [ ] Selection method explained
|
||||
- [ ] Compute resources specified
|
||||
- [ ] Code availability stated
|
||||
|
||||
#### Statistical Reporting
|
||||
|
||||
- [ ] Error bars on all figures
|
||||
- [ ] Standard deviation vs standard error specified
|
||||
- [ ] Number of runs stated
|
||||
- [ ] Significance tests if comparing methods
|
||||
|
||||
#### Anonymization
|
||||
|
||||
- [ ] No author names in paper
|
||||
- [ ] No acknowledgments
|
||||
- [ ] No grant numbers
|
||||
- [ ] Prior work cited in third person
|
||||
- [ ] No identifiable repository URLs
|
||||
|
||||
---
|
||||
|
||||
## ICLR Requirements
|
||||
|
||||
### LLM Disclosure Policy (New for 2026)
|
||||
|
||||
ICLR has a specific LLM disclosure requirement:
|
||||
|
||||
> "If LLMs played a significant role in research ideation and/or writing to the extent that they could be regarded as a contributor, authors must describe their precise role in a separate appendix section."
|
||||
|
||||
**When disclosure is required:**
|
||||
- LLM used for significant research ideation
|
||||
- LLM used for substantial writing
|
||||
- LLM could be considered a contributor
|
||||
|
||||
**When disclosure is NOT required:**
|
||||
- Grammar checking
|
||||
- Minor editing assistance
|
||||
- Code completion tools
|
||||
|
||||
**Consequences of non-disclosure:**
|
||||
- Desk rejection
|
||||
- Potential post-publication issues
|
||||
|
||||
### ICLR Specific Requirements
|
||||
|
||||
#### Reproducibility Statement (Optional but Recommended)
|
||||
|
||||
Add a statement referencing:
|
||||
- Supporting materials
|
||||
- Code availability
|
||||
- Data availability
|
||||
- Model checkpoints
|
||||
|
||||
#### Ethics Statement (Optional)
|
||||
|
||||
Address potential concerns in ≤1 page. Does not count toward page limit.
|
||||
|
||||
#### Reciprocal Reviewing
|
||||
|
||||
- Authors on 3+ papers must serve as reviewers for ≥6 papers
|
||||
- Each submission needs ≥1 author registered to review ≥3 papers
|
||||
|
||||
---
|
||||
|
||||
## ACL Requirements
|
||||
|
||||
### Limitations Section (Mandatory)
|
||||
|
||||
ACL specifically requires a Limitations section:
|
||||
|
||||
**What to include:**
|
||||
- Strong assumptions made
|
||||
- Scope limitations
|
||||
- When method may fail
|
||||
- Generalization concerns
|
||||
|
||||
**Important:** The Limitations section does NOT count toward the page limit.
|
||||
|
||||
### ACL Specific Checklist
|
||||
|
||||
#### Responsible NLP
|
||||
|
||||
- [ ] Bias considerations addressed
|
||||
- [ ] Fairness evaluated if applicable
|
||||
- [ ] Dual-use concerns discussed
|
||||
|
||||
#### Multilingual Considerations
|
||||
|
||||
If applicable:
|
||||
- [ ] Language diversity addressed
|
||||
- [ ] Non-English languages included
|
||||
- [ ] Translation quality verified
|
||||
|
||||
#### Human Evaluation
|
||||
|
||||
If applicable:
|
||||
- [ ] Annotator details provided
|
||||
- [ ] Agreement metrics reported
|
||||
- [ ] Compensation documented
|
||||
|
||||
---
|
||||
|
||||
## Universal Pre-Submission Checklist
|
||||
|
||||
### Before Every Submission
|
||||
|
||||
#### Paper Content
|
||||
|
||||
- [ ] Abstract ≤ word limit (usually 250-300 words)
|
||||
- [ ] Main content within page limit
|
||||
- [ ] References complete and verified
|
||||
- [ ] Limitations section included
|
||||
- [ ] All figures/tables have captions
|
||||
- [ ] Captions are self-contained
|
||||
|
||||
#### Formatting
|
||||
|
||||
- [ ] Correct template used (venue + year specific)
|
||||
- [ ] Margins not modified
|
||||
- [ ] Font sizes not modified
|
||||
- [ ] Double-blind requirements met
|
||||
- [ ] Page numbers (for review) or none (camera-ready)
|
||||
|
||||
#### Technical
|
||||
|
||||
- [ ] All claims supported by evidence
|
||||
- [ ] Error bars included
|
||||
- [ ] Baselines appropriate
|
||||
- [ ] Hyperparameters documented
|
||||
- [ ] Compute resources stated
|
||||
|
||||
#### Reproducibility
|
||||
|
||||
- [ ] Code will be available (or justification)
|
||||
- [ ] Data will be available (or justification)
|
||||
- [ ] Environment documented
|
||||
- [ ] Commands to reproduce provided
|
||||
|
||||
#### Ethics
|
||||
|
||||
- [ ] Broader impacts considered
|
||||
- [ ] Limitations honestly stated
|
||||
- [ ] Licenses respected
|
||||
- [ ] IRB obtained if needed
|
||||
|
||||
#### Final Checks
|
||||
|
||||
- [ ] PDF compiles without errors
|
||||
- [ ] All figures render correctly
|
||||
- [ ] All citations resolve
|
||||
- [ ] Supplementary material organized
|
||||
- [ ] Conference checklist completed
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference: Page Limits
|
||||
|
||||
| Conference | Main Content | References | Appendix |
|
||||
|------------|-------------|------------|----------|
|
||||
| NeurIPS 2025 | 9 pages | Unlimited | Unlimited (checklist separate) |
|
||||
| ICML 2026 | 8 pages (+1 camera) | Unlimited | Unlimited |
|
||||
| ICLR 2026 | 9 pages (+1 camera) | Unlimited | Unlimited |
|
||||
| ACL 2025 | 8 pages (long) | Unlimited | Unlimited |
|
||||
| AAAI 2026 | 7 pages (+1 camera) | Unlimited | Unlimited |
|
||||
| COLM 2025 | 9 pages (+1 camera) | Unlimited | Unlimited |
|
||||
|
||||
---
|
||||
|
||||
## Template Locations
|
||||
|
||||
All conference templates are in the `templates/` directory:
|
||||
|
||||
```
|
||||
templates/
|
||||
├── icml2026/ # ICML 2026 official
|
||||
├── iclr2026/ # ICLR 2026 official
|
||||
├── neurips2025/ # NeurIPS 2025
|
||||
├── acl/ # ACL style files
|
||||
├── aaai2026/ # AAAI 2026
|
||||
└── colm2025/ # COLM 2025
|
||||
```
|
||||
562
skills/mlops/ml-paper-writing/references/citation-workflow.md
Normal file
562
skills/mlops/ml-paper-writing/references/citation-workflow.md
Normal file
|
|
@ -0,0 +1,562 @@
|
|||
# Citation Management & Hallucination Prevention
|
||||
|
||||
This reference provides a complete workflow for managing citations programmatically, preventing AI-generated citation hallucinations, and maintaining clean bibliographies.
|
||||
|
||||
---
|
||||
|
||||
## Contents
|
||||
|
||||
- [Why Citation Verification Matters](#why-citation-verification-matters)
|
||||
- [Citation APIs Overview](#citation-apis-overview)
|
||||
- [Verified Citation Workflow](#verified-citation-workflow)
|
||||
- [Python Implementation](#python-implementation)
|
||||
- [BibTeX Management](#bibtex-management)
|
||||
- [Common Citation Formats](#common-citation-formats)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
|
||||
---
|
||||
|
||||
## Why Citation Verification Matters
|
||||
|
||||
### The Hallucination Problem
|
||||
|
||||
Research has documented significant issues with AI-generated citations:
|
||||
- **~40% error rate** in AI-generated citations (Enago Academy research)
|
||||
- NeurIPS 2025 found **100+ hallucinated citations** slipped through review
|
||||
- Common errors include:
|
||||
- Fabricated paper titles with real author names
|
||||
- Wrong publication venues or years
|
||||
- Non-existent papers with plausible metadata
|
||||
- Incorrect DOIs or arXiv IDs
|
||||
|
||||
### Consequences
|
||||
|
||||
- Desk rejection at some venues
|
||||
- Loss of credibility with reviewers
|
||||
- Potential retraction if published
|
||||
- Wasted time chasing non-existent sources
|
||||
|
||||
### Solution
|
||||
|
||||
**Never generate citations from memory—always verify programmatically.**
|
||||
|
||||
---
|
||||
|
||||
## Citation APIs Overview
|
||||
|
||||
### Primary APIs
|
||||
|
||||
| API | Coverage | Rate Limits | Best For |
|
||||
|-----|----------|-------------|----------|
|
||||
| **Semantic Scholar** | 214M papers | 1 RPS (free key) | ML/AI papers, citation graphs |
|
||||
| **CrossRef** | 140M+ DOIs | Polite pool with mailto | DOI lookup, BibTeX retrieval |
|
||||
| **arXiv** | Preprints | 3-second delays | ML preprints, PDF access |
|
||||
| **OpenAlex** | 240M+ works | 100K/day, 10 RPS | Open alternative to MAG |
|
||||
|
||||
### API Selection Guide
|
||||
|
||||
```
|
||||
Need ML paper search? → Semantic Scholar
|
||||
Have DOI, need BibTeX? → CrossRef content negotiation
|
||||
Looking for preprint? → arXiv API
|
||||
Need open data, bulk access? → OpenAlex
|
||||
```
|
||||
|
||||
### No Official Google Scholar API
|
||||
|
||||
Google Scholar has no official API. Scraping violates ToS. Use SerpApi ($75-275/month) only if Semantic Scholar coverage is insufficient.
|
||||
|
||||
---
|
||||
|
||||
## Verified Citation Workflow
|
||||
|
||||
### 5-Step Process
|
||||
|
||||
```
|
||||
1. SEARCH → Query Semantic Scholar with specific keywords
|
||||
↓
|
||||
2. VERIFY → Confirm paper exists in 2+ sources
|
||||
↓
|
||||
3. RETRIEVE → Get BibTeX via DOI content negotiation
|
||||
↓
|
||||
4. VALIDATE → Confirm the claim appears in source
|
||||
↓
|
||||
5. ADD → Add verified entry to .bib file
|
||||
```
|
||||
|
||||
### Step 1: Search
|
||||
|
||||
Use Semantic Scholar for ML/AI papers:
|
||||
|
||||
```python
|
||||
from semanticscholar import SemanticScholar
|
||||
|
||||
sch = SemanticScholar()
|
||||
results = sch.search_paper("transformer attention mechanism", limit=10)
|
||||
|
||||
for paper in results:
|
||||
print(f"Title: {paper.title}")
|
||||
print(f"Year: {paper.year}")
|
||||
print(f"DOI: {paper.externalIds.get('DOI', 'N/A')}")
|
||||
print(f"arXiv: {paper.externalIds.get('ArXiv', 'N/A')}")
|
||||
print(f"Citation count: {paper.citationCount}")
|
||||
print("---")
|
||||
```
|
||||
|
||||
### Step 2: Verify Existence
|
||||
|
||||
Confirm paper exists in at least two sources:
|
||||
|
||||
```python
|
||||
import requests
|
||||
|
||||
def verify_paper(doi=None, arxiv_id=None, title=None):
|
||||
"""Verify paper exists in multiple sources."""
|
||||
sources_found = []
|
||||
|
||||
# Check Semantic Scholar
|
||||
sch = SemanticScholar()
|
||||
if doi:
|
||||
paper = sch.get_paper(f"DOI:{doi}")
|
||||
if paper:
|
||||
sources_found.append("Semantic Scholar")
|
||||
|
||||
# Check CrossRef (via DOI)
|
||||
if doi:
|
||||
resp = requests.get(f"https://api.crossref.org/works/{doi}")
|
||||
if resp.status_code == 200:
|
||||
sources_found.append("CrossRef")
|
||||
|
||||
# Check arXiv
|
||||
if arxiv_id:
|
||||
resp = requests.get(
|
||||
f"http://export.arxiv.org/api/query?id_list={arxiv_id}"
|
||||
)
|
||||
if "<entry>" in resp.text:
|
||||
sources_found.append("arXiv")
|
||||
|
||||
return len(sources_found) >= 2, sources_found
|
||||
```
|
||||
|
||||
### Step 3: Retrieve BibTeX
|
||||
|
||||
Use DOI content negotiation for guaranteed accuracy:
|
||||
|
||||
```python
|
||||
import requests
|
||||
|
||||
def doi_to_bibtex(doi: str) -> str:
|
||||
"""Get verified BibTeX from DOI via CrossRef content negotiation."""
|
||||
response = requests.get(
|
||||
f"https://doi.org/{doi}",
|
||||
headers={"Accept": "application/x-bibtex"},
|
||||
allow_redirects=True
|
||||
)
|
||||
response.raise_for_status()
|
||||
return response.text
|
||||
|
||||
# Example: "Attention Is All You Need"
|
||||
bibtex = doi_to_bibtex("10.48550/arXiv.1706.03762")
|
||||
print(bibtex)
|
||||
```
|
||||
|
||||
### Step 4: Validate Claims
|
||||
|
||||
Before citing a paper for a specific claim, verify the claim exists:
|
||||
|
||||
```python
|
||||
def get_paper_abstract(doi):
|
||||
"""Get abstract to verify claims."""
|
||||
sch = SemanticScholar()
|
||||
paper = sch.get_paper(f"DOI:{doi}")
|
||||
return paper.abstract if paper else None
|
||||
|
||||
# Verify claim appears in abstract
|
||||
abstract = get_paper_abstract("10.48550/arXiv.1706.03762")
|
||||
claim = "attention mechanism"
|
||||
if claim.lower() in abstract.lower():
|
||||
print("Claim appears in paper")
|
||||
```
|
||||
|
||||
### Step 5: Add to Bibliography
|
||||
|
||||
Add verified entry to your .bib file with consistent key format:
|
||||
|
||||
```python
|
||||
def generate_citation_key(bibtex: str) -> str:
|
||||
"""Generate consistent citation key: author_year_firstword."""
|
||||
import re
|
||||
|
||||
# Extract author
|
||||
author_match = re.search(r'author\s*=\s*\{([^}]+)\}', bibtex, re.I)
|
||||
if author_match:
|
||||
first_author = author_match.group(1).split(',')[0].split()[-1]
|
||||
else:
|
||||
first_author = "unknown"
|
||||
|
||||
# Extract year
|
||||
year_match = re.search(r'year\s*=\s*\{?(\d{4})\}?', bibtex, re.I)
|
||||
year = year_match.group(1) if year_match else "0000"
|
||||
|
||||
# Extract title first word
|
||||
title_match = re.search(r'title\s*=\s*\{([^}]+)\}', bibtex, re.I)
|
||||
if title_match:
|
||||
first_word = title_match.group(1).split()[0].lower()
|
||||
first_word = re.sub(r'[^a-z]', '', first_word)
|
||||
else:
|
||||
first_word = "paper"
|
||||
|
||||
return f"{first_author.lower()}_{year}_{first_word}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Python Implementation
|
||||
|
||||
### Complete Citation Manager Class
|
||||
|
||||
```python
|
||||
"""
|
||||
Citation Manager - Verified citation workflow for ML papers.
|
||||
"""
|
||||
|
||||
import requests
|
||||
import time
|
||||
from typing import Optional, List, Dict, Tuple
|
||||
from dataclasses import dataclass
|
||||
|
||||
try:
|
||||
from semanticscholar import SemanticScholar
|
||||
except ImportError:
|
||||
print("Install: pip install semanticscholar")
|
||||
SemanticScholar = None
|
||||
|
||||
@dataclass
|
||||
class Paper:
|
||||
title: str
|
||||
authors: List[str]
|
||||
year: int
|
||||
doi: Optional[str]
|
||||
arxiv_id: Optional[str]
|
||||
venue: Optional[str]
|
||||
citation_count: int
|
||||
abstract: Optional[str]
|
||||
|
||||
class CitationManager:
|
||||
"""Manage citations with verification."""
|
||||
|
||||
def __init__(self, api_key: Optional[str] = None):
|
||||
self.sch = SemanticScholar(api_key=api_key) if SemanticScholar else None
|
||||
self.verified_papers: Dict[str, Paper] = {}
|
||||
|
||||
def search(self, query: str, limit: int = 10) -> List[Paper]:
|
||||
"""Search for papers using Semantic Scholar."""
|
||||
if not self.sch:
|
||||
raise RuntimeError("Semantic Scholar not available")
|
||||
|
||||
results = self.sch.search_paper(query, limit=limit)
|
||||
papers = []
|
||||
|
||||
for r in results:
|
||||
paper = Paper(
|
||||
title=r.title,
|
||||
authors=[a.name for a in (r.authors or [])],
|
||||
year=r.year or 0,
|
||||
doi=r.externalIds.get('DOI') if r.externalIds else None,
|
||||
arxiv_id=r.externalIds.get('ArXiv') if r.externalIds else None,
|
||||
venue=r.venue,
|
||||
citation_count=r.citationCount or 0,
|
||||
abstract=r.abstract
|
||||
)
|
||||
papers.append(paper)
|
||||
|
||||
return papers
|
||||
|
||||
def verify(self, paper: Paper) -> Tuple[bool, List[str]]:
|
||||
"""Verify paper exists in multiple sources."""
|
||||
sources = []
|
||||
|
||||
# Already found in Semantic Scholar via search
|
||||
sources.append("Semantic Scholar")
|
||||
|
||||
# Check CrossRef if DOI available
|
||||
if paper.doi:
|
||||
try:
|
||||
resp = requests.get(
|
||||
f"https://api.crossref.org/works/{paper.doi}",
|
||||
timeout=10
|
||||
)
|
||||
if resp.status_code == 200:
|
||||
sources.append("CrossRef")
|
||||
except:
|
||||
pass
|
||||
|
||||
# Check arXiv if ID available
|
||||
if paper.arxiv_id:
|
||||
try:
|
||||
resp = requests.get(
|
||||
f"http://export.arxiv.org/api/query?id_list={paper.arxiv_id}",
|
||||
timeout=10
|
||||
)
|
||||
if "<entry>" in resp.text and "<title>" in resp.text:
|
||||
sources.append("arXiv")
|
||||
except:
|
||||
pass
|
||||
|
||||
return len(sources) >= 2, sources
|
||||
|
||||
def get_bibtex(self, paper: Paper) -> Optional[str]:
|
||||
"""Get BibTeX for verified paper."""
|
||||
if paper.doi:
|
||||
try:
|
||||
resp = requests.get(
|
||||
f"https://doi.org/{paper.doi}",
|
||||
headers={"Accept": "application/x-bibtex"},
|
||||
timeout=10,
|
||||
allow_redirects=True
|
||||
)
|
||||
if resp.status_code == 200:
|
||||
return resp.text
|
||||
except:
|
||||
pass
|
||||
|
||||
# Fallback: generate from paper data
|
||||
return self._generate_bibtex(paper)
|
||||
|
||||
def _generate_bibtex(self, paper: Paper) -> str:
|
||||
"""Generate BibTeX from paper metadata."""
|
||||
# Generate citation key
|
||||
first_author = paper.authors[0].split()[-1] if paper.authors else "unknown"
|
||||
first_word = paper.title.split()[0].lower().replace(',', '').replace(':', '')
|
||||
key = f"{first_author.lower()}_{paper.year}_{first_word}"
|
||||
|
||||
# Format authors
|
||||
authors = " and ".join(paper.authors) if paper.authors else "Unknown"
|
||||
|
||||
bibtex = f"""@article{{{key},
|
||||
title = {{{paper.title}}},
|
||||
author = {{{authors}}},
|
||||
year = {{{paper.year}}},
|
||||
{'doi = {' + paper.doi + '},' if paper.doi else ''}
|
||||
{'eprint = {' + paper.arxiv_id + '},' if paper.arxiv_id else ''}
|
||||
{'journal = {' + paper.venue + '},' if paper.venue else ''}
|
||||
}}"""
|
||||
return bibtex
|
||||
|
||||
def cite(self, query: str) -> Optional[str]:
|
||||
"""Full workflow: search, verify, return BibTeX."""
|
||||
# Search
|
||||
papers = self.search(query, limit=5)
|
||||
if not papers:
|
||||
return None
|
||||
|
||||
# Take top result
|
||||
paper = papers[0]
|
||||
|
||||
# Verify
|
||||
verified, sources = self.verify(paper)
|
||||
if not verified:
|
||||
print(f"Warning: Could only verify in {sources}")
|
||||
|
||||
# Get BibTeX
|
||||
bibtex = self.get_bibtex(paper)
|
||||
|
||||
# Cache
|
||||
if bibtex:
|
||||
self.verified_papers[paper.title] = paper
|
||||
|
||||
return bibtex
|
||||
|
||||
|
||||
# Usage example
|
||||
if __name__ == "__main__":
|
||||
cm = CitationManager()
|
||||
|
||||
# Search and cite
|
||||
bibtex = cm.cite("attention is all you need transformer")
|
||||
if bibtex:
|
||||
print(bibtex)
|
||||
```
|
||||
|
||||
### Quick Functions
|
||||
|
||||
```python
|
||||
def quick_cite(query: str) -> str:
|
||||
"""One-liner citation."""
|
||||
cm = CitationManager()
|
||||
return cm.cite(query)
|
||||
|
||||
def batch_cite(queries: List[str], output_file: str = "references.bib"):
|
||||
"""Cite multiple papers and save to file."""
|
||||
cm = CitationManager()
|
||||
bibtex_entries = []
|
||||
|
||||
for query in queries:
|
||||
print(f"Processing: {query}")
|
||||
bibtex = cm.cite(query)
|
||||
if bibtex:
|
||||
bibtex_entries.append(bibtex)
|
||||
time.sleep(1) # Rate limiting
|
||||
|
||||
with open(output_file, 'w') as f:
|
||||
f.write("\n\n".join(bibtex_entries))
|
||||
|
||||
print(f"Saved {len(bibtex_entries)} citations to {output_file}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## BibTeX Management
|
||||
|
||||
### BibTeX vs BibLaTeX
|
||||
|
||||
| Feature | BibTeX | BibLaTeX |
|
||||
|---------|--------|----------|
|
||||
| Unicode support | Limited | Full |
|
||||
| Entry types | Standard | Extended (@online, @dataset) |
|
||||
| Customization | Limited | Highly flexible |
|
||||
| Backend | bibtex | Biber (recommended) |
|
||||
|
||||
**Recommendation**: Use BibLaTeX with Biber for new papers.
|
||||
|
||||
### LaTeX Setup
|
||||
|
||||
```latex
|
||||
% In preamble
|
||||
\usepackage[
|
||||
backend=biber,
|
||||
style=numeric,
|
||||
sorting=none
|
||||
]{biblatex}
|
||||
\addbibresource{references.bib}
|
||||
|
||||
% In document
|
||||
\cite{vaswani_2017_attention}
|
||||
|
||||
% At end
|
||||
\printbibliography
|
||||
```
|
||||
|
||||
### Citation Commands
|
||||
|
||||
```latex
|
||||
\cite{key} % Numeric: [1]
|
||||
\citep{key} % Parenthetical: (Author, 2020)
|
||||
\citet{key} % Textual: Author (2020)
|
||||
\citeauthor{key} % Just author name
|
||||
\citeyear{key} % Just year
|
||||
```
|
||||
|
||||
### Consistent Citation Keys
|
||||
|
||||
Use format: `author_year_firstword`
|
||||
|
||||
```
|
||||
vaswani_2017_attention
|
||||
devlin_2019_bert
|
||||
brown_2020_language
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Citation Formats
|
||||
|
||||
### Conference Paper
|
||||
|
||||
```bibtex
|
||||
@inproceedings{vaswani_2017_attention,
|
||||
title = {Attention Is All You Need},
|
||||
author = {Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and
|
||||
Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and
|
||||
Kaiser, Lukasz and Polosukhin, Illia},
|
||||
booktitle = {Advances in Neural Information Processing Systems},
|
||||
volume = {30},
|
||||
year = {2017},
|
||||
publisher = {Curran Associates, Inc.}
|
||||
}
|
||||
```
|
||||
|
||||
### Journal Article
|
||||
|
||||
```bibtex
|
||||
@article{hochreiter_1997_long,
|
||||
title = {Long Short-Term Memory},
|
||||
author = {Hochreiter, Sepp and Schmidhuber, J{\"u}rgen},
|
||||
journal = {Neural Computation},
|
||||
volume = {9},
|
||||
number = {8},
|
||||
pages = {1735--1780},
|
||||
year = {1997},
|
||||
publisher = {MIT Press}
|
||||
}
|
||||
```
|
||||
|
||||
### arXiv Preprint
|
||||
|
||||
```bibtex
|
||||
@misc{brown_2020_language,
|
||||
title = {Language Models are Few-Shot Learners},
|
||||
author = {Brown, Tom and Mann, Benjamin and Ryder, Nick and others},
|
||||
year = {2020},
|
||||
eprint = {2005.14165},
|
||||
archiveprefix = {arXiv},
|
||||
primaryclass = {cs.CL}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Issue: Semantic Scholar returns no results**
|
||||
- Try more specific keywords
|
||||
- Check spelling of author names
|
||||
- Use quotation marks for exact phrases
|
||||
|
||||
**Issue: DOI doesn't resolve to BibTeX**
|
||||
- DOI may be registered but not linked to CrossRef
|
||||
- Try arXiv ID instead if available
|
||||
- Generate BibTeX from metadata manually
|
||||
|
||||
**Issue: Rate limiting errors**
|
||||
- Add delays between requests (1-3 seconds)
|
||||
- Use API key if available
|
||||
- Cache results to avoid repeat queries
|
||||
|
||||
**Issue: Encoding problems in BibTeX**
|
||||
- Use proper LaTeX escaping: `{\"u}` for ü
|
||||
- Ensure file is UTF-8 encoded
|
||||
- Use BibLaTeX with Biber for better Unicode
|
||||
|
||||
### Verification Checklist
|
||||
|
||||
Before adding a citation:
|
||||
|
||||
- [ ] Paper found in at least 2 sources
|
||||
- [ ] DOI or arXiv ID verified
|
||||
- [ ] BibTeX retrieved (not generated from memory)
|
||||
- [ ] Entry type correct (@inproceedings vs @article)
|
||||
- [ ] Author names complete and correctly formatted
|
||||
- [ ] Year and venue verified
|
||||
- [ ] Citation key follows consistent format
|
||||
|
||||
---
|
||||
|
||||
## Additional Resources
|
||||
|
||||
**APIs:**
|
||||
- Semantic Scholar: https://api.semanticscholar.org/api-docs/
|
||||
- CrossRef: https://www.crossref.org/documentation/retrieve-metadata/rest-api/
|
||||
- arXiv: https://info.arxiv.org/help/api/basics.html
|
||||
- OpenAlex: https://docs.openalex.org/
|
||||
|
||||
**Python Libraries:**
|
||||
- `semanticscholar`: https://pypi.org/project/semanticscholar/
|
||||
- `arxiv`: https://pypi.org/project/arxiv/
|
||||
- `habanero` (CrossRef): https://github.com/sckott/habanero
|
||||
|
||||
**Verification Tools:**
|
||||
- Citely: https://citely.ai/citation-checker
|
||||
- ReciteWorks: https://reciteworks.com/
|
||||
367
skills/mlops/ml-paper-writing/references/reviewer-guidelines.md
Normal file
367
skills/mlops/ml-paper-writing/references/reviewer-guidelines.md
Normal file
|
|
@ -0,0 +1,367 @@
|
|||
# Reviewer Guidelines & Evaluation Criteria
|
||||
|
||||
This reference documents how reviewers evaluate papers at major ML/AI conferences, helping authors anticipate and address reviewer concerns.
|
||||
|
||||
---
|
||||
|
||||
## Contents
|
||||
|
||||
- [Universal Evaluation Dimensions](#universal-evaluation-dimensions)
|
||||
- [NeurIPS Reviewer Guidelines](#neurips-reviewer-guidelines)
|
||||
- [ICML Reviewer Guidelines](#icml-reviewer-guidelines)
|
||||
- [ICLR Reviewer Guidelines](#iclr-reviewer-guidelines)
|
||||
- [ACL Reviewer Guidelines](#acl-reviewer-guidelines)
|
||||
- [What Makes Reviews Strong](#what-makes-reviews-strong)
|
||||
- [Common Reviewer Concerns](#common-reviewer-concerns)
|
||||
- [How to Address Reviewer Feedback](#how-to-address-reviewer-feedback)
|
||||
|
||||
---
|
||||
|
||||
## Universal Evaluation Dimensions
|
||||
|
||||
All major ML conferences assess papers across four core dimensions:
|
||||
|
||||
### 1. Quality (Technical Soundness)
|
||||
|
||||
**What reviewers ask:**
|
||||
- Are claims well-supported by theoretical analysis or experimental results?
|
||||
- Are the proofs correct? Are the experiments properly controlled?
|
||||
- Are baselines appropriate and fairly compared?
|
||||
- Is the methodology sound?
|
||||
|
||||
**How to ensure high quality:**
|
||||
- Include complete proofs (main paper or appendix with sketches)
|
||||
- Use appropriate baselines (not strawmen)
|
||||
- Report variance/error bars with methodology
|
||||
- Document hyperparameter selection process
|
||||
|
||||
### 2. Clarity (Writing & Organization)
|
||||
|
||||
**What reviewers ask:**
|
||||
- Is the paper clearly written and well organized?
|
||||
- Can an expert in the field reproduce the results?
|
||||
- Is notation consistent? Are terms defined?
|
||||
- Is the paper self-contained?
|
||||
|
||||
**How to ensure clarity:**
|
||||
- Use consistent terminology throughout
|
||||
- Define all notation at first use
|
||||
- Include reproducibility details (appendix acceptable)
|
||||
- Have non-authors read before submission
|
||||
|
||||
### 3. Significance (Impact & Importance)
|
||||
|
||||
**What reviewers ask:**
|
||||
- Are the results impactful for the community?
|
||||
- Will others build upon this work?
|
||||
- Does it address an important problem?
|
||||
- What is the potential for real-world impact?
|
||||
|
||||
**How to demonstrate significance:**
|
||||
- Clearly articulate the problem's importance
|
||||
- Connect to broader research themes
|
||||
- Discuss potential applications
|
||||
- Compare to existing approaches meaningfully
|
||||
|
||||
### 4. Originality (Novelty & Contribution)
|
||||
|
||||
**What reviewers ask:**
|
||||
- Does this provide new insights?
|
||||
- How does it differ from prior work?
|
||||
- Is the contribution non-trivial?
|
||||
|
||||
**Key insight from NeurIPS guidelines:**
|
||||
> "Originality does not necessarily require introducing an entirely new method. Papers that provide novel insights from evaluating existing approaches or shed light on why methods succeed can also be highly original."
|
||||
|
||||
---
|
||||
|
||||
## NeurIPS Reviewer Guidelines
|
||||
|
||||
### Scoring System (1-6 Scale)
|
||||
|
||||
| Score | Label | Description |
|
||||
|-------|-------|-------------|
|
||||
| **6** | Strong Accept | Groundbreaking, flawless work; top 2-3% of submissions |
|
||||
| **5** | Accept | Technically solid, high impact; would benefit the community |
|
||||
| **4** | Borderline Accept | Solid work with limited evaluation; leans accept |
|
||||
| **3** | Borderline Reject | Solid but weaknesses outweigh strengths; leans reject |
|
||||
| **2** | Reject | Technical flaws or weak evaluation |
|
||||
| **1** | Strong Reject | Well-known results or unaddressed ethics concerns |
|
||||
|
||||
### Reviewer Instructions
|
||||
|
||||
Reviewers are explicitly instructed to:
|
||||
|
||||
1. **Evaluate the paper as written** - not what it could be with revisions
|
||||
2. **Provide constructive feedback** - 3-5 actionable points
|
||||
3. **Not penalize honest limitations** - acknowledging weaknesses is encouraged
|
||||
4. **Assess reproducibility** - can the work be verified?
|
||||
5. **Consider ethical implications** - potential misuse or harm
|
||||
|
||||
### What Reviewers Should Avoid
|
||||
|
||||
- Superficial, uninformed reviews
|
||||
- Demanding unreasonable additional experiments
|
||||
- Penalizing authors for honest limitation acknowledgment
|
||||
- Rejecting for missing citations to reviewer's own work
|
||||
|
||||
### Timeline (NeurIPS 2025)
|
||||
|
||||
- Bidding: May 17-21
|
||||
- Reviewing period: May 29 - July 2
|
||||
- Author rebuttals: July 24-30
|
||||
- Discussion period: July 31 - August 13
|
||||
- Final notifications: September 18
|
||||
|
||||
---
|
||||
|
||||
## ICML Reviewer Guidelines
|
||||
|
||||
### Review Structure
|
||||
|
||||
ICML reviewers provide:
|
||||
|
||||
1. **Summary** - Brief description of contributions
|
||||
2. **Strengths** - Positive aspects
|
||||
3. **Weaknesses** - Areas for improvement
|
||||
4. **Questions** - Clarifications for authors
|
||||
5. **Limitations** - Assessment of stated limitations
|
||||
6. **Ethics** - Any concerns
|
||||
7. **Overall Score** - Recommendation
|
||||
|
||||
### Scoring Guidelines
|
||||
|
||||
ICML uses a similar 1-6 scale with calibration:
|
||||
- Top 25% of accepted papers: Score 5-6
|
||||
- Typical accepted paper: Score 4-5
|
||||
- Borderline: Score 3-4
|
||||
- Clear reject: Score 1-2
|
||||
|
||||
### Key Evaluation Points
|
||||
|
||||
1. **Reproducibility** - Are there enough details?
|
||||
2. **Experimental rigor** - Multiple seeds, proper baselines?
|
||||
3. **Writing quality** - Clear, organized, well-structured?
|
||||
4. **Novelty** - Non-trivial contribution?
|
||||
|
||||
---
|
||||
|
||||
## ICLR Reviewer Guidelines
|
||||
|
||||
### OpenReview Process
|
||||
|
||||
ICLR uses OpenReview with:
|
||||
- Public reviews (after acceptance decisions)
|
||||
- Author responses visible to reviewers
|
||||
- Discussion between reviewers and ACs
|
||||
|
||||
### Scoring
|
||||
|
||||
ICLR reviews include:
|
||||
- **Soundness**: 1-4 scale
|
||||
- **Presentation**: 1-4 scale
|
||||
- **Contribution**: 1-4 scale
|
||||
- **Overall**: 1-10 scale
|
||||
- **Confidence**: 1-5 scale
|
||||
|
||||
### Unique ICLR Considerations
|
||||
|
||||
1. **LLM Disclosure** - Reviewers assess whether LLM use is properly disclosed
|
||||
2. **Reproducibility** - Emphasis on code availability
|
||||
3. **Reciprocal Reviewing** - Authors must also serve as reviewers
|
||||
|
||||
---
|
||||
|
||||
## ACL Reviewer Guidelines
|
||||
|
||||
### ACL-Specific Criteria
|
||||
|
||||
ACL adds NLP-specific evaluation:
|
||||
|
||||
1. **Linguistic soundness** - Are linguistic claims accurate?
|
||||
2. **Resource documentation** - Are datasets/models properly documented?
|
||||
3. **Multilingual consideration** - If applicable, is language diversity addressed?
|
||||
|
||||
### Limitations Section
|
||||
|
||||
ACL specifically requires a Limitations section. Reviewers check:
|
||||
- Are limitations honest and comprehensive?
|
||||
- Do limitations undermine core claims?
|
||||
- Are potential negative impacts addressed?
|
||||
|
||||
### Ethics Review
|
||||
|
||||
ACL has a dedicated ethics review process for:
|
||||
- Dual-use concerns
|
||||
- Data privacy issues
|
||||
- Bias and fairness implications
|
||||
|
||||
---
|
||||
|
||||
## What Makes Reviews Strong
|
||||
|
||||
### Following Daniel Dennett's Rules
|
||||
|
||||
Good reviewers follow these principles:
|
||||
|
||||
1. **Re-express the position fairly** - Show you understand the paper
|
||||
2. **List agreements** - Acknowledge what works well
|
||||
3. **List what you learned** - Credit the contribution
|
||||
4. **Only then critique** - After establishing understanding
|
||||
|
||||
### Review Structure Best Practices
|
||||
|
||||
**Strong Review Structure:**
|
||||
```
|
||||
Summary (1 paragraph):
|
||||
- What the paper does
|
||||
- Main contribution claimed
|
||||
|
||||
Strengths (3-5 bullets):
|
||||
- Specific positive aspects
|
||||
- Why these matter
|
||||
|
||||
Weaknesses (3-5 bullets):
|
||||
- Specific concerns
|
||||
- Why these matter
|
||||
- Suggestions for addressing
|
||||
|
||||
Questions (2-4 items):
|
||||
- Clarifications needed
|
||||
- Things that would change assessment
|
||||
|
||||
Minor Issues (optional):
|
||||
- Typos, unclear sentences
|
||||
- Formatting issues
|
||||
|
||||
Overall Assessment:
|
||||
- Clear recommendation with reasoning
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Reviewer Concerns
|
||||
|
||||
### Technical Concerns
|
||||
|
||||
| Concern | How to Pre-empt |
|
||||
|---------|-----------------|
|
||||
| "Baselines too weak" | Use state-of-the-art baselines, cite recent work |
|
||||
| "Missing ablations" | Include systematic ablation study |
|
||||
| "No error bars" | Report std dev/error, multiple runs |
|
||||
| "Hyperparameters not tuned" | Document tuning process, search ranges |
|
||||
| "Claims not supported" | Ensure every claim has evidence |
|
||||
|
||||
### Novelty Concerns
|
||||
|
||||
| Concern | How to Pre-empt |
|
||||
|---------|-----------------|
|
||||
| "Incremental contribution" | Clearly articulate what's new vs prior work |
|
||||
| "Similar to [paper X]" | Explicitly compare to X in Related Work |
|
||||
| "Straightforward extension" | Highlight non-obvious aspects |
|
||||
|
||||
### Clarity Concerns
|
||||
|
||||
| Concern | How to Pre-empt |
|
||||
|---------|-----------------|
|
||||
| "Hard to follow" | Use clear structure, signposting |
|
||||
| "Notation inconsistent" | Review all notation, create notation table |
|
||||
| "Missing details" | Include reproducibility appendix |
|
||||
| "Figures unclear" | Self-contained captions, proper sizing |
|
||||
|
||||
### Significance Concerns
|
||||
|
||||
| Concern | How to Pre-empt |
|
||||
|---------|-----------------|
|
||||
| "Limited impact" | Discuss broader implications |
|
||||
| "Narrow evaluation" | Evaluate on multiple benchmarks |
|
||||
| "Only works in restricted setting" | Acknowledge scope, explain why still valuable |
|
||||
|
||||
---
|
||||
|
||||
## How to Address Reviewer Feedback
|
||||
|
||||
### Rebuttal Best Practices
|
||||
|
||||
**Do:**
|
||||
- Thank reviewers for their time
|
||||
- Address each concern specifically
|
||||
- Provide evidence (new experiments if possible)
|
||||
- Be concise—reviewers are busy
|
||||
- Acknowledge valid criticisms
|
||||
|
||||
**Don't:**
|
||||
- Be defensive or dismissive
|
||||
- Make promises you can't keep
|
||||
- Ignore difficult criticisms
|
||||
- Write excessively long rebuttals
|
||||
- Argue about subjective assessments
|
||||
|
||||
### Rebuttal Template
|
||||
|
||||
```markdown
|
||||
We thank the reviewers for their thoughtful feedback.
|
||||
|
||||
## Reviewer 1
|
||||
|
||||
**R1-Q1: [Quoted concern]**
|
||||
[Direct response with evidence]
|
||||
|
||||
**R1-Q2: [Quoted concern]**
|
||||
[Direct response with evidence]
|
||||
|
||||
## Reviewer 2
|
||||
|
||||
...
|
||||
|
||||
## Summary of Changes
|
||||
If accepted, we will:
|
||||
1. [Specific change]
|
||||
2. [Specific change]
|
||||
3. [Specific change]
|
||||
```
|
||||
|
||||
### When to Accept Criticism
|
||||
|
||||
Some reviewer feedback should simply be accepted:
|
||||
- Valid technical errors
|
||||
- Missing important related work
|
||||
- Unclear explanations
|
||||
- Missing experimental details
|
||||
|
||||
Acknowledge these gracefully: "The reviewer is correct that... We will revise to..."
|
||||
|
||||
### When to Push Back
|
||||
|
||||
You can respectfully disagree when:
|
||||
- Reviewer misunderstood the paper
|
||||
- Requested experiments are out of scope
|
||||
- Criticism is factually incorrect
|
||||
|
||||
Frame disagreements constructively: "We appreciate this perspective. However, [explanation]..."
|
||||
|
||||
---
|
||||
|
||||
## Pre-Submission Reviewer Simulation
|
||||
|
||||
Before submitting, ask yourself:
|
||||
|
||||
**Quality:**
|
||||
- [ ] Would I trust these results if I saw them?
|
||||
- [ ] Are all claims supported by evidence?
|
||||
- [ ] Are baselines fair and recent?
|
||||
|
||||
**Clarity:**
|
||||
- [ ] Can someone reproduce this from the paper?
|
||||
- [ ] Is the writing clear to non-experts in this subfield?
|
||||
- [ ] Are all terms and notation defined?
|
||||
|
||||
**Significance:**
|
||||
- [ ] Why should the community care about this?
|
||||
- [ ] What can people do with this work?
|
||||
- [ ] Is the problem important?
|
||||
|
||||
**Originality:**
|
||||
- [ ] What specifically is new here?
|
||||
- [ ] How does this differ from closest related work?
|
||||
- [ ] Is the contribution non-trivial?
|
||||
159
skills/mlops/ml-paper-writing/references/sources.md
Normal file
159
skills/mlops/ml-paper-writing/references/sources.md
Normal file
|
|
@ -0,0 +1,159 @@
|
|||
# Source Bibliography
|
||||
|
||||
This document lists all authoritative sources used to build this skill, organized by topic.
|
||||
|
||||
---
|
||||
|
||||
## Writing Philosophy & Guides
|
||||
|
||||
### Primary Sources (Must-Read)
|
||||
|
||||
| Source | Author | URL | Key Contribution |
|
||||
|--------|--------|-----|------------------|
|
||||
| **Highly Opinionated Advice on How to Write ML Papers** | Neel Nanda | [Alignment Forum](https://www.alignmentforum.org/posts/eJGptPbbFPZGLpjsp/highly-opinionated-advice-on-how-to-write-ml-papers) | Narrative framework, "What/Why/So What", time allocation |
|
||||
| **How to Write ML Papers** | Sebastian Farquhar (DeepMind) | [Blog](https://sebastianfarquhar.com/on-research/2024/11/04/how_to_write_ml_papers/) | 5-sentence abstract formula, structure templates |
|
||||
| **A Survival Guide to a PhD** | Andrej Karpathy | [Blog](http://karpathy.github.io/2016/09/07/phd/) | Paper structure recipe, contribution framing |
|
||||
| **Heuristics for Scientific Writing** | Zachary Lipton (CMU) | [Blog](https://www.approximatelycorrect.com/2018/01/29/heuristics-technical-scientific-writing-machine-learning-perspective/) | Word choice, section balance, intensifier warnings |
|
||||
| **Advice for Authors** | Jacob Steinhardt (UC Berkeley) | [Blog](https://jsteinhardt.stat.berkeley.edu/blog/advice-for-authors) | Precision over brevity, consistent terminology |
|
||||
| **Easy Paper Writing Tips** | Ethan Perez (Anthropic) | [Blog](https://ethanperez.net/easy-paper-writing-tips/) | Micro-level tips, apostrophe unfolding, clarity tricks |
|
||||
|
||||
### Foundational Scientific Writing
|
||||
|
||||
| Source | Author | URL | Key Contribution |
|
||||
|--------|--------|-----|------------------|
|
||||
| **The Science of Scientific Writing** | Gopen & Swan | [PDF](https://cseweb.ucsd.edu/~swanson/papers/science-of-writing.pdf) | Topic/stress positions, old-before-new, 7 principles |
|
||||
| **Summary of Science of Scientific Writing** | Lawrence Crowl | [Summary](https://www.crowl.org/Lawrence/writing/GopenSwan90.html) | Condensed version of Gopen & Swan |
|
||||
|
||||
### Additional Resources
|
||||
|
||||
| Source | URL | Key Contribution |
|
||||
|--------|-----|------------------|
|
||||
| How To Write A Research Paper In ML | [Blog](https://grigorisg9gr.github.io/machine%20learning/research%20paper/how-to-write-a-research-paper-in-machine-learning/) | Practical walkthrough, LaTeX tips |
|
||||
| A Recipe for Training Neural Networks | [Karpathy Blog](http://karpathy.github.io/2019/04/25/recipe/) | Debugging methodology that translates to paper structure |
|
||||
| ICML Paper Writing Best Practices | [ICML](https://icml.cc/Conferences/2022/BestPractices) | Official venue guidance |
|
||||
| Bill Freeman's Writing Slides | [MIT](https://billf.mit.edu/sites/default/files/documents/cvprPapers.pdf) | Visual guide to paper structure |
|
||||
|
||||
---
|
||||
|
||||
## Official Conference Guidelines
|
||||
|
||||
### NeurIPS
|
||||
|
||||
| Document | URL | Purpose |
|
||||
|----------|-----|---------|
|
||||
| Paper Checklist Guidelines | [NeurIPS](https://neurips.cc/public/guides/PaperChecklist) | 16-item mandatory checklist |
|
||||
| Reviewer Guidelines 2025 | [NeurIPS](https://neurips.cc/Conferences/2025/ReviewerGuidelines) | Evaluation criteria, scoring |
|
||||
| Style Files | [NeurIPS](https://neurips.cc/Conferences/2025/PaperInformation/StyleFiles) | LaTeX templates |
|
||||
|
||||
### ICML
|
||||
|
||||
| Document | URL | Purpose |
|
||||
|----------|-----|---------|
|
||||
| Paper Guidelines | [ICML](https://icml.cc/Conferences/2024/PaperGuidelines) | Submission requirements |
|
||||
| Reviewer Instructions 2025 | [ICML](https://icml.cc/Conferences/2025/ReviewerInstructions) | Review form, evaluation |
|
||||
| Style & Author Instructions | [ICML](https://icml.cc/Conferences/2022/StyleAuthorInstructions) | Formatting specifications |
|
||||
|
||||
### ICLR
|
||||
|
||||
| Document | URL | Purpose |
|
||||
|----------|-----|---------|
|
||||
| Author Guide 2026 | [ICLR](https://iclr.cc/Conferences/2026/AuthorGuide) | Submission requirements, LLM disclosure |
|
||||
| Reviewer Guide 2025 | [ICLR](https://iclr.cc/Conferences/2025/ReviewerGuide) | Review process, evaluation |
|
||||
|
||||
### ACL/EMNLP
|
||||
|
||||
| Document | URL | Purpose |
|
||||
|----------|-----|---------|
|
||||
| ACL Style Files | [GitHub](https://github.com/acl-org/acl-style-files) | LaTeX templates |
|
||||
| ACL Rolling Review | [ARR](https://aclrollingreview.org/) | Submission process |
|
||||
|
||||
### AAAI
|
||||
|
||||
| Document | URL | Purpose |
|
||||
|----------|-----|---------|
|
||||
| Author Kit 2026 | [AAAI](https://aaai.org/authorkit26/) | Templates and guidelines |
|
||||
|
||||
### COLM
|
||||
|
||||
| Document | URL | Purpose |
|
||||
|----------|-----|---------|
|
||||
| Template | [GitHub](https://github.com/COLM-org/Template) | LaTeX templates |
|
||||
|
||||
---
|
||||
|
||||
## Citation APIs & Tools
|
||||
|
||||
### APIs
|
||||
|
||||
| API | Documentation | Best For |
|
||||
|-----|---------------|----------|
|
||||
| **Semantic Scholar** | [Docs](https://api.semanticscholar.org/api-docs/) | ML/AI papers, citation graphs |
|
||||
| **CrossRef** | [Docs](https://www.crossref.org/documentation/retrieve-metadata/rest-api/) | DOI lookup, BibTeX retrieval |
|
||||
| **arXiv** | [Docs](https://info.arxiv.org/help/api/basics.html) | Preprints, PDF access |
|
||||
| **OpenAlex** | [Docs](https://docs.openalex.org/) | Open alternative, bulk access |
|
||||
|
||||
### Python Libraries
|
||||
|
||||
| Library | Install | Purpose |
|
||||
|---------|---------|---------|
|
||||
| `semanticscholar` | `pip install semanticscholar` | Semantic Scholar wrapper |
|
||||
| `arxiv` | `pip install arxiv` | arXiv search and download |
|
||||
| `habanero` | `pip install habanero` | CrossRef client |
|
||||
|
||||
### Citation Verification
|
||||
|
||||
| Tool | URL | Purpose |
|
||||
|------|-----|---------|
|
||||
| Citely | [citely.ai](https://citely.ai/citation-checker) | Batch verification |
|
||||
| ReciteWorks | [reciteworks.com](https://reciteworks.com/) | In-text citation checking |
|
||||
|
||||
---
|
||||
|
||||
## Visualization & Formatting
|
||||
|
||||
### Figure Creation
|
||||
|
||||
| Tool | URL | Purpose |
|
||||
|------|-----|---------|
|
||||
| PlotNeuralNet | [GitHub](https://github.com/HarisIqbal88/PlotNeuralNet) | TikZ neural network diagrams |
|
||||
| SciencePlots | [GitHub](https://github.com/garrettj403/SciencePlots) | Publication-ready matplotlib |
|
||||
| Okabe-Ito Palette | [Reference](https://jfly.uni-koeln.de/color/) | Colorblind-safe colors |
|
||||
|
||||
### LaTeX Resources
|
||||
|
||||
| Resource | URL | Purpose |
|
||||
|----------|-----|---------|
|
||||
| Overleaf Templates | [Overleaf](https://www.overleaf.com/latex/templates) | Online LaTeX editor |
|
||||
| BibLaTeX Guide | [CTAN](https://ctan.org/pkg/biblatex) | Modern citation management |
|
||||
|
||||
---
|
||||
|
||||
## Research on AI Writing & Hallucination
|
||||
|
||||
| Source | URL | Key Finding |
|
||||
|--------|-----|-------------|
|
||||
| AI Hallucinations in Citations | [Enago](https://www.enago.com/academy/ai-hallucinations-research-citations/) | ~40% error rate |
|
||||
| Hallucination in AI Writing | [PMC](https://pmc.ncbi.nlm.nih.gov/articles/PMC10726751/) | Types of citation errors |
|
||||
| NeurIPS 2025 AI Report | [ByteIota](https://byteiota.com/neurips-2025-100-ai-hallucinations-slip-through-review/) | 100+ hallucinated citations |
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference by Topic
|
||||
|
||||
### For Narrative & Structure
|
||||
→ Start with: Neel Nanda, Sebastian Farquhar, Andrej Karpathy
|
||||
|
||||
### For Sentence-Level Clarity
|
||||
→ Start with: Gopen & Swan, Ethan Perez, Zachary Lipton
|
||||
|
||||
### For Word Choice & Style
|
||||
→ Start with: Zachary Lipton, Jacob Steinhardt
|
||||
|
||||
### For Conference-Specific Requirements
|
||||
→ Start with: Official venue guidelines (NeurIPS, ICML, ICLR, ACL)
|
||||
|
||||
### For Citation Management
|
||||
→ Start with: Semantic Scholar API, CrossRef, citation-workflow.md
|
||||
|
||||
### For Reviewer Expectations
|
||||
→ Start with: Venue reviewer guidelines, reviewer-guidelines.md
|
||||
476
skills/mlops/ml-paper-writing/references/writing-guide.md
Normal file
476
skills/mlops/ml-paper-writing/references/writing-guide.md
Normal file
|
|
@ -0,0 +1,476 @@
|
|||
# ML Paper Writing Philosophy & Best Practices
|
||||
|
||||
This reference compiles writing advice from prominent ML researchers including Neel Nanda, Andrej Karpathy, Sebastian Farquhar, Zachary Lipton, and Jacob Steinhardt.
|
||||
|
||||
---
|
||||
|
||||
## Contents
|
||||
|
||||
- [The Narrative Principle](#the-narrative-principle)
|
||||
- [Time Allocation](#time-allocation)
|
||||
- [Abstract Writing Formula](#abstract-writing-formula)
|
||||
- [Introduction Structure](#introduction-structure)
|
||||
- [Sentence-Level Clarity](#sentence-level-clarity)
|
||||
- [Word Choice and Precision](#word-choice-and-precision)
|
||||
- [Mathematical Writing](#mathematical-writing)
|
||||
- [Figure Design](#figure-design)
|
||||
- [Common Mistakes to Avoid](#common-mistakes-to-avoid)
|
||||
|
||||
---
|
||||
|
||||
## The Narrative Principle
|
||||
|
||||
### From Neel Nanda
|
||||
|
||||
"A paper is a short, rigorous, evidence-based technical story with a takeaway readers care about."
|
||||
|
||||
The narrative rests on three pillars that must be crystal clear by the end of your introduction:
|
||||
|
||||
**The "What"**: One to three specific novel claims fitting within a cohesive theme. Vague contributions like "we study X" fail immediately—reviewers need precise, falsifiable claims.
|
||||
|
||||
**The "Why"**: Rigorous empirical evidence that convincingly supports those claims, including strong baselines honestly tuned and experiments that distinguish between competing hypotheses rather than merely showing "decent results."
|
||||
|
||||
**The "So What"**: Why readers should care, connecting your contribution to problems the community recognizes as important.
|
||||
|
||||
### From Andrej Karpathy
|
||||
|
||||
"A paper is not a random collection of experiments you report on. The paper sells a single thing that was not obvious or present before. The entire paper is organized around this core contribution with surgical precision."
|
||||
|
||||
This applies whether you're presenting a new architecture, a theoretical result, or improved understanding of existing methods—NeurIPS explicitly notes that "originality does not necessarily require an entirely new method."
|
||||
|
||||
**Practical Implication**: If you cannot state your contribution in one sentence, you don't yet have a paper. Everything else—experiments, related work, discussion—exists only to support that core claim.
|
||||
|
||||
---
|
||||
|
||||
## Time Allocation
|
||||
|
||||
### From Neel Nanda
|
||||
|
||||
Spend approximately **the same amount of time** on each of:
|
||||
1. The abstract
|
||||
2. The introduction
|
||||
3. The figures
|
||||
4. Everything else combined
|
||||
|
||||
This isn't hyperbole—most reviewers form preliminary judgments before reaching your methods section. Readers encounter your paper in a predictable pattern: **title → abstract → introduction → figures → maybe the rest.**
|
||||
|
||||
### Reviewer Reading Patterns
|
||||
|
||||
Studies of reviewer behavior show:
|
||||
- Abstract is read 100% of the time
|
||||
- Introduction is skimmed by 90%+ of reviewers
|
||||
- Figures are examined before methods by most reviewers
|
||||
- Full methods are read only if interest is established
|
||||
|
||||
**Implication**: Front-load your paper's value. Don't bury the contribution.
|
||||
|
||||
---
|
||||
|
||||
## Abstract Writing Formula
|
||||
|
||||
### Sebastian Farquhar's 5-Sentence Formula
|
||||
|
||||
1. **What you achieved**: "We introduce...", "We prove...", "We demonstrate..."
|
||||
2. **Why this is hard and important**
|
||||
3. **How you do it** (with specialist keywords for discoverability)
|
||||
4. **What evidence you have**
|
||||
5. **Your most remarkable number/result**
|
||||
|
||||
### Example (Good Abstract)
|
||||
|
||||
```
|
||||
We prove that gradient descent on overparameterized neural networks
|
||||
converges to global minima at a linear rate. [What]
|
||||
This resolves a fundamental question about why deep learning works
|
||||
despite non-convex optimization landscapes. [Why hard/important]
|
||||
Our proof relies on showing that the Neural Tangent Kernel remains
|
||||
approximately constant during training, reducing the problem to
|
||||
kernel regression. [How with keywords]
|
||||
We validate our theory on CIFAR-10 and ImageNet, showing that
|
||||
predicted convergence rates match experiments within 5%. [Evidence]
|
||||
This is the first polynomial-time convergence guarantee for
|
||||
networks with practical depth and width. [Remarkable result]
|
||||
```
|
||||
|
||||
### What to Avoid
|
||||
|
||||
From Zachary Lipton: "If the first sentence can be pre-pended to any ML paper, delete it."
|
||||
|
||||
**Delete these openings**:
|
||||
- "Large language models have achieved remarkable success..."
|
||||
- "Deep learning has revolutionized..."
|
||||
- "In recent years, neural networks have..."
|
||||
|
||||
**Start with your specific contribution instead.**
|
||||
|
||||
---
|
||||
|
||||
## Introduction Structure
|
||||
|
||||
### Requirements
|
||||
|
||||
- **1-1.5 pages maximum** (in two-column format)
|
||||
- **Methods should start by page 2-3**
|
||||
- Must include **2-4 bullet contribution list** (max 1-2 lines each)
|
||||
|
||||
### Structure Template
|
||||
|
||||
```markdown
|
||||
1. Opening Hook (2-3 sentences)
|
||||
- State the problem your paper addresses
|
||||
- Why it matters RIGHT NOW
|
||||
|
||||
2. Background/Challenge (1 paragraph)
|
||||
- What makes this problem hard?
|
||||
- What have others tried? Why is it insufficient?
|
||||
|
||||
3. Your Approach (1 paragraph)
|
||||
- What do you do differently?
|
||||
- Key insight that enables your contribution
|
||||
|
||||
4. Contribution Bullets (2-4 items)
|
||||
- Be specific and falsifiable
|
||||
- Each bullet: 1-2 lines maximum
|
||||
|
||||
5. Results Preview (2-3 sentences)
|
||||
- Most impressive numbers
|
||||
- Scope of evaluation
|
||||
|
||||
6. Paper Organization (optional, 1-2 sentences)
|
||||
- "Section 2 presents... Section 3 describes..."
|
||||
```
|
||||
|
||||
### Contribution Bullets: Good vs Bad
|
||||
|
||||
**Good:**
|
||||
- We prove that X converges in O(n log n) time under assumption Y
|
||||
- We introduce Z, a 3-layer architecture that reduces memory by 40%
|
||||
- We demonstrate that A outperforms B by 15% on benchmark C
|
||||
|
||||
**Bad:**
|
||||
- We study the problem of X (not a contribution)
|
||||
- We provide extensive experiments (too vague)
|
||||
- We make several contributions to the field (says nothing)
|
||||
|
||||
---
|
||||
|
||||
## Sentence-Level Clarity
|
||||
|
||||
### From Gopen & Swan: "The Science of Scientific Writing"
|
||||
|
||||
The seminal 1990 paper by George Gopen and Judith Swan establishes that **readers have structural expectations** about where information appears in prose. Violating these expectations forces readers to spend energy on structure rather than content.
|
||||
|
||||
> "If the reader is to grasp what the writer means, the writer must understand what the reader needs."
|
||||
|
||||
#### The 7 Principles of Reader Expectations
|
||||
|
||||
**Principle 1: Subject-Verb Proximity**
|
||||
|
||||
Keep grammatical subject and verb close together. Anything intervening reads as interruption of lesser importance.
|
||||
|
||||
**Weak**: "The model, which was trained on 100M tokens and fine-tuned on domain-specific data using LoRA with rank 16, achieves state-of-the-art results"
|
||||
|
||||
**Strong**: "The model achieves state-of-the-art results after training on 100M tokens and fine-tuning with LoRA (rank 16)"
|
||||
|
||||
**Principle 2: Stress Position (Save the Best for Last)**
|
||||
|
||||
Readers naturally emphasize the **last words of a sentence**. Place your most important information there.
|
||||
|
||||
**Weak**: "Accuracy improves by 15% when using attention"
|
||||
**Strong**: "When using attention, accuracy improves by **15%**"
|
||||
|
||||
**Principle 3: Topic Position (First Things First)**
|
||||
|
||||
The beginning of a sentence establishes perspective. Put the "whose story" element first—readers expect the sentence to be about whoever shows up first.
|
||||
|
||||
**Weak**: "A novel attention mechanism that computes alignment scores is introduced"
|
||||
**Strong**: "To address the alignment problem, we introduce a novel attention mechanism"
|
||||
|
||||
**Principle 4: Old Information Before New**
|
||||
|
||||
Put familiar information (old) in the topic position for backward linkage; put new information in the stress position for emphasis.
|
||||
|
||||
**Weak**: "Sparse attention was introduced by Child et al. The quadratic complexity of standard attention motivates this work."
|
||||
**Strong**: "Standard attention has quadratic complexity. To address this, Child et al. introduced sparse attention."
|
||||
|
||||
**Principle 5: One Unit, One Function**
|
||||
|
||||
Each unit of discourse (sentence, paragraph, section) should serve a single function. If you have two points, use two units.
|
||||
|
||||
**Principle 6: Articulate Action in the Verb**
|
||||
|
||||
Express the action of each sentence in its verb, not in nominalized nouns.
|
||||
|
||||
**Weak**: "We performed an analysis of the results" (nominalization)
|
||||
**Strong**: "We analyzed the results" (action in verb)
|
||||
|
||||
**Principle 7: Context Before New Information**
|
||||
|
||||
Provide context before asking the reader to consider anything new. This applies at all levels—sentence, paragraph, section.
|
||||
|
||||
**Weak**: "Equation 3 shows that convergence is guaranteed when the learning rate satisfies..."
|
||||
**Strong**: "For convergence to be guaranteed, the learning rate must satisfy the condition in Equation 3..."
|
||||
|
||||
#### Summary Table
|
||||
|
||||
| Principle | Rule | Mnemonic |
|
||||
|-----------|------|----------|
|
||||
| Subject-Verb Proximity | Keep subject and verb close | "Don't interrupt yourself" |
|
||||
| Stress Position | Emphasis at sentence end | "Save the best for last" |
|
||||
| Topic Position | Context at sentence start | "First things first" |
|
||||
| Old Before New | Familiar → unfamiliar | "Build on known ground" |
|
||||
| One Unit, One Function | Each paragraph = one point | "One idea per container" |
|
||||
| Action in Verb | Use verbs, not nominalizations | "Verbs do, nouns sit" |
|
||||
| Context Before New | Explain before presenting | "Set the stage first" |
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
## Micro-Level Writing Tips
|
||||
|
||||
### From Ethan Perez (Anthropic)
|
||||
|
||||
These practical micro-level tips improve clarity at the sentence and word level.
|
||||
|
||||
#### Pronoun Management
|
||||
|
||||
**Minimize pronouns** ("this," "it," "these," "that"). When pronouns are necessary, use them as adjectives with a noun:
|
||||
|
||||
**Weak**: "This shows that the model converges."
|
||||
**Strong**: "This result shows that the model converges."
|
||||
|
||||
**Weak**: "It improves performance."
|
||||
**Strong**: "This modification improves performance."
|
||||
|
||||
#### Verb Placement
|
||||
|
||||
**Position verbs early** in sentences for better parsing:
|
||||
|
||||
**Weak**: "The gradient, after being computed and normalized, updates the weights."
|
||||
**Strong**: "The gradient updates the weights after being computed and normalized."
|
||||
|
||||
#### Apostrophe Unfolding
|
||||
|
||||
Transform possessive constructions for clarity:
|
||||
|
||||
**Original**: "X's Y" → **Unfolded**: "The Y of X"
|
||||
|
||||
**Before**: "The model's accuracy on the test set"
|
||||
**After**: "The accuracy of the model on the test set"
|
||||
|
||||
This isn't always better, but when sentences feel awkward, try unfolding.
|
||||
|
||||
#### Words to Eliminate
|
||||
|
||||
Delete these filler words in almost all cases:
|
||||
- "actually"
|
||||
- "a bit"
|
||||
- "fortunately" / "unfortunately"
|
||||
- "very" / "really"
|
||||
- "quite"
|
||||
- "basically"
|
||||
- "essentially"
|
||||
- Excessive connectives ("however," "moreover," "furthermore" when not needed)
|
||||
|
||||
#### Sentence Construction Rules
|
||||
|
||||
1. **One idea per sentence** - If struggling to express an idea in one sentence, it needs two
|
||||
2. **No repeated sounds** - Avoid similar-sounding words in the same sentence
|
||||
3. **Every sentence adds information** - Delete sentences that merely restate
|
||||
4. **Active voice always** - Specify the actor ("We find..." not "It is found...")
|
||||
5. **Expand contractions** - "don't" → "do not" for formality
|
||||
|
||||
#### Paragraph Architecture
|
||||
|
||||
- **First sentence**: State the point clearly
|
||||
- **Middle sentences**: Support with evidence
|
||||
- **Last sentence**: Reinforce or transition
|
||||
|
||||
Don't bury key information in the middle of paragraphs.
|
||||
|
||||
---
|
||||
|
||||
## Word Choice and Precision
|
||||
|
||||
### From Zachary Lipton
|
||||
|
||||
**Eliminate hedging** unless genuine uncertainty exists:
|
||||
- Delete "may" and "can" unless necessary
|
||||
- "provides *very* tight approximation" drips with insecurity
|
||||
- "provides tight approximation" is confident
|
||||
|
||||
**Avoid vacuous intensifiers**:
|
||||
- Delete: very, extremely, highly, significantly (unless statistical)
|
||||
- These words signal insecurity, not strength
|
||||
|
||||
### From Jacob Steinhardt
|
||||
|
||||
**Precision over brevity**: Replace vague terms with specific ones.
|
||||
|
||||
| Vague | Specific |
|
||||
|-------|----------|
|
||||
| performance | accuracy, latency, throughput |
|
||||
| improves | increases accuracy by X%, reduces latency by Y |
|
||||
| large | 1B parameters, 100M tokens |
|
||||
| fast | 3x faster, 50ms latency |
|
||||
| good results | 92% accuracy, 0.85 F1 |
|
||||
|
||||
**Consistent terminology**: Referring to the same concept with different terms creates confusion.
|
||||
|
||||
**Choose one and stick with it**:
|
||||
- "model" vs "network" vs "architecture"
|
||||
- "training" vs "learning" vs "optimization"
|
||||
- "sample" vs "example" vs "instance"
|
||||
|
||||
### Vocabulary Signaling
|
||||
|
||||
**Avoid words signaling incremental work**:
|
||||
- Never: "combine," "modify," "expand," "extend"
|
||||
- Instead: "develop," "propose," "introduce"
|
||||
|
||||
**Why**: "We combine X and Y" sounds like you stapled two existing ideas together. "We develop a method that leverages X for Y" sounds like genuine contribution.
|
||||
|
||||
---
|
||||
|
||||
## Mathematical Writing
|
||||
|
||||
### From Ethan Perez
|
||||
|
||||
**Unfold apostrophes** for clarity:
|
||||
- Weak: "X's Y"
|
||||
- Strong: "The Y of X"
|
||||
|
||||
Example: "the model's accuracy" → "the accuracy of the model"
|
||||
|
||||
### General Principles
|
||||
|
||||
1. **State all assumptions formally** before theorems
|
||||
2. **Provide intuitive explanations** alongside proofs
|
||||
3. **Use consistent notation** throughout the paper
|
||||
4. **Define symbols at first use**
|
||||
|
||||
### Notation Conventions
|
||||
|
||||
```latex
|
||||
% Scalars: lowercase italic
|
||||
$x$, $y$, $\alpha$, $\beta$
|
||||
|
||||
% Vectors: lowercase bold
|
||||
$\mathbf{x}$, $\mathbf{v}$
|
||||
|
||||
% Matrices: uppercase bold
|
||||
$\mathbf{W}$, $\mathbf{X}$
|
||||
|
||||
% Sets: uppercase calligraphic
|
||||
$\mathcal{X}$, $\mathcal{D}$
|
||||
|
||||
% Functions: roman for named functions
|
||||
$\mathrm{softmax}$, $\mathrm{ReLU}$
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Figure Design
|
||||
|
||||
### From Neel Nanda
|
||||
|
||||
Figures should tell a coherent story even if the reader skips the text. Many readers DO skip the text initially.
|
||||
|
||||
### Design Principles
|
||||
|
||||
1. **Figure 1 is crucial**: Often the first thing readers examine after abstract
|
||||
2. **Self-contained captions**: Reader should understand figure without main text
|
||||
3. **No title inside figure**: The caption serves this function (ICML/NeurIPS rule)
|
||||
4. **Vector graphics**: PDF/EPS for plots, PNG (600 DPI) only for photographs
|
||||
|
||||
### Accessibility Requirements
|
||||
|
||||
8% of men have color vision deficiency. Your figures must work for them.
|
||||
|
||||
**Solutions**:
|
||||
- Use colorblind-safe palettes: Okabe-Ito or Paul Tol
|
||||
- Avoid red-green combinations
|
||||
- Verify figures work in grayscale
|
||||
- Use different line styles (solid, dashed, dotted) in addition to colors
|
||||
|
||||
### Tools
|
||||
|
||||
```python
|
||||
# SciencePlots: Publication-ready styles
|
||||
import matplotlib.pyplot as plt
|
||||
plt.style.use(['science', 'ieee'])
|
||||
|
||||
# Or for Nature-style
|
||||
plt.style.use(['science', 'nature'])
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Mistakes to Avoid
|
||||
|
||||
### Structure Mistakes
|
||||
|
||||
| Mistake | Solution |
|
||||
|---------|----------|
|
||||
| Introduction too long (>1.5 pages) | Move background to Related Work |
|
||||
| Methods buried (after page 3) | Front-load contribution, cut intro |
|
||||
| Missing contribution bullets | Add 2-4 specific, falsifiable claims |
|
||||
| Experiments without explicit claims | State what each experiment tests |
|
||||
|
||||
### Writing Mistakes
|
||||
|
||||
| Mistake | Solution |
|
||||
|---------|----------|
|
||||
| Generic abstract opening | Start with your specific contribution |
|
||||
| Inconsistent terminology | Choose one term per concept |
|
||||
| Passive voice overuse | Use active voice: "We show" not "It is shown" |
|
||||
| Hedging everywhere | Be confident unless genuinely uncertain |
|
||||
|
||||
### Figure Mistakes
|
||||
|
||||
| Mistake | Solution |
|
||||
|---------|----------|
|
||||
| Raster graphics for plots | Use vector (PDF/EPS) |
|
||||
| Red-green color scheme | Use colorblind-safe palette |
|
||||
| Title inside figure | Put title in caption |
|
||||
| Captions require main text | Make captions self-contained |
|
||||
|
||||
### Citation Mistakes
|
||||
|
||||
| Mistake | Solution |
|
||||
|---------|----------|
|
||||
| Paper-by-paper Related Work | Organize methodologically |
|
||||
| Missing relevant citations | Reviewers authored papers—cite generously |
|
||||
| AI-generated citations | Always verify via APIs |
|
||||
| Inconsistent citation format | Use BibLaTeX with consistent keys |
|
||||
|
||||
---
|
||||
|
||||
## Pre-Submission Checklist
|
||||
|
||||
Before submitting, verify:
|
||||
|
||||
**Narrative**:
|
||||
- [ ] Can state contribution in one sentence
|
||||
- [ ] Three pillars (What/Why/So What) clear in intro
|
||||
- [ ] Every experiment supports a specific claim
|
||||
|
||||
**Structure**:
|
||||
- [ ] Abstract follows 5-sentence formula
|
||||
- [ ] Introduction ≤1.5 pages
|
||||
- [ ] Methods start by page 2-3
|
||||
- [ ] 2-4 contribution bullets included
|
||||
- [ ] Limitations section present
|
||||
|
||||
**Writing**:
|
||||
- [ ] Consistent terminology throughout
|
||||
- [ ] No generic opening sentences
|
||||
- [ ] Hedging removed unless necessary
|
||||
- [ ] All figures have self-contained captions
|
||||
|
||||
**Technical**:
|
||||
- [ ] All citations verified via API
|
||||
- [ ] Error bars included with methodology
|
||||
- [ ] Compute resources documented
|
||||
- [ ] Code/data availability stated
|
||||
Loading…
Add table
Add a link
Reference in a new issue