--- name: do-research description: "Call this skill when user asks you to 'do research'. Call it to get public information about your problem, and when the user asks you to look it up." --- # Do Research You must do research in order to solve problems. This skill documents why and how to do that research. ## Why do research Most problems have been studied before. Other people already found solutions. Some solutions succeeded, but some failed. Most solutions are different, and the reasons for their success and failure likely different. Find that work before you solve the problem yourself for two clear benefits: 1. **Steal successes:** You can copy the solutions that have succeeded and make incremental improvements quickly. 2. **Avoid failures:** You can observe and reason about solutions that have failed to avoid making similar mistakes. **Most Importantly:** Your training data stopped in the past, people have continued to publish new work after it. ## Tools Use these tools to do research. - **Your own notes first.** If you keep a searchable notes folder (for example with [qmd](https://github.com/tobi/qmd)), search it before the web. - **WebSearch** - **Reading a page:** - `lightpanda fetch --dump markdown --dump-max-bytes 20000 `: a fast headless browser that runs the page's JavaScript and gives back Markdown. Use it first ([lightpanda](https://github.com/lightpanda-io/browser)). - **WebFetch**, when lightpanda can't load the page. - `agent-browser open `, then `agent-browser eval "document.body.innerText"`: a full headless Chromium, for pages that need clicks, logins or scrolling. The `agent-browser` skill has the commands. - `pinchtab`: a headless Chrome server (start it with `pinchtab`, port 9867), then `pinchtab nav ` and `pinchtab text`, or `pinchtab snap -i -c` for a compact tree of the links and buttons. Use it when you visit many pages in one session. Stop it after: `kill $(lsof -ti tcp:9867 -sTCP:LISTEN)`. - **Papers and their citations: OpenAlex** (free, no key). - Search: `curl -s 'https://api.openalex.org/works?search=&per-page=10&select=id,display_name,publication_year,cited_by_count'` - Newer work that cites a paper: `curl -s 'https://api.openalex.org/works?filter=cites:&sort=cited_by_count:desc&per-page=10&select=display_name,publication_year,cited_by_count'` - **Newest preprints: arXiv**: `curl -sL 'https://export.arxiv.org/api/query?search_query=all:+AND+all:&max_results=10&sortBy=submittedDate'` - **More places to find papers.** All of these answered a plain `curl` with no key. The list is open: for example these, et cetera. Your field probably has its own archive or index, so search for it. - **Semantic Scholar** (citation counts, links to free PDFs): `curl -s 'https://api.semanticscholar.org/graph/v1/paper/search?query=&limit=10&fields=title,year,citationCount,openAccessPdf'`. Without a key the limit is shared and low (about 100 requests per 5 minutes); slow down on a 429. - **Crossref** (any DOI's metadata, licence and reference list): `curl -s 'https://api.crossref.org/works?query=&rows=10&select=DOI,title,issued,is-referenced-by-count&mailto='`. The `mailto` puts you in the faster "polite" pool. - **Unpaywall** (does a legal free copy of this DOI exist?): `curl -s 'https://api.unpaywall.org/v2/?email='`. Read `is_oa` and `best_oa_location`. - **Medicine and life science: PubMed**: `curl -s 'https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pubmed&term=&retmax=10&retmode=json'`, then `esummary.fcgi?db=pubmed&id=&retmode=json` for titles. A free NCBI API key (`&api_key=`) lifts the limit from 3 to 10 requests a second. - **Europe PMC** (the same literature with full text for the open-access part): `curl -s 'https://www.ebi.ac.uk/europepmc/webservices/rest/search?query=&format=json&pageSize=10&resultType=lite'`, and `.../rest//fullTextXML` for a whole paper. - **bioRxiv and medRxiv** (new biology and medicine preprints, by date): `curl -s 'https://api.biorxiv.org/details/biorxiv///0/json'`. It lists by date range, so search elsewhere and use this to see what is new. - **Open-access journals: DOAJ**: `curl -s 'https://doaj.org/api/search/articles/?pageSize=10'`. - **Peer review and ML papers: OpenReview**: `curl -s 'https://api2.openreview.net/notes/search?term=&limit=10'`. Reviews and rebuttals are findings too. - **Hugging Face papers** (ML papers with models and datasets attached): `curl -s 'https://huggingface.co/api/papers/search?q=&limit=10'`. - **Datasets, code and reports from any field: Zenodo**: `curl -s 'https://zenodo.org/api/records?q=&size=10'`. - **Background on a term: Wikipedia**: `curl -s 'https://en.wikipedia.org/w/api.php?action=query&list=search&srsearch=&format=json&srlimit=10'`. Use it to learn the field's own names for your problem, then follow its references. - **JSTOR** has no API an agent can use; it needs a browser and the user's say-so. SSRN blocks plain requests (HTTP 403). CORE needs a key. dblp answered with a bot check. - **What practitioners say: Hacker News** (Algolia, free, no key): `curl -s 'https://hn.algolia.com/api/v1/search?query=&tags=story&hitsPerPage=20'`. Use `tags=comment` for comments, and `search_by_date` in place of `search` for the newest. - **What people built: GitHub**: `gh search repos "" --limit 10` and `gh search code "" --limit 10`. ## Tool Usage ### Your notes You are probably not the first agent to research this. Search what you and other agents already saved before you search the web. ### WebSearch Default: `mode: "standard"` Recent / Niche: `mode: "extended"` Strategy: - **WebSearch is full of marketing** designed to trick you. There are many results that look like research and are not research. Avoid vendors unless specifically researching for vendors or for details about a specific vendor. - **Use the lever principle** to optimize results. Use search to find directories and collections of information density - exploration with lightpanda is fast and cheap. - **You do not know what you do not know** - today is later than your training cut off and the world has not stopped. There is always new information available. ## Conditions for success You are searching for information that allows you to understand the deeper principle of your problem space. You are trying to update your axioms so you can improve your understanding of your task. Tasks that are difficult are often so because they are large and poorly understood. Your true goal with research is to inventory the shape of success and failure and deduce the principles that created those shapes. Once you understand the principles, you can safely break your task into smaller chunks and build plans to complete each step. ## Wrapping up Store your results where the next agent can find them, for example a `research/` folder in your repo. 1. **Save every source you relied on into `raw/`.** Raw files are never edited. - Web page: `lightpanda fetch --dump markdown > raw/web/.md`, with the URL and today's date at the top. - Paper: the PDF in `raw/papers/`, with a markdown copy beside it. 2. **Write an `answer` page for your problem:** what succeeded, what failed, the principles you deduced, and what is still unknown. 3. **Record dead ends.** A search that found nothing, or a route you ruled out, goes on the answer page. It saves the next agent the same search. 4. **Commit by path,** one commit per piece of research.