Sources and method
Each week a scheduled job takes fresh data from the sources, computes every figure again, and publishes the result as a dated snapshot. Nobody edits the figures by hand. A snapshot never changes after it is published.
The weekly run
- 01Check the sourcesOne request to each source. If GBIF or the data store does not answer, the run stops.
- 02DownloadGBIF downloads, one after another, each with its own DOI. iNaturalist species lists per state. The checklists and the IUCN Red List when they change.
- 03PlaceEach record goes to its LGD district by its coordinates.
- 04ComputeThe master list: every species from every source, with which species have an open record, which observations wait for a name, and the IUCN status, for India and each state.
- 05CheckEvery download has a DOI. Large changes from the last run stop the run.
- 06PublishThe files go to the data store. The site reads them in your browser.
The job can run again safely. It does not repeat finished steps, and it does not send a file again if the data store already has the same file. If GBIF reports no change since the last run, the week downloads nothing.
What the words mean
- Documented
- A checklist, a paper or an open record gives the species for India: a ZSI or BSI checklist, Kew's plant list, Plazi, BSI Plant Discoveries, ZSI Animal Discoveries, or an open record. A listing by BSI or ZSI counts. Most species with no open record are known from papers and from specimens that are not online yet.
- Open record
- A record of the species from India that anyone can check online: a GBIF record (a specimen, a photo, a sighting, a sound, a DNA sample, or a record from a paper), or an iNaturalist research-grade observation. This is the rule of GBIF, which takes only research-grade observations from iNaturalist. An iNaturalist observation that waits for a name is not an open record of its species yet.
- Described
- The year the species was first described. For a species described from India since 2007: the year of the paper (Plazi, BSI Plant Discoveries, ZSI Animal Discoveries). Else the year in the Catalogue of Life author citation, else in the ZSI checklist, else in Kew's plant list (for a plant renamed later, the year of its first name). Most fungi, lichens and mosses have no year in these sources.
- No record since it was described
- No GBIF record of the species from India after the year of its description. This is checked only for species described since 2007 that the yearly BSI Plant Discoveries and ZSI Animal Discoveries editions or Plazi list. For a species that an edition lists in a later year than its description (for example a new record for India), it is not checked.
- IUCN status
- The category on the IUCN Red List (CR, EN, VU, NT, LC, DD), or not assessed. Threatened means CR, EN or VU.
- Recorded in the last 10 years
- The species has an open record placed in a state, from the year 10 years before the list or later (from 2016 for a list of 2026). A record with no place, or an iNaturalist count with no year, does not count here.
- Seven gaps in what we know
- Biologists name seven gaps ("shortfalls") in what we know about species (Hortal and others, 2015, Seven shortfalls that beset large-scale knowledge of biodiversity, Annual Review of Ecology, Evolution, and Systematics 46: 523–549). The home page and each species page show the seven as a pyramid: a species needs a name before a map, and a map before the other five can be studied. On the home page, each block has a bar: the share of the documented species of the chosen groups where that gap is known. On a species page, the block shows the gap for that species, and the bar the share for its group. For India, Overlooked measures: Which species exist (Linnean gap): many species are not described yet. The bar counts documented species with a known year of description. We also count the species on the India list described in the last 10 years: a list that still grows each year has species still to find. With it we count three signs that a name rests on evidence people can reach: a type specimen from India on GBIF (a record with a type status such as holotype or syntype), a photo from India on GBIF, and a name in an Indian language on Wikidata (labels and common names in 26 languages of India, found by the Catalogue of Life ID; a label that is only the scientific name does not count). Where species live (Wallacean gap): for described species, we know too little about where they live. The pyramid's bar counts documented species with a record placed in a state in the last 10 years. The larger bar below it on the home page counts each species once, at the furthest step we know: only in checklists and papers, an open record with no place, a place but no record in the last 10 years, or a record in the last 10 years. The names not on the India list, known only from observations that wait for a name, are part of this gap too. How many there are (Prestonian gap): how many of each species there are, and how that changes. Documented species with a known population trend that is rising, stable or falling: in the latest IUCN Red List assessment ("Increasing", "Stable" or "Decreasing"; "Unknown" does not count), or for birds in State of India's Birds 2023, long-term or current ("Trend Inconclusive" and "Insufficient Data" do not count). A group with no species in either source is not measured. Each species page shows IUCN's trend. How species are related (Darwinian gap): each species' place in the tree of life. The bar counts the species with a DNA sequence or sample from India on GBIF, or a DNA barcode from India on BOLD Systems. We also count the species with any public DNA barcode on BOLD Systems (the Barcode of Life Data System): any public barcode, from BOLD's records on GBIF (the International Barcode of Life dataset), and, of them, a barcode from India (BOLD's own summary for India, matched by name). What species are like (Raunkiæran gap): traits such as size, life cycle, and when a species flowers or breeds. Measured: documented species with a record placed in India that gives a life stage, a sex, or flowers or fruits (GBIF lifeStage, sex and reproductiveCondition; on iNaturalist these are annotations). The values "Unknown", "Indeterminate" and "no flowers or fruits" do not count. This counts records with a trait, not measured traits such as size. What conditions species need (Hutchinsonian gap): the climate and habitat a species can live in. Measured: documented species with GBIF records placed in India to 1 km or better (the coordinate uncertainty the publisher gives) at 10 or more places, about 110 m apart. Ten places is a common minimum to relate a species to climate maps. A record with no stated uncertainty does not count. Each species page also gives the area of occupancy: the 2 × 2 km cells (IUCN's grid size, in an equal-area grid centred on India) with an open record placed in India. How species interact (Eltonian gap): what a species eats, pollinates or lives on. We count the species with an interaction recorded in GloBI (the Global Biotic Interactions database), anywhere and in India: a record with a point inside an Indian district, or with no point and a locality with the word India. A species counts as source or as partner. Names that GloBI gives under an older name count for the species, through the GBIF name match. Each species page draws its web: its partner species by kind of link (eats, eaten by, pollination, lives on or in, lives on it, other), the 30 with most records of each kind, a dot filled when a record of the link is from India. A partner on the India list links to its own page. We publish the partner names and counts, not GloBI's records; each page links to its interactions on GloBI. Later papers add more, for example the Racovitzan gap (life below the surface: caves, soil, ground water, the deep sea) and the Haeckelian gap (species known from one life stage only).
- Red List assessments
- From IUCN's category and the year of each assessment, with this list's open records. Assessed: the species has a category on the IUCN Red List. Not assessed, with open records in 5 or more states: enough open records for an assessor to start. Assessed more than 10 years ago: IUCN aims to assess each species again every 5 to 10 years. Data Deficient, with newer records: records with a place after the year of the assessment. Threatened, with no recent record: Critically Endangered, Endangered or Vulnerable, with no record placed in a state in the last 10 years. Endemic to India, not assessed. IUCN's terms let anyone use the categories; the other data of an assessment we use only through IUCN's open copy on GBIF.
- Waiting for a name
- An iNaturalist observation with no agreed name yet. The observer may have named it to species, to genus only, or only to a family or a higher level. A species that only such observations name is not on the India list yet; when someone confirms the name, it goes on the list.
- Name not matched
- A source name that does not match a Catalogue of Life name. It stays on the list under its own name, with this label.
- District and state
- LGD boundaries. A GBIF record goes to a district by its coordinates, rounded to 3 decimals (about 110 m). A point less than 5 km from land goes to the nearest district. A record that cannot go in a district goes to a state when its state name is clear, else it counts only for India. These records have no coordinates, coordinates that are uncertain by more than 25 km, or coordinates that GBIF marks as possibly wrong. GBIF records with coordinates outside India are not used. Records whose state name is a place outside India (for example Nepal, Ceylon or Indiana), and datasets that are not from India although GBIF has them as India, are not used; the lists are in the source code (pipeline/ref). DNA sequences with no coordinates are not used, because their dates are often the date the sequence was sent.
- No exact location
- The species has an open record from India, but no record of it can go in a district (see District and state).
- Only GBIF records with no exact location
- No exact location, and GBIF is the only source that lists the species for India. Most are old museum specimens. One source and no map point is weak evidence, so check the records before you use the species.
- Records with no coordinates
- Some GBIF records of the species from India have no coordinates. The publisher can add them.
- Records with coordinates uncertain by more than 25 km
- Some GBIF records give coordinates, but the publisher says they are uncertain by more than 25 km: too coarse for a district.
- Records with coordinates that GBIF marks as possibly wrong
- GBIF found a problem with the coordinates of some records, for example swapped or zero values. Such a record may not be from India. Check it with the publisher.
- Recorded under the name as published
- GBIF has records of the species from India after its description, but only under the name as the paper gives it: GBIF has not added the name to its species list yet. These records are an open record of the species in India, in no state.
Records used
- Country: India, with coordinates and no known problem with the place
- Status: present
- Uncertainty of the place: 25 km or less
- No fossils, and no living specimens in zoos or gardens
- Some datasets are not used, for example gene-bank and compiled crop data
- DNA sequence records count. A species known only from DNA sequences has the label "DNA only". Host-organism records from sequence archives are not used
Names follow the Catalogue of Life, as GBIF uses it. Species links go to the Catalogue of Life. A short reviewed list corrects where the Catalogue of Life places a name that Indian sources use for another species: for example, Pteropus giganteus, the old name of the Indian flying fox, which the Catalogue of Life places under the Southeast Asian Pteropus vampyrus. Each correction has its reason, in name_overrides.csv.
Shortlists: the test Experimental
The Shortlists page gives a short list of names for observations that wait on iNaturalist. A model makes each list: BioCLIP 2.5, with a classifier trained on research-grade photos from India. A person checks every name on iNaturalist. We post nothing. A list never changes a status, a count or a label on this site.
We tested the lists on 2,545 research-grade observations from India, from observers whose photos the model did not learn from. Each waited 30 days or more for research grade. Overall, the first name was right for 84.4%. A list made to hold the right name 90% of the time held it 92.0% of the time, with 1 name in the middle case. On observations that still wait the lists do less well, because these are harder.
Which groups are on the page
We wrote these rules on 20 October 2026, before we drew the test data, and we did not change them after.
- A group needs 100 test observations or more. Algae, ferns, fish, lichens, and mosses and liverworts have fewer, so they are not on the page.
- Quick to confirm (up to 3 species): the list holds the right species 88% of the time or more, the middle list has 2 names or fewer, and the first name is right 80% of the time or more.
- Genus first (up to 5 genera): else, the genus list holds the right genus 88% of the time or more, with 2 genera or fewer in the middle case.
- Else the group is for an expert, and it is not on the page.
- The model may have learned from older photos. So for each group we check the observations that became research grade in 2025 or later. If there are 40 or more, and the list holds the right name less than 85% of the time, the group goes down one band. Grasses and sedges went down (81.3% on 50 observations). Bees, wasps and ants (85.3%) and moths (87.6%) are close to the line.
| Group | Band on the page | Test observations | First name right | Right name in the list |
|---|---|---|---|---|
| Bats | Quick to confirm | 176 | 92.0% | 90.9% |
| Bees, wasps and ants | Quick to confirm | 119 | 85.7% | 88.2% |
| Beetles | Quick to confirm | 148 | 89.2% | 92.1% |
| Dragonflies | Quick to confirm | 108 | 88.0% | 93.4% |
| Flies | Quick to confirm | 148 | 87.8% | 92.9% |
| Flowering plants (dicots) | Quick to confirm | 156 | 88.5% | 92.5% |
| Fungi | Quick to confirm | 115 | 85.2% | 90.7% |
| Molluscs | Quick to confirm | 144 | 89.6% | 91.8% |
| Moths | Quick to confirm | 100 | 88.0% | 91.2% |
| Orchids | Quick to confirm | 109 | 80.7% | 91.6% |
| Spiders and kin | Quick to confirm | 149 | 90.6% | 91.3% |
| True bugs | Quick to confirm | 121 | 90.1% | 94.0% |
| Amphibians | Genus first | 154 | 77.9% | 92.7% |
| Birds | Genus first | 107 | 75.7% | 93.1% |
| Butterflies | Genus first | 115 | 76.5% | 94.3% |
| Grasses and sedges | Genus first | 129 | 82.2% | 91.1% |
| Large mammals | Genus first | 104 | 74.0% | 92.3% |
| Reptiles | Genus first | 126 | 73.0% | 90.9% |
| Rodents and shrews | Genus first | 126 | 73.8% | 92.7% |
"First name right": the right species is first in the list. "Right name in the list": the list holds the right species (Quick to confirm) or the right genus (Genus first). The lists are made to hold the right name 90% of the time.
The test that comes next
On 20 October 2026 we saved the lists for 8,044 observations that waited in the India queue (made in the last 120 days, up to 400 per group). These observations are never on the Shortlists page. We check them again on or after 19 November 2026, 19 December 2026 and 18 January 2027: how many got a confirmed name, and how often that name was in the list. Then we compare them with the observations that the page shows. The difference is the effect of the page. The saved file has the SHA-256 checksum c395de2a16730e31338b02953f371fe8435cd20d606cb99724d0661d2e523398, recorded before the first check.
To see if the page helps, we count the links that people follow from it to iNaturalist, per group, as one number per day. We keep no data about a visitor.
| Download | DOI | Created | Rows |
|---|---|---|---|
| Loading… | |||
Some species are at risk when people know where they live: collectors, traders and hunters can use location data. We follow the practice of GBIF, eBird and iNaturalist.
- We publish no coordinates. The smallest area we show is a district, on species pages and in the open data. Half of India's districts are larger than 3,100 km².
- Our MCP server for AI assistants answers by state or biogeographic province only. It does not list species by district.
- Our records come from GBIF. Before they reach us, iNaturalist moves each record of a threatened species to a random point in a cell of about 500 km², and many publishers make sensitive records less exact.
- If you think a species should show only by state, tell us. Give the species and the reason, and do not write any location. We record the reason and a date to review it.
AI assistants can read the list through our MCP server at https://overlooked-india.pages.dev/mcp (Streamable HTTP, no key, read only). It can find a species by name, give one species in full, count and list species by group, state, biogeographic region, IUCN category, endemic mark, order or family, and list new species. Each answer gives the week of the list, the licence, the credits and the disclaimer. The data guide tells agents about the server and the open files.
| Source | Used for | Answer | Time |
|---|---|---|---|
| Loading… | |||
Sources and credit
| Source | Used for | Licence or terms |
|---|---|---|
| GBIF.org | Observations and specimens with a place in India | Each record keeps the licence of its dataset (CC0, CC BY or CC BY-NC). Cite the download DOIs above. |
| Catalogue of Life | Species names, as GBIF uses them; the year each species was described | CC BY 4.0 |
| Local Government Directory, via bharatlas | District boundaries | CC0 1.0 / CC BY 4.0 |
| Wildlife Institute of India (Rodgers, Panwar and Mathur), via bharatlas | Biogeographic zones and provinces of India (10 zones, 26 provinces). A GBIF record takes the province its point is in. | GODL-India |
| iNaturalist | Species with research-grade observations per state; observations that wait for a name, by species, genus and group | Counts from api.inaturalist.org, within its request limits. Each observation keeps its own licence. Open-licence research-grade observations also reach GBIF. |
| State of India's Birds | Birds with a known population trend | The SoIB Partnership's main table (SoIB 2023; the 2025 and 2026 updates keep its trend categories), MIT licence, Copyright (c) 2023 The SoIB Partnership. We publish only counts. Cite: SoIB (2023). State of India's Birds, 2023: Range, trends, and conservation status. The SoIB Partnership. doi:10.5281/zenodo.11124589. |
| GloBI | Species with a recorded interaction, anywhere and in India | GloBI's weekly snapshot of all its interactions, compiled from about 460 datasets; each interaction keeps the licence of its dataset. We publish only counts per species. Cite: Poelen, J.H., Simons, J.D. and Mungall, C.J. (2014). Global Biotic Interactions: an open infrastructure to share and analyze species-interaction datasets. Ecological Informatics 24: 148–159, and the datasets at globalbioticinteractions.org. |
| Species photos | One photo per species and an English common name, on the species page | A photo from India first: research-grade iNaturalist observations from India (via GBIF). Else iNaturalist's taxon photos, else the Wikimedia Commons image that Wikidata gives. Only photos with a Creative Commons licence (CC BY-NC included); never "all rights reserved". Each photo keeps its own licence and credit, shown under it; we link to the image and do not host it. No photo: the page asks for one on iNaturalist, where others confirm the name. |
| IUCN Red List | Red List category of each species, the year of its assessment, and its population trend | Categories and years: the Red List checklist that IUCN publishes on GBIF (CC BY 4.0), doi:10.15468/0qnb58, accessed via GBIF.org. Population trends, and whether a species' range map includes India (to order the names to check): the IUCN Red List API, version 2026-1. Cite: IUCN (2026). The IUCN Red List of Threatened Species. Version 2026-1. https://www.iucnredlist.org. Each species page links to its assessment. |
| India Biodiversity Portal | Its records in GBIF | CC BY 4.0. Its GBIF dataset has records until 2024. |
| Botanical Survey of India (national checklists; Plant Discoveries; regional and family books) | Plant species of India; new plant species and new records for India; lichens, algae and fungi from seven free books (Macrolichens of Sikkim, Algal Flora of Dehradun District, Phycologia Indica 1 and 2, Meliolales of India, Hymenochaetaceae of India, Russulaceae of Kumaon Himalaya). These books are scans: we read them with OCR and keep a name only when GBIF knows it as a species of the right kingdom. | Reproduced free of charge, accurately, with acknowledgement. Source: Botanical Survey of India, Ministry of Environment, Forest and Climate Change, Government of India. |
| Zoological Survey of India (Fauna of India Checklist; Animal Discoveries) | Animal species of India (Fauna of India checklists); new animal species (Animal Discoveries) | Reproduced free of charge, accurately, with acknowledgement. Source: Zoological Survey of India, Ministry of Environment, Forest and Climate Change, Government of India. |
| Bibliography of South Asian Ornithology | Papers on each bird of South Asia since 1713 (about 41,000 papers), and the year of the latest | Compiled by Aasheesh Pittie (southasiaornith.in), a public Zotero library. We count the papers tagged with each bird name and keep the years; we publish only these counts, not the bibliography. A tag counts for a species when the name matches it, or GBIF's name match gives it (an older name counts for the species); tags with no bird genus on the list, such as places, are left out. |
| Bryophyte checklist of India (source "Bryophytes") | Mosses, liverworts and hornworts of India (2,489 taxa by the paper's count) | Dandotiya, D., Govindapyari, H., Suman, S. and Uniyal, P.L. (2011). Checklist of the bryophytes of India. Archive for Bryology 88: 1-126. From the University of Frankfurt's repository. We use only the names, as facts for the India list, and publish counts; we do not copy the paper's list. Read from the paper's text: 1,780 mosses, 672 liverworts and 25 hornworts. The paper leaves out 12 numbers; 30 rows name only a genus ("Bryum sp.") and 4 rows have a shortened genus with no heading above them, so they are not used. Each name is matched with GBIF. |
| Lichen checklist of India (source "Lichens") | Lichen species of India (3,236 taxa), and which are endemic | Sinha, G.P., Nayaka, S. and Mishra, G.K. (2024). A comprehensive checklist of lichens from India - 2024. Cryptogam Biodiversity and Assessment 8(2): 1-134. doi:10.21756/cab.v8i2.01. The paper is under CC BY-NC 4.0 (non-commercial). We use only the names and their endemic mark, as facts for the India list, and publish counts; we do not copy the paper's table. Read from the paper's text, row by row; every row from 1 to 3,236 is checked. |
| Plazi (TreatmentBank) | Species described from India since 2007 | CC0 |
| Royal Botanic Gardens, Kew (WCVP) | Plant species of India, and which are endemic; the year each plant was described | CC BY 4.0 |
Download the data
Parquet files of this snapshot. You can read them with DuckDB, pandas or R. Older snapshots stay available.
- Loading…
Licence
Data and text on this site: CC BY 4.0, Sathya Sankaran. Code: MIT.
Each source keeps its own licence. When you use these figures, cite this site, the GBIF download DOIs, and the sources above.