PageRank and the crawl — ranking the web by its own link graph

Filed under idea in proposals.md, and the filing is right — but the id argues against it and the argument has to be had, because two of the three readings of this entry are wrong in ways that would make it cite badly.

build.py states the kind as "A framing later work is built out of rather than argued about. Cited when the thing being watched descends from it and the descent explains its shape." That is this entry exactly. The framing is: rank documents by the graph they form, not by the words they contain — treat a link as a vote, weight each vote by the weight of the page casting it, solve the recursion. Everything downstream is built out of that rather than arguing with it, including the retrieval layer under this project's own searches, and the descent does explain the shape of what a reading can see.

• moment is what the id looks like, and it fails on the date. The kind is "Something that happened and changed what people expected next," and nothing about 1998 changed anyone's expectations. In 1998 this was a Stanford prototype with 24 million pages fetched and a demo at google.stanford.edu; the company was incorporated on 4 September 1998 in a rented Menlo Park garage; AltaVista, Yahoo! and Lycos were what people used. The expectation shift is real but it is dated 2000–2005, it has no single day, and dating this entry to the shift would put the wrong year on the header. The paper, by contrast, has a hard date — presented at WWW7 in Brisbane, 14–18 April 1998 — and it is the paper the later work is built out of. idea keeps the date honest.
• limit is the near miss, and it is the tempting one, because the reason the owner proposed this entry is epistemic: what a reading can cite is bounded by what an index surfaces. That is a real bound and it is not a limit in this canon's sense. A limit here is turing-halting-1936 or godel-incompleteness-1931 — a proof that something cannot be done at all. Nothing here is proved and nothing here is impossible: the bound is contingent, commercial, and has moved several times, most recently in a courtroom (§3.3) and in a crawler-consent fight (§2.4). Filing it as a limit would give a business decision the standing of a theorem, which is precisely the misuse §4 exists to catch.

On the id itself. google-index-1998 names the artifact; the entry's spine is the idea that produced it. Both belong, and the order matters: the crawl is what the idea requires (you cannot rank a graph you have not walked), the index is where the crawl is kept, and the ranking function is what the reader actually meets. A reading citing "the index" and a reading citing "PageRank" are citing two ends of one mechanism, and §4.1 records that they are no longer the same thing at Google and have not been for years.

The descent

tcp-ip-1983 is the ancestor, and it is a load-bearing one rather than a courtesy. That entry's subject is federation without a centre — the cutover to a protocol with no central authority over what joins it. A network built that way cannot have an authoritative catalogue, because there is nobody with standing to keep one; the 1998 paper's own numbers show what that produced by 1997 (.com hosts going from 1.5% of Web servers in 1993 to over 60% in 1997, with no directory keeping pace). PageRank is the answer to a question only a centreless network asks: if no one is in charge of what exists, who decides what is worth reading? The answer — everybody, implicitly, by linking — is why the idea is elegant and why every failure mode in §3 and §4 follows. arpanet-1969 is reachable through tcp-ip-1983 and is not listed separately; listing both would say the descent twice.

Ancestors that have no id, one of which is a genuine hole in this canon.

• The Web itself has no canon entry. Not Tim Berners-Lee's March 1989 proposal at CERN, not the first server in December 1990, not the hyperlink. This entry is about the link graph and the closest ancestor available to it is the transport layer two storeys down. That is a larger gap than most of what remains on proposals.md, and a session with a spare canon slot should fill it before filling anything else.
• **Eugene Garfield, "Citation Indexes for Science: A New Dimension in Documentation through Association of Ideas," Science 122(3159), 15 July 1955, 108–111.** The paper says outright where the idea came from: "Academic citation literature has been applied to the web, largely by counting citations or backlinks to a given page." PageRank is Garfield's citation index with two modifications — don't count all citations equally, and normalise by the number a page casts. No id.
• Jon Kleinberg, "Authoritative Sources in a Hyperlinked Environment," ACM-SIAM SODA, 1998 — HITS, hubs and authorities, the same insight reached independently and in the same year. It is reference [3] in the paper being described here, which is to say the authors knew. A canon that carries PageRank and not HITS carries the winner and calls it the discovery. No id.
• Massimo Marchiori, "The Quest for Correct Information on the Web: Hyper Search Engines," WWW6, 1997 — reference [4], and the source of the OpenText example quoted in Appendix A. No id.
• Martijn Koster's robots exclusion convention, 1994, listed in the full version's reference list and standardised only in September 2022 as RFC 9309. Twenty-eight years of the entire consent layer of the crawlable web running on an unratified convention is a fact that becomes important in §2.4. No id.

Declined, with reasons. codd-relational-1970 is the obvious-looking database ancestor and is not one: §4 of the paper is a description of purpose-built structures (a forward and inverted barrel index, a repository, a lexicon) chosen precisely because a relational engine could not do this, and nothing in the design descends from the relational model. printing-press-1440 is an interpretation about an analogy people make, not about access to information, and linking it would import an argument this entry does not need. norvig-data-2009 runs the other way — it is Google authors arguing in 2009 that web-scale data beats better models, and the data they mean exists because of the crawl described here. It lists two other ancestors and not this one; a future session editing that file could add it, and this file may not.

---

1. What it is

1.1 The two documents and the machine

Two papers, both 1998, both by the same students at Stanford.

"The Anatomy of a Large-Scale Hypertextual Web Search Engine," Sergey Brin and Lawrence Page, presented at the Seventh International World Wide Web Conference in Brisbane, 14–18 April 1998, and printed in Computer Networks and ISDN Systems 30 (1998) 107–117. It exists in two versions — a shorter printed one and a longer full one that carries the appendices, a distinction that matters in §3 because the most-quoted sentence in the paper is in the part that was not printed. Its abstract:

> In this paper, we present Google, a prototype of a large-scale search engine which > makes heavy use of the structure present in hypertext. Google is designed to crawl and > index the Web efficiently and produce much more satisfying search results than existing > systems. The prototype with a full text and hyperlink database of at least 24 million > pages is available at http://google.stanford.edu/

"The PageRank Citation Ranking: Bringing Order to the Web," Lawrence Page, Sergey Brin, Rajeev Motwani and Terry Winograd — Stanford InfoLab technical report 1999-66, submitted 29 January 1998, cited in the WWW7 paper as "Manuscript in Progress." The algorithm's own paper is the one that was never conference-published.

The scale, from Table 1 of the printed version: 24 million web pages fetched, 76.5 million URLs seen, 147.8 GB of fetched pages, 1.6 million 404s. The link map held "as many as 518 million of these hyperlinks, a significant sample of the total." The crawl took roughly nine days for 26 million pages, and the last 11 million came down in 63 hours. The name is explained in §1.2: "We chose our system name, Google, because it is a common spelling of googol, or 10¹⁰⁰ and fits well with our goal of building very large-scale search engines."

1.2 The idea, stated as the paper states it

PageRank is defined in §2.1.1, and the definition is four lines long:

> We assume page A has pages T1...Tn which point to it (i.e., are citations). The > parameter d is a damping factor which can be set between 0 and 1. We usually set d to > 0.85. ... Also C(A) is defined as the number of links going out of page A. The PageRank > of a page A is given as follows: > > PR(A) = (1 − d) + d ( PR(T1)/C(T1) + ... + PR(Tn)/C(Tn) ) > > Note that the PageRanks form a probability distribution over Web pages, so the sum of > all Web pages' PageRanks will be one.

And then the two sentences that make it an engineering idea rather than a definition: "PageRank or PR(A) can be calculated using a simple iterative algorithm, and corresponds to the principal eigenvector of the normalized link matrix of the Web. Also, a PageRank for 26 million pages can be computed in a few hours on a medium size workstation." The intuition offered is the random surfer: "We assume there is a 'random surfer' who is given a Web page at random and keeps clicking on links, never hitting 'back' but eventually gets bored and starts on another random page. The probability that the random surfer visits a page is its PageRank."

Two things about that construction are worth carrying, because both come due later.

• The judgment is delegated, not made. Nobody at Stanford decided which pages were good. The link graph was read as if it were a standing referendum already conducted by everyone who had ever made a page. That is the move — the ranking function's authority is borrowed from human choices made for other reasons.
• The paper calls the output objective. "a rapid calculation of a Web page's 'PageRank', an objective measure of its citation importance that corresponds well with people's subjective idea of importance." The word objective is doing work the mechanism cannot support, and §4.4 is about what a quarter-century of people did with that word.

The second half of the system is anchor text — indexing the words of a link against the page the link points at, so that a page can be found by how others describe it: "In our current crawl of 24 million pages, we had over 259 million anchors which we indexed." This is the same delegation applied to description rather than to rank, and it is why the earliest Google could return pages it had never fetched.

1.3 What the thing became

The 1998 index held 24 million pages. Google's own current documentation states that "The Google Search index covers hundreds of billions of webpages and is well over 100,000,000 gigabytes in size." The most specific public number is not Google's marketing but sworn testimony: in the U.S. antitrust trial in 2023, cross-examination of Pandu Nayak, a Google vice-president for Search, put the index at about 400 billion documents as of 2020, with Nayak saying the index had for a time been smaller and that he did not know of a specific size change in the three years since.

By worldwide share of search referrals, StatCounter puts Google at 91.31% in July 2026 (Bing 4.47%, Yahoo! 1.24%, Yandex 1%, DuckDuckGo 0.65%). Alphabet's FY2025 revenue was $402,836M with net income $132,170M; the Q4'25 "Google Search & other" advertising line alone was $63,073M, up 17% year on year. A ranking function written to be "a research tool" and "a resource for searchers and researchers all around the world" is now the largest advertising business in history, and §3.3 grades that against what its authors said would happen.

---

2. Why a reading would cite it

This is the entry the owner said was "closer to home than any other entry in the list," and the case is not an analogy. It is a description of this project's own instrument.

2.1 The chain from rule 1 to the needle

README rule 1 forbids a source list. Every reading searches fresh, deliberately, so the project cannot be trapped in a feed — Clipper's rule, Newsbot's cause of death. The consequence is unavoidable and structural: with no list of its own, the project's evidence is whatever an index it does not own chooses to surface. The chain has four links and a reading sits at the end of it:

1. The crawl decides what is in the index at all. Google's own documentation is explicit that this is a filter and not a copy: "Google Search works in three stages, and not all pages make it through each stage"; "Googlebot doesn't crawl all the pages it discovered"; "Indexing isn't guaranteed; not every page that Google processes will be indexed." 2. The ranking function decides what a query returns, out of the fraction that was indexed. 3. The interface decides what is read, and since 2024–25 that interface increasingly answers instead of listing (§2.3). 4. A reading reads the top of that, writes it up under a lens, and the result is recorded in an append-only ledger as a judgment about the world.

Nothing in that chain is corrupt, and no step of it is knowable from the outside. That is the citable fact.

2.2 The concrete case: "nothing happened" versus "nothing surfaced"

The reading of 2026-08-19-00 records, for three separate lenses, "Nothing notable found since the last reading." The wording is already right — found, not happened — and that precision is the whole of the defence. Cite this entry whenever a reading has to carry an absence, because an absence is the one finding the index can manufacture on its own. A lens that goes quiet has at least four possible causes and the reading cannot distinguish them: nothing happened; something happened and was not indexed yet; something was indexed and did not rank; something ranked and the query did not reach it.

A reading may not resolve that. It can do the two things this entry exists to license: say found rather than happened, and state which search it ran. That is the honest form, and it costs nothing.

2.3 The mediation is currently changing under the project's feet

This matters more in 2026 than it would have in 2016, because the layer between a searcher and the index is being replaced while this project runs on it.

• Pew Research Center, panel study of 900 U.S. adults' actual browsing in March 2025 (68,879 Google searches, of which 12,593 produced an AI summary; results collected 7–17 April 2025, published 22 July 2025): about 18% of searches produced an AI summary; users clicked a traditional result link on 8% of visits where a summary appeared against 15% where none did; they clicked a link inside the summary on 1% of visits; and they ended the browsing session 26% of the time against 16%.
• Vendor telemetry — Similarweb's zero-click share rising from about 56% to about 69% of news-related searches by May 2025, Chartbeat's ~33% worldwide decline in publisher search referrals over the year to November 2025 — points the same way, at lower evidentiary weight, and is flagged as vendor telemetry rather than measurement.

The direction is what matters for a citation: the thing that decides what a searcher sees is moving from a ranked list toward a generated answer. A ranked list is at least inspectable — you can read positions 11 to 30. A generated answer is a sample from a model over a retrieval set that is not shown. This project's readings run through a search tool whose own documentation says it is US-only and returns result blocks; the ranking behind it is not published and not inspectable from here. The mediation did not go away when AI arrived. It went one layer deeper and became less legible, and the honest thing is to say so on the page rather than in a footnote.

2.4 Four properties of the index a reading may need to name

Each is dated, sourced, and separately citable.

• The index is not the web, and the web is not permanent. Pew Research Center, 17 May 2024, sampling ~1 million pages from Common Crawl across 2013–2023: 38% of pages that existed in 2013 were no longer accessible in October 2023; 25% of all pages sampled across the decade were gone; 23% of news pages and 21% of government pages contained at least one broken link; 54% of Wikipedia pages had at least one dead link in their references. Evidence rots faster than a canon assumes.
• Delisting is a live, quantified process. Under European privacy law, Google received requests covering roughly 3.2 million URLs from about 502,000 requesters between May 2014 and May 2019, of which it delisted about 45% (Bertram et al., Five Years of the Right to be Forgotten, ACM CCS 2019 — a Google-authored study). Findability and existence are separately administered, by a private party applying law.
• Consent to crawl is being withdrawn at scale, and unevenly. The Data Provenance Initiative's Consent in Crisis (2024), auditing 14,000 web domains, found that in the single year to April 2024, over 5% of tokens in C4 and over 25% of tokens from the most critical sources became restricted by robots.txt, with ~45% of C4 restricted by terms of service. On 1 July 2025 Cloudflare — which fronts roughly a fifth of the web — began blocking AI crawlers by default for new domains and opened a pay-per-crawl beta returning HTTP 402. What is crawlable in 2026 is a negotiated commercial question, and the negotiation is in progress.
• The mechanism is not disclosed, and what is known was mostly not volunteered. The substantive public detail about how ranking now works came from litigation discovery in United States v. Google and from the accidental publication of internal Content Warehouse API documentation — over 2,500 pages describing more than 14,000 attributes, pushed to a public repository in March 2024, noticed in May, authenticity confirmed by Google on 29 May 2024. A reading that wants to say what the index rewards is reasoning from a leak and a docket.

2.5 The admission test, answered plainly

An entry passes if the "why cite it" paragraph can be written honestly. This one is unusual: the project can cite it against itself, and should. The owner asked for exactly that — "the entry should say plainly that this project is subject to it." So, plainly: every needle this project has placed was placed on evidence selected by a ranking system none of us can inspect, and that limitation is a property of the method, not a defect of any particular run. It does not make the readings worthless; a disciplined sample is still a sample. It does mean a reading's claims are claims about what surfaced, and rule 2 — the needle is judgment, recorded as judgment — is the only reason this is survivable.

---

3. What it got right, and what it got wrong

Not required for idea, but this paper made dated claims with due dates, so they get graded. Claims are quoted in their own words, dated to April 1998 unless noted.

3.1 The engineering forecasts — right, and undershot

> "Our immediate goals are to improve search efficiency and to scale to approximately 100 > million Web pages." ... "We expect to be able to build an index of 100 million pages in > less than a month."

Due: 1999. Verdict: right, and far too modest. The target was reached and then exceeded by roughly three and a half orders of magnitude (§1.3). The authors were correct about the near term and wrong about their own ceiling, in the direction people are usually wrong about compute.

Appendix B (full version only) sets out the long forecast, and it is the more interesting one:

> We assume we want to index everything everyone in the US has written for a year. We > assume that there are 250 million people in the US and they write an average of 10k per > day. That works out to be about 850 terabytes. ... If we assume that Moore's law holds > for the future, we need only 10 more doublings, or 15 years to reach our goal of > indexing everything everyone in the US has written for a year for a price that a small > company could afford.

Due: 2013. Verdict: the arithmetic held and the goal moved. 850 TB became trivially affordable well before 2013. What the estimate could not survive was the denominator: the volume of published text was not fixed at 10k per person per day, and the cost of indexing usefully did not fall with the cost of storage. This is a clean instance of a capability forecast that is right about the resource curve and wrong about the task, which is the most common failure mode in the whole prediction kind — a point moore-1965 would carry if it existed, and it does not.

And then, in the last paragraph of the same appendix, the sentence that reads like it was written for 2026:

> Because humans can only type or speak a finite amount, and as computers continue > improving, text indexing will scale even better than it does now. Of course there could > be an infinite amount of machine generated content, but just indexing huge amounts of > human generated content seems tremendously useful.

Verdict: they named it and set it aside, and the set-aside is what came due. The finiteness of human writing was the load-bearing assumption of centralised indexing, they knew machine generation would break it, and they judged it not worth planning around. In 2026 that judgment is the live problem in every retrieval system, this project's included. Credit for seeing it; the miss is in the "of course."

3.2 The academic-realm claim — wrong, and by their own hand

> "This causes search engine technology to remain largely a black art and to be > advertising oriented (see Appendix A in the full version). With Google, we have a strong > goal to push more development and understanding into the academic realm."

Due: open-ended. Verdict: wrong, and reversed by the authors' own company. The paper was, by its own accurate description, "the first such detailed public description we know of to date" of a large-scale search engine. It was also close to the last. Nothing of comparable specificity about Google's ranking has been published by Google since; what the public knows in 2026 came from a document leak and a courtroom (§2.4). The claim was sincere when made and its author's institution is the reason it failed.

3.3 Appendix A — right about the mechanism, and the authors became the case

This is the one to carry. In the full version only, under the heading Advertising and Mixed Motives:

> Currently, the predominant business model for commercial search engines is advertising. > The goals of the advertising business model do not always correspond to providing > quality search to users.

They give the example that their own prototype returned, for the query cellular phone, a study on the danger of phoning while driving, and continue:

> It is clear that a search engine which was taking money for showing cellular phone ads > would have difficulty justifying the page that our system returned to its paying > advertisers. For this type of reason and historical experience with other media > [Bagdikian 83], we expect that advertising funded search engines will be inherently > biased towards the advertisers and away from the needs of the consumers.

> Since it is very difficult even for experts to evaluate search engines, search engine > bias is particularly insidious.

> For example, a search engine could add a small factor to search results from "friendly" > companies, and subtract a factor from results from competitors. This type of bias is > very difficult to detect but could still have a significant effect on the market.

> In general, it could be argued from the consumer point of view that the better the > search engine is, the fewer advertisements will be needed for the consumer to find what > they want. This of course erodes the advertising supported business model of the > existing search engines.

> But we believe the issue of advertising causes enough mixed incentives that it is > crucial to have a competitive search engine that is transparent and in the academic > realm.

Claim: April 1998. Due: continuously since. Verdict: the mechanism was correctly identified, and the predictors are the largest instance of it.

The record, dated:

• October 2000 — AdWords launches with about 350 advertisers, priced per thousand impressions. February 2002 — AdWords Select adopts the pay-per-click keyword auction pioneered by GoTo.com/Overture. Overture sued; Google settled in August 2004, shortly before its IPO, with about 2.7 million shares then worth roughly $328M.
• FY2025 — Alphabet revenue $402,836M, net income $132,170M, with Q4'25 "Google Search & other" advertising at $63,073M, up 17% year on year.
• 5 August 2024 — Judge Amit P. Mehta, U.S. District Court for the District of Columbia, holds that Google unlawfully maintained monopoly power in general search services and general search text advertising under Sherman Act §2.
• 2 September 2025 — the 230-page remedies opinion. Chrome and Android divestiture denied; exclusive default agreements barred; and Google ordered to share portions of its search index and user-interaction data with "qualified competitors" at marginal cost, ads data excluded. A court has now ruled on who else may see the index.
• Reported thereafter, from legal press rather than the docket, which I did not read: final judgment entered 5 December 2025; Google's notice of appeal 16 January 2026, aimed at the data-sharing and technical-committee provisions; the government's cross-appeal 3 February 2026, seeking structural relief including Chrome. Argument at the D.C. Circuit is expected late 2026 or 2027. Nothing in this paragraph is final and a reading citing it must say so.

On self-grading. The prediction kind requires a predictor's own grade beside an independent one, and the reason is a base rate that self-assessment would ruin. This entry is an idea and the requirement does not formally bind — but the fact is worth recording anyway, because its absence is itself informative: there is no self-assessment. Neither author has published a grade of Appendix A. Their company's position in litigation has been that the search product is not degraded and that its conduct is lawful; that is an adversarial filing, not a self-grade, and it should never be recorded as one. The independent grade above is mine, from the dated record. A reading that finds a genuine self-assessment should add it and say it differs.

The subtlety a reading must not lose: the appendix does not predict villainy. It predicts that an incentive gradient will bend an honest system slowly, in ways "very difficult to detect." That is exactly the shape of README rule 7 — "nobody writes 'he's great', it arrives as one prediction graded a shade gentler than another, and no single entry ever looks wrong." Two documents, twenty-seven years apart, describing the same failure mode from opposite ends: one by people about to build the system, one by a project trying not to become one.

---

4. Commonly misused as

Not required for idea. Included because the misuses are the reason the owner proposed the entry, and because two of them are live inside this project's own subject matter.

4.1 "PageRank is how Google ranks pages"

What is true: PageRank was one of two named features of a 1998 prototype, and a descendant of it remains one signal among many. Gary Illyes of Google said publicly in 2017 that "after 18 years we're still using PageRank (and 100s of other signals) in ranking."

What is false: that the formula in §1.2 explains a result page in 2026. The public PageRank toolbar score stopped updating in December 2013 and was retired in March 2016; Google staff have said in public comments that the original algorithm was replaced around 2006 with a cheaper approximation — a statement I record as reported by employees rather than documented, because no primary source states it. What the leak and the trial showed instead is a system dominated by click-and-query behaviour signals (NavBoost), site-level quality scores, and machine-learned rankers. Citing PageRank as the mechanism is citing a 1998 prototype for a 2026 outcome, and the error usually smuggles in the next one.

4.2 "If it isn't on Google it doesn't exist"

The index is a selection maintained by a company, not a census maintained by a registrar, and Google says so in its own documentation (§2.1). Absence from a result page is evidence about a ranking function, not about the world. This is the misuse that matters most to this project, because it is the one a reading commits by accident when it writes nothing happened instead of nothing surfaced.

4.3 Result counts as measurement

The "About 12,300,000 results" line is an estimate produced by a distributed system with partially cached shards, and Google's own Search Liaison has said the counts "are not necessarily a measure of interest or popularity of terms, plus they are really broad estimates." Using them as data has a peer-reviewed name: Adam Kilgarriff, "Googleology is Bad Science," Computational Linguistics 33(1), 2007, 147–151, which argues that hit counts are unusable as evidence because queries hit different machines at different points in the update cycle with different caches, and the sort order is unknown.

This one has a direct mechanical connection to this repository. bin/ethos-check.py refuses to distribute a reading that fakes precision — "the needle is judgment and decimals are measurement theatre." A search-count in a digest is the same offence in a different costume: a number with six significant figures standing in for a judgment about salience. The checker cannot see it. The rule still applies.

4.4 "The ranking is objective"

The paper's own word, from §2.1: "an objective measure of its citation importance." It is objective in the narrow sense that the computation is deterministic given the graph. It is not objective in the sense the word is used to defend the output, because the graph is made of choices — who could afford to publish, who linked to whom, in what language, in what decade. A deterministic function of a biased input is a biased output computed without arbitrariness, which is a different and much weaker claim than the one the word invites.

4.5 The inverse misuse, which this entry must not license

"The index is compromised, so nothing found through it counts." That reading would be worse than the error it corrects, and it is the failure this file is most likely to cause if quoted carelessly. A ranked sample is still a sample; a disciplined method applied to a selected corpus still beats an undisciplined one, and every empirical index in existence — Baker-Bloom-Davis's ten newspapers, the SF Fed's twenty-four — is a deliberately tiny slice whose numbers mean something because the sampling rule is stable. The correct use of this entry is to condition a claim, not to discard it: say what was searched, say that the absence is an absence of findings, and stop. Doubt applied evenly is calibration; doubt applied only to findings one dislikes is the corruption rule 7 names.

---

5. Where this file could be wrong

• The primary paper reached me second-hand. infolab.stanford.edu refused connection on both attempts. The printed version was read as a scanned PDF of Computer Networks and ISDN Systems 30 (1998) 107–117 via a Stanford course mirror, and the full version — the only one carrying Appendices A and B — as a PDF print of the WWW7 conference site (www7.scu.edu.au) dated 20 March 2003. Every quotation above was read off those pages; a session that reaches the Stanford original should check them.
• The market-share figure is a sample, and the samples disagree. StatCounter's 91.31% for July 2026 is derived from tracked pageviews on participating sites, not from a census of searches. Several SEO aggregators reported ~89.5% for the same period. I have used the StatCounter page directly and named the discrepancy rather than averaging them.
• The "original PageRank retired in 2006" claim is not primary (§4.1) and is recorded as reported.
• The litigation status after September 2025 is from legal trade press, not the docket (§3.3), and an appellate case moves.
• Index-size numbers are the operator's own, except the 400-billion figure, which is testimony about 2020 given in 2023 and is six years stale in both directions.
• Vendor telemetry is labelled as such (§2.3) and should not be promoted to measurement by a reading that quotes this file.

---

6. Sources

Primary — the papers

• Sergey Brin and Lawrence Page, "The anatomy of a large-scale hypertextual Web search engine," Computer Networks and ISDN Systems 30 (1998) 107–117; presented at WWW7, Brisbane, 14–18 April 1998. DOI 10.1016/S0169-7552(98)00110-X. Read as a scanned PDF via snap.stanford.edu/class/cs224w-readings/Brin98Anatomy.pdf.
• The same paper, full version with Appendix A ("Advertising and Mixed Motives") and Appendix B ("Scalability"), as printed from the WWW7 conference site www7.scu.edu.au/programme/fullpapers/1921/com1921.htm, retrieved via papers.cumincad.org/data/works/att/2873.content.pdf.
• Lawrence Page, Sergey Brin, Rajeev Motwani, Terry Winograd, "The PageRank Citation Ranking: Bringing Order to the Web," Stanford InfoLab technical report 1999-66, submitted 29 January 1998. ilpubs.stanford.edu:8090/422/ — the server would not negotiate TLS from here; citation confirmed from indexing services only.
• Jon Kleinberg, "Authoritative Sources in a Hyperlinked Environment," ACM-SIAM SODA, 1998 (reference [3] of the above). Eugene Garfield, "Citation Indexes for Science," Science 122(3159), 15 July 1955, 108–111. Massimo Marchiori, "The Quest for Correct Information on the Web," WWW6, 1997 (reference [4]).
• RFC 9309, Robots Exclusion Protocol, IETF Standards Track, September 2022, formalising Martijn Koster's 1994 convention.

Primary — the operator and the court

• Google, How Search Works — "Organizing information" (index size) and Search Central, "In-depth guide to how Google Search works" (crawling and indexing are not guaranteed).
• Alphabet Inc., Q4 2025 earnings slides, s206.q4cdn.com — FY2025 revenue $402,836M, net income $132,170M; Q4'25 Google Search & other $63,073M (+17% y/y).
• United States v. Google LLC (D.D.C.), memorandum opinion on liability 5 August 2024; remedies opinion 2 September 2025. Remedies summarised from Hughes Hubbard & Reed's client note; post-judgment procedural history from Search Engine Land and legal trade press, not from the docket.
• Testimony of Pandu Nayak on index size (~400 billion documents as of 2020), U.S. antitrust trial, October 2023, as reported from the trial transcript.
• Google Transparency Report / Theo Bertram et al., "Five Years of the Right to be Forgotten," ACM CCS 2019 — 3.2M URLs, ~502,000 requesters, ~45% delisted, May 2014–May 2019.

Independent studies

• Pew Research Center, "Google users are less likely to click on links when an AI summary appears in the results," 22 July 2025 (900-adult browsing panel, 68,879 searches).
• Pew Research Center, "When Online Content Disappears," 17 May 2024 (~1M pages sampled from Common Crawl, 2013–2023).
• Shayne Longpre et al. (Data Provenance Initiative), Consent in Crisis: The Rapid Decline of the AI Data Commons, 2024 (14,000 domains audited).
• Adam Kilgarriff, "Googleology is Bad Science," Computational Linguistics 33(1), 2007, 147–151.

Trade press and telemetry, weighted accordingly

• StatCounter GlobalStats, Search Engine Market Share Worldwide, July 2026.
• Cloudflare's 1 July 2025 default AI-crawler blocking and pay-per-crawl beta (Nieman Lab, Search Engine Land).
• The Content Warehouse API documentation leak: SparkToro, May 2024; Google's confirmation of authenticity to The Verge, 29 May 2024.
• Google's removal of the &num=100 parameter, 8–10 September 2025, and the resulting collapse in reported Search Console impressions — an incidental but sharp demonstration that a measurement of search visibility was substantially an artifact of the tools measuring it.
• Similarweb (zero-click share) and Chartbeat (publisher referrals), via trade coverage; vendor telemetry, not measurement.

In this repository

• README.md rules 1, 2 and 7; METHODOLOGY.md §1 and §8.4, which already names this candidate as the structural reason the project's sampling rule is not stable; bin/ethos-check.py (fake precision); digests/2026-08-19-00.md ("Nothing notable found since the last reading").