AI crawler

AEO - Answer Engine Optimization - making a page readable, parseable and quotable by a system that answers the question instead of returning links.
GEO - Generative Engine Optimization - the same work framed around how you are represented inside generated text rather than whether you are cited.
LLMO - Large Language Model Optimization - a third label for the same territory, framed model-first.
SEO - Search Engine Optimization - optimising for a ranking that produces a click.
Answer engine - A system that answers a question directly - ChatGPT, Perplexity, Claude, Google AI Overviews - instead of returning a list of links.
AI visibility - Whether language models such as ChatGPT, Claude and Perplexity can find, parse and cite a page.
llms.txt - A Markdown file at the root of a domain that gives an AI crawler a curated map of the site.
llms-full.txt - The full-text variant of llms.txt - every page of the site concatenated into one file.
Markdown mirror - A plain-Markdown twin of an HTML page, served at the same URL with .md appended.
AGENTS.md - A file at the root of a repository describing it to coding agents - conventions, commands, constraints.
sitemap.xml - A machine-readable XML index of every public URL on a site.
Content-Type header - The HTTP header declaring the media type of a response - what decides whether llms.txt is text or an HTML page.
JSON-LD - Structured data written as a JSON block in a script tag, decoupled from the visible markup.
Schema.org - The shared vocabulary JSON-LD is written in - Article, FAQPage, Person, Organization, BreadcrumbList.
Article schema - The schema.org type for a written article, carrying headline, author, datePublished and dateModified.
FAQPage - Schema.org markup declaring question-and-answer pairs on a page.
@graph - A JSON-LD container holding several related entities in one script, connected by @id references.
@id - A stable unique identifier for an entity inside JSON-LD markup.
datePublished - The ISO 8601 timestamp marking when a page first went live. It never changes.
dateModified - The ISO 8601 timestamp of the last meaningful change to the content - the freshness signal a model reads.
author field - The structured Person or Organization object in JSON-LD that says who is responsible for the content.
sameAs - A JSON-LD array linking an entity to its profiles elsewhere - GitHub, LinkedIn, Wikipedia, a company register.
Rich results - Enhanced search result displays - stars, FAQs, breadcrumbs - triggered by valid schema markup.
Microdata and RDFa - Attribute-based syntaxes for structured data - still valid, but coupled to the markup.
Entity - A person, organisation or thing with a persistent identity that can be referenced across sites.
Semantic HTML - Markup whose elements state what the content is rather than how it looks.
Heading hierarchy - The nesting of h1 to h6 that gives a document its outline and tells a chunker where a section begins.
H1 - The document's topic declaration for machine readers - exactly one per page.
Landmark elements - The semantic tags marking major page regions - header, nav, main, article, aside, footer.
Canonical - The declared preferred URL for a piece of content, when several URLs serve it.
Meta description - The 50 to 160 character summary in the document head, used to build a snippet.
Open Graph - The og: meta tags that standardise how a page is represented in previews and shares.
og:image - The Open Graph tag naming the preview image, 1200 by 630 pixels.
Citation card - The block an answer engine renders for a source - title, description, preview image.
AI crawler - Any automated client that fetches web pages on behalf of a large language model product.
Training crawler - A bot collecting text to train the next generation of models - GPTBot, ClaudeBot, CCBot, Bytespider.
Search indexer - A bot building the retrieval index an assistant cites from - OAI-SearchBot, PerplexityBot, Claude-SearchBot.
User-triggered fetcher - An agent that fetches a single URL because a person asked the assistant to read it.
GPTBot - OpenAI's training crawler. Honours robots.txt.
ClaudeBot - Anthropic's training crawler. Honours robots.txt and crawl-delay.
PerplexityBot - Perplexity's search indexer. Honours robots.txt; its user-triggered sibling generally does not.
Google-Extended - A control token that blocks Google's AI training use without affecting Google Search inclusion.
robots.txt - The file at the root of a domain telling crawlers which paths they may request.
noindex - A directive in a robots meta tag or an X-Robots-Tag header that keeps a page out of an index.
X-Robots-Tag - The HTTP response header carrying robots directives - the invisible twin of the robots meta tag.
nosnippet - A directive suppressing text fragments from a page in results, and apparently direct quoting by AI systems.
Content-Signal - A robots.txt line, proposed by Cloudflare, expressing preference by use case rather than by bot name.
RFC 9309 - The IETF standard for the Robots Exclusion Protocol - it defines robots.txt as voluntary guidance.
User-agent string - The free-text header a client uses to identify itself - trivially spoofable.
IP verification - Confirming a crawler's identity by matching its source IP against the vendor's published address ranges.
Embedding - A fixed-length list of numbers representing the meaning of a text as a direction in space.
Vector - The embedding itself, treated as a direction in space - and, in the plural, what a vector database indexes.
Cosine similarity - The measure of the angle between two vectors, from -1 to 1.
Dimension - One numeric position inside an embedding - 384 of them in all-MiniLM-L6-v2.
Chunking - Splitting a page into passages that are embedded and retrieved independently.
Hierarchical chunking - Chunking that carries the heading path into each passage, so a fragment keeps its context.
Retrieval - The stage where relevant passages are found, before any generation happens.
RAG - Retrieval-Augmented Generation - retrieve relevant chunks, then generate an answer from them.
Grounding - Constraining a model's answer to retrieved sources rather than to its trained weights.
Hallucination - A confident, fluent, false statement produced by a model.
Tokenizer - The component that splits text into pieces the model knows, breaking unknown words into sub-words.
Sentence transformer - A model trained on sentence pairs so that semantically close sentences produce close vectors.
Client-side rendering - An empty container in the HTML, filled by JavaScript after the page loads.
Server-side rendering - HTML generated per request on the server, complete in the first response.
Static site generation - HTML generated at build time and served as a static file from a CDN.
Incremental static regeneration - Static files that rebuild on a schedule or via a trigger.
Streaming SSR - Server-rendered HTML sent in pieces - structure first, slow content after.
Hydration - The step where client JavaScript takes over a server-rendered page and makes it interactive.
Server components - Components that render on the server by default in the Next.js App Router, until marked with 'use client'.
Text-to-HTML ratio - Visible text divided by raw HTML size - a fast proxy for whether content arrives in the first response.
AI Overviews - Google's generated answer above the organic results, with cited sources.
Citation - A named linked reference to a page inside a generated answer.
Zero-click - A query answered without any visit to a source.
AEO score - The 0 to 100 summary computed for a page from its parseability signals.
AEO Checker - A Chrome extension that audits the page you are on for AI-readiness and scores it out of 100.
AEO Adoption Scan 2026 - A scan of 93 pages across 78 domains measuring how often each AEO signal actually appears.
Question headings - Section headings phrased as questions, mirroring how people prompt an answer engine.
Featured snippet - A passage lifted verbatim from a page and shown above the results.
Embedding Playground - A browser tool that embeds up to five texts and shows the cosine similarity matrix between them.

На alexturik.com

AI Crawler List 2026: Every User-Agent, robots.txt per Bot