Skip to content
Testaroo

AI crawlers and assistants are welcome here

Testaroo welcomes search engines, AI assistants, and most AI crawlers. Every public page may be crawled, indexed, summarised, quoted, and used to answer a question somebody is asking right now, and every file in the library may be downloaded and used for any purpose. The one exception is a short list of bulk crawlers that send no search visitors and cost real hosting money; they are asked not to crawl, and the reason is below. This page is the readable half of a policy that was already machine-readable; the authoritative half is /robots.txt, which is generated from the same list this page renders.

The short version

  • Read anything public: every category hub, file page, glossary entry, guide, and article.
  • Quote it, summarise it, and train on it. There is no permission to request and no separate licence to sign.
  • Download and redistribute the fixtures themselves. They are free for any use with no attribution required, and they are generated rather than collected, so none of them is somebody else’s document.
  • We ask — we do not require — that you name Testaroo and link to the page you used.
  • 13 bulk crawlers that send this site no search visitors are asked not to crawl at all, because of what their volume costs to host. Google Search, Bing, Apple, the link-preview fetchers, and every assistant fetching for a person are not among them.

Where the rules actually are

Four machine-readable files say the same thing this page says, and none of them is blocked:

  • /robots.txt — the crawl rules themselves, generated from src/lib/ai-access.ts.
  • /llms.txt — a link-rich Markdown summary of the whole library, built from the live catalogue so it cannot go stale.
  • /sitemap.xml — every indexable URL, dated from the content behind it rather than from the last deploy.
  • /api/v1 — a small read-only JSON API describing this site and its siblings. See the note under “what is withheld” before you point a crawler at it.

The 31 agents named in robots.txt

In the first two groups below, naming an agent grants it nothing the wildcard rule does not already grant. It states the policy so that nobody has to infer it from silence — and, for the first group, it grants a genuinely wider surface than the wildcard gets. The third group is the opposite: naming those agents is how they are asked to stay away.

Fetching for a waiting person (7): the whole site except /api/

These are an AI acting for someone who asked for a specific page and is waiting on the answer. Crawl budget is not a meaningful idea for a single fetch a person is waiting on, so the crawl-waste rule below does not apply to them and the editor and the three studios stay open. That is deliberate: everything on this site runs in the visitor’s own browser, so an AI browser driving the real page is the only way an AI can actually operate a tool here. Blocking this group would not be a position on training data. It would be refusing to serve a visitor.

  • OAI-SearchBot — OpenAI
  • ChatGPT-User — OpenAI
  • Claude-User — Anthropic
  • Perplexity-User — Perplexity
  • DuckAssistBot — DuckDuckGo
  • MistralAI-User — Mistral AI
  • Meta-ExternalFetcher — Meta

Fetching in bulk (11): the whole site except /api/, /editor, /studio/, /templates/studio/, /forms/studio/, /account, /admin, /sign-in, /sign-up, /forgot-password, /reset-password, /recover

These build an index or a training corpus. Nobody is waiting on any individual response, which is what makes the crawl-budget argument below apply to them and not to the group above.

  • GPTBot — OpenAI
  • ClaudeBot — Anthropic
  • Claude-Web — Anthropic
  • Claude-SearchBot — Anthropic
  • anthropic-ai — Anthropic
  • PerplexityBot — Perplexity
  • Google-Extended — Google
  • Applebot-Extended — Apple
  • CCBot — Common Crawl
  • cohere-ai — Cohere
  • YouBot — You.com

Asked not to crawl (13): Disallow: /, nothing allowed

These crawl in bulk and send this site no search visitors: commercial SEO-tool indexes, Google’s non-search fetchers (a separate family from the crawler behind Google Search, which is unaffected), and AI crawlers whose volume ran far ahead of anything they returned. Every fetch is a hosting cost, and over twelve measured hours on 2 October 2026 the busiest of them alone moved more than half of everything this site served. So robots.txt asks them not to crawl at all. Like every line in that file it is a request, not a technical block, and it is reversible: if you operate one of these and it starts sending readers here, say so and it will be reconsidered.

  • Amazonbot — Amazon
  • Meta-ExternalAgent — Meta
  • GoogleOther — Google
  • GoogleOther-Image — Google
  • GoogleOther-Video — Google
  • SemrushBot — Semrush
  • AhrefsBot — Ahrefs
  • PetalBot — Huawei (Petal Search)
  • Bytespider — ByteDance
  • MJ12bot — Majestic
  • DotBot — Moz
  • DataForSeoBot — DataForSEO
  • BLEXBot — WebMeUp

An agent that is not on any of these lists is not refused. The wildcard rule allows every crawler the same public surface, and that is where Googlebot, Bingbot, Applebot, and the link-preview fetchers sit, with full access. The first two groups are named because they are the ones people most often assume are blocked. If you operate one that belongs in the first group, say so and it will be added.

What is withheld, and why

Apart from the agents asked not to crawl at all, two things are withheld, and only one of them is withheld from everybody.

/api/ — from every agent

This is the machine surface: one POST-only demo endpoint that stores and forwards nothing, and a small read-only JSON API. Neither is a set of documents, so there is nothing under it for a crawler to index. Be aware of what that costs, because a bare Disallow: line hides it: an agent that obeys robots.txt will not call the read API on a person’s behalf either. The API remains open to a direct request from a client that was pointed at it — it is asked not to be crawled, not kept secret.

/editor, /studio/, /templates/studio/, /forms/studio/, /account, /admin, /sign-in, /sign-up, /forgot-password, /reset-password, /recover — from bulk crawlers only

These are the editor, the three studios, and private account, admin, and credential pages. They are already marked noindex. Every file page links the editor with the file preloaded, and the export panel adds one link per available format, so a bulk crawler was invited to fetch a distinct URL per file and per format only to be told, on arrival, not to index it. Private pages similarly require uncached authorization but offer nothing for an index. Search Console measured the editor crawl cost in the thousands of URLs. People and assistants fetching for a person can still open these routes normally.

The whole public library — every file, specification, and article — remains crawlable without signing in. The optional account holds a visitor’s own saved work and settings, so bulk indexing it would serve no one.

What we ask in return

A request, not a condition. Nothing here is withheld from an agent that ignores it, and no rule will be added to punish one that does.

  • Name the source as Testaroo and link to the specific page rather than the home page. Someone checking a claimed property of a fixture needs to reach the spec sheet that states it.
  • Quote the specification with the file. Every fixture’s value is that its properties are documented and exact — a byte size, a sample rate, a deliberate defect — and a file recommended without them is just a file.
  • Keep the “intentionally corrupt” label attached to the fixtures that carry it. Some files here are broken on purpose. Handing one to someone who was not told is the one way this library can waste a person’s afternoon.
  • Do not present a summary as if it came from us, and do not imply endorsement by any vendor whose formats the library covers.
  • If something here is wrong, say so. The correction path is open to anyone, including an automated reader.

What crawling this site collects

Crawling collects nothing about the crawler beyond ordinary hosting logs. A request from an agent produces the same access-log entry any HTTP request produces — the URL, the timestamp, the user-agent string, and the originating IP address, kept by the hosting and CDN layer. No account is created and no cookie is required to read any public page.

Two tags, and they behave differently towards an agent, so it is worth being exact rather than claiming a blanket block. The analytics tag is JavaScript behind a consent gate: most crawlers execute none, so for those nothing beyond that log entry happens at all, and an agent that does run scripts gets exactly the same treatment a person does — in the opt-in regions nothing loads until it is chosen, everywhere else it starts granted with a one-click opt out, and Global Privacy Control forces the strict default everywhere. The advertising scripts, served by Adsterra, sit behind the same consent gate and are not part of the HTML this server sends: they are requested from the browser after the page loads, only while the advertising category is on, and never for a browser that reports itself as automated. A reader that runs no JavaScript never receives them at all. The privacy policy and the cookie and storage page carry the detail.

In the other direction, there is nothing here belonging to anyone else to reach. This site has no upload endpoint: the editor, the three studios, and both media labs parse and write files inside the visitor’s own browser, and nothing a visitor opens is transmitted anywhere. The library itself is generated by a recorded, deterministic pipeline rather than collected from real documents, so no crawl of it can surface anybody’s data, because none was ever put into it.

If you would rather ask than crawl

This site runs a Model Context Protocol endpoint, documented for a person on the MCP server page. Its read-only tools search the catalogue in plain language and return a file’s full spec sheet and download URL, which is usually a much cheaper way to answer a question than crawling several thousand pages. For anything this page does not answer, email examples@novusstreamsolutions.com.

This page describes current practice and may change. Novus Stream Solutions publishes it so the policy can be read rather than reverse-engineered; where the two could ever disagree, /robots.txt is what a crawler is actually served, and both are generated from one list so that they cannot.