Our crawler
How EurekaGPT reads the web
EurekaGPT uses two declared bots to read public web pages. Both obey robots.txt, run from published IP addresses and never use page text to train a model.
EurekaGPT is operated by LOGICANCE S.L. (NIF B93934537), Valencia, Spain. Questions about our bots, and requests to remove pages, go to bot@eurekagpt.eu or to the form below.
Our two bots
| Bot | What it does | User agent |
|---|---|---|
| EurekaSearchBot | Builds the Eureka Search index that EurekaGPT cites in its answers. | Mozilla/5.0 (compatible; EurekaSearchBot/1.0; +https://eurekagpt.eu/bot) |
| EurekaGPT-Bot | Reads one page when a user's question needs it. | Mozilla/5.0 (compatible; EurekaGPT-Bot/1.0; +https://eurekagpt.eu/bot) |
Both run only from the IP addresses we publish. EurekaSearchBot also has matching reverse DNS:
- EurekaSearchBot: eurekagpt.eu/eurekasearchbot.json
- EurekaGPT-Bot: eurekagpt.eu/eurekagpt-bot.json
A request that claims to be one of our bots from any other address is not ours.
For documented APIs and open-data files we use a third client, EurekaSearch/1.0, from EurekaSearchBot's addresses.
What our bots respect
- Text-and-data-mining opt-outs.
tdm-reservationor TDMRep: we do not index the page. - robots.txt. We follow the group that carries our bot's name, and the
*group when there is none. We read robots.txt again at least every 24 hours, and we honourCrawl-delay. - Content signals.
search=no: we do not index the page.ai-input=no: we keep the title, the link and an excerpt of at most 200 characters, and the AI sees only that excerpt. We never train models on page text, soai-trainneeds nothing from us. - Page directives.
noindexin a meta tag or anX-Robots-Tagheader: we do not index the page.nosnippet: we keep the title and the link only. - Public pages only. No logins, no paywall bypass, no CAPTCHA solving, no scripts run to reveal hidden text.
- A block is final. If your site answers 401, 403 or a challenge page, we stop. We do not retry from another address, under another name or through another service.
- Light load. At most one request per second per site by default, slower when you set
Crawl-delay. We back off on errors.
How we use what we read
- We store pages in short passages so we can find and cite them. The user sees a short excerpt and a link to your page.
- The AI summarises and quotes briefly, at most about 25 words per quote. It does not reproduce articles. Every citation points to a passage we retrieved.
- For news, we keep the headline, the first paragraph (at most 300 characters) and the link. We never store the article body. After 90 days only the title, the date and the link remain.
- Official and open-data sources follow their own licence, and each passage carries its licence and attribution.
- We never use page text to train or fine-tune any model, ours or anyone else's. We tune our ranking on numbers only, such as page length, dates and links.
- We do not build profiles of people. Names stay on the pages that publish them.
- We find pages through feeds, sitemaps and public URL lists such as Common Crawl and GDELT. The text itself is always fetched by EurekaSearchBot, so your current robots.txt applies.
- We find the websites of public bodies, universities and research organisations in open registers: Wikidata (CC0), IndicePA (AgID, CC BY 4.0), the French Annuaire de l'administration (DILA, Licence Ouverte 2.0), the Dutch Register van Overheidsorganisaties (CC0) and CORDIS (© European Union).
- We find the sites people in each European country visit most in the Chrome UX Report (Google, CC BY 4.0).
- We leave out whole categories of sites: adult, dating, gambling, malware, phishing, link shorteners, family-history services, health forums and sites made for children. To sort sites we use our own lists and the UT1 blacklists of Université Toulouse Capitole (CC BY-SA 4.0).
- A new site starts with a few pages. We read more only when those pages are useful.
How to block us
To keep EurekaSearchBot out of your whole site, add this to your robots.txt:
User-agent: EurekaSearchBot Disallow: /
To keep EurekaGPT-Bot out as well, add its own group:
User-agent: EurekaGPT-Bot Disallow: /
To stay in search but keep your text away from the AI, add a content signal:
User-agent: * Content-Signal: search=yes, ai-input=no
A change in robots.txt takes effect within 24 hours.
Ask us to remove content
Anyone can ask us to remove content about them, and any site owner can ask us to remove their pages. We handle every request within 2 working days. We remove the pages or the whole site from our index, and our bots and sources refuse them from then on.
If the page is an official record, such as a law or a register, and its official source still publishes it, we may keep the record and remove it from searches by a person's name where data protection law requires it.
We received your request. We will handle it within 2 working days and write to you at the address you gave.
We could not send your request. Check the form and try again, or write to bot@eurekagpt.eu.
Contact us
Write to bot@eurekagpt.eu to ask us to remove pages, to report a problem with our bots or to get a report of what we hold from your site.
LOGICANCE S.L.Calle Jesús 81, Esc. A, 2º, Pta. 6
46007 Valencia, Spain
Last updated: 3 November 2026