Lesezeichen

Robots.txt meant for search engines don’t work well for web archives | Internet Archive Blogs


Beschreibung

We see the future of web archiving relying less on robots.txt file declarations geared toward search engines, and more on representing the web as it really was, and is, from a user’s perspective.

Vorschau

Tags

Nutzer

  • @schneeschmelze

Kommentare und Rezensionen