The Archival Crisis: Why The New York Times and Reddit are Purging The Wayback Machine

Several U.S. publishers and platforms are systematically blocking the Internet Archive’s Wayback Machine (by blacklisting the ia_archiver bot and updating robots.txt) to prevent AI companies from scraping archived content for model training; the move is framed as a defensive response to unauthorized high-frequency data harvesting and commercial licensing deals that change how platforms monetize their data.