Dead link finder

Links break without anyone touching them

When you add a link, you’re counting on someone else’s server, or on a file you might rename next year. Your page only stores the address. The target moves, a domain doesn’t get renewed, an image is deleted from the media library, and your page still looks exactly the same in your editor. The only person who finds out is the one who clicks.

This slow decay is called link rot, and it never stops. A 2024 study by the Pew Research Center found that 38% of the web pages that existed in 2013 could no longer be reached ten years later. To know where your site stands, somebody has to try every link. That’s a miserable job for a person and an easy one for a program.

How the crawl proceeds

1. We fetch the start page and read robots.txt. Paths it disallows are left alone.
2. Every <a href> and <img src> is collected, including lazy-loaded images in data-src. mailto:, tel: and javascript: links are skipped, and the #fragment is dropped.
3. Links to pages on the same host go into the queue to be read in turn. example.com and www.example.com count as one site. Other subdomains don’t.
4. Everything else (other sites, PDFs, images, downloads) is only tested. We start with a HEAD request, which asks for the status without the body, and fall back to a normal GET if the server refuses HEAD.
5. Redirects are followed by hand, up to five hops. A sixth hop, or an address that redirects to itself, is reported as a loop.

The crawl stops after 100 pages read or 600 requests. It opens at most four connections to a host at a time, waits six seconds for an answer, and tests each address once no matter how often it’s linked. When a path shows up with more than five different query strings, the extra variants are ignored. Otherwise a calendar or a filter page would eat the whole budget.

Reading the four lists

Broken links

The target answered with an error, or didn’t answer. 404 and 410 mean the page is gone. 500 to 504 point to a failing server and may be temporary. “Domain name not found” often means the whole site has closed. “Invalid SSL certificate” means browsers show a warning before the page.

Broken images

Same causes, but what the visitor sees is an empty frame or a broken-picture icon.

Internal redirects

Links to your own pages that only work through a redirect. Nothing is broken, but each click costs an extra round trip, and somebody may remove the redirect one day.

To check by hand

The target answered 401, 403, 429 or 999, so it refuses robots or wants a login. LinkedIn, Facebook and many sites behind bot protection do this. You’ll have to open the link yourself to settle it.

Each row lists up to five pages where the address appears, and the failures repeated most often come first. A dead link in a footer shows up on every page, and you fix it in one place.

Repairing what the crawl finds

• Your own page moved. Correct the link, then add a permanent redirect for people arriving from elsewhere. Apache: Redirect 301 /old-page/ /new-page/. nginx: location = /old-page/ { return 301 /new-page/; }.
• Another site moved its page. Search that site for the new address. If the content is gone, link to a copy on the Wayback Machine (web.archive.org) or remove the link.
• Missing image. Upload it again under the same name, or edit the page.
• Internal redirect. Replace the address in the link with the final one shown in the report.