The site
393 static pages, English and Chinese twins under /en/ and /zh/, generated from a 68-film and 60-camera data set plus hand-written guides and tools.
What the first pass found
- 397 JSON-LD blocks, 43 invalid or missing across tools and dossiers.
- One page with no meta description and several duplicate description strings.
- 15,152 internal links to check, plus 641 external links that could rot.
- No
lastmodand a broken sitemap pointer:robots.txtadvertised/sitemap.xml, which did not exist.
What the crawler surfaced that a headless check would miss
Cloudflare Pages was answering HTTP 200 with the site root for every unknown path. /.env.production, /.git/config, /admin/login and any typo all returned the homepage. That is a soft-404: duplicate content for search engines, and wasted crawl budget on a new domain.
Once that was fixed with a real 404.html, the same fix exposed a second bug that had been hidden by the 200s: the sitemap pointer in robots.txt had been broken the whole time and nobody could see it.
Separately, www.openfilm.cc was serving the entire site as duplicate hostnames because a host-level redirect rule never fired.
What we changed
- Generated JSON-LD for every tool, dossier and service page, then validated all 397 blocks.
- Wrote a crawler that re-checks every internal and external link on each build.
- Added a real 404 page and a
robotsprop so it can opt out of indexing. - Pointed
robots.txtatsitemap-index.xmland added a 301 from/sitemap.xml. - Added a zone-level redirect for
wwwto the apex domain. - Automated IndexNow so every new page is submitted to Bing and Yandex within 30 minutes of publication.
Verification
- 397/397 JSON-LD blocks valid.
- 15,152 internal links, 0 broken after the last build.
- 1 real 404 out of 641 external links, found and repaired.
- 392/392 sitemap URLs return HTTP 200 when fetched live.
- Unknown paths now return 404 with a useful page instead of a 200 copy of the homepage.
- Googlebot is crawling the content: 18
robots.txtfetches and dozens of article fetches in a 24-hour window, all 200.
What this means if you hire us
You get the same treatment: a crawl of your site, a prioritised list of what is actually wrong, the fixes written out in a form your developer can apply, and a second pass to prove each one landed. Scope is fixed at 150 USDC and delivery is five business days.