06.01 / 60 min

Crawl, discover directories, and detect soft 404s

Combine links, scripts, robots, sitemaps, OpenAPI, and bounded path discovery into an endpoint map.

By the endRun a same-origin crawlInspect discovery filesRecognize soft 404s
Controlled exercise

Use only the Academy Lab on 127.0.0.1:5177. Do not copy these tests to a public system without explicit written authorization.

01

Prove the target boundary

Create or reopen the Academy Lab project and send GET /health. Continue only when the target is 127.0.0.1 and local_only is true. The public-recon lessons require a separately authorized target and are never run against an unrelated domain.

GET http://127.0.0.1:5177/health
Expected result

200 with "local_only": true.

02

Seed the crawler

Set /crawl/start as the only seed, same-origin only, depth 3, request cap 50, and one worker. Disable form submission during discovery.

GET http://127.0.0.1:5177/crawl/start
Expected result

Account, API docs, preference form, script, and API references are discovered.

03

Read machine indexes

Fetch robots.txt, sitemap.xml, openapi.json, swagger.json, and .well-known/security.txt. Mark whether each URL was linked, documented, or guessed.

04

Run the directory scanner

Use a tiny lab list: admin, api, backup, graphql, missing-path. Calibrate with at least two random nonexistent paths before accepting matches.

05

Handle soft 404s

Compare status, length, normalized title, body hash, and redirect destination. Do not report a directory when it matches the nonexistent baseline.

MASTERY CHECKPOINT

Prove it before moving on.

Your map identifies every endpoint source and excludes the nonexistent control.