How to Find Hidden Web and API Endpoints on Kali Linux

Modern applications expose more than linked HTML pages. REST routes, GraphQL handlers, and admin folders often sit outside navigation and documentation. After banner grabbing shows HTTP on port 80, you build a URL and endpoint inventory before parameter fuzzing or login attacks.

This walkthrough finds hidden web and API endpoints on Metasploitable 2 in an authorized pentest lab. You probe API specs, review browser traffic, crawl JavaScript with Katana, and brute-force directories with Gobuster, ffuf, and dirb.

Hidden endpoint discovery quick start

Export these variables once per shell session. These exports last for the current shell only — open a new terminal and re-run the block.

export TARGET=192.168.56.127
export WEB_URL="http://${TARGET}"
export LAB=/tmp/hidden-endpoints-lab
mkdir -p "$LAB"
# Proxmox Group 2 lab: export TARGET=10.10.10.30

Before you run the commands below:

• Lab target — TARGET defaults to 192.168.56.127 (Metasploitable 2 on the VirtualBox host-only lab). Confirm the address in your pentest lab guide if your layout differs.
• Base URL — WEB_URL is http://${TARGET} on the default lab image unless your engagement uses HTTPS or a non-standard port.
• Root vs non-root — The commands here run as a normal Kali user. Package installs need sudo.
• Missing tools — Install the enumeration packages in the tools section if katana, gobuster, or ffuf is not found.

Once network reconnaissance shows port 80 open on $TARGET, probe common API documentation paths before you send a full wordlist:

for path in openapi.json swagger.json swagger-ui api-docs v3/api-docs graphql graphiql; do
  curl -s -o /dev/null -w "%{http_code} /${path}\n" "${WEB_URL}/${path}"
done

404 /openapi.json
404 /swagger.json
404 /swagger-ui
404 /api-docs
404 /v3/api-docs
404 /graphql
404 /graphiql

Metasploitable returns 404 for every spec path on the default image. On modern targets, 200 means inspect the file, 401 or 403 often means the route exists but access is restricted, and 404 means no route at that exact location.

Directory brute force still finds classic lab paths Gobuster reports with 200 or 301:

gobuster dir -u "$WEB_URL" -w /usr/share/wordlists/dirb/common.txt -q -t 20 --timeout 5s

dav                  (Status: 301) [Size: 319] [--> http://192.168.56.127/dav/]
phpMyAdmin           (Status: 301) [Size: 326] [--> http://192.168.56.127/phpMyAdmin/]
phpinfo.php          (Status: 200) [Size: 48026]
test                 (Status: 301) [Size: 320] [--> http://192.168.56.127/test/]
twiki                (Status: 301) [Size: 321] [--> http://192.168.56.127/twiki/]

Combine API spec probes, browser traffic, JavaScript crawling, and wordlist scans. Each method surfaces different endpoint types.

Web paths, files, and API endpoints

Beginners often treat every URL as a "directory." In penetration testing you record what each route is:

Directory brute force can discover some hidden endpoints, especially on older PHP apps. Modern single-page applications also expose API routes through JavaScript bundles, browser network requests, OpenAPI specifications, and archived content. OWASP API reconnaissance guidance recommends combining documentation review, proxy traffic, crawling, and source analysis rather than relying on wordlists alone.

Install endpoint discovery tools

Install the enumeration packages if your Kali image is missing a tool. Use apt to refresh indexes and install the stack:

sudo apt update
sudo apt install -y katana gobuster ffuf dirb nikto curl wget

apt exits silently when the install succeeds.

Compare endpoint discovery methods

Start with API documentation and browser traffic on modern apps. Add robots.txt, JavaScript crawling, and directory brute force. Escalate to recursive fuzzing in the URL fuzzer tools guide when scope allows larger wordlists.

Verify HTTP port 80 on $TARGET

Discovery tools waste time when Apache is down. Confirm port 80 responds before you probe endpoints:

nc -zv "$TARGET" 80

(UNKNOWN) [192.168.56.127] 80 (http) open

Port 80 is open on the lab target. Probe API documentation paths next, then run crawling and brute-force tools.

Probe API documentation paths

OpenAPI and Swagger files can expose most of an API surface in one response. Older specifications sometimes list deprecated routes that still answer on the server.

Check common documentation locations with curl:

for path in openapi.json swagger.json swagger-ui api-docs v3/api-docs graphql graphiql; do
  curl -s -o /dev/null -w "%{http_code} /${path}\n" "${WEB_URL}/${path}"
done

404 /openapi.json
404 /swagger.json
404 /swagger-ui
404 /api-docs
404 /v3/api-docs
404 /graphql
404 /graphiql

Read the status codes this way:

• 200 — download and parse the spec for paths and methods
• 401 or 403 — route likely exists; try authenticated access when scope allows
• 404 — no resource at that exact path; try versioned prefixes such as /api/v1/openapi.json

When a spec returns 200, save it for your report and add every listed path to your validation queue.

Capture endpoints from browser DevTools

On single-page applications, the browser Network tab shows API calls the frontend already makes. These endpoints tend to be higher confidence than random wordlist guesses because the application uses them in normal operation.

Use this workflow on any authorized web target (Metasploitable's older apps show fewer XHR calls, but the steps are the same on modern sites):

1. Open the site in Firefox or Chrome.
2. Press F12 and select the Network tab.
3. Enable Fetch/XHR filtering.
4. Reload the page and click through login, search, profile, filters, and checkout flows.
5. Record each request URL, HTTP method, and status code.
6. Look for patterns such as /api/, /v1/, /graphql, or hostnames on API subdomains.

Example traffic you might log from a modern application:

GET  /api/v1/profile
POST /api/v1/search
GET  /api/v2/orders/123
POST /graphql

Burp Suite or OWASP ZAP captures the same requests when the browser routes traffic through a proxy. Browse as much functionality as the engagement allows — lazy loading and code splitting often expose endpoints only after you interact with a feature.

Extract endpoints from JavaScript with Katana

Frontend bundles often embed /api/v1/... strings, GraphQL URLs, internal route names, and deprecated calls. Katana crawls pages and parses JavaScript when you pass -jc.

Run a shallow crawl against the lab web root:

katana -u "$WEB_URL" -d 1 -jc -silent

Trimmed output from the lab run:

http://192.168.56.127
http://192.168.56.127/dav/
http://192.168.56.127/twiki/
http://192.168.56.127/twiki/bin/view/Main/WebHome
http://192.168.56.127/dvwa/
http://192.168.56.127/dvwa/login.php
http://192.168.56.127/mutillidae/
http://192.168.56.127/mutillidae/index.php?page=login.php

Katana surfaced dvwa and mutillidae paths linked from JavaScript or embedded references — routes a short wordlist pass might miss. Save longer crawls when you need a file for reporting:

katana -u "$WEB_URL" -d 2 -jc -silent -o "$LAB/katana.txt"

Katana writes discovered URLs to $LAB/katana.txt with no extra terminal output when the run finishes cleanly.

Read robots.txt and sitemap.xml

robots.txt and sitemap.xml are passive-first checks. They cost one HTTP request each and sometimes list admin paths or API prefixes operators forgot to hide elsewhere.

Request robots.txt from the web root:

curl -s "$WEB_URL/robots.txt" | head -15

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /robots.txt was not found on this server.</p>
<hr>
<address>Apache/2.2.8 (Ubuntu) DAV/2 Server at 192.168.56.127 Port 80</address>

Metasploitable returns 404 for robots.txt on the default image. On production sites you may see Disallow rules that name sensitive folders — add each path to your validation list.

Check whether a sitemap exists:

curl -s -o /dev/null -w 'sitemap.xml %{http_code}\n' "$WEB_URL/sitemap.xml"

sitemap.xml 404

A 404 here means there is no sitemap to mine. When a site returns 200, pipe the XML through grep -oE '<loc>[^<]+' to extract URLs before you send new traffic.

Brute-force paths with Gobuster

Gobuster walks a wordlist against $WEB_URL and prints HTTP status codes for each guess. Directory brute force is one endpoint-discovery technique — use it after API and JavaScript passes on modern targets, or earlier on classic Apache apps like Metasploitable.

Run a quiet directory scan against the lab web root:

gobuster dir -u "$WEB_URL" -w /usr/share/wordlists/dirb/common.txt -q -t 20 --timeout 5s

Trimmed output from the lab run:

dav                  (Status: 301) [Size: 319] [--> http://192.168.56.127/dav/]
phpMyAdmin           (Status: 301) [Size: 326] [--> http://192.168.56.127/phpMyAdmin/]
phpinfo.php          (Status: 200) [Size: 48026]
phpinfo              (Status: 200) [Size: 48014]
test                 (Status: 301) [Size: 320] [--> http://192.168.56.127/test/]
twiki                (Status: 301) [Size: 321] [--> http://192.168.56.127/twiki/]

phpMyAdmin and phpinfo.php are classic Metasploitable findings. A small wordlist will not list every lab application — escalate to larger lists when scope allows.

Save results when you need a file for reporting:

gobuster dir -u "$WEB_URL" -w /usr/share/wordlists/dirb/common.txt -q -t 20 --timeout 5s -o "$LAB/gobuster.txt"

Gobuster writes the hit list to $LAB/gobuster.txt with no extra terminal output when the run finishes cleanly.

Fuzz directories with ffuf

ffuf sends the same wordlist with flexible match rules. Use it when you want to filter noise or tune threads without switching tools.

Fuzz the first path segment with common status codes enabled:

ffuf -u "${WEB_URL}/FUZZ" -w /usr/share/wordlists/dirb/common.txt -mc 200,301,302,403 -t 20 -timeout 5 -s

Trimmed silent output:

cgi-bin/
dav
phpMyAdmin
phpinfo
phpinfo.php
server-status
test
twiki

ffuf listed the same high-value paths Gobuster found. Paths ending in 403 such as server-status still prove the route exists even when the server blocks your request.

Enumerate paths with dirb

dirb predates Gobuster and ffuf but remains useful for classic reporting and optional recursion. The -S flag hides 404 lines so you can read hits faster.

Run dirb against the lab host with silent mode:

dirb "$WEB_URL" /usr/share/wordlists/dirb/common.txt -S -r -z 20

Trimmed output:

+ http://192.168.56.127/cgi-bin/ (CODE:403|SIZE:295)
==> DIRECTORY: http://192.168.56.127/dav/
+ http://192.168.56.127/phpinfo.php (CODE:200|SIZE:48104)
==> DIRECTORY: http://192.168.56.127/phpMyAdmin/
==> DIRECTORY: http://192.168.56.127/test/
==> DIRECTORY: http://192.168.56.127/twiki/

dirb marked directory entries with ==> DIRECTORY: and file hits with +. Compare its list with Gobuster, ffuf, and Katana before you move to validation.

Validate paths with curl

A scanner hit is a lead until you confirm the response. Use curl with a write-out format to record status codes without dumping HTML bodies.

Check paths the scanners reported:

curl -s -o /dev/null -w 'phpMyAdmin %{http_code}\n' "$WEB_URL/phpMyAdmin/"

phpMyAdmin 200

Confirm a second discovered directory the same way:

curl -s -o /dev/null -w 'twiki %{http_code}\n' "$WEB_URL/twiki/"

twiki 200

200 means the path answered. 401 and 403 still prove the route exists but blocked your request. 404 may mean the path is gone or hidden by routing rules.

Crawl linked pages with wget

Brute force finds unlinked folders. Crawling follows anchors on pages you already reach and surfaces paths wordlists miss.

Run a shallow spider against the lab home page with the wget command:

wget --spider -r -l 1 -p -e robots=off "$WEB_URL" 2>&1 | head -12

Spider mode enabled. Check if remote file exists.
--2026-08-30 22:18:06--  http://192.168.56.127/
Connecting to 192.168.56.127:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Remote file exists and could contain links to other resources -- retrieving.

The Metasploitable home page is small, so the spider confirms connectivity more than it expands the path list. On larger applications, increase -l depth carefully and stay inside scope.

Fingerprint common files with Nikto

Nikto checks hundreds of known sensitive paths and version headers. It complements brute force by flagging configuration issues around paths you already found.

Run a time-limited pass against the lab web server:

nikto -h "$WEB_URL" -maxtime 60s 2>&1 | head -15

+ Target IP:          192.168.56.127
+ Target Port:        80
+ Server: Apache/2.2.8 (Ubuntu) DAV/2
+ [999986] /: Retrieved x-powered-by header: PHP/5.2.4-2ubuntu5.10.
+ [750500] /icons/: Directory indexing found.
+ [600625] PHP/5.2.4-2ubuntu5.10 appears to be outdated (current is at least 8.5.1).
+ [600050] Apache/2.2.8 appears to be outdated (current is at least 2.4.66).

Nikto flagged outdated PHP and Apache builds plus directory indexing on /icons/. Cross-check those paths with your endpoint inventory before you research CVEs in scope.

Plan scans around WAF and origin IP

Production sites often sit behind a WAF or CDN. Directory brute force against the public hostname hits the edge network first. Responses may differ from the origin Apache or nginx server behind it.

On Metasploitable, your scans connect directly to Apache on $TARGET — there is no CDN in the lab path. On customer engagements, compare these signals before you interpret 403 floods as dead ends:

• DNS and certificates — historical A records or certificate transparency may reveal a direct origin IP inside scope
• Response headers — Server, cf-ray, and X-CDN style headers show whether you reached an edge node
• Dual testing — when your rules of engagement include a confirmed origin IP, repeat a small wordlist against http://ORIGIN_IP with the Host header set to the site name

Stay inside written scope. Passive OSINT platforms such as Shodan can help map exposure, but never point active brute force at IPs you are not authorized to test.

Troubleshooting

Summary

You built a hidden endpoint inventory on Metasploitable 2 using the layered workflow real engagements expect: probe API documentation paths, capture live traffic in browser DevTools, crawl JavaScript with Katana, then brute force directories with Gobuster, ffuf, and dirb. Validate every hit with curl, spider shallow links with wget, and fingerprint the host with Nikto.

Metasploitable returned 404 for OpenAPI, Swagger, and GraphQL paths, which is typical for this lab image. Katana still surfaced dvwa and mutillidae routes that wordlists alone might miss. On modern targets, browser Network traffic and JavaScript analysis usually outperform blind directory brute force for REST and GraphQL discovery.

Plan WAF versus origin testing on production targets so you do not misread edge 403 responses as missing endpoints. Run these steps only on authorized applications. When you need recursive forced browsing, custom filters, or authenticated crawling, continue with URL fuzzer tools and manual testing in Burp Suite.

References

• OWASP API Security — API reconnaissance
• Katana (GitHub)
• Gobuster (GitHub)
• ffuf (GitHub)
• DIRB project
• Nikto
• GNU Wget manual
• Google — robots.txt specification