Web Requests — HTTP, cURL and APIs for Web Pentesting (HTB CWES)

This is a study summary built from the Web Requests module of the HTB Academy Web Penetration Tester path (the one that leads to the HTB Certified Web Exploitation Specialist — CWES exam). It is written as revision material: concepts condensed, every command kept, and an operational cheatsheet at the bottom. The original module, its exercises and its lab targets live on HTB Academy — this is my own recap, not a copy of their text.

Understanding web requests is the floor you build everything else on. Before you can attack or defend a web application you have to be fluent in how the browser and the server actually talk — the request/response shape, the headers that carry the interesting state, the methods that trigger actions, and the status codes that tell you what happened. This module delivers all of that through two tools you will never put down: cURL on the command line and the browser DevTools.

1. HTTP, HTTPS and the URL

HTTP (HyperText Transfer Protocol) is the application-layer protocol used to request resources over the web. It is a client/server exchange: the client asks for a resource, the server processes the request and returns it. Default port is 80, though a server can be configured to listen anywhere.

HTTPS (HTTP Secure) is the same protocol wrapped in encryption (TLS). Everything travels as a single encrypted stream, so an attacker sitting on the path — a shared Wi-Fi, a malicious proxy — sees ciphertext instead of credentials. Default port is 443. Plain HTTP is being phased out; browsers increasingly refuse it.

Pentest note. Even over HTTPS the destination can leak via clear-text DNS. Encrypted DNS (8.8.8.8, 1.1.1.1) or a VPN closes that gap. And a Man-in-the-Middle can attempt an HTTP downgrade attack to strip TLS — modern browsers and HSTS defend against it (see the Strict-Transport-Security header below).

The URL, component by component

A URL carries far more than "the site I want". Example:

http://admin:password@inlanefreight.com:80/dashboard.php?login=true#status

Component	Example	What it is
Scheme	http:// / https://	Protocol to use. Ends in ://.
User info	admin:password@	Optional credentials, user:pass, separated from the host by @.
Host	inlanefreight.com	Hostname or IP of the resource.
Port	:80	Defaults to 80 (http) / 443 (https) if omitted.
Path	/dashboard.php	The resource — a file or folder. Empty path → server returns the default index.
Query string	?login=true	parameter=value pairs, started by ?, joined with &.
Fragment	#status	Handled client-side by the browser to jump to a section; never sent to the server.

Only scheme and host are strictly required — without them there is nothing to request.

How an HTTP request actually flows

The browser first checks the local /etc/hosts file, then asks a DNS server to turn the domain into an IP (a server can't be reached without one). It then sends a GET to port 80 for /, the server returns its default index file with a 200 OK, and the browser renders it.

/etc/hosts is yours to edit — add IP hostname lines to force resolution. Essential on labs where the target is given as a hostname (e.g. mapping a box IP to inlanefreight.htb).

The HTTPS handshake

If you type http:// for a site that enforces HTTPS, the server answers 301 Moved Permanently and pushes you to :443. A TLS handshake follows (client hello, server hello, certificate and key exchange, verification), an encrypted handshake confirms everything works, and only then does normal — now encrypted — HTTP traffic continue.

2. cURL — your command-line browser

cURL (client URL) is a command-line HTTP client. Unlike a browser it does not render HTML/JS/CSS — it prints the raw response — which is exactly what you want when the request/response is the thing you are testing. It scripts and automates trivially, so it's the backbone of command-line web testing.

curl http://info.cern.ch/          # fetch a URL, print the raw body
curl -O http://info.cern.ch/index.html   # save using the remote filename
curl -o page.html http://info.cern.ch/   # save under a name you choose
curl -s -O http://info.cern.ch/index.html # -s silences the progress meter
curl -h            # short help; --help all for everything; man curl for the manual

Key flags introduced across the module: -O/-o (save output), -s (silent), -v/-vvv (verbose — show the full request and response), -i (include response headers in output), -I (headers only, via a HEAD request), -A (set User-Agent), -H (set an arbitrary header), -u (basic-auth credentials), -k (skip TLS certificate checks), -X (set the method), -d (request body data), -b (send a cookie), -L (follow redirects). They are collected in the cheatsheet.

Invalid certificates

cURL refuses to talk to a site with a bad/expired/self-signed certificate — the same protection a browser gives you. On labs and local apps that's common; -k tells cURL to proceed anyway:

curl https://inlanefreight.com        # curl: (60) SSL certificate problem...
curl -k https://inlanefreight.com     # ignore the cert, continue

3. Requests, responses and headers

Anatomy of a request and response

The first line of a request has three space-separated fields: the method (GET), the path (/users/login.html, optionally with a ?query), and the version (HTTP/1.1). Then come the headers (one per line: Name: value), a blank line, and an optional body.

The first line of a response has two fields: the version (HTTP/1.1) and the status code (200 OK). Then headers, a blank line, and an optional body (HTML, JSON, an image, a PDF…).

HTTP/1.x is clear text with newline-separated fields; HTTP/2 sends the same information as binary. Use -v to see both sides of the exchange:

curl inlanefreight.com -v

> GET / HTTP/1.1
> Host: inlanefreight.com
> User-Agent: curl/7.65.3
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Server: Apache/X.Y.ZZ (Ubuntu)
< WWW-Authenticate: Basic realm="Restricted Content"
...

Headers worth knowing

Headers carry the state that matters in web attacks. The classic five-way split below comes from the old RFC 2616; the current spec (RFC 9110, 2022) groups fields by function instead, but these buckets are still the easiest way to learn them. The authoritative, always-current list is the IANA HTTP Field Name Registry.

General / entity (describe the message or its content)

Header	Example	Why it matters
Connection	close / keep-alive	Whether the connection stays open.
Cache-Control	no-store	Responses with secrets should set no-store so they aren't cached to disk.
Content-Type	text/html, application/json	Media type of the body; charset sets the encoding.
Content-Length	385	Body size in bytes — the server reads exactly this many.
Content-Encoding	gzip	Compression applied to the body.
Transfer-Encoding	chunked	Stream a body of unknown length. Inconsistent handling of this vs Content-Length between a proxy and a back-end is the root of HTTP request smuggling.

Request headers (sent by the client)

Header	Example	Why it matters
Host	www.inlanefreight.com	Which virtual host you want. Prime enumeration target; abused in Host header injection.
User-Agent	curl/7.77.0	Describes the client (browser, version, OS).
Referer	http://site/	Where the request came from. Easily forged — don't trust it.
Accept	*/*	Media types the client accepts.
Cookie	PHPSESSID=b4e4...	Client-stored identifier(s), name=value, multiple separated by ;.
Authorization	Basic cGFzc3dvcmQK	Credentials the client attaches every request — Basic, Bearer (JWT), Digest.
X-Forwarded-For	203.0.113.5	Added by proxies; client-controllable, so often spoofed to bypass IP allow-lists / rate limits.

Response headers (sent by the server)

Header	Example	Why it matters
Server	Apache/2.4.41 (Ubuntu)	Server software + version → enumeration. Often genericized for hardening.
Location	https://site/login	Redirect target (with 3xx). Unvalidated input here → open redirect.
Set-Cookie	PHPSESSID=...; Secure; HttpOnly; SameSite=Strict	Issues a cookie. Secure = HTTPS only, HttpOnly = hidden from JS (anti-XSS theft), SameSite = CSRF defense.
WWW-Authenticate	Basic realm="localhost"	Advertises the auth scheme; comes with 401.

Security headers (instruct the browser)

Header	Example	Protects against
Content-Security-Policy	script-src 'self'	Restricts resource origins → XSS.
Strict-Transport-Security	max-age=31536000	Forces HTTPS → sniffing / downgrade.
X-Content-Type-Options	nosniff	Stops MIME sniffing → upload-to-stored-XSS tricks.
X-Frame-Options	DENY	Clickjacking (superseded by CSP frame-ancestors).
Access-Control-Allow-Origin	https://site	CORS; wildcards / reflected Origin → CORS misconfig.
Referrer-Policy	origin	Limits what the Referer leaks.

Viewing and setting headers with cURL:

curl -I https://site          # response headers only (HEAD request)
curl -i https://site          # headers + body for whatever method you send
curl https://site -A 'Mozilla/5.0'   # set User-Agent with -A
curl -H 'Header: value' https://site # set any header; repeat -H for more

4. Methods and status codes

Methods (verbs)

Method	Does	Attack angle
GET	Request a resource; params in the URL query.	Reflected input, IDOR via params.
POST	Send data in the body (forms, uploads, JSON).	Logins, file upload, injection.
HEAD	Like GET but headers only, no body.	Cheap probing / length checks.
PUT	Create/replace a resource.	Unrestricted → arbitrary file upload / RCE.
DELETE	Remove a resource.	Unrestricted → data loss / DoS.
OPTIONS	List methods the server allows.	Discover PUT/DELETE/PATCH.
PATCH	Partial update of a resource.	Same surface as PUT.

Modern apps lean on GET and POST; REST APIs add PUT and DELETE.

Status code classes

Class	Meaning	Common examples
1xx	Informational	—
2xx	Success	200 OK
3xx	Redirect	301 Moved Permanently, 302 Found
4xx	Client error	400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found
5xx	Server error	500 Internal Server Error

As a tester you read these constantly: 401 vs 403 (auth vs authorization), 302 after a login (success + redirect), 500 (you broke something — often a sign an injection landed).

5. GET in practice — basic auth and parameters

HTTP Basic Auth

Some pages are protected by the web server itself (not the app), prompting for a username/password. The server answers unauthenticated requests with 401 and a WWW-Authenticate: Basic header. Three ways to authenticate with cURL:

curl -i http://TARGET/                       # 401 Authorization Required
curl -u admin:admin http://TARGET/           # credentials via -u
curl http://admin:admin@TARGET/              # credentials in the URL
curl -H 'Authorization: Basic YWRtaW46YWRtaW4=' http://TARGET/   # raw header

Basic auth just base64-encodes user:pass into the Authorization header — it is encoding, not encryption. YWRtaW46YWRtaW4= is simply admin:admin. Craft or crack it by hand:

echo -n 'admin:admin' | base64        # -> YWRtaW46YWRtaW4=
echo 'YWRtaW46YWRtaW4=' | base64 -d    # -> admin:admin

GET parameters

When a search box hits search.php?search=le, you can replay it straight from the command line — GET params live in the URL:

curl 'http://TARGET/search.php?search=le' -H 'Authorization: Basic YWRtaW46YWRtaW4='

Pulling the exact request out of the browser is faster than rebuilding it: in DevTools → Network, right-click the request → Copy → Copy as cURL (whole command, headers included — trim to the ones that matter) or Copy as Fetch (paste into the Console to replay in-browser).

6. POST in practice — forms, cookies and JSON

POST puts parameters in the body instead of the URL. That means less logging, fewer encoding constraints (binary is fine), and no ~2000-char URL limit — which is why uploads and logins use it.

Login forms

A PHP login posts username=admin&password=admin as the body. Replay it manually:

curl -X POST -d 'username=admin&password=admin' http://TARGET/
curl -X POST -d 'username=admin&password=admin' http://TARGET/ -L   # follow a post-login redirect

Authenticated cookies

A successful login returns Set-Cookie. Grab it, then reuse it so you never log in again:

curl -X POST -d 'username=admin&password=admin' http://TARGET/ -i   # read Set-Cookie
curl -b 'PHPSESSID=c1nsa6op7vtk7kdis7bcnbadf1' http://TARGET/       # send it back with -b
curl -H 'Cookie: PHPSESSID=c1nsa6op7vtk7kdis7bcnbadf1' http://TARGET/   # same, as a header

In the browser you can do the same by hand: DevTools → Storage (Firefox: SHIFT+F9) → Cookies → edit/add name + value → refresh. A valid session cookie alone is often enough to be "logged in" — a fact that matters a lot for XSS and session attacks.

JSON bodies

APIs usually expect JSON and a matching Content-Type. Copy the request headers from DevTools (Copy → Copy Request Headers) to confirm Content-Type: application/json, then:

curl -X POST -d '{"search":"london"}' \
     -b 'PHPSESSID=c1nsa6op7vtk7kdis7bcnbadf1' \
     -H 'Content-Type: application/json' \
     http://TARGET/search.php

Hitting the endpoint directly — no front-end, no clicking — is the whole point: it's faster and it's where the bugs are.

7. CRUD / REST APIs

Many APIs map HTTP methods onto database operations against a table/row addressed in the path (/api.php/city/london):

Operation	Method	Effect
Create	POST	Add a row to the table.
Read	GET	Read matching rows.
Update	PUT (or PATCH)	Modify an existing row.
Delete	DELETE	Remove a row.

PATCH does a partial update, PUT a full one; OPTIONS tells you which the server accepts. Pipe JSON output through jq to read it.

# READ — one row, a prefix match, or the whole table
curl -s http://TARGET/api.php/city/london | jq
curl -s http://TARGET/api.php/city/le | jq
curl -s http://TARGET/api.php/city/ | jq

# CREATE
curl -X POST http://TARGET/api.php/city/ \
     -d '{"city_name":"HTB_City", "country_name":"HTB"}' \
     -H 'Content-Type: application/json'

# UPDATE
curl -X PUT http://TARGET/api.php/city/london \
     -d '{"city_name":"New_HTB_City", "country_name":"HTB"}' \
     -H 'Content-Type: application/json'

# DELETE
curl -X DELETE http://TARGET/api.php/city/New_HTB_City

If any of these work unauthenticated, that's the finding: anonymous create/update/delete is a vulnerability. Real APIs gate write operations behind a cookie or an Authorization/JWT header — exactly the mechanisms from the GET and POST sections.

8. What to carry into the CWES exam

• cURL is the muscle memory. Build requests by hand — method (-X), body (-d), headers (-H), cookies (-b), auth (-u), follow redirects (-L), ignore certs (-k). You'll do this hundreds of times.
• DevTools → Copy as cURL turns any browser action into a reproducible command. Capture, trim, weaponize.
• Read the signal in headers and codes. Set-Cookie, WWW-Authenticate, Location, Server; 401 vs 403, 302 on login, 500 on a broken query.
• Talk to endpoints directly. Skip the UI and hit search.php / api.php with the right body and headers — faster, and where the vulnerabilities live.
• Reproduce every example. The module ends with a hands-on skills assessment; the only way through it (and the exam) is to have actually run these commands, not just read them.

---

Cheatsheet — Web Requests

cURL flags

curl URL                     # GET, print raw body
curl -s URL                  # silent (no progress meter)
curl -O URL                  # save as the remote filename
curl -o name URL             # save under a chosen name
curl -v URL                  # verbose: full request + response
curl -vvv URL                # even more verbose
curl -i URL                  # include response headers in output
curl -I URL                  # response headers only (HEAD request)
curl -k URL                  # skip TLS certificate validation
curl -L URL                  # follow redirects (3xx / Location)
curl -A 'Mozilla/5.0' URL    # set User-Agent
curl -H 'Name: value' URL    # set a header (repeatable)
curl -u user:pass URL        # HTTP basic auth
curl -b 'K=V' URL            # send a cookie
curl -X METHOD URL           # set the HTTP method
curl -d 'data' URL           # send a request body (implies POST)
curl -h  |  curl --help all  |  man curl   # help

Auth with cURL

curl -u admin:admin http://TARGET/
curl http://admin:admin@TARGET/
curl -H 'Authorization: Basic YWRtaW46YWRtaW4=' http://TARGET/
echo -n 'admin:admin' | base64        # build a Basic token
echo 'YWRtaW46YWRtaW4=' | base64 -d    # decode one

POST / cookies / JSON

curl -X POST -d 'username=admin&password=admin' http://TARGET/
curl -X POST -d 'username=admin&password=admin' http://TARGET/ -i   # read Set-Cookie
curl -b 'PHPSESSID=VALUE' http://TARGET/
curl -X POST -d '{"search":"london"}' -H 'Content-Type: application/json' \
     -b 'PHPSESSID=VALUE' http://TARGET/search.php

CRUD API (+ jq)

curl -s http://TARGET/api.php/city/london | jq           # Read
curl -X POST   http://TARGET/api.php/city/ -d '{"city_name":"X","country_name":"Y"}' -H 'Content-Type: application/json'   # Create
curl -X PUT    http://TARGET/api.php/city/london -d '{"city_name":"Z"}' -H 'Content-Type: application/json'                # Update
curl -X DELETE http://TARGET/api.php/city/Z                                                                                # Delete

/etc/hosts

# Add a line so a lab hostname resolves to the target IP:
#   <TARGET_IP>   inlanefreight.htb
sudo sh -c 'echo "10.10.10.10  inlanefreight.htb" >> /etc/hosts'

Browser DevTools

F12  /  CTRL+SHIFT+I      open DevTools
CTRL+SHIFT+E (Firefox)    jump to the Network tab
CTRL+SHIFT+K (Firefox)    jump to the Console (replay Fetch here)
SHIFT+F9    (Firefox)     Storage tab (view / edit cookies)
Network → right-click a request → Copy → Copy as cURL      replay on the CLI
Network → right-click a request → Copy → Copy as Fetch     replay in the Console
Network → right-click a request → Copy → Copy Request Headers
Response tab → Raw        view the unrendered response body

Built from HTB Academy's Web Requests module — if you want the labs, the full lessons and the exam, they're on HTB Academy.