Lets breakdown a HTTP request — line by line (med-advanced)
Believe it or not, there are incredibly amazing and a (large) amount of things (parameters being set, routes and next hops calculated, all sorts of other things) actually go into you just visiting a site like www.twitter.com (lets just say on a computer/laptop and avoid the mobile discussion) for now.
Command Issued in Linux: curl -svLI http://twitter.com
Instead of doing this the “typical” way and just visiting twitter on a GUI (graphic) browser lets do it command line and watch what is happening behind the scenes. The more you know, the better off you are.
Quick Recap: If you have never used or need to brush up on the Curl command. it is without a doubt one of the 10 most essential Linux (or any OS) commands I could not live without. It is denoted as a “Headless Browser” basically, a googlebot or spider would be considered a headless browser. Picture “Headless” as “Monitor less” (no screen still have brain).
You can be a regular user and issue the curl command (in most cases), the flags I used above are
-s — You almost always want the -s (silent) flag or curl gives status gets annoy
-v — verbose
-L — *Critical* often overlooked, Follow Redirects
-I — Either ive found I am looking at headers (like with -I) in this case, or I am looking at markup or code (if you dont specify -I you will see the page not rendered)
Output of -svLI http://twitter.com
* Trying 104.244.42.193:80...
[ notice we have already DNS queried the IP for Twitter]
[ Since we did not specify https:// port 80 (nonsecure)
[ Always try both www. and without to see diff. behavior* Connected to twitter.com (104.244.42.193) port 80 (#0)
> HEAD / HTTP/1.1
> Host: twitter.com
> User-Agent: curl/7.80.0
> Accept: */* |- The data we see here is know as
-----------------------------| an HTTP Request, curl client sent
* Mark bundle as not supporting multiuse
^ When you see these * lines they are from Curl verbose msgs< HTTP/1.1 301 Moved Permanently
^ This is *exactly* why we use curl with -L to follow redirects
- So far we have 200->301 this is pointing us to (see below)[*] location: https://twitter.com/
< cache-control: no-cache, no-store, max-age=0
< content-length: 0
[?] x-response-time: 1
[?] x-connection-hash: 54744d4f7c93092a0b2d1cd46b471
< date: Wed, 23 Feb 2012 13:53:12 GMT
[?]
-------------------------------------
Note: the items you see above are part of a response in this case from twitter.com. The actual criterial from what you see above (such as server: tsa_b) are known as Web "Headers". They are always KEY=>VALUE pairs. Sometimes their may be no "value", but it would be set to null or 0 in that case.One thing hackers are always looking for is *extra information* aka Information Exposure. This has happened already, if you did not notice, keep in mind the info we get back from Twitter's servers could be anything to throw a hacker off, but we must also keep in mind what seems like a "simple" connection to twitter.com, in reality behind the scenes is probably something quite mind blowing when it comes to how they load balance, rev proxy, auto scale, etc. Lets Keep it simple, and I will point out where I feel they are leaking info:>Header response : server: tsa_b
Without knowing anything at all about this (other then, its Not standard. A hacker may goto https://shodan.io, https://publicwww.com, or https://yandex.ru and search on just "twitter tsa_b" .. I have a feeling its something to do with load balancing or Cloud availability zones.This info above is known as an HTTP response and it is the response to curl's request to originally hit twitter.com (http), which redirected us to https://twitter.com as you can see above.
There are 2 ways to redirect a client in a response such as you see above.
Location: https://twitter.com
Host: https://twitter.comI have seen both used, with very little difference. however, although beyond the scope of this whitepaper, if you have BurpProxy, or Fiddler, or any other mitm proxy, try intercepting a response and if you see a 301,302 and see Location try putting Host: www.msn.com below it , It is possible to bypass WAF and FW Rules as well a WAF's but that discussion is for another time.
Carrying on. We have finally made it to https://twitter.com (port 443/tcp) but have not established a secure connection yet. Time to exchange certificates.*Connection #0 to host twitter.com left intact
* Issue another request to this URL: 'https://twitter.com/'
* Trying 104.244.42.65:443...
* Connected to twitter.com (104.244.42.65) port 443 (#1)
* ALPN, offering h2 [ This is typical cert negotiation]
* ALPN, offering http/1.1
<snip>
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* subject: C=US; ST=California; L=San Francisco; O=Twit
ter, Inc.; CN=twitter.com
* start date: Dec 13 00:00:00 2021 GMT
* expire date: Dec 12 23:59:59 2022 GMT
* subjectAltName: host "twitter.com" matched cert's "twitter.com"
* issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
* SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x7fa217b12a90)
} [5 bytes data]
> HEAD / HTTP/2
> Host: twitter.com
> user-agent: curl/7.80.0
> accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
< HTTP/2 200
< date: Wed, 23 Feb 2022 13:53:13 GMT
< expiry: Tue, 31 Mar 1981 05:00:00 GMT
< pragma: no-cache
< server: tsa_b
< set-cookie: guest_id=v1%3A164562439313578936; Max-Age=34214400; Expires=Sun, 26 Mar 2023 13:53:13 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=None
< content-type: text/html; charset=utf-8
< x-powered-by: Express
< cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
< last-modified: Wed, 23 Feb 2022 13:53:13 GMT
< x-frame-options: DENY
< x-xss-protection: 0
< x-content-type-options: nosniff
[ The next part of the reponse is called the CSP Content Security (Policy set forward by Twitter to help secure the site ]
I am not going into details on this but i will quickly state that the Content-Security-Policy (case insensitive) is setup with "directives" such a source-src, or script-src, and then hostnames, script locations etc to "whitelist" That way if an evil hacker can run scripts on twitter.com unless its whitelisted, it will deny)
< content-security-policy: connect-src 'self' blob: https://*.pscp.tv https://*.video.pscp.tv https://*.twimg.com https://api.twitter.com https://api-stream.twitter.com https://ads-api.twitter.com https://aa.twitter.com https://caps.twitter.com https://pay.twitter.com https://sentry.io https://ton.twitter.com https://twitter.com https://upload.twitter.com https://www.google-analytics.com https://accounts.google.com/gsi/status https://accounts.google.com/gsi/log https://app.link https://api2.branch.io https://bnc.lt wss://*.pscp.tv https://vmap.snappytv.com https://vmapstage.snappytv.com https://vmaprel.snappytv.com https://vmap.grabyo.com https://dhdsnappytv-vh.akamaihd.net https://pdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://dwo3ckksxlb0v.cloudfront.net ; default-src 'self'; form-action 'self' https://twitter.com https://*.twitter.com; font-src 'self' https://*.twimg.com; frame-src 'self' https://twitter.com https://mobile.twitter.com https://pay.twitter.com https://cards-frame.twitter.com https://accounts.google.com/ https://recaptcha.net/recaptcha/ https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/; img-src 'self' blob: data: https://*.cdn.twitter.com https://ton.twitter.com https://*.twimg.com https://analytics.twitter.com https://cm.g.doubleclick.net https://www.google-analytics.com https://www.periscope.tv https://www.pscp.tv https://media.riffsy.com https://*.giphy.com https://media.tenor.com https://c.tenor.com https://*.pscp.tv https://*.periscope.tv https://prod-periscope-profile.s3-us-west-2.amazonaws.com https://platform-lookaside.fbsbx.com https://scontent.xx.fbcdn.net https://scontent-sea1-1.xx.fbcdn.net https://*.googleusercontent.com https://imgix.revue.co; manifest-src 'self'; media-src 'self' blob: https://twitter.com https://*.twimg.com https://*.vine.co https://*.pscp.tv https://*.video.pscp.tv https://dhdsnappytv-vh.akamaihd.net https://pdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://dwo3ckksxlb0v.cloudfront.net; object-src 'none'; script-src 'self' 'unsafe-inline' https://*.twimg.com https://recaptcha.net/recaptcha/ https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/ https://www.google-analytics.com https://twitter.com https://app.link https://accounts.google.com/gsi/client https://appleid.cdn-apple.com/appleauth/static/jsapi/appleid/1/en_US/appleid.auth.js 'nonce-Y2RkZGMyODAtMWVhYi00YzdhLWFhNmItYjU1MGYyYjRhNDgx'; style-src 'self' 'unsafe-inline' https://accounts.google.com/gsi/style https://*.twimg.com; worker-src 'self' blob:; report-uri https://twitter.com/i/csp_report?a=O5RXE%3D%3D%3D&ro=false
[End of CSP] still getting the response< strict-transport-security: max-age=631138519 (google these)
< cross-origin-opener-policy: same-origin-allow-popups
< cross-origin-embedder-policy: unsafe-none
< x-response-time: 71
< x-connection-hash: 185a5e6ca2d8b97a225bc2910cae73145257335c0b1dcfbce12c7e77f3ba86e1
<
* Connection #1 to host twitter.com left intact
> ""CONNECTEDYeap thats it. to make a connection to Twitter you just need to go thru all that ;)Its good time to discuss some amazing online resources. Remember let other systems and online tools do recon for you. expose your IP never if possible.dj substance [9x]
https://9x.network/sl1de
Some resources that will blow your mind —
DNSVIZ-_-Twitter-Online_Map
https://dnssec-debugger.verisignlabs.com/twitter.com
Mozilla Observatory — Online web security scan of target
Deep Dive into the Crypto of the targets SSL/TLS ciphers
HSTS Preload Online Test on twitter.com
CRT.SH — All historical and current cert info on Twitter.com
