Guide to Using httpx

Published: at 12:00 PM

Introduction

httpx is a powerful tool designed for HTTP probing and reconnaissance. It allows users to identify and gather information about web services efficiently. Below, we dive into various commands and usage scenarios of httpx, providing detailed examples and descriptions for each feature.

TL;DR

You can find a shorter cheat sheet version of this article here.

Table of contents

Open Table of contents

Basic Usage

Probe Host

To probe a single host, use the following command:

httpx -u example.com -probe

This command initiates an HTTP probe on example.com. This is useful for quickly checking the availability of the web page.

Scan Hosts from Input File

To scan multiple hosts listed in a file, use:

httpx -l hosts.txt

Here, hosts.txt contains the list of target hosts, each on a new line. The output will include a list of hosts where the HTTP probe was successful. This is particularly helpful when dealing with large sets of hosts, such as those generated from subdomain enumeration tools.

Scan Targets from Other Program Output

httpx can also take input from other programs using pipes:

cat hosts.txt | grep example.com | httpx

This method reads hosts from hosts.txt, grep for specific domain and pipes them directly into httpx for scanning.

Tool Chain Example

Combining httpx with other tools, like subfinder:

subfinder -d example.com -silent | httpx -title -tech-detect -status-code

This command uses subfinder to find subdomains of example.com and pipes them into httpx to get the page title, detect technologies, and retrieve HTTP status codes.

Testing Multiple Ports

To scan multiple common HTTP ports:

httpx -u example.com -ports 80,443,8009,8080,8081,8090,8180,9443

This command tests the specified ports for HTTP services on example.com. Testing multiple ports is crucial because web services can run on non-standard ports, and identifying these can reveal additional attack surfaces.

Path Brute-force

Test different paths or files:

httpx -l urls.txt -sc -path "/,/path1,/path2,/path3"

This command checks the specified paths on the targets from urls.txt file and displays the status codes. It might be useful when you want to check multiple paths or files on a single host or as in this example list of URLs.

Probes

httpx offers a variety of probes that allow you to extract detailed information from your HTTP scans. These probes are essential for gaining deeper insights into the target’s properties, such as the response status, server details, and content types. By using these probes, you can tailor your scans to capture the most relevant data for your analysis.

httpx -status-code -content-type -content-length -location \
  -title -web-server -tech-detect -ip -cname -word-count -line-count -response-time \
  -cdn -hash sha256 -include-response -silent -stats -follow-host-redirects -max-redirects 2

This command collects a wide range of data, including status code, content type, server name, and more, following redirects up to two times. This allows you to get a detailed overview of the target’s HTTP properties.

Save responses and filter HTTP redirects and error pages:

httpx -silent -l urls.txt -j -o httpx.json -sr \
  -sc -title -ct -cl -bp -server -td -ip -cname -word-count -hash sha256 -fep -fc 301 \
  -tlsi -random-agent -stats -t 5 -rl 10 -timeout 5 -maxhr 3

This command saves the scan results in httpx.json, displaying various headers and applying filters to ignore specific status codes and error pages. This is useful for post-processing and analyzing the results offline - for example looking for secrets like hard-coded API keys.

Interesting Probes

Here are some of the key probes available in httpx:

Rate Limits

Rate limits in httpx allow you to control the number of requests sent per second or minute, ensuring that your scanning process does not overwhelm the target server or your network. This feature is essential for maintaining a respectful and efficient scanning. By configuring rate limits, you can balance the thoroughness of your scans with the need to avoid detection and throttling by the target server. Here’s how you can set and manage rate limits in httpx:

httpx -u example.com -t 10 -rate-limit 50

This sets the number of threads to 10 and limits the rate to 50 requests per second.

Rate Limit Options

Matchers

Matchers in httpx enable you to filter and process responses based on specific criteria, such as HTTP status codes, the presence of certain strings, or regular expressions. This functionality is crucial for narrowing down results to those that meet particular conditions, making your scans more focused and effective. By using matchers, you can quickly identify and act upon relevant responses, optimizing your workflow and ensuring that you capture the data that matters most to your analysis or security assessments. Here’s how you can utilize matchers in httpx to streamline your scanning process:

Match Specific HTTP Codes

cat hosts.txt | httpx -mc 200,302

This matches responses with HTTP status codes 200 or 302.

Match Responses with Specific String

cat hosts.txt | httpx -ms admin

This matches responses containing the string “admin”.

Match Responses with Regex

cat hosts.txt | httpx -mr 'admin*'

This matches responses that fit the regex pattern admin*.

Filters

Filters in httpx allow you to exclude unwanted responses based on specific criteria, such as HTTP status codes or text content. This feature is essential for refining your scan results, enabling you to focus only on the relevant data and reducing noise from your output. By applying filters, you can efficiently manage large volumes of data, streamline your analysis, and ensure that you are only working with the most pertinent information. Here’s how you can leverage filters in httpx to optimize your scanning and data processing workflows:

Filter Responses with Specific HTTP Codes

httpx -l urls.txt -fc 404,403,401,400,500

This filters out responses with the specified status codes.

Filter Responses Based on ML Error Page Detection

httpx -l urls.txt -sc -fep

This uses machine learning to filter error pages.

Filter Responses with Specific Text

httpx -l urls.txt -fs error

This filters out responses containing the text “error”.

Filter Responses Based on Regex

httpx -l urls.txt -fe '.*Error.*'

This filters out responses that match the regex pattern .*Error.*.

Extractors

Extractors in httpx are powerful tools designed to pull specific parts of the response content based on patterns or regular expressions. This feature is particularly useful for isolating and retrieving valuable data from HTTP responses, such as usernames, emails, or other critical information embedded within the response body. By using extractors, you can automate the process of data extraction, making it more efficient and accurate. When you are performing security assessments, extractors help you focus on the essential elements of your scan results, ensuring that you gather the necessary information quickly and effectively. Here’s how you can utilize extractors in httpx to enhance your data extraction processes:

Extract Part of the Response with Regex

cat hosts.txt | httpx -er 'admin*'

This extracts parts of the response that match the regex admin*.

Optimizations

Optimizations in httpx are designed to enhance the performance and efficiency of your scans. By fine-tuning various parameters, you can control how httpx handles timeouts, retries, error counts, and host exclusions. These optimizations are crucial for large-scale scanning operations, where performance and accuracy are paramount. Leveraging these settings allows you to minimize resource usage, avoid unnecessary retries, and ensure that your scans run smoothly without interruption. Here are some key optimization techniques to improve your scanning process:

Probe with Protocol Scheme Supplied in the Input

httpx -l urls.txt -nfs

This probes using the protocol scheme supplied in the input without falling back to another scheme.

Set Timeouts, Max Error Count, and Retries

httpx -l urls.txt -timeout 5 -maxhr 3 -retries 1

This sets a timeout of 5 seconds, a maximum error count of 3 per host, and 1 retry.

Output

httpx provide versatile options for storing and displaying scan results. Here’s how you can utilize httpx’s output options to enhance your scanning and analysis processes:

Save Output to a File

httpx -l urls.txt -o httpx.log

This saves the scan output to httpx.log.

httpx -l urls.txt -j

This prints the output in JSON Lines format.

httpx -l urls.txt -stats

This command prints scan statistics during execution. You can change how often stats are displayed using -si parameter followed by the number of seconds. The default interval is 5 seconds.

Store Responses

httpx -l urls.txt -sr http-responses/

This saves the HTTP responses in a given directory.

Screenshots

The Screenshots feature in httpx allows you to capture visual representations of web pages during your scanning process. This capability is invaluable for visually inspecting how web pages appear and verifying their content and layout. httpx screenshots are made using headless browser. Here are some examples:

Create a Screenshot of the Website

echo https://example.com | httpx -ss -st

This takes a screenshot of the specified URL.

Screenshot Options

Configuration File

httpx offers a robust configuration file feature that allows users to customize and streamline their scanning operations according to specific requirements. This feature is particularly useful for defining scan parameters, such as headers, timeouts, retries, and output preferences, in a structured YAML format. By utilizing httpx configuration files, users can standardize scan settings across multiple sessions, enhance scanning efficiency, and tailor scans to different environments or targets.

Use Custom Configuration File

httpx -config httpx-config.yaml

This uses a custom configuration file located at httpx-config.yaml.

Example Configuration File

status-code: true
content-length: true
content-type: true
location: true

line-count: true
word-count: true

title: true
body-preview: true
web-server: true
tech-detect: true
ip: true
cname: true

filter-code: 302,401,403
filter-error-page: true

threads: 10
rate-limit: 20

update: false
disable-update-check: true

store-response: true
store-response-dir: httpx-responses

json: true
include-response-header: true
include-response: true

random-agent: true
#header: Custom Global Headers

follow-redirects: false
follow-host-redirects: false

tls-impersonate: true
version: false
stats: true
silent: true
stats-interval: 5

max-host-error: 3
retries: 0
timeout: 5

This configuration file sets various default parameters for httpx scans.

Updates

httpx simplifies updates with options to check for and install the latest version. It also allows users to disable automated update checks on startup for better control of the scan.

Disable Update Checks

httpx -l targets.txt -duc

This disables automatic update checks.

Update httpx

httpx -up

This updates httpx to the latest version.

Processing JSONL Results with jq

As a bonus here are some examples of how to filter and extract specific information from httpx JSON output. Learn how to selectively process JSON results based on status codes, technology, titles, and more:

Select Results with status_code == 200

cat httpx.json | jq 'select(.status_code == 200)'

This filters results where the status code is 200.

Select Results with Ruby in tech

cat httpx.json | jq 'select(.tech[] | contains("Ruby"))'

This selects results that have “Ruby” in the technology list.

Select URL, tech where tech contains Ruby

cat httpx.json | jq 'select(.tech[] | contains("Ruby")) | .tech,.url' 2>/

dev/null

This extracts URLs and technologies where the tech list contains “Ruby”.

The Same for PHP

cat httpx.json | jq 'select(.tech[] | contains("PHP")) | .tech,.url' 2>/dev/null

This extracts URLs and technologies where the tech list contains “PHP”.

Search Results by Title

cat httpx.json | jq 'select(.title | contains("Index of"))'

This searches results for titles containing “Index of”.

Search All Results for Nginx (Case-Insensitive)

cat httpx.json | jq 'select(.tech[] | ascii_downcase| contains("nginx")) | .tech,.url' 2>/dev/null

This searches for Nginx in the technology list, case-insensitively.

Extract Basic Info About Each Request

cat httpx.json| jq '{url: .url, host: .host, method: .method, status_code: .status_code, content_type: .content_type, words: .words, webserver: .webserver, tech: .tech, hash: .hash}'

This extracts basic information about each request.

The Same, but Only for 200 Responses

cat httpx-* |  jq 'select(.status_code == 200) | {url: .url, host: .host, method: .method, status_code: .status_code, content_type: .content_type, words: .words, webserver: .webserver, tech: .tech, hash: .hash}'

This extracts basic information for requests with a status code of 200.

Select 200 Status Code and Content Type application/json

cat httpx.json| jq 'select((.status_code == 200) and (.content_type == "application/json"))'

This selects results where the status code is 200 and the content type is application/json.

Conclusion

In summary, httpx is a powerful tool tailored for security assessments, web reconnaissance, and HTTP probing tasks. httpx enables users to efficiently scan hosts, test multiple ports, perform path bruteforcing, and employ advanced filtering and extraction on the results. httpx is useful for identifying vulnerabilities and gaining insights into the technological landscape of target systems. By integrating httpx into your toolkit, you gain a reliable HTTP scanning solution for conducting infrastructure analysis.

References