Wget cheatsheet

Basic usage

Download the URL

wget https://example.com

Concatenate the output and write to a single file

wget -O file https://learntheshell.com/

Download file to a specific directory

wget -P /home/user/Downloads https://learntheshell.com/sample.zip

Resume interrupted download

wget -c https://learntheshell.com/sample.zip

Use wget to download all the files listed in a file

wget -i urls.txt

Download in the background

wget -b https://learntheshell.com/sample.zip

Ignore certificate errors

wget --no-check-certificate http://learntheshell.com

Ignore robots.txt file

wget -e robots=off http://learntheshell.com

Set maximum number of redirects

wget --max-redirect=number http://learntheshell.com

Limit download speed

wget --limit-rate=200k https://learntheshell.com/sample.zip

Recursive download

wget -r https://learntheshell.com/

You can limit the depth of recursion by adding the -l

wget -r -l 2 https://learntheshell.com/

Download website for offline viewing

wget -r -p --convert-links https://learntheshell.com/

Check links without downloading

wget --spider https://learntheshell.com/

Cookies

Disable cookies

wget --no-cookies https://learntheshell.com/

Set custom cookie value

wget --no-cookies --header "Cookie: name=value"

Save cookies to file

wget --save-cookies cookies.txt https://example.com

Load cookies from file

wget --load-cookies cookies.txt https://example.com

Headers

Send additional HTTP header (can be specified multiple times)

wget --header='Custom-header: test' https://example.com

Set referer

wget --referer=url https://learntheshell.com/

User agent

wget -U user-agent-string https://learntheshell.com/

Disable cache (send Cache-Control and Pragma headers with no-cache value)

wget --no-cache https://example.com

Save headers at the beginning of the file

wget --save-headers https://learntheshell.com/

Authentication

Basic HTTP authentication

wget --user=username --password=password https://learntheshell.com/protected-file.zip

Prompt for password

wget --user=username --ask-password https://learntheshell.com/protected-file.zip

Timeouts

Define timeout of the request

wget -T 10 https://learntheshell.com/

The -T combines all the options below:

  • --dns-timeout - DNS loookup
  • --connect-timeout - TCP connection
  • --read-timeout - idle time

Download files

Download all PDF files, ignore robots.txt file

wget -e robots=off -r -l1 -A.pdf -nd http://server

Mirror the website

wget --mirror --convert-links --adjust-extension --page-requisites --no-parent https://learntheshell.com/

or shorter version

wget -m -k -E -p -np https://learntheshell.com/