Dump HTTP header using wget

wget --server-response --spider http://www.example.com/
Let me suggest using wget for obtaining the HTTP header only as the last resort because it generates considerable textual overhead. The first ellipsis of the sample output stands for Spider mode enabled. Check if remote file exists. --2009-03-31 20:42:46-- http://www.example.com/ Resolving www.example.com... Connecting to www.example.com||:80... connected. HTTP request sent, awaiting response... and the second one looks for Length: 438 [text/html] Remote file exists and could contain further links, but recursion is disabled -- not retrieving.
Sample Output
  HTTP/1.1 200 OK
  Date: Tue, 31 Mar 2009 18:42:46 GMT
  Server: Apache/2.2.3 (CentOS)
  Last-Modified: Tue, 15 Nov 2005 13:24:10 GMT
  ETag: "b80f4-1b6-80bfd280"
  Accept-Ranges: bytes
  Content-Length: 438
  Connection: close
  Content-Type: text/html; charset=UTF-8

By: penpen
2009-03-31 18:49:14

These Might Interest You

  • Without the -dump option the header is displayed in lynx. You can also use w3m, the command then is w3m -dump_head http://www.example.com/ Show Sample Output

    lynx -dump -head http://www.example.com/
    penpen · 2009-03-31 18:41:36 1
  • I have a server with a php requiring basic authentication, like this: header('WWW-Authenticate: Basic realm="do auth"'); header('HTTP/1.0 401 Unauthorized'); ...?> And the basic authentication in wget do not worked: wget --auth-no-challenge --http-user="username" --http-password="password" -O- "http://url" wget --keep-session-cookies --save-cookies=cookies.txt --load-cookies=cokies.txt --http-user="username" --http-password="password" -O- "http://url" I always received the 401 Authorization failed. The saved cookie is always empty. With my way, I received the header from the server and save the cookie, then resend the session cookie with authentication data Show Sample Output

    wget --auth-no-challenge --server-response -O- $url 2>&1 | grep "Cookie" | sed "s/^ Set-//g" > cookie.txt; wget --auth-no-challenge --server-response --http-user="user" --http-password="pw" --header="$(cat cookie.txt)" -O- $url
    glaudiston · 2010-12-01 11:24:35 0
  • Also removes translator comments. You can remove the header by omitting --keep-header, but if your msgids contain non-ASCII characters you will need the header to specify a suitable charset.

    msgfilter --keep-header --input input.po awk '{}' | sed '/^#$/d; /^#[^\:\~,\.]/d' >empty.po
    seanf · 2013-02-08 08:05:32 0
  • Show disk space info, grepping out the uninteresting ones beginning with ^none while we're at it. The main point of this submission is the way it maintains the header row with the command grouping, by removing it from the pipeline before it gets fed into the sort command. (I'm surprised sort doesn't have an option to skip a header row, actually..) It took me a while to work out how to do this, I thought of it as I was drifting off to sleep last night! Show Sample Output

    df -h | grep -v ^none | ( read header ; echo "$header" ; sort -rn -k 5)
    purpleturtle · 2011-03-16 14:25:45 1

What Others Think

slightly easier with curl tho: curl -I http://www.example.com
realgt · 441 weeks ago
realgt, that issues a HEAD request though, which may not be supported by the server whereas what I really want is the GET request, but display only the headers.
sgornick · 406 weeks ago
curl --head http://www.example.com
sgornick · 406 weeks ago
curl is easer. in general curl is always simpler to use then any other like wget, lynx, etc
jonnys · 339 weeks and 3 days ago
Agreed curl is the best. I always do: curl -sv http://www.example.com > /dev/null to get just the client and server headers.
fancylad · 306 weeks ago
This is a perfect command CURL uses more memory and resources than wget and is less commond on routers and minilinux.
alecuba16 · 295 weeks ago

What do you think?

Any thoughts on this command? Does it work on your machine? Can you do the same thing with only 14 characters?

You must be signed in to comment.

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: