Commands using wget sorted by votes

Commands using wget (286)

sorted by

Wget Command to Download Full Recursive Version of Web Page

The example will create a directory called " Caveats: @imports of css files will not be converted.
This is sample output - yours may be different.
8

wget -p --convert-links http://www.foo.com

damniel · 2009-02-13 19:02:08 11
Poke a Webserver to see what it's powered by.

the good: Server: Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.4 with Suhosin-Patch the bad: Server: Microsoft-IIS/6.0 and the ugly: Server: Apache/2.2.10 (Win32) mod_ssl/2.2.10 OpenSSL/0.9.8i PHP/5.2.6
This is sample output - yours may be different.
8

wget -S -O/dev/null "INSERT_URL_HERE" 2>&1 | grep Server

asmoore82 · 2009-03-09 06:54:54 7
Download free e-books

Mask the user agent as firefox, recursively download 2 levels deep from a span host with a maximum of 1 redirection, use random wait time and dump all pdf files to myBooksFolder without creating any other directories. Host will have no way of knowing that this is a grabber script.
This is sample output - yours may be different.
8

wget -erobots=off --user-agent="Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3" -H -r -l2 --max-redirect=1 -w 5 --random-wait -PmyBooksFolder -nd --no-parent -A.pdf http://URL

unixmonkey8504 · 2010-03-01 20:22:44 29
Google Translate

allow multiword translations Show Sample Output
This is sample output - yours may be different.
```
$ translate en de I am looking for a table
```
8

translate() { lng1="$1";lng2="$2";shift;shift; wget -qO- "http://ajax.googleapis.com/ajax/services/language/translate?v=1.0&q=${@// /+}&langpair=$lng1|$lng2" | sed 's/.*"translatedText":"$[^"]*$".*}/\1\n/'; }

bandie91 · 2010-05-02 22:15:30 36
Get all files of particular type (say, PDF) listed on some wegpage (say, example.com)

See man wget if you want linked files and not only those hosted on the website.
This is sample output - yours may be different.
8

wget -r -A .pdf -l 5 -nH --no-parent http://example.com

houghi · 2011-06-09 17:17:03 6
Download an entire ftp directory using wget

If the username includes an @ you can use this one: wget -r --user=username_here --password=pass_here ftp://ftp.example.com
This is sample output - yours may be different.
7

wget -r ftp://user:pass@ftp.example.com

movaxes · 2009-03-09 19:39:30 13
Dump HTTP header using wget

Let me suggest using wget for obtaining the HTTP header only as the last resort because it generates considerable textual overhead. The first ellipsis of the sample output stands for Spider mode enabled. Check if remote file exists. --2009-03-31 20:42:46-- http://www.example.com/ Resolving www.example.com... 208.77.188.166 Connecting to www.example.com|208.77.188.166|:80... connected. HTTP request sent, awaiting response... and the second one looks for Length: 438 [text/html] Remote file exists and could contain further links, but recursion is disabled -- not retrieving. Show Sample Output
This is sample output - yours may be different.
```
&#8230;
  HTTP/1.1 200 OK
  Date: Tue, 31 Mar 2009 18:42:46 GMT
  Server: Apache/2.2.3 (CentOS)
  Last-Modified: Tue, 15 Nov 2005 13:24:10 GMT
  ETag: "b80f4-1b6-80bfd280"
  Accept-Ranges: bytes
  Content-Length: 438
  Connection: close
  Content-Type: text/html; charset=UTF-8
&#8230;
```
7

wget --server-response --spider http://www.example.com/

penpen · 2009-03-31 18:49:14 15
Block the 6700 worst spamhosts

The above url contains over 6700 of the common ad websites. The command just pastes these into your /etc/hosts. Show Sample Output
This is sample output - yours may be different.
```
-
```
7

wget -q -O - http://someonewhocares.org/hosts/ | grep ^127 >> /etc/hosts

torrid · 2009-12-14 17:11:16 8
Limit bandwidth usage by any program

Trickle is a voluntary, cooperative bandwidth shaper. it works entirely in userland and is very easy to use. The most simple application is to limit the bandwidth usage of programs.
This is sample output - yours may be different.
7

trickle -d 60 wget http://very.big/file

point_to_null · 2010-03-29 06:55:30 5
Lets Tux say the random fact. [add it to .bashrc to see it in new terminal window]
This is sample output - yours may be different.
7

wget randomfunfacts.com -O - 2>/dev/null|grep \<strong\>|sed "s;^.*<i>$.*$</i>.*$;\1;"|cowsay -f tux

unixmonkey10612 · 2010-07-01 18:31:36 3
Retrieve a random command from the commandlinefu.com API

Seeing that we get back plain text anyway we don't need lynx. Also the sed-part removes the credit line.
This is sample output - yours may be different.
7

wget -qO - http://www.commandlinefu.com/commands/random/plaintext | sed -n '1d; /./p'

dramaturg · 2010-12-05 15:32:14 11
Look for English words in /dev/urandom

Little faster alternative.
This is sample output - yours may be different.
7

head -100000 /dev/urandom | strings|tr '[A-Z]' '[a-z]'|sort >temp.txt && wget -q http://www.mavi1.org/web_security/wordlists/webster-dictionary.txt -O-|tr '[A-Z]' '[a-z]'|sort >temp2.txt&&comm -12 temp.txt temp2.txt

marssi · 2010-12-26 11:04:42 6
Download from Rapidshare Premium using wget - Part 2

The download content part. NOTE: the '-c' seems to not work very well and the download stuck at 99% sometimes. Just finish wget with no problem. Also, the download may restart after complete. You can also cancel. I don't know if it is a wget or Rapidshare glitch since I don't have problems with Megaupload, for example. UPDATE: as pointed by roebek the restart glitch can be solved by the "-t 1" option. Thanks a lot.
This is sample output - yours may be different.
6

wget -c -t 1 --load-cookies ~/.cookies/rapidshare <URL>

o6291408 · 2009-03-28 09:13:35 5
background a wget download
This is sample output - yours may be different.
6

wget -b http://dl.google.com/android/android-sdk_r14-linux.tgz

zlemini · 2011-10-19 21:55:25 5
Find out how old a web page is

I used to use the Firefox "View page info" feature a lot to determine how stale the web page I was looking at was. Now that I use mostly Chrome I miss that feature, so here is a command line alternative using wget. The -S says to display the server response, the --spider says to not download any files/pages, just fetch the header. The output goes to stderr, so to grep it you use 2>&1 to combine the stderr stream with stdout, the pipe that to grep for Last-Modified. You can use curl instead if you have it installed, like this: curl --head -s http://osswin.sourceforge.net | grep Mod Show Sample Output
This is sample output - yours may be different.
```
  Last-Modified: Wed, 08 Feb 2012 20:14:53 GMT
```
6

wget -S --spider http://osswin.sourceforge.net/ 2>&1 | grep Mod

dmmst19 · 2012-04-18 03:43:33 6
download 10 random wallpapers from google

you may want &hl=en for &hl=es for the language you may want imgsz=xxlarge for imgsz=large or whatever filter you may want q=apples or whatever
This is sample output - yours may be different.
6

for i in {1..10};do wget $(wget -O- -U "" "http://images.google.com/images?imgsz=xxlarge&hl=en&q=wallpaper+HD&start=$(($RANDOM%900+100))" --quiet | grep -oe 'http://[^"]*\.jpg' | head -1);done

dzup · 2012-07-26 10:42:13 6
Update your OpenDNS network ip

Intended for dynamic ip OpenDNS users, this command will update your OpenDNS network IP. For getting your IP, you can use one of the many one-liners here on commandlinefu. Example: I use this in a script which is run by kppp after it has successfully connected to my ISP: --- #!/bin/bash IP="`curl -s http://checkip.dyndns.org/ | grep -o '[[:digit:].]\+'`" PW="hex-obfuscated-pw-here" if [ "$IP" == "" ] ; then echo 'Not online.' ; exit 1 else wget -q --user=topsecret --password="`echo $PW | xxd -ps -r`" 'https://updates.opendns.com/nic/update?hostname=myhostname&myip='"$IP" -O - /etc/init.d/ntp-client restart & fi --- PS: DynDNS should use a similar method, if you know the URL, please post a comment. (Something with members.dyndns.org, if I recall correctly) Show Sample Output
This is sample output - yours may be different.
```
alanceil@kvirasim:19:55:0:~> wget -q --user=topsecret --password=topsecret 'https://updates.opendns.com/nic/update?hostname=myhostname&myip=12.34.56.78' -O -
good 12.34.56.78
```
5

wget -q --user=<username> --password=<password> 'https://updates.opendns.com/nic/update?hostname=your_opendns_hostname&myip=your_ip' -O -

Alanceil · 2009-06-22 18:08:42 24
use wget to check if a remote file exists
This is sample output - yours may be different.
5

wget --spider -v http://www.server.com/path/file.ext

loetje · 2009-08-04 22:27:54 4
return external ip

I dont have curl or links installed, so I use wget with write file as standard out. Show Sample Output
This is sample output - yours may be different.
```
150.162.64.156
```
5

wget -O - -q icanhazip.com

andrepuel · 2009-08-14 10:24:30 14
Download Apple movie trailers

Copy the link to an HD movie trailer in to this command. It's more eleganant if it's put in a to a script, taking the URL as input.
This is sample output - yours may be different.
5

wget -U "QuickTime/7.6.2 (qtver=7.6.2;os=Windows NT 5.1Service Pack 3)" `echo http://movies.apple.com/movies/someHDmovie_720p.mov | sed 's/$[0-9][0-9]$0p/h\10p/'`

deadrabbit · 2009-08-29 00:29:40 21
download file1 file2 file3 file4 .... file 100
This is sample output - yours may be different.
5

wget http://domain.com/file{1..100}

wincus · 2010-01-12 14:28:04 3
Pronounce an English word using Merriam-Webster.com

The original was a little bit too complicated for me. This one does not use any variables.
This is sample output - yours may be different.
5

pronounce(){ wget -qO- $(wget -qO- "http://www.m-w.com/dictionary/$@" | grep 'return au' | sed -r "s|.*return au$'([^']*)', '([^'])[^']*'$.*|http://cougar.eb.com/soundc11/\2/\1|") | aplay -q; }

matthewbauer · 2010-03-12 17:44:16 6
translate what is in the clipboard in english and write it to the terminal

Uses google api to translate, you can modify the language in which translate modifying the parameter "langpair=|en", the format is language input|language output.
This is sample output - yours may be different.
5

wget -qO - "http://ajax.googleapis.com/ajax/services/language/translate?langpair=|en&v=1.0&q=`xsel`" |cut -d \" -f 6

fain182 · 2010-06-04 17:20:17 11
Google Translate

substitute "example" with desired string; tl = target language (en, fr, de, hu, ...); you can leave sl parameter as-is (autodetection works fine) Show Sample Output
This is sample output - yours may be different.
```
exemple de traduction
```
5

wget -U "Mozilla/5.0" -qO - "http://translate.google.com/translate_a/t?client=t&text=translation+example&sl=auto&tl=fr" | sed 's/\[\[\[\"//' | cut -d \" -f 1

sairon · 2011-03-06 13:46:16 109
Retrieve the size of a file on a server

- Where $URL is the URL of the file. - Replace the $2 by $3 at the end to get a human-readable size. Credits to svanberg @ ArchLinux forums for original idea. Edit: Replaced command with better version by FRUiT. (removed unnecessary grep)
This is sample output - yours may be different.
5

wget --spider $URL 2>&1 | awk '/Length/ {print $2}'

d3Xt3r · 2011-07-03 00:14:58 10
< 1 2 3 4 > Last ›

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands

Check These Out

Download all files from podcast RSS feed

Neither of the others worked for me. This does.

Email an svn dump

Dumps a compressed svn backup to a file, and emails the files along with any messages as the body of the email

List apache2 virtualhosts

Lists virtualhosts currently enabled for apache2, showing the ServerName:port, conf file and DocumentRoot

Get the IP address of a machine. Just the IP, no junk.

Why use many different utilities all piped together, when you only need two?

Convert seconds to [DD:][HH:]MM:SS

Converts any number of seconds into days, hours, minutes and seconds. sec2dhms() { declare -i SS="$1" D=$(( SS / 86400 )) H=$(( SS % 86400 / 3600 )) M=$(( SS % 3600 / 60 )) S=$(( SS % 60 )) [ "$D" -gt 0 ] && echo -n "${D}:" [ "$H" -gt 0 ] && printf "%02g:" "$H" printf "%02g:%02g\n" "$M" "$S" }

Enter parameter if empty (script becomes interactive when parameters are missing)

Can be used for command line parameters too. If you have a more complicated way of entering values (validation, GUI, ...), then write a function i.e. EnterValue() that echoes the value and then you can write: $ param=${param:-$(EnterValue)}

Stream system sounds over rtmp

sox (SOund eXchange) can capture the system audio be it a browser playing youtube or from hardware mic and can pipe it to ffmpeg which encodes it into flv and send it over rtmp. Tested using Red5 rtmp server.

Find usb device in realtime

Using this command you can track a moment when usb device was attached.

Create a bunch of dummy files for testing

Sometimes I need to create a directory of files to operate on to test out some commandlinefu I am cooking up. The main thing is the range ({1..N}) expansion.

Convert a bunch of HTML files from ISO-8859-1 to UTF-8 file encoding in a folder and all sub-folders

This is my first attempt at converting all HTML files to UTF-8 file encoding, including all subfolders. Theres probably a much more compact way to do it, but I'm quite proud of it with my windows background ;)

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for:

» all commands
» commands with 3 up-votes (commandlinefu3)
» commands with 10 up-votes (commandlinefu10)
» commands using wget

Commands using wget (286) the last day the last week the last month all time sorted by date votes

What's this?

Check These Out

Stay in the loop…

Commands using wget (286)

sorted by