Commands using wget (286)

  • Get gzip compressed web page using wget.

    Like the original command, but the -f allows this one to succeed even if the website returns uncompressed data. From gzip(1) on the -f flag: If the input data is not in a format recognized by gzip, and if the --stdout is also given, copy the input data without change to the standard output: let zcat behave as cat.


    1
    wget -q -O- --header="Accept-Encoding: gzip" <url> | gzip -cdf > out.html
    tempusername · 2014-11-29 20:42:21 8
  • Download all Red Hat Manuals - A better way by user Flatcap

    Let's give Flatcap credit for this elegant solution, instead of leaving it hidden as a comment. Tested on RHEL6 and it works. Nice and clean.


    1
    curl -s https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/ | grep -o '[^"]*Linux/7/pdf[^"]*' | xargs -I{} wget https://access.redhat.com{}
    SuperFly · 2015-05-11 11:57:20 11
  • Gets WAN ip address directly from DGN2200 router.


    1
    wget -O - -o /dev/null -q --user=$user --password=$pass "http://$ip/ADV_home2.htm" | awk -r '/Internet Port/, /Domain/ {if ($0 ~ /([[:digit:]]+\.){3}[[:digit:]]+/ && ($3 !~ /^>(0|255)/)) {match($3, /([[:digit:]]+\.){3}[[:digit:]]+/, ar); print ar[0]; }}'
    phranz · 2015-07-09 22:55:57 9
  • Download and install the OpenStore on the Ubuntu Phone

    This will download and install the latest version of the open store on the ubuntu phone, this store includes unconfined applications such as the TweakGeek and the Ubuntu Touch Tweak Tool. You can see the install instructions from here: https://open.uappexplorer.com/docs#install Show Sample Output


    1
    wget https://open.uappexplorer.com/api/download/openstore.openstore-team/openstore.*_*_armhf.click && pkcon install-local --allow-untrusted openstore.*_*_armhf.click
    bugmenot · 2016-02-04 14:24:46 16
  • Check syntax of remote ruby file


    1
    wget --quiet 'https://raw.githubusercontent.com/rahult/books/master/well_grounded_rubyist/threads/rps.rb' - | ruby -c
    swarzynski · 2016-02-18 11:14:55 14
  • Download all files from podcast RSS feed

    Neither of the others worked for me. This does.


    1
    curl http://url/rss | grep -o '<enclosure url="[^"]*' | grep -o '[^"]*$' | xargs wget -c
    dakira · 2016-05-29 12:07:21 21
  • Download latest released gitlab docker container

    Download latest released gitlab docker container


    1
    wget -qO- 'https://github.com'$(curl -s 'https://github.com'$(curl -s https://github.com/sameersbn/docker-gitlab/releases | grep -m 1 -o '<a.*[0-9\.]</a>' | cut -d '"' -f 2) | grep -o '<a.* rel="nofollow">' | grep 'tar.gz' | cut -d '"' -f 2)
    BigZ · 2016-08-23 21:36:57 14
  • wget download with multiple simultaneous connections


    1
    cat url.list | parallel -j 8 wget -O {#}.html {}
    arthurwayne · 2018-12-22 08:14:06 33
  • my command for downloading delicious web links,


    0
    wget -H -r -nv --level=1 -k -p -erobots=off -np -N --exclude-domains=del.icio.us,doubleclick.net --exclude-directories=
    bbelt16ag · 2009-05-18 18:05:19 4
  • get events from google calendar for a given dates range

    substitute the URL with your private/public XML url from calendar sharing settings substitute the dates YYYY-mm-dd adjust the perl parsing part for your needs Show Sample Output


    0
    wget -q -O - 'URL/full?orderby=starttime&singleevents=true&start-min=2009-06-01&start-max=2009-07-31' | perl -lane '@m=$_=~m/<title type=.text.>(.+?)</g;@a=$_=~m/startTime=.(2009.+?)T/g;shift @m;for ($i=0;$i<@m;$i++){ print $m[$i].",".$a[$i];}';
    unixmonkey4704 · 2009-07-23 14:48:54 4
  • print latest (top 10, top 3 or *) commandlinefu.com commands

    This lengthy cryptic line will print the latest top 10 commandlinefu.com posts without their summaries. To print also their respective summaries use the following (even bigger) command line: wget -qO - http://www.commandlinefu.com/feed/tenup | xmlstarlet sel -T -t -o '<doc>' -n -t -m rss/channel/item -o '<item>' -n -o '<title>' -v title -o '</title>' -n -o '<description>' -v description -o '</description>' -n -o '</item>' -n -t -o '</doc>' | xmlstarlet sel -T -t -m doc/item -v description/code -n -v title -n -n It is recommended to include this line into a shell script to be easily run, as I do myself. You could also use the following URLs to browse the top 3 commands: wget -qO - http://www.commandlinefu.com/feed/threeup | xmlstarlet ... .. or all others: wget -qO - http://feeds2.feedburner.com/Command-line-fu | xmlstarlet ... PS: You need to install "xmlstarlet" to run it. It is found in Debian APT repositories (apt-get install xmlstarlet) or under the http://xmlstar.sourceforge.net/ URL. Show Sample Output


    0
    wget -qO - http://www.commandlinefu.com/feed/tenup | xmlstarlet sel -T -t -o '&lt;x&gt;' -n -t -m rss/channel/item -o '&lt;y&gt;' -n -v description -o '&lt;/y&gt;' -n -t -o '&lt;/x&gt;' | xmlstarlet sel -T -t -m x/y -v code -n
    fsilveira · 2009-08-14 02:44:00 3
  • count the appearance of a word or a string in a given webpage

    Can be used to help perform some SEO optimizations. Show Sample Output


    0
    wget -q -O- PAGE_URL | grep -o 'WORD_OR_STRING' | wc -w
    evalinux · 2009-08-17 13:08:46 4
  • Download all Phrack .tar.gzs

    This will download all the phracks! Enjoy!


    0
    for ((i=1; i<67; i++)) do wget http://www.phrack.org/archives/tgz/phrack${i}.tar.gz -q; done
    Abiden · 2009-08-20 23:27:01 6
  • Run remote web page, but don't save the results

    I don't know if the --spider option works to execute a script, but it might be worth trying. Note that the Drupal project uses the following in a cron job. wget -O - -q http://localhost/drupal/cron.php The output is sent to standard out so it can be logged by cron.


    0
    wget -q --spider http://server/cgi/script
    ashawley · 2009-09-11 05:33:48 3
  • Get your external IP address


    0
    wget -qO - http://www.sputnick-area.net/ip;echo
    cfajohnson · 2009-11-20 23:10:31 4
  • Make ABBA better (requires firefox)

    ABBA would be more entertaining if they sang this.


    0
    wget -O - -q http://www.azlyrics.com/lyrics/abba/takeachanceonme.html | sed -e 's/[cC]hance/dump/g' > ~/tdom.htm && firefox ~/tdom.htm
    tighe · 2009-12-04 22:56:00 5
  • View a random xkcd comic

    Only need to install Image Magick package. Display a xkcd comic with its title and save it in /tmp directory If you prefer to view the newest xkcd, use this command: wget -q http://xkcd.com/ -O-| sed -n '/<img src="http:\/\/imgs.xkcd.com\/comics/{s/.*\(http:.*\)" t.*/\1/;p}' | awk '{system ("wget -q " $1 " -O- | display -title $(basename " $1") -write /tmp/$(basename " $1")");}'


    0
    wget -q http://dynamic.xkcd.com/comic/random/ -O-| sed -n '/<img src="http:\/\/imgs.xkcd.com\/comics/{s/.*\(http:.*\)" t.*/\1/;p}' | awk '{system ("wget -q " $1 " -O- | display -title $(basename " $1") -write /tmp/$(basename " $1")");}'
    laugg · 2009-12-09 13:41:25 7
  • Define words with google. (busybox version)

    This is a minimalistic version of the ubiquitious Google definition screen scraper. This version was designed not only to run fast, but to work using BusyBox. BusyBox is a collection of basic Unix tools that have been compiled into a single binary to save space on tiny installations of Unix. For example, although my phone doesn't have perl or the GNU utilities, it does have BusyBox's stripped down versions of wget, tr, and sed. It turns out that those tools suffice for many tasks. Known Bugs: This script does not handle HTML entities at all. I don't think there's an easy way to do that within BusyBox, but I'd love to see it if someone could do it. Also, this script can only define a single word, not phrases. (Well, you could if you typed in %20, but that'd be gross.) Lastly, this script does not show the URL where definitions were found. Given the randomness of the Net, that last bit of information is often key. Show Sample Output


    0
    wget -q -U busybox -O- "http://www.google.com/search?ie=UTF8&q=define%3A$1" | tr '<' '\n' | sed -n 's/^li>\(.*\)/\1\n/p'
    hackerb9 · 2010-02-01 13:01:47 9
  • download and run script from trusted webserver

    This is a simple command that you can run complex shell scripts via ssh. For instance if you would have to run the same process on several hundred hosts. There is no security so you have to trust the server that is sourcing this script.


    0
    wget -qO - sometrusted.web.site/tmp/somecommand | sh
    UnixSage · 2010-06-01 01:25:21 3
  • download with checksum

    other options: * replace md5sum with sha1sum for SHA1 checksum * replace '>' with '| tar zx' for extracting tarball Show Sample Output


    0
    wget -qO - http://www.google.com | tee >(md5sum) > /tmp/index.html
    jianingy · 2010-07-23 06:29:29 4
  • View the newest xkcd comic.

    This function displays the latest comic from xkcd.com. One of the best things about xkcd is the title text when you hover over the comic, so this function also displays that after you close the comic. To get a random xkcd comic use the following: xkcdrandom() { wget -qO- http://dynamic.xkcd.com/comic/random | sed -n 's#^<img src="\(http://imgs.[^"]\+\)"\s\+title="\(.\+\?\)"\salt.\+$#eog "\1"\necho '"'\2'#p" | bash; } These are just a bit shorter than the ones eigthmillion wrote, however his version didn't work as expected on my laptop for some reason (I got the title-tag first), so these build a command which is executed by bash.


    0
    xkcd() { wget -qO- http://xkcd.com/ | sed -n 's#^<img src="\(http://imgs.[^"]\+\)"\s\+title="\(.\+\?\)"\salt.\+$#eog "\1"\necho '"'\2'#p" | bash ; }
    John_W · 2010-08-25 15:44:31 6
  • Get the latest Geek and Poke comic


    0
    wget -q $(lynx --dump 'http://geekandpoke.typepad.com/' | grep '\/.a\/' | grep '\-pi' | head -n 1 | awk '{print $2}') -O geekandpoke.jpg
    tersmitten · 2010-09-07 12:15:36 3
  • Geolocate a given IP address

    Grabs the ip2location site and removes everything but the span tag containing the country value. Place it inside your .bashrc or .bash_aliases file. Show Sample Output


    0
    ip2loc() { wget -qO - www.ip2location.com/$1 | grep "<span id=\"dgLookup__ctl2_lblICountry\">" | sed 's/<[^>]*>//g; s/^[\t]*//; s/&quot;/"/g; s/</</g; s/>/>/g; s/&amp;/\&/g'; }
    bkuri · 2010-10-13 00:19:35 4
  • Stream and save Youtube video

    Watch a video while it's downloading. It's additionally saved to the disk for later viewing.


    0
    wget `youtube-dl -g 'http://www.youtube.com/watch?v=-S3O9qi2E2U'` -O - | tee -a parachute-ending.flv | mplayer -cache 8192 -
    artagnon · 2010-10-28 13:51:59 3
  • Install unrar on Linux box from sources


    0
    cd /usr/src ; wget http://www.rarlab.com/rar/unrarsrc-4.0.2.tar.gz ; tar xvfz unrarsrc-4.0.2.tar.gz ; cd unrar ; ln -s makefile.unix Makefile ; make clean ; make ; make install
    yababay · 2010-12-09 10:35:28 6
  • ‹ First  < 5 6 7 8 9 >  Last ›

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands

Check These Out

Translates a phrase from English to Portuguese
Translates a string from English to Portuguese by using google translator web service.

batch crop images whit ImageMagick
Just starting to get in love with mogrify.

Display summary of git commit ids and messages for a given branch
In color. Additionally you may define in your ~/.gitconfig and run it just as 'git one': one = log --pretty='format:%Cgreen%H %Cred%ai %Creset- %s'

Terminal Escape Code Zen - Strace and Tput
Depending on the TERM, the terminfo version, ncurses version, etc.. you may be using a varied assortment of terminal escape codes. With this command you can easily find out exactly what is going on.. This is terminal escape zen! $ ( 2>&2 strace -f -F -e write -s 1000 sh -c 'echo -e "initc\nis2\ncnorm\nrmso\nsgr0" | tput -S' 2>&1 ) | grep -o '"\\[^"]*"' --color=always "\33]4;%p1%d;rgb:%p2%{255}%*%{1000}%/%2.2X/%p3%{255}%*%{1000}%/%2.2X/%p4%{255}%*%{1000}%/%2.2X\33\\\33[!p\33[?3;4l\33[4l\33>\33[?12l\33[?25h\33[27m\33(B\33[m" Lets say you want to find out what you need to echo in order to get the text to blink.. $ echo -e "`tput blink`This will blink`tput sgr0` This wont" Now you can use this function instead of calling tput (tput is much smarter for portable code because it works differently depending on the current TERM, and tput -T anyterm works too.) to turn that echo into a much faster executing code. tput queries files, opens files, etc.. but echo is very strait and narrow. So now you can do this: $ echo -e "\33[5mThis will blink\33(B\33[m This wont" More at http://www.askapache.com/linux-unix/bash_profile-functions-advanced-shell.html

for all flv files in a dir, grab the first frame and make a jpg.
This is handy for making screenshots of all your videos for referring to in your flv player.

Use GNU info with a pager
This makes GNU info output menu items recursively and pipe its contents to less, allowing one to use GNU info in a manner similar to 'man'.

Which processes are listening on a specific port (e.g. port 80)
swap out "80" for your port of interest. Can use port number or named ports e.g. "http"

Advanced python tracing
Trace python statement execution and syscalls invoked during that simultaneously

Send remote command output to your local clipboard
This command will copy command's output into your local clipboard

back ssh from firewalled hosts
host B (you) redirects a modem port (62220) to his local ssh. host A is a remote machine (the ones that issues the ssh cmd). once connected port 5497 is in listening mode on host B. host B just do a ssh 127.0.0.1 -p 5497 -l user and reaches the remote host'ssh. This can be used also for vnc and so on.

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: