Commands tagged wget (102)

  • This will download and install the latest version of the open store on the ubuntu phone, this store includes unconfined applications such as the TweakGeek and the Ubuntu Touch Tweak Tool. You can see the install instructions from here: https://open.uappexplorer.com/docs#install Show Sample Output


    1
    wget https://open.uappexplorer.com/api/download/openstore.openstore-team/openstore.*_*_armhf.click && pkcon install-local --allow-untrusted openstore.*_*_armhf.click
    bugmenot · 2016-02-04 14:24:46 16
  • Neither of the others worked for me. This does.


    1
    curl http://url/rss | grep -o '<enclosure url="[^"]*' | grep -o '[^"]*$' | xargs wget -c
    dakira · 2016-05-29 12:07:21 21
  • Download latest released gitlab docker container


    1
    wget -qO- 'https://github.com'$(curl -s 'https://github.com'$(curl -s https://github.com/sameersbn/docker-gitlab/releases | grep -m 1 -o '<a.*[0-9\.]</a>' | cut -d '"' -f 2) | grep -o '<a.* rel="nofollow">' | grep 'tar.gz' | cut -d '"' -f 2)
    BigZ · 2016-08-23 21:36:57 14

  • 1
    cat url.list | parallel -j 8 wget -O {#}.html {}
    arthurwayne · 2018-12-22 08:14:06 35
  • Directly download all mp3 files of the desired podcast


    1
    curl http://radiofrance-podcast.net/podcast09/rss_14726.xml | grep -Eo "(http|https)://[a-zA-Z0-9./?=_%:-]*mp3" | sort -u | xargs wget
    pascalvaucheret · 2021-08-09 13:40:26 216
  • substitute the URL with your private/public XML url from calendar sharing settings substitute the dates YYYY-mm-dd adjust the perl parsing part for your needs Show Sample Output


    0
    wget -q -O - 'URL/full?orderby=starttime&singleevents=true&start-min=2009-06-01&start-max=2009-07-31' | perl -lane '@m=$_=~m/<title type=.text.>(.+?)</g;@a=$_=~m/startTime=.(2009.+?)T/g;shift @m;for ($i=0;$i<@m;$i++){ print $m[$i].",".$a[$i];}';
    unixmonkey4704 · 2009-07-23 14:48:54 4
  • I don't know if the --spider option works to execute a script, but it might be worth trying. Note that the Drupal project uses the following in a cron job. wget -O - -q http://localhost/drupal/cron.php The output is sent to standard out so it can be logged by cron.


    0
    wget -q --spider http://server/cgi/script
    ashawley · 2009-09-11 05:33:48 3
  • Mostly for Norwegians, but easily adoptable to others. Very handy if you are brainstorming for a new domainname. Will only display the available ones.. You can usually do this better with dig, but if you dont have dig, or the TLD only have an online service to check with, this will be usefull.. Show Sample Output


    0
    check_dns_no() { for i in $* ; do if `wget -O - -q http://www.norid.no/domenenavnbaser/whois/?query=$i.no | grep "no match" &>/dev/null` ; then echo $i.no "available" ; fi ; sleep 1 ;done }
    xeor · 2009-09-30 21:17:33 8
  • Only need to install Image Magick package. Display a xkcd comic with its title and save it in /tmp directory If you prefer to view the newest xkcd, use this command: wget -q http://xkcd.com/ -O-| sed -n '/<img src="http:\/\/imgs.xkcd.com\/comics/{s/.*\(http:.*\)" t.*/\1/;p}' | awk '{system ("wget -q " $1 " -O- | display -title $(basename " $1") -write /tmp/$(basename " $1")");}'


    0
    wget -q http://dynamic.xkcd.com/comic/random/ -O-| sed -n '/<img src="http:\/\/imgs.xkcd.com\/comics/{s/.*\(http:.*\)" t.*/\1/;p}' | awk '{system ("wget -q " $1 " -O- | display -title $(basename " $1") -write /tmp/$(basename " $1")");}'
    laugg · 2009-12-09 13:41:25 8
  • This is a minimalistic version of the ubiquitious Google definition screen scraper. This version was designed not only to run fast, but to work using BusyBox. BusyBox is a collection of basic Unix tools that have been compiled into a single binary to save space on tiny installations of Unix. For example, although my phone doesn't have perl or the GNU utilities, it does have BusyBox's stripped down versions of wget, tr, and sed. It turns out that those tools suffice for many tasks. Known Bugs: This script does not handle HTML entities at all. I don't think there's an easy way to do that within BusyBox, but I'd love to see it if someone could do it. Also, this script can only define a single word, not phrases. (Well, you could if you typed in %20, but that'd be gross.) Lastly, this script does not show the URL where definitions were found. Given the randomness of the Net, that last bit of information is often key. Show Sample Output


    0
    wget -q -U busybox -O- "http://www.google.com/search?ie=UTF8&q=define%3A$1" | tr '<' '\n' | sed -n 's/^li>\(.*\)/\1\n/p'
    hackerb9 · 2010-02-01 13:01:47 10
  • yt2mp3(){ for j in `seq 1 301`;do i=`curl -s gdata.youtube.com/feeds/api/users/$1/uploads\?start-index=$j\&max-results=1|grep -o "watch[^&]*"`;ffmpeg -i `wget youtube.com/$i -qO-|grep -o 'url_map"[^,]*'|sed -n '1{s_.*|__;s_\\\__g;p}'` -vn -ab 128k "`youtube-dl -e ${i#*=}`.mp3";done;} squeezed the monster (and nifty ☺) command from 7776 from 531 characters to 284 characters, but I don't see a way to get it down to 255. This is definitely a kludge!


    0
    Command in description (Your command is too long - please keep it to less than 255 characters)
    __ · 2011-02-03 08:25:42 6
  • On a machine behind a firewall, it's possible to pass the proxy server address in as a prefix to wget to avoid having to set it as an environment variable first.


    0
    http_proxy=<proxy.server:port> wget <url>
    rdc · 2011-03-30 13:06:19 4
  • This will visit recursively all linked urls starting from the specified URL. It won't save anything locally and it will produce a detailed log. Useful to find broken links in your site. It ignores robots.txt, so just use it on a site you own!


    0
    wget --spider -o wget.log -e robots=off --wait 1 -r -p http://www.example.com
    lele · 2011-04-05 13:42:14 5

  • 0
    for i in `seq -w 1 50`; do wget --continue \ http://commandline.org.uk/images/posts/animal/$i.jpg; done
    totti · 2011-08-19 20:06:16 6

  • 0
    curl -sm1 http://www.website.com/ | grep -o 'http://[^"]*jpg' | sort -u | wget -qT1 -i-
    kev · 2011-09-10 19:21:13 5
  • Recursively download all files of a certain type down to two levels, ignoring directory structure and local duplicates. Usage: wgetall mp3 http://example.com/download/


    0
    wgetall () { wget -r -l2 -nd -Nc -A.$@ $@ }
    peterRepeater · 2011-09-28 09:43:25 4
  • This shell function uses wget(1) to show what site a shortened URL is pointing to, even if there are many nested shortened URLs. This is a great way to test whether or not the shortened URL is sending you to a malicious site, or somewhere nasty that you don't want to visit. The sample output is from: expandurl http://t.co/LDWqmtDM Show Sample Output


    0
    expandurl() { wget -S $1 2>&1 | grep ^Location; }
    atoponce · 2011-10-18 18:50:54 12
  • Gets the IP and sticks it into the middle-mouse-click buffer


    0
    echo -n $(curl -Ss http://icanhazip.com) | xclip
    red_five · 2012-02-17 16:58:40 4
  • Tries to avoid the fragile nature of scrapers by looking for user-input in the output as opposed to markup or headers on the web site. Show Sample Output


    0
    function ip-where { wget -qO- -U Mozilla http://www.ip-adress.com/ip_tracer/$1 | html2text -nobs -style pretty | sed -n /^$1/,/^$/p;}
    tox2ik · 2012-10-22 21:39:53 6
  • Updated to the new version of the MW webpage (seems MW does not use cougar anymore, so the other commands do not work nowadays), and using Xidel to parse the page with a html parser instead regex. Example usage: pronounce onomatopoetic I'm not sure how well Xidel works with binary streams (although it seems to work great in tests), so using wget to download the actual wav file might be safer, i.e.: pronounce(){ wget -qO- $(xidel "http://www.m-w.com/dictionary/$*" -f "replace(css('.au')[1]/@onclick,\".*'([^']+)', *'([^']+)'.*\", '/audio.php?file=\$1&word=\$2')" -e 'css("embed")[1]/@src') | aplay -q;} Xidel is not a standard cli tool and has to be downloaded from xidel.sourceforge.net


    0
    pronounce(){ xidel "http://www.m-w.com/dictionary/$*" -f "replace(css('.au')[1]/@onclick,\".*'([^']+)', *'([^']+)'.*\", '/audio.php?file=\$1&word=\$2')" -f 'css("embed")[1]/@src' --download - | aplay -q;}
    BeniBela · 2013-04-18 13:03:16 5
  • On Linux substitute pbpaste with `xsel --clipboard --output` or `xclip -selection clipboard -o` (untested)


    0
    pbpaste | xargs wget
    loopkid · 2013-08-11 23:12:10 7
  • This script can be used to download enclosed files from a RSS feed. For example, it can be used to download mp3 files from a podcasts RSS feed. Show Sample Output


    0
    wget -q -O- http://example-podcast-feed.com/rss | grep -o "<enclosure[ -~][^>]*" | grep -o "http://[ -~][^\"]*" | xargs wget -c
    talha131 · 2013-09-24 12:38:08 23

  • 0
    wget -q -O- http://bitinfocharts.com/markets/btc-e/btc-usd.html |grep -o -P 'lastTrade">([0-9]{1,})(.){0,1}[0-9]{0,}' |grep -o -P '(\d)+(\.){0,1}(\d)*' |head -n 1
    peter1337 · 2014-01-25 23:40:00 22
  • This is the command line I use to get my IP address in order to update my zoneedit account. Full script on my blog http://akim.sissaoui.com/linux-attitude/script-de-mise-a-jour-ddns-zoneedit-com-en-bashsh/ Show Sample Output


    0
    $ wget --no-check-certificate -q checkip.dyndns.org -O index.html && cat index.html|cut -d ' ' -f 6 | cut -d '<' -f 1
    Superkikim · 2014-05-12 07:10:29 8
  • Just pulls a quote for each day and displays it in a notification bubble... or you can change it a bit and just have it run in the terminal wget -q -O "quote" https://www.goodreads.com/quotes_of_the_day;echo "Quote of the Day";cat quote | grep '&ldquo;\|/author/show' | sed -e 's/<[a-zA-Z\/][^>]*>//g' | sed 's/&ldquo;//g' | sed 's/&rdquo;//g'; rm -f quote Show Sample Output


    0
    wget -q -O "quote" https://www.goodreads.com/quotes_of_the_day;notify-send "$(echo "Quote of the Day";cat quote | grep '&ldquo;\|/author/show' | sed -e 's/<[a-zA-Z\/][^>]*>//g' | sed 's/&ldquo;//g' | sed 's/&rdquo;//g')"; rm -f quote
    nowhereman88 · 2014-06-15 03:17:19 61
  •  < 1 2 3 4 5 > 

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands


Check These Out

debian/ubuntu get installed nvidia driver version from terminal
i've been writing a bash script where i needed to query the installed version number of the nvidia driver when it's not loaded. Unfortunately i have not found a easy way of doing this. If i'm a stupid person, please enlighten me of a better way ;)

High resolution video screen recording
$ gorecord foo.mp4 I've tried all of the screen recorders available for Linux and this is easily the best. xvidcap segfaults; VNC is too much hassle. There are alternatives of this command already here that I am just too lazy to reply to. Messing with the frames per second option, -r, 25 seems to be the best. Any lower and the video will look like a flipbook, if it records at all - -r 10 won't - any faster is the same, oddly enough. Edit: CLF doesn't like my long command to add audio, so here it is in the description. $ goaddaudio() ${ $if [ $# != 3 ]; then $ echo 'goaddaudio < audio > < src video > < dst video >' $ return $ fi $ $ f=goaddaudio$RANDOM $ ffmpeg -i "$2" &> $f $ d=$( grep Duration $f | awk '{print $2}' | tr -d ',' ) && $ rm $f && $ ffmpeg -i "$1" -i "$2" -r 25 -ab 192k -ar 44100 -sameq -t $d "$3" $}

Download an Entire website with wget

Convert a bunch of HTML files from ISO-8859-1 to UTF-8 file encoding in a folder and all sub-folders
This is my first attempt at converting all HTML files to UTF-8 file encoding, including all subfolders. Theres probably a much more compact way to do it, but I'm quite proud of it with my windows background ;)

Grab all .flv files from a webpage to the current working directory
I wanted all the 'hidden' .flv files from the http link in the command line; wget seemed appropriate, fed with output from lynx, grep the flv files and the normalised via sed (to remove the numeric bullet). Similar to the 'Grab mp3 files' fu. Replace link with your own, grep arg with something more interesting ;) See here for something along the same lines... http://www.commandlinefu.com/commands/view/1006/grab-mp3-files-from-your-favorite-netcasts-mp3blog-or-sites-that-often-have-good-mp3s Hope you find it useful! Improvements welcome, naturally.

return external ip

Find the top 10 directories containing the highest number of files
It can be used to pinpoint the path(s) where the largest number of files resides when running out of free i-nodes

Which processes are listening on a specific port (e.g. port 80)
swap out "80" for your port of interest. Can use port number or named ports e.g. "http"

recursively change file name from uppercase to lowercase (or viceversa)
Example of zsh globing, glob qualifier, and substitution: -Q state that the parameter will contain a glob qualifier. (**/)(*) is recursive (.) is our glob qualifier, with states the match is a file "." The first parameter $1, is then substituted with $2 but with lowercasing '(L)' ... a (U) would of course be from lower to upper.

Convert CSV to JSON
Replace 'csv_file.csv' with your filename.


Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: