Commands using wget (270)

  • This will download all files of the type specified after "-A" from a website. Here is a breakdown of the options: -r turns on recursion and downloads all links on page -l1 goes only one level of links into the page(this is really important when using -r) -H spans domains meaning it will download links to sites that don't have the same domain -nd means put all the downloads in the current directory instead of making all the directories in the path -A mp3 filters to only download links that are mp3s(this can be a comma separated list of different file formats to search for multiple types) -e robots=off just means to ignore the robots.txt file which stops programs like wget from crashing the site... sorry http://example/url lol..


    3
    wget -r -l1 -H -nd -A mp3 -e robots=off http://example/url
    trizko · 2013-07-13 02:00:23 0
  • This command will help you to get Ip address origin of city and country. I will be happy if someone can shrink the sed command fb.me/josenirmal Show Sample Output


    0
    wget -qO - whatismyipaddress.com/ip/<type ip address> | grep -E "City:|Country:" | sed 's:<tr><th>::'| sed 's</th>::' | sed 's:</td>::' | sed 's:</tr>::' | sed 's:<img*::'
    pentester · 2013-06-21 03:27:09 0
  • Need to find a Mageia Linux mirror server providing Mageia 4 via rsync? Modify the "url=" string for the version you want. This shows i586 which is the 32bit version. If you want the 64bit version it is: url=http://mirrors.mageia.org/api/mageia.4.x86_64.list; wget -q ${url} -O - | grep rsync: Show Sample Output


    1
    url=http://mirrors.mageia.org/api/mageia.4.i586.list; wget -q ${url} -O - | grep rsync:
    mpb · 2013-05-20 16:19:05 0
  • in place of "output-filename.mp4" put the name you want the file to be named with. in place of "youtube-video-link" put the link of the Video page eg: http://www.youtube.com/watch?v=AclA-7YntvE in place of "format-number" put the number of the file format you would like How to get the "format-number" to get format number type in below command before running this command youtube-dl -F "youtube-video-link" and it will list all the available formats with the format number, like to download in 360p mp4 use the number "18" To automatically let it fetch the best quality available just remove the -f "format-number" and you are good to go. Show Sample Output


    2
    wget -O "output-filename.mp4" $( youtube-dl -g -f "format-number" "youtube-video-link" )
    unixmonkey57804 · 2013-05-19 16:25:30 1
  • remove old index.html if you download it again and organiaz the java script tag on the file index.html


    0
    rm index.html | wget www.google.com;cat index.html | sed 's/<script>/\n\n\<script>\n\n/g' | sed 's/<\/script>/>\n\n/g'
    lbhack · 2013-04-10 04:05:30 0
  • Reason can be: taken, available, contains_banned_word


    0
    if wget https://twitter.com/users/username_available?username=xmuda -q -O - | grep -q "\"reason\":\"taken\""; then echo "Username taken"; else echo "Free / Banned Name"; fi
    Joschasa · 2013-03-23 17:39:15 0

  • 0
    wget -O - "[PICASA ALBUM RSS LINK]" |sed 's/</\n</g' | grep media:content |sed 's/.*url='"'"'\([^'"'"']*\)'"'"'.*$/\1/' |awk -F'/' '{gsub($NF,"d/"$NF); print $0}'|wget -i -
    mxac · 2013-03-16 15:48:41 0
  • alias speedtest='wget --output-document=/dev/null http://speedtest.wdc01.softlayer.com/downloads/test500.zip'


    1
    wget --output-document=/dev/null http://speedtest.wdc01.softlayer.com/downloads/test500.zip
    opexxx · 2013-03-15 13:25:07 0
  • Very similar as doing "wget http://example.com/mytarball|tar xzv", this one involves the "tee" command between both, which will simultaneously write the tarball and copy it to stdout. So this command will locally save the tarball and extract it - both at the same time while it downloads.


    0
    wget -O- http://example.com/mytarball.tgz | tee mytarball.tgz | tar xzv
    unixmonkey48888 · 2013-03-06 11:11:28 0
  • 1.- Enter into the playlist path. 2.- Run the command. 3.- Playlists created!


    2
    wget -q -O - http://listen.di.fm/public2 | sed 's/},{/\n/g' | perl -n -e '/"key":"([^"]*)".*"playlist":"([^"]*)"/; print "$1\n"; system("wget -q -O - $2 | grep -E '^File' | cut -d= -f2 > di_$1.m3u")'
    Zort · 2013-02-20 03:37:41 1
  • I couldn't find this on the site and it's a useful switch. Great for large files. Show Sample Output


    2
    wget -c or wget --continue
    sonic · 2013-02-17 21:12:00 0
  • If you have to deal with MS Sharepoint which is (rarely, let's hope) used in e.g. certain corporate environments). This uses Cntlm. For single files, just use cURL -- its NTLM authentication works quite well. # /etc/cntlm.conf: # Username account # Domain domain # Password ############ # Proxy 10.20.30.40 (IP of the sharepoint site) # NoProxy * # Listen 3128


    1
    http_proxy=http://127.0.0.1:3128 wget --http-user='domain\account' --http-password='###' -p -r -l 8 --no-remove-listing -P . 'http://sp.corp.com/teams/Team/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2fteams%2fTeam%2fShared%20Documents%2fFolder'
    mhs · 2012-12-26 09:03:55 0

  • 0
    wget --recursive --page-requisites --convert-links www.moyagraphix.co.za
    bayu_aritedjo · 2012-12-21 08:23:54 0
  • In this example, where the users gpg keyring has a password, the user will be interactively prompted for the keyring password. If the keyring has no password, same as above, sans the prompt. Suitable for cron jobs. ~/.gnupg/passwd/http-auth.gpg is the encrypted http auth password, for this particular wget use case. This approach has many use cases. example bash functions: function http_auth_pass() { gpg2 --decrypt ~/.gnupg/passwd/http-auth.gpg 2>/dev/null; } function decrypt_pass() { gpg2 --decrypt ~/.gnupg/passwd/"$1" 2>/dev/null; }


    1
    wget --input-file=~/donwloads.txt --user="$USER" --password="$(gpg2 --decrypt ~/.gnupg/passwd/http-auth.gpg 2>/dev/null)"
    kyle0r · 2012-12-13 00:14:55 0
  • This command should be copy-pasted in Windows, but very similar one will work on Linux. It uses wget and sed.


    -1
    wget --no-check-certificate https://code.google.com/p/msysgit/downloads/list -O - 2>nul | sed -n "0,/.*\(\/\/msysgit.googlecode.com\/files\/Git-.*\.exe\).*/s//http:\1/p" | wget -i - -O Git-Latest.exe
    michfield · 2012-11-14 08:17:50 0
  • Tries to avoid the fragile nature of scrapers by looking for user-input in the output as opposed to markup or headers on the web site. Show Sample Output


    0
    function ip-where { wget -qO- -U Mozilla http://www.ip-adress.com/ip_tracer/$1 | html2text -nobs -style pretty | sed -n /^$1/,/^$/p;}
    tox2ik · 2012-10-22 21:39:53 0
  • Like i said, i havent test it yet, all becouse my internet its soo slow, if you try and works please share, also be nice to do it using the direct url link. Show Sample Output


    -2
    ompload() { wget -O- - "$1" --quiet|curl -# -F file1=@- http://ompldr.org/upload|awk '/Info:|File:|Thumbnail:|BBCode:/{gsub(/<[^<]*?\/?>/,"");$1=$1;print}';}
    dzup · 2012-10-19 22:54:55 1
  • This will take the packages matching a given `apt-cache search` query (a collection of AND'd words or regexps) and tell you how popular they are. This is particularly nice for those times you have to figure out which solution to use for e.g. a PDF reader or a VNC client. Substitute "ubuntu.com" for "debian.org" if you want this to use Ubuntu's data instead. Everything else will work perfectly. Show Sample Output


    4
    apt-popcon() { (echo \#rank; apt-cache search "$@" |awk '$1 !~ /^lib/ {print " "$1" "}') |grep -Ff- <(wget -qqO- http://popcon.debian.org/by_inst.gz |gunzip); }
    adamhotep · 2012-09-08 00:29:31 0

  • 0
    wget -mk -w 20 http://www.example.com/
    evandrix · 2012-08-27 14:33:48 0

  • -3
    wb(){ for i in $(wget -O- -U "" "http://wallbase.cc/random/23/eqeq/1920x1080/0/" --quiet|grep wallpaper/|grep -oe 'http://wallbase.[^"]*'); do if (( n > "$1" )); then break;fi;let n++;wget $(wget -O- -U "" $i --quiet|grep -oe 'http://[^"]*\.jpg');done;}
    dzup · 2012-08-21 17:00:44 0

  • -1
    for i in $(wget -O- -U "" "http://wallbase.cc/random/23/e..." --quiet|grep wallpaper/|grep -oe 'http://wallbase.cc[^"]*'); do wget $(wget -O- -U "" $i --quiet|grep -oe 'http://[^"]*\.jpg');done
    mama21mama · 2012-08-19 11:06:30 0

  • 0
    wget --no-use-server-timestamps $(curl $(curl http://wallbase.cc/random/23/eqeq/1920x1080/0/100/20 | grep 'wallpaper/' | awk -F'"' '{print $2}' | head -n1) | grep -A4 bigwall | grep img | awk -F'"' '{print $2}'); feh --bg-center $(ls -1t | head -n1)
    yosh · 2012-08-14 22:06:27 0
  • you may want &hl=en for &hl=es for the language you may want imgsz=xxlarge for imgsz=large or whatever filter you may want q=apples or whatever


    6
    for i in {1..10};do wget $(wget -O- -U "" "http://images.google.com/images?imgsz=xxlarge&hl=en&q=wallpaper+HD&start=$(($RANDOM%900+100))" --quiet | grep -oe 'http://[^"]*\.jpg' | head -1);done
    dzup · 2012-07-26 10:42:13 2

  • 1
    while pgrep wget || sudo shutdown -P now; do sleep 1m; done
    kev · 2012-05-20 17:49:56 1

  • 1
    cat urls.txt | wget -i- -T 10 -t 3 --waitretry 1
    kev · 2012-05-14 06:41:14 2
  •  < 1 2 3 4 5 >  Last ›

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands



Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: