Commands using wget (264)


  • 0
    curl -sm1 http://www.website.com/ | grep -o 'http://[^"]*jpg' | sort -u | wget -qT1 -i-
    kev · 2011-09-10 19:21:13 1
  • No need to install additional packages eg: say hello For multiword say how+are+you


    0
    say() { wget -q -U Mozilla -O output.mp3 "http://translate.google.com/translate_tts?tl=en&q=$1"; gnome-terminal -x bash -c "totem output.mp3"; sleep 4; totem --quit;}
    totti · 2011-09-07 19:48:53 0
  • -r ? указывает на то, что нужно рекурсивно переходить по ссылкам на сайте, чтобы скачивать страницы. -k ? используется для того, чтобы wget преобразовал все ссылки в скаченных файлах таким образом, чтобы по ним можно было переходить на локальном компьютере (в автономном режиме). -p ? указывает на то, что нужно загрузить все файлы, которые требуются для отображения страниц (изображения, css и т.д.). -l ? определяет максимальную глубину вложенности страниц, которые wget должен скачать (по умолчанию значение равно 5, в примере мы установили 7). В большинстве случаев сайты имеют страницы с большой степенью вложенности и wget может просто ?закопаться?, скачивая новые страницы. Чтобы этого не произошло можно использовать параметр -l. -E ? добавлять к загруженным файлам расширение .html. -nc ? при использовании данного параметра существующие файлы не будут перезаписаны. Это удобно, когда нужно продолжить загрузку сайта, прерванную в предыдущий раз.


    -4
    wget -r -k -l 7 -p -E -nc http://site.com/
    realjkeee · 2011-08-20 10:16:06 0
  • This uses wget instead of curl


    -1
    wget -q -O - http://www.perl.org/get.html | grep -m1 '\.tar\.gz' | sed 's/.*perl-//; s/\.tar\.gz.*//'
    dbbolton · 2011-08-19 23:38:10 0

  • 0
    for i in `seq -w 1 50`; do wget --continue \ http://commandline.org.uk/images/posts/animal/$i.jpg; done
    totti · 2011-08-19 20:06:16 2
  • ?mirror : turn on options suitable for mirroring. -p : download all files that are necessary to properly display a given HTML page. ?convert-links : after the download, convert the links in document for local viewing. -P ./LOCAL-DIR : save all the files and directories to the specified directory.


    1
    $ wget --mirror -p --convert-links -P ./<LOCAL-DIR> <WEBSITE-URL>
    tkembo · 2011-08-18 08:27:28 2
  • I wanted to play a song from the shell and get the shell back, I also dont want to store the file if it is not needed. edit, not sure if I need to mention it... killall vlc to stop it Show Sample Output


    -1
    wget http://somesite.com/somestream.pls; cvlc somestream.pls&sleep 5; rm somestream.pls*
    tomjrace · 2011-08-04 19:24:18 0

  • 1
    NAME=`wget --quiet URL -O - | grep util-vserver | tail -n 1 | sed 's|</a>.*||;s/.*>//'`; wget URL$UTILVSERVER;
    WMP · 2011-07-17 13:01:20 1
  • - Where $URL is the URL of the file. - Replace the $2 by $3 at the end to get a human-readable size. Credits to svanberg @ ArchLinux forums for original idea. Edit: Replaced command with better version by FRUiT. (removed unnecessary grep)


    5
    wget --spider $URL 2>&1 | awk '/Length/ {print $2}'
    d3Xt3r · 2011-07-03 00:14:58 6
  • This wgets the iso directly to the USB device, replace /dev/sdb with the device name of the USB stick. After wget finishes you will be able to boot the .iso file from the USB stick.


    0
    wget -O/dev/sdb ftp://ftp.debian.org/debian/dists/stable/main/installer-amd64/current/images/netboot/mini.iso
    unixmonkey22563 · 2011-06-12 21:58:13 0
  • See man wget if you want linked files and not only those hosted on the website.


    8
    wget -r -A .pdf -l 5 -nH --no-parent http://example.com
    houghi · 2011-06-09 17:17:03 1
  • This example command fetches 'example.com' webpage and then fetches+saves all PDF files listed (linked to) on that webpage. [*Note: of course there are no PDFs on example.com. This is just an example]


    1
    curl -s http://example.com | grep -o -P "<a.*href.*>" | grep -o "http.*.pdf" | xargs -d"\n" -n1 wget -c
    b_t · 2011-06-09 14:42:46 1
  • This one-liner greps first 30 direct URLs for .torrent files matching your search querry, ordered by number of seeds (descending; determined by the second number after your querry, in this case 7; for other options just check the site via your favorite web-browser). You don't have to care about grepping the torrent names as well, because they are already included in the .torrent URL (except for spaces and some other characters replaced by underscores, but still human-readable). Be sure to have some http://isup.me/ macro handy (someone often kicks the ethernet cables out of their servers ;) ). I've also coded a more user-friendly ash (should be BASH compatible) script, which also lists the total size of download and number of seeds/peers (available at http://saironiq.blogspot.com/2011/04/my-shell-scripts-4-thepiratebayorg.html - may need some tweaking, as it was written for a router running OpenWrt and transmission). Happy downloading!


    4
    wget -U Mozilla -qO - "http://thepiratebay.org/search/your_querry_here/0/7/0" | grep -o 'http\:\/\/torrents\.thepiratebay\.org\/.*\.torrent'
    sairon · 2011-04-15 15:01:16 0
  • put your link [url] to check if exist the remote file Show Sample Output


    1
    wget -O/dev/null -q URLtoCheck && echo exists || echo not exist
    xeonproject · 2011-04-07 20:55:33 0
  • This will visit recursively all linked urls starting from the specified URL. It won't save anything locally and it will produce a detailed log. Useful to find broken links in your site. It ignores robots.txt, so just use it on a site you own!


    0
    wget --spider -o wget.log -e robots=off --wait 1 -r -p http://www.example.com
    lele · 2011-04-05 13:42:14 1
  • On a machine behind a firewall, it's possible to pass the proxy server address in as a prefix to wget to avoid having to set it as an environment variable first.


    0
    http_proxy=<proxy.server:port> wget <url>
    rdc · 2011-03-30 13:06:19 0
  • Check if a site is down with downforeveryoneorjustme.com


    13
    down4me() { wget -qO - "http://www.downforeveryoneorjustme.com/$1" | sed '/just you/!d;s/<[^>]*>//g' ; }
    vando · 2011-03-11 14:38:38 0
  • wget -qO - "http://www.google.com/dictionary/json?callback=dict_api.callbacks.id100&q=steering+wheel&sl=en&tl=en&restrict=pr,de&client=te" this does the actual google dictionary query, returns a JSON string encapsulated in some fancy tag sed 's/dict_api\.callbacks.id100.//' here we remove the tag beginning sed 's/,200,null)//' and here the tag end There are also some special characters which could cause problems with some JSON parsers, so if you get some errors, this is probably the case (sed is your friend). I laso like to trim the "webDefinitions" part, because it (sometimes) contains misleading information. sed 's/\,\"webDefinitions.*//' (but remember to append a "}" at the end, because the JSON string will be invalid) The output also contains links to mp3 files with pronounciation. As of now, this is only usable in the English language. If you choose other than English, you will only get webDefinitions (which are crap).


    1
    wget -qO - "http://www.google.com/dictionary/json?callback=dict_api.callbacks.id100&q=steering+wheel&sl=en&tl=en&restrict=pr,de&client=te" | sed 's/dict_api\.callbacks.id100.//' | sed 's/,200,null)//'
    sairon · 2011-03-08 15:00:39 0
  • EDIT: command updated to support accented characters! Works in any of 58 google supported languages (some sound like crap, english is the best IMO). You get a mp3 file containing your query in spoken language. There is a limit of 100 characters for the "q" parameter, so be careful. The "tl" parameter contains target language.


    37
    wget -q -U Mozilla -O output.mp3 "http://translate.google.com/translate_tts?ie=UTF-8&tl=en&q=hello+world
    sairon · 2011-03-08 14:05:36 12
  • The FLAC audio must be encoded at 16000Hz sampling rate (SoX is your friend). Outputs a short JSON string, the actual speech is in the hypotheses->utterance, the accuracy is stored in hypotheses->confidence (ranging from 0 to 1). Google also accepts audio in some special speex format (audio/x-speex-with-header-byte), which is much smaller in comparison with losless FLAC, but I haven't been able to encode such a sample. Show Sample Output


    3
    wget -q -U "Mozilla/5.0" --post-file speech.flac --header="Content-Type: audio/x-flac; rate=16000" -O - "http://www.google.com/speech-api/v1/recognize?lang=en-us&client=chromium"
    sairon · 2011-03-08 13:39:01 0
  • substitute "example" with desired string; tl = target language (en, fr, de, hu, ...); you can leave sl parameter as-is (autodetection works fine) Show Sample Output


    5
    wget -U "Mozilla/5.0" -qO - "http://translate.google.com/translate_a/t?client=t&text=translation+example&sl=auto&tl=fr" | sed 's/\[\[\[\"//' | cut -d \" -f 1
    sairon · 2011-03-06 13:46:16 0

  • 0
    wget http://cmyip.com -O - -o /dev/null | grep -Po '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+'
    ruslan · 2011-02-28 16:46:14 0
  • Download a bunch of random animated gifs from http://gifbin.com/


    -3
    site="http://gifbin.com/"; for i in $(wget -qO- "$site"random| sed -r "s/^.*(bin\/.+\.gif).*$/\1/m" | grep "^bin"); do wget -c "$site$i"; filename=`basename $i`; [ `identify $filename | wc -l` -gt 1 ] || rm -f $filename; done
    az · 2011-02-15 10:05:37 0
  • Get external IP of the current machine via http://cmyip.com Show Sample Output


    -3
    wget http://cmyip.com -O - -o /dev/null | awk '/\<title/ {print $4}'
    ruslan · 2011-02-15 05:58:30 1

  • -2
    wget --mirror -A.jpg http://www.xs4all.nl/~dassel/wall/
    madrasi · 2011-02-08 03:15:35 0
  • ‹ First  < 3 4 5 6 7 >  Last ›

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands



Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: