Commands tagged wget (102)
-
No curl and no sed, just wget :)
2isgd() { /usr/bin/wget -qO - "http://is.gd/create.php?format=simple&url=$1" ;} -
in place of "output-filename.mp4" put the name you want the file to be named with. in place of "youtube-video-link" put the link of the Video page eg: http://www.youtube.com/watch?v=AclA-7YntvE in place of "format-number" put the number of the file format you would like How to get the "format-number" to get format number type in below command before running this command
youtube-dl -F "youtube-video-link"and it will list all the available formats with the format number, like to download in 360p mp4 use the number "18" To automatically let it fetch the best quality available just remove the -f "format-number" and you are good to go. Show Sample Output2wget -O "output-filename.mp4" $( youtube-dl -g -f "format-number" "youtube-video-link" ) -
Optionally, pipe the output into http://sed.sourceforge.net/grabbag/scripts/html2iso.sed Or: wget -qO - http://www.asciiartfarts.com/random.cgi | sed -n '//,//p' | sed -n '/ Show Sample Output
2wget -qO - http://www.asciiartfarts.com/random.cgi | sed -n '/<pre>/,/<\/pre>/p' | sed -n '/<table*/,/<\/table>/p' | sed '1d' | sed '$d' | recode html..ascii -
2wget --recursive --no-clobber --page-requisites --html-extension --convert-links --domains website.org --no-parent www.website.com/folder
-
The difference between the original version provided and this one is that this one works rather than outputting a wget error
2curl $1 | grep -E "http.*\.mp3" | sed "s/.*\(http.*\.mp3\).*/\1/" | xargs wget -
[Note: This command needs to be run as root]. If you are downloading something large at night, you can start wget as a normal user and issue the above command as root. When the download is done, the computer will automatically go to sleep. If at any time you feel the computer should not go to sleep automatically(like if you find the download still continuing in the morning), just create an empty file called nosleep in /tmp directory.
1while [ -n "`pgrep wget`" ]; do sleep 2 ;done; [ -e "/tmp/nosleep"] || echo mem >/sys/power/state -
1wget `lynx --dump http://xkcd.com/|grep png`
-
Prompts the user for username and password, that are then exported to http_proxy for use by wget, yum etc Default user, webproxy and port are used. Using this script prevent the cleartext user and pass being in your bash_history and on-screen Show Sample Output
1set-proxy () { P=webproxy:1234; DU="fred"; read -p "username[$DU]:" USER; printf "%b"; UN=${USER:-$DU}; read -s -p "password:" PASS; printf "%b" "\n"; export http_proxy="http://${UN}:${PASS}@$P/"; export ftp_proxy="http://${UN}:${PASS}@$P/"; } -
To learn more about Google Ngram Viewer: http://ngrams.googlelabs.com/info
1wget -qO - http://ngrams.googlelabs.com/datasets | grep -E href='(.+\.zip)' | sed -r "s/.*href='(.+\.zip)'.*/\1/" | uniq | while read line; do `wget $line`; done -
This example command fetches 'example.com' webpage and then fetches+saves all PDF files listed (linked to) on that webpage. [*Note: of course there are no PDFs on example.com. This is just an example]
1curl -s http://example.com | grep -o -P "<a.*href.*>" | grep -o "http.*.pdf" | xargs -d"\n" -n1 wget -c -
1wget -A mp3,mpg,mpeg,avi -r -l 3 http://www.site.com/
-
1wget -nd -r -l 2 -A jpg,jpeg,png,gif http://website-url.com
-
A simple script for download all the MegaTokyo strips from the first to the last one
1for i in $(seq 1 `curl http://megatokyo.com 2>/dev/null|grep current|cut -f6 -d\"`);do wget http://megatokyo.com/`curl http://megatokyo.com/strip/${i} 2>/dev/null|grep src=\"strips\/|cut -f4 -d\"`;done -
I is for headers only s is for silence curl -Is outputs ONLY headers the pipe and grep is to filter them to Modified only.. Show Sample Output
1curl -Is http://osswin.sourceforge.net/ | grep Mod -
1cat urls.txt | wget -i- -T 10 -t 3 --waitretry 1
-
1while pgrep wget || sudo shutdown -P now; do sleep 1m; done
-
1telnet v4address.com
-
If you have to deal with MS Sharepoint which is (rarely, let's hope) used in e.g. certain corporate environments). This uses Cntlm. For single files, just use cURL -- its NTLM authentication works quite well. # /etc/cntlm.conf: # Username account # Domain domain # Password ############ # Proxy 10.20.30.40 (IP of the sharepoint site) # NoProxy * # Listen 3128
1http_proxy=http://127.0.0.1:3128 wget --http-user='domain\account' --http-password='###' -p -r -l 8 --no-remove-listing -P . 'http://sp.corp.com/teams/Team/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2fteams%2fTeam%2fShared%20Documents%2fFolder' -
alias speedtest='wget --output-document=/dev/null http://speedtest.wdc01.softlayer.com/downloads/test500.zip'
1wget --output-document=/dev/null http://speedtest.wdc01.softlayer.com/downloads/test500.zip -
first grep all href images then sed the url part then wget
1curl -s $1 | grep -o -i '<a href="//images.4chan.org/[^>]*>' | sed -r 's%.*"//([^"]*)".*%\1%' | xargs wget -
First (and only) argument should be a 4chan thread URL.
1function 4chandl () { wget -e robots=off -nvcdp -t 0 -Hkrl 0 -I \*/src/ -P . "$1" } -
Returns your external IP address to the command line using only wget Show Sample Output
1wget http://ipecho.net/plain -O - -q ; echo -
Download latest NVIDIA Geforce x64 Windows7-8 driver from Nvidia's website. Pulls the latest download version (which includes beta). This is the "English" version. The following command includes a 'sed' line to replace "english" with "international" if needed. You can also replace the starting subdomain with "eu." "uk." and others. Enjoy this one liner! 1 character under the max :)
wget "us.download.nvidia.com$(wget -qO- "$(wget -qO- "nvidia.com/Download/processFind.aspx?psid=95&pfid=695&osid=19&lid=1&lang=en-us" | awk '/driverResults.aspx/ {print $4}' | cut -d "'" -f2 | head -n 1)" | awk '/url=/ {print $2}' | sed -e "s/english/international/" | cut -d '=' -f3 | cut -d '&' -f1)"Show Sample Output1wget "us.download.nvidia.com$(wget -qO- "$(wget -qO- "nvidia.com/Download/processFind.aspx?psid=95&pfid=695&osid=19&lid=1&lang=en-us"|awk '/driverResults.aspx/ {print $4}'|cut -d "'" -f2|head -n 1)"|awk '/url=/ {print $2}'|cut -d '=' -f3|cut -d '&' -f1)" -
Directly send the content of a url to standard out. This command is most convenient for sending the output of a download directly to another command. Show Sample Output
1alias echourl="wget -qO -" -
Like the original command, but the -f allows this one to succeed even if the website returns uncompressed data. From gzip(1) on the -f flag: If the input data is not in a format recognized by gzip, and if the --stdout is also given, copy the input data without change to the standard output: let zcat behave as cat.
1wget -q -O- --header="Accept-Encoding: gzip" <url> | gzip -cdf > out.html - < 1 2 3 4 > Last ›
What's this?
commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.
Check These Out
Stay in the loop…
- Follow the Tweets.
-
Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.
» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10 - Subscribe to the feeds.
-
Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):
Subscribe to the feed for:
- » all commands
- » commands with 3 up-votes (commandlinefu3)
- » commands with 10 up-votes (commandlinefu10)
- » commands tagged wget