Commands using wget

the good:

Server: Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.4 with Suhosin-Patch

the bad:

Server: Microsoft-IIS/6.0

and the ugly:

Server: Apache/2.2.10 (Win32) mod_ssl/2.2.10 OpenSSL/0.9.8i PHP/5.2.6

This download a complete audio podcast

This is a quick line to stream in the latest offerings of your favorite netcasts/podcasts. You will need to have a file named netcast.txt in the directory you run this from. This file should have one and only one of your netcast's/podcst's url per line.

When run the line grabs the offering on the top of the netcast/podcast stack and end it over , quietly, to vlc.

Since I move around computers during the day I wanted an easy way to listen to my daily dose of news and such without having to worry about downloading to whatever machine I am on. This is just a quick grab and stream of whats current.

Future plans... have the list of netcasts be read from the web. possibly an rss or such. I use greader so there might be a way to use it as the source so as not to have to muck with multiple lists

This was gotten from http://www.veen.com/jeff/archives/000573.html. The line will grab all the mp3 files on the urls listed in text file sourceurls.txt (one url per line) . A much more complete breakdown of the line can be found at the web site mentioned above.

Blacklisted is a compiled list of all known dirty hosts (botnets, spammers, bruteforcers, etc.) which is updated on an hourly basis. This command will get the list and create the rules for you, if you want them automatically blocked, append |sh to the end of the command line. It's a more practical solution to block all and allow in specifics however, there are many who don't or can't do this which is where this script will come in handy. For those using ipfw, a quick fix would be {print "add deny ip from "$1" to any}. Posted in the sample output are the top two entries. Be advised the blacklisted file itself filters out RFC1918 addresses (10.x.x.x, 172.16-31.x.x, 192.168.x.x) however, it is advisable you check/parse the list before you implement the rules

I've got this posted in one of my .bash_profiles for humor whenever I log in.

-p parameter tells wget to include all files, including images.

-e robots=off you don't want wget to obey by the robots.txt file

-U mozilla as your browsers identity.

--random-wait to let wget chose a random number of seconds to wait, avoid get into black list.

Other Useful wget Parameters:

--limit-rate=20k limits the rate at which it downloads files.

-b continues wget after logging out.

-o $HOME/wget_log.txt logs the output

Very nice command when you want to download a site locally to your machine, including images, css and javascript

The example will create a directory called "

Caveats: @imports of css files will not be converted.

Seq allows you to define printf like formating by specified with -f, %03g is actually tells seq I got three digits, fill the blank digits with 0, and the range is from 176 to 240.

prompts for a search term and then pulls down the first result from google images

Self-referential use of wget.

This spiders the given site without downloading the HTML content. The resulting directory structure is then parsed to output a list of the URLs to url-list.txt. Note that this can take a long time to run and you make want to throttle the spidering so as to play nicely.

This recursively downloads all images from a given website to your /tmp directory. The -nH and -nd switches disable downloading of the directory structure.