check broken links using wget as a spider

wget --spider -nd -e robots=off -prb -o wget-log -nv http://example.com

This command crawls a domain with the typical WGET output. It logs every thing to a WGET-LOG file with any errors repeated at the end. It also had the added benefit of not flooding your terminal without ouput, so it is safe to run in the background.

By: anEffingChamp

2015-01-08 16:14:19

wget wget broken links spider

1 Alternatives + Submit Alt

check broken links using wget as a spider

This will visit recursively all linked urls starting from the specified URL. It won't save anything locally and it will produce a detailed log. Useful to find broken links in your site. It ignores robots.txt, so just use it on a site you own!
This is sample output - yours may be different.
0

wget --spider -o wget.log -e robots=off --wait 1 -r -p http://www.example.com

lele · 2011-04-05 13:42:14 5

What Others Think

where to get shrooms dried mushrooms african pyramid mushroom african transkei mushroom albino penis envy b+ mushrooms blue meanies mushroom brazilian magic mushroom burmese mushroom golden teacher mushroom ko samui super strain malabar coast martinique mexican mushroom mushroom british columbia super thai mushroom nepalese chitwan mushroom penis envy texas penis envy 6 texas yellow caps vietnamese white jedi mind fuck tri coloured ecuadorian mushroom magic mushsroom edibles mexican cubensis psychedelic chocolate bar where to get shrooms dried mushrooms african pyramid mushroom african transkei mushroom albino penis envy b+ mushrooms blue meanies mushroom brazilian magic mushroom burmese mushroom golden teacher mushroom ko samui super strain malabar coast martinique mexican mushroom mushroom british columbia super thai mushroom nepalese chitwan mushroom penis envy texas penis envy 6 texas yellow caps vietnamese white jedi mind fuck tri coloured ecuadorian mushroom magic mushsroom edibles mexican cubensis psychedelic chocolate bar midnight mint psychedelic chocolate bar origin mango psilo gummies 0.25g-1g origin strawberry psilo gummies 0.25g-1g penis envy psychedelic chocolate bar psilocybin cube gummies vegan dark chocolate psychedelic chocolate bar microdoses brain capsules 50mg-200mg chill capsules 50mg-200mg clarity capsules 50mg-200mg health capsules 50mg-200mg love capsules 50mg-200mg mood capsules 50mg-200mg pure capsules 50mg-200mg social capsules 30 pack 350mg social capsules 5 pack 350mg buy weed online without medical card marijuana flowers types of concentrates cannabis cartridges buy edibles online legal marijuana shatter cannabis wax marijuana hash buy psychedelic online tinctures for anxiety cannabis seeds bank weed smoker accessories birthday cake kush acdc strain ace killer og strain ak 47 strain alaskan thunder fuck alien kush strain ambrosia kush strain atomik moon rocks cbd 40.57% atomik moon rocks diamond 58.77% banana og strain bc big bud strain birthday cake kush blue cheese strain blue dream og kush blue moonrock kush blue rhino strain blueberry kush strain bruce banner strain cannabis trim cherry pie strain cannatonic strain citrix strain dark star kush durban poison strain eleven roses strain g13 haze strain gelato strain girl scout cookies flavors girl scout cookies online granddad purple strain grape nana weed strains grape pie strain green crack strain guava kush strain hawaiian skunk jack herer strain mendo breath weed mighty mango mr nice guy weed blue mystic strain northern lights strain northern lights moon rock platinum og kush gas og strain pink og kush platinum hawaiian strain pootie tang strain purple mimosa strain purple white widow strain smart buds sour diesel weed strain sour moonrock strain space monkey tins strawberry banana weed sun rocks super citrus haze strain super silver haze wedding cake strain exotic white widows strain zombie kush strain all natural pistachio crunch apple fritter vape cartridge aura edibles gummy-candy banana og cartridge birthday cake dank vape pen cartridg biscotti vape cartridge blackberry kush cartridge bliss edibles 300mg thc blue dream cartridge 180mlg berry blue raspberry 500mg-chocolate chip- cookies atomic wheelchair activated thc distillate strike gold blue dream shatter bubba kush shatter cbd dried organic ginger blueberry kush vape cartridges chemdawg shatter 50mg choco chunk cookies cookies and cream vape delta 8 thc capsules do-si-dos cartridges durban poison vape cartridge eds and bills edible candy bags forbidden fruit cartridge fruity pebbles cartridge gelato cartridge 1000mg girl scout cookies shatter girl scout cookies wax gorilla glue #4 wax gorilla glue cartridge gorilla glue shatter grand daddy purple kush wax gsc cartridge hardcore og cartridge jack herer cartridge jack herer wax jet fuel cartridge juicy melon cartridge khalifa kush shatter kosher kush cartridge labanese camel hash lemon berry cartridge

johnson20007 · 148 weeks and 1 day ago

Amazing! This blog looks just like my old one! It's on a completely different subject but it has pretty much the same layout and design. Wonderful choice of colors! Pug Puppies for Sale Near Me PUG PUPPY FOR SALE NEAR ME PUG PUPPIES FOR SALE pug puppies for sale in kentucky Pug Puppies for Sale Under $500 Near Me pug puppies for sale in texas pugs puppies for sale teacup pugs for sale pug puppies for sale by owner pug puppies ohio

rahimhh21 · 119 weeks and 4 days ago

AnthonyLewis9201 · 113 weeks and 5 days ago

Perfecthomepugs · 109 weeks and 6 days ago

Princesmith · 103 weeks and 1 day ago

pugpuppies95 · 71 weeks and 6 days ago

liltommy · 71 weeks and 6 days ago

donperi1 · 64 weeks and 6 days ago

What do you think?

Any thoughts on this command? Does it work on your machine? Can you do the same thing with only 14 characters?

You must be signed in to comment.

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands

Similar Commands

Dump HTTP header using wget

use wget to check if a remote file exists

Run remote web page, but don't save the results

Find out how old a web page is

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for:

» all commands
» commands with 3 up-votes (commandlinefu3)
» commands with 10 up-votes (commandlinefu10)