Retrieve a list of all webpages on a site

URL=www.example.com && wget -rq --spider --force-html "http://$URL" && find $URL -type d > url-list.txt && rm -rf $URL
This spiders the given site without downloading the HTML content. The resulting directory structure is then parsed to output a list of the URLs to url-list.txt. Note that this can take a long time to run and you make want to throttle the spidering so as to play nicely.

1
By: root
2009-01-27 17:59:08

What Others Think

house cleaning olympia wa you are soo inspiring! thanks for this!
yeoyeo · 36 weeks and 6 days ago
I have read your article couple of times because your views are on my own for the most part. It is great content for every reader. ??
KayGasaway · 21 weeks ago
Great job for publishing such a beneficial web site. Your web log isn’t only useful but it is additionally really creative too. There tend to be not many people who can certainly write not so simple posts that artistically. Continue the nice writing ???????
KayGasaway · 20 weeks and 1 day ago
Great job for publishing such a beneficial web site. Your web log isn’t only useful but it is additionally really creative too. There tend to be not many people who can certainly write not so simple posts that artistically. Continue the nice writing ???????
KayGasaway · 20 weeks and 1 day ago
Particular interviews furnish firsthand message on mart size, industry trends, ontogeny trends, capitalist landscape and outlook, etc. Colin Farell Tracksuit
KayGasaway · 19 weeks and 5 days ago
I love visiting sites in my free time. I have visited many sites but did not find any site more efficient than yours. Thanks for the nudge! ??
KayGasaway · 19 weeks and 3 days ago
I know your aptitude on this. I should say we ought to have an online discourse on this. Composing just remarks will close the talk straight away! What's more, will confine the advantages from this data. safestore auto
KayGasaway · 19 weeks and 3 days ago
No doubt this is an excellent post I got a lot of knowledge after reading good luck. Theme of blog is excellent there is almost everything to read, Brilliant post. safestore auto
KayGasaway · 19 weeks and 1 day ago
You there, this is really good post here. Thanks for taking the time to post such valuable information. Quality content is what always gets the visitors coming. ??? 1?
KayGasaway · 19 weeks ago
This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information. Keep it up. Keep blogging. Looking to reading your next post. Legal Writing Workshops & Programs
KayGasaway · 18 weeks and 4 days ago
You there, this is really good post here. Thanks for taking the time to post such valuable information. Quality content is what always gets the visitors coming. http://www.fpoc.co.uk/walmartone-wire-login-guide-to-the-walmart-gift-card/">walmart one
KayGasaway · 18 weeks and 1 day ago
It turns out that even the hottest port has a few places where you can get off the beaten path. Here are some recommendations that will make you feel like you're in the know web design
KayGasaway · 18 weeks ago
I was surfing net and fortunately came across this site and found very interesting stuff here. Its really fun to read. I enjoyed a lot. Thanks for sharing this wonderful information. best chinese PCB manufacturer
KayGasaway · 17 weeks and 5 days ago
I'm constantly searching on the internet for posts that will help me. Too much is clearly to learn about this. I believe you created good quality items in Functions also. Keep working, congrats! Onewheel Accessories
KayGasaway · 17 weeks and 4 days ago
Buy healing crystal bracelets for chakra balancing, reiki energy healing, meditation, yoga, stress, anxiety depression Personalized to your healing needs and intentions Crystal bracelets to attract abundance, luck, money, love, remove negativity, enhance psychic power, increase willpower, motivation, and more! healing crystal bracelets
KayGasaway · 16 weeks and 6 days ago
This article is an engaging abundance of enlightening information that is intriguing and elegantly composed. I praise your diligent work on this and thank you for this data. You have what it takes to get consideration. Onewheel Accessory
KayGasaway · 16 weeks and 5 days ago
This is my first time i visit here. I found such a substantial number of interesting stuff in your blog especially its examination. Really its inconceivable article. Keep it up. Managed IT Support companies Century City Los Angeles, California
KayGasaway · 16 weeks ago
Buy Pharmacy grade drugs to boost stamina and enhance performance. Buy HGH, mens sexual health products, weight loss remedy, EPO and Clenbuterol Buy Online. sibutramine buy
KayGasaway · 15 weeks and 5 days ago
Meetursugar makes dating easy and fun for everyone. With our first date tool, our members discover high-quality, frequent first dates. We ensure that all people active on our site are serious about dating. http://www.meetursugar.com/">Dating in New York
KayGasaway · 15 weeks and 4 days ago
This is such a great resource that you are providing and you give it away for free. I love seeing blog that understand the value of providing a quality resource for free. https://reduslim.international/ro/romania/
jackson2321 · 14 weeks and 1 day ago
This is such a great resource that you are providing and you give it away for free. I love seeing blog that understand the value of providing a quality resource for free. http://thepurpletreeshop.com/">http://thepurpletreeshop.com/
jackson2321 · 14 weeks and 1 day ago
HOVERBOARD UK with 1 year warranty and FREE next day delivery Made from British Design Buy genuine Hoverboards for sale online at HOVERBOARD PRO HOVERBOARD
ayesha · 11 weeks and 2 days ago
Lavington Designs - Visit our website for Premium & High Quality Personalized Gifts for Adults and Children, Quality Personalized Clothing, Totes, Travel Bags and Kids Gifts - Get the latest updates on new products and upcoming sales. Personalized Gifts
KayGasaway · 11 weeks ago
I want to say thanks for beautiful blog sharing with us. Your blog really great resource to update my knowledge. eye doctor Collinsville
KayGasaway · 10 weeks and 4 days ago
Winslot Casino : Slot Online Uang Asli Populer Terpercaya 167.172.7.163 ?, Winslot merupakan situs judi online dengan winrate tinggi yang menyediakan permainan judi slot online uang asli paling populer di Indonesia.
ayesha · 8 weeks and 4 days ago
Find the Maldives Holidays Package & right Luxury hotel for your honeymoon or Holidays Rawnaq Tourism recommends the Best Maldives Hotels depending on your individual needs
KayGasaway · 6 weeks ago
Free Free Slots No Deposit Sites 2021 ??are the most popular choices online all the time,http://www.krov.fm/broadcasts/415">Free Slots No Deposit Sites 2021 Online slots are one of the ways that players enjoy their place to play at the most.
jackson2321 · 3 weeks and 1 day ago
The #1 trusted Polaris UTV by land owners, farmers, and hunters for over a decade. Polaris Ranger For Sale UK 0 Finance Available Full range available at Beckside Machinery Official Polaris Dealers UK. Beckside Machinery Ltd, registered as a limited company in England and Wales, Please visit our Website for more details. Thank You.
ayesha · 3 days ago

What do you think?

Any thoughts on this command? Does it work on your machine? Can you do the same thing with only 14 characters?

You must be signed in to comment.

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands



Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: