Commands tagged wget

alias speedtest='wget --output-document=/dev/null http://speedtest.wdc01.softlayer.com/downloads/test500.zip'

If you have to deal with MS Sharepoint which is (rarely, let's hope) used in e.g. certain corporate environments).

This uses Cntlm.

For single files, just use cURL -- its NTLM authentication works quite well.

# /etc/cntlm.conf:

# Username account

# Domain domain

# Password ############

# Proxy 10.20.30.40 (IP of the sharepoint site)

# NoProxy *

# Listen 3128

This command should be copy-pasted in Windows, but very similar one will work on Linux.

It uses wget and sed.

Tries to avoid the fragile nature of scrapers by looking for user-input in the output as opposed to markup or headers on the web site.

I is for headers only

s is for silence

curl -Is outputs ONLY headers the pipe and grep is to filter them to Modified only..

I used to use the Firefox "View page info" feature a lot to determine how stale the web page I was looking at was. Now that I use mostly Chrome I miss that feature, so here is a command line alternative using wget. The -S says to display the server response, the --spider says to not download any files/pages, just fetch the header. The output goes to stderr, so to grep it you use 2>&1 to combine the stderr stream with stdout, the pipe that to grep for Last-Modified.

You can use curl instead if you have it installed, like this:

A simple script for download all the MegaTokyo strips from the first to the last one

Gets the IP and sticks it into the middle-mouse-click buffer

This shell function uses wget(1) to show what site a shortened URL is pointing to, even if there are many nested shortened URLs. This is a great way to test whether or not the shortened URL is sending you to a malicious site, or somewhere nasty that you don't want to visit. The sample output is from:

No curl and no sed, just wget :)

Recursively download all files of a certain type down to two levels, ignoring directory structure and local duplicates.

Usage:

wgetall mp3 http://example.com/download/

This is just a "cut" addicted variant of the previous unixmonkey24730 command...

This uses wget instead of curl

useful to use after with the --load-cookies option of wget

- Where $URL is the URL of the file.

- Replace the $2 by $3 at the end to get a human-readable size.

Credits to svanberg @ ArchLinux forums for original idea.

Edit: Replaced command with better version by FRUiT. (removed unnecessary grep)

See man wget if you want linked files and not only those hosted on the website.