You must have smartmontools installed for this to work. This also assumes you 512 byte sector sizes, this is pretty standard.

Finds duplicates based on MD5 sum. Compares only files with the same size. Performance improvements on:

The new version takes around 3 seconds where the old version took around 17 minutes. The bottle neck in the old command was the second find. It searches for the files with the specified file size. The new version keeps the file path and size from the beginning.

Mirror a remote directory using some tricks to maximize network speed.

lftp:: coolest file transfer tool ever

-u: username and password (pwd is merely a placeholder if you have ~/.ssh/id_rsa)

-e: execute internal lftp commands

set sftp:connect-program: use some specific command instead of plain ssh

ssh::

-a -x -T: disable useless things

-c arcfour: use the most efficient cipher specification

-o Compression=no: disable compression to save CPU

mirror: copy remote dir subtree to local dir

-v: be verbose (cool progress bar and speed meter, one for each file in parallel)

-c: continue interrupted file transfers if possible

--loop: repeat mirror until no differences found

--use-pget-n=3: transfer each file with 3 independent parallel TCP connections

-P 2: transfer 2 files in parallel (totalling 6 TCP connections)

sftp://remotehost:22: use sftp protocol on port 22 (you can give any other port if appropriate)

You can play with values for --use-pget-n and/or -P to achieve maximum speed depending on the particular network.

If the files are compressible removing "-o Compression=n" can be beneficial.

Better create an alias for the command.

I'm sure there's a more elegant sed version for the tr + grep section.

Download video files from a bunch of sites (here is a list https://rg3.github.io/youtube-dl/supportedsites.html).

The options say: base filename on title, ignores errors and continue partial downloads. Also, stores some metadata into a .json file plz.

Paste youtube users and playlists for extra fun.

Protip: git-annex loves these files

Google Cloud SDK comes with a package manager `gcloud components` but it needs a bit of `sed` to work. Modify the "^| Not" bit to change the package selection. (The gcloud --format option is currently broken)

By this command you can take the snapshot of you harddisk (full) and create the image , the image will be directly store on remote server through ssh. Here i am creating the image of /dev/hda and saving it at 4.2.2.2 as /root/server.img.

The command will read the apache log file and fetch the virtual host requested and the number of requests.

This sed command will search for 4.2.2.2 in all lines of test.txt and replace comment symbol ";" . You can use it for other purpose also.

Replace the credentials to psql if necessary, and the my-query part with your query.

Sure, it's dirty, but it's quick, it only displays the excuse, and it works.

Converts all monkey audio files below currently directory to FLAC.

For only current directory, use `for f in *.ape; do avconv -i "$f" "${f%.ape}.flac"; done`

To remove APE files afterward, use `rm */*.ape`

This command will use grep to read the shortcut (which in the above examle is file.url), and filter out all but the only important line, which contains the website URL, and some extra characters that will need to be removes (for example, URL=http://example.com). The cut command is then used to get rid of the URL= at the beginning. The output is then piped into Firefox, which should interpret the it as a web URL to be opened. Of course, you can replace Firefox with any other broswer. Tested in bash and sh.

Using egrep to search multiple patterns.

This alias is useful if you need to use some text often. Executing the alias will copy the text into your clipboard and then remove it after X seconds.

Works in Gnome