Check These Out
This one-liner will the *delete* without any further confirmation all 100% duplicates but one based on their md5 hash in the current directory tree (i.e including files in its subdirectories).
Good for cleaning up collections of mp3 files or pictures of your dog|cat|kids|wife being present in gazillion incarnations on hd.
md5sum can be substituted with sha1sum without problems.
The actual filename is not taken into account-just the hash is used.
Whatever sort thinks is the first filename is kept.
It is assumed that the filename does not contain 0x00.
As per the good suggestion in the first comment, this one does a hard link instead:
$ find . -xdev -type f -print0 | xargs -0 md5sum | sort | perl -ne 'chomp; $ph=$h; ($h,$f)=split(/\s+/,$_,2); if ($h ne $ph) { $k = $f; } else { unlink($f); link($k, $f); }'
Extracts the model name of the CPU and displays it on screen.
I used this command (in addition to a code formatting tool) to "cleanup" a bunch of PHP files
You might want to secure your AWS operations requiring to use a MFA token. But then to use API or tools, you need to pass credentials generated with a MFA token.
This commands asks you for the MFA code and retrieves these credentials using AWS Cli. To print the exports, you can use:
`awk '{ print "export AWS_ACCESS_KEY_ID=\"" $1 "\"\n" "export AWS_SECRET_ACCESS_KEY=\"" $2 "\"\n" "export AWS_SESSION_TOKEN=\"" $3 "\"" }'`
You must adapt the command line to include:
* $MFA_IDis ARN of the virtual MFA or serial number of the physical one
* TTL for the credentials
The command (above) will remove any duplicate rows based on the FIRST column of data in an un-sorted file.
The '$1' represents a positional parameter. You can change both instances of '$1' in the command to remove duplicates based on a different column, for instance, the third:
$ awk '{ if ($3 in stored_lines) x=1; else print; stored_lines[$3]=1 }' infile.txt > outfile.txt
Or you can change it to '$0' to base the removal on the whole row:
$ awk '{ if ($0 in stored_lines) x=1; else print; stored_lines[$0]=1 }' infile.txt > outfile.txt
** Note: I wouldn't use this on a MASSIVE file, unless you're RAM-rich ;) **
The command will read the apache log file and fetch the virtual host requested and the number of requests.
Trace python statement execution and syscalls invoked during that simultaneously
Uses history to get the last n+1 commands (since this command will appear as the most recent), then strips out the line number and this command using sed, and appends the commands to a file.
