Commands by unixmonkey8046

Commands by unixmonkey8046 (13)

sorted by

List files accessed by a command

Can be run as a script `ftrace` if my_command is substrituted with "$@" It is useful when running a command that fails and you have the feeling it is accessing a file you are not aware of. Show Sample Output
This is sample output - yours may be different.
```
$ strace -ff -e trace=file chmod 755 ~/bin/ftrace 2>&1 | perl -ne 's/^[^"]+"(([^\\"]|\\[\\"nt])*)".*/$1/ && print'
/bin/chmod
/etc/ld.so.nohwcap
/etc/ld.so.preload
/etc/ld.so.cache
/etc/ld.so.nohwcap
/lib/x86_64-linux-gnu/libc.so.6
/usr/lib/locale/locale-archive
/home/tange/bin/ftrace
/home/tange/bin/ftrace
```
17

strace -ff -e trace=file my_command 2>&1 | perl -ne 's/^[^"]+"(([^\\"]|\\[\\"nt])*)".*/$1/ && print'

unixmonkey8046 · 2011-08-16 15:00:18 10
Compute the average number of KB per file for each dir

Use this to find identify if dirs mostly contain large or small files. Show Sample Output
This is sample output - yours may be different.
```
yeslab.org 300.32786885245901639344
theyesmen.org 679.16801687505732016752
```
2

parallel echo -n {}"\ "\;echo '$(du -s {} | awk "{print \$1}") / $(find {} | wc -l)' \| bc -l ::: *

unixmonkey8046 · 2011-07-28 12:21:34 5
Use acpi and notify-send to report current temperature every five minutes.

Use acpi and notify-send to report current temperature every five minutes. Works best in a shell script run at startup. acpi is called for temperature and fed to notify-send for a tooltip. After waiting five minutes, it will start over.
This is sample output - yours may be different.
1

while notify-send "`acpi -t`"; do sleep 300; done

unixmonkey8046 · 2011-07-05 18:22:06 4
grep (or anything else) many files with multiprocessor power

Parallel does not suffer from the risk of mixing of output that xargs suffers from. -j+0 will run as many jobs in parallel as you have cores. With parallel you only need -0 (and -print0) if your filenames contain a '\n'. Parallel is from https://savannah.nongnu.org/projects/parallel/
This is sample output - yours may be different.
5

find . -type f | parallel -j+0 grep -i foobar

unixmonkey8046 · 2010-01-30 02:08:46 18
List top ten files/directories sorted by size

If a directory name contains space xargs will do the wrong thing. Parallel https://savannah.nongnu.org/projects/parallel/ deals better with that. Show Sample Output
This is sample output - yours may be different.
```
33M     a b
17M     libxnee
3.6M    doc
3.0M    gnee
```
-2

du -s * | sort -nr | head | cut -f2 | parallel -k du -sh

unixmonkey8046 · 2010-01-28 12:59:14 7
backup and remove files with access time older than 5 days.

This deals nicely with files having special characters in the file name (space ' or "). Parallel is from https://savannah.nongnu.org/projects/parallel/
This is sample output - yours may be different.
-3

tar -zcvpf backup_`date +"%Y%m%d_%H%M%S"`.tar.gz `find <target> -atime +5 -type f` 2> /dev/null | parallel -X rm -f

unixmonkey8046 · 2010-01-28 12:41:41 4
Delete all but latest file in a directory

xargs deals badly with special characters (such as space, ' and "). To see the problem try this: touch important_file touch 'not important_file' ls not* | xargs rm Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.
This is sample output - yours may be different.
-1

ls -t1 | sed 1d | parallel -X rm

unixmonkey8046 · 2010-01-28 12:28:18 4
archive all files containing local changes (svn)

xargs deals badly with special characters (such as space, ' and "). In this case if you have a file called '12" record'. Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem. Both solutions work bad if the number of files is more than the allowed line length of the shell.
This is sample output - yours may be different.
-2

svn st | cut -c 9- | parallel -X tar -czvf ../backup.tgz

unixmonkey8046 · 2010-01-28 11:43:16 4
add all files not under version control to repository

xargs deals badly with special characters (such as space, ' and "). To see the problem try this: touch important_file touch 'not important_file' ls not* | xargs rm Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.
This is sample output - yours may be different.
-2

svn status |grep '\?' |awk '{print $2}'| parallel -Xj1 svn add

unixmonkey8046 · 2010-01-28 08:47:54 4
recursive search and replace old with new string, inside files

xargs deals badly with special characters (such as space, ' and "). To see the problem try this: touch important_file touch 'not important_file' ls not* | xargs rm Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.
This is sample output - yours may be different.
4

grep -rl oldstring . | parallel sed -i -e 's/oldstring/newstring/'

unixmonkey8046 · 2010-01-28 08:44:16 9
Find Duplicate Files (based on size first, then MD5 hash)

A bit shorter and parallelized. Depending on the speed of your cpu and your disk this may run faster. Parallel is from https://savannah.nongnu.org/projects/parallel/
This is sample output - yours may be different.
0

find -not -empty -type f -printf "%s\n" | sort | uniq -d | parallel find -type f -size {}c | parallel md5sum | sort | uniq -w32 --all-repeated=separate

unixmonkey8046 · 2010-01-28 08:40:18 4
Remove all files previously extracted from a tar(.gz) file.

xargs deals badly with special characters (such as space, ' and "). To see the problem try this: touch important_file touch 'not important_file' ls not* | xargs rm Parallel https://savannah.nongnu.org/projects/parallel/ does not have this problem.
This is sample output - yours may be different.
-2

tar -tf <file.tar.gz> | parallel rm

unixmonkey8046 · 2010-01-28 08:28:16 4
run command on a group of nodes in parallel

Parallel is from https://savannah.nongnu.org/projects/parallel/ Other examples would be: (echo foss.org.my; echo www.debian.org; echo www.freenetproject.org) | parallel traceroute seq -f %04g 0 9999 | parallel -X rm pict{}.jpg
This is sample output - yours may be different.
3

seq 1 5 | parallel ssh {}.cluster.net uptime

unixmonkey8046 · 2010-01-28 08:18:50 7

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands

Check These Out

Find files and calculate size of result in shell

Use find's internal stat to get the file size then let the shell add up the numbers.

Remove old kernel packages

Remove annoying files from recently extracted zip archive

Inspired by http://www.commandlinefu.com/commands/view/2573/remove-all-files-previously-extracted-from-a-tar.gz-file. .... yet for zip files

tar the current directory wihtout the absolute path

tars the current directory (and its children) in an archive of the same name (plus ".tar" :)) in the parent directory without the absolute path, so that when the archive is extracted, only the current directory name is created for the path. Assumes bash/zsh.

urldecoding with one pure BASH builtin

You can use ordinary printf to convert "%23%21%2fbin%2fbash" into "#!/bin/bash" with no external utilities, by using a little known printf feature -- the "%b" specifier converts shell escapes. Replace % with \x and printf will understand the urlencoded string. BASH's printf has an extension to set a variable directly, too. So you get to convert urlencoded strings from garble to plaintext in one step with no externals and no backticks.

Advanced python tracing

Trace python statement execution and syscalls invoked during that simultaneously

Get AWS temporary credentials ready to export based on a MFA virtual appliance

You might want to secure your AWS operations requiring to use a MFA token. But then to use API or tools, you need to pass credentials generated with a MFA token. This commands asks you for the MFA code and retrieves these credentials using AWS Cli. To print the exports, you can use: `awk '{ print "export AWS_ACCESS_KEY_ID=\"" $1 "\"\n" "export AWS_SECRET_ACCESS_KEY=\"" $2 "\"\n" "export AWS_SESSION_TOKEN=\"" $3 "\"" }'` You must adapt the command line to include: * $MFA_IDis ARN of the virtual MFA or serial number of the physical one * TTL for the credentials

Printout a list of field numbers (awk index) from a CSV file with headers as first line.

Useful to identify the field number in big CSV files with large number of fields. The index is the reference to use in processing with commands like 'cut' or 'awk' involved.

Get the IP address of a machine. Just the IP, no junk.

Why use many different utilities all piped together, when you only need two?

Download and install the newest dropbox beta

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for:

» all commands
» commands with 3 up-votes (commandlinefu3)
» commands with 10 up-votes (commandlinefu10)
» commands by unixmonkey8046