top 10 of access log

parallel can be installed on your central node and can be used to run a command multiple times.

In this example, multiple ssh connections are used to run commands. (-j is the number of jobs to run at the same time). The result can then be piped to commands to perform the "reduce" stage. (sort then uniq in this example).

This example assumes "keyless ssh login" has been set up between the central node and all machines in the cluster.

bashreduce may also do what you want.

make usable on OSX with filenames containing spaces. note: will still break if filenames contain newlines... possible, but who does that?!

Output contains also garbage (text parts from netstat's output) but it's good enough for quick check who's overloading your server.

Get the longest match of file extension (Ex. For 'foo.tar.gz', you get '.tar.gz' instead of '.gz')

If you have GNU findutils, you can get only the file name with

instead of

grabbed from Andrew Aylett post:

http://superuser.com/questions/133313/can-i-speed-up-cygwins-fork

Avoids the nested 'find' commands but doesn't seem to run any faster than syssyphus's solution.

I used to do this sorting with:

But this would cause the line (2nd column) to be sorted in descending (reverse) order as well sa the 1st column. So this will ensure the 2nd column is in ascending alphabetical order.

This is a modified version of the OP, wrapped into a bash function.

This version handles newlines and other whitespace correctly, the original has problems with the thankfully rare case of newlines in the file names.

It also allows checking an arbitrary number of directories against each other, which is nice when the directories that you think might have duplicates don't have a convenient common ancestor directory.

# find assumes email files start with a number 1-9

# sed joins the lines starting with " " to the previous line

# gawk print the received and from lines

# sort according to the second field (received+from)

# uniq print the duplicated filename

# a message is viewed as duplicate if it is received at the same time as another message, and from the same person.

The command was intended to be run under cron. If run in a terminal, mutt can be used:

mutt -e "push otD~=xq" -f $folder

count & sort one field of the log files , such as nginx/apache access log files .

#_connects src_IP dst_IP When_It_Happened_Secs

The following displays only the entries that are duplicates.

cut -f1,2 - IP range 16

cut -f1,2,3 - IP range 24

cut -f1,2,3,4 - IP range 24

Uses line-porcelain in git blame, which makes it easier to parse the output.

shows number of mysql bin log events (which are mysql server events) per minute, useful to check stress times postmortem

See the summary.

Per country GET report, based on access log. Easy to transform to unique IP