Commands by dopeman (11)

  • The command (above) will remove any duplicate rows based on the FIRST column of data in an un-sorted file. The '$1' represents a positional parameter. You can change both instances of '$1' in the command to remove duplicates based on a different column, for instance, the third: awk '{ if ($3 in stored_lines) x=1; else print; stored_lines[$3]=1 }' infile.txt > outfile.txt Or you can change it to '$0' to base the removal on the whole row: awk '{ if ($0 in stored_lines) x=1; else print; stored_lines[$0]=1 }' infile.txt > outfile.txt ** Note: I wouldn't use this on a MASSIVE file, unless you're RAM-rich ;) **


    4
    awk '{ if ($1 in stored_lines) x=1; else print; stored_lines[$1]=1 }' infile.txt > outfile.txt
    dopeman · 2010-12-15 17:08:47 3
  • buf myfile.txt This is useful when you are making small but frequent changes to a file. It keeps things organised and clear for another administrator to see what changed and at what time. An overview of changes can be deduced using a simple: ls -ltr


    1
    buf () { filename=$1; filetime=$(date +%Y%m%d_%H%M%S); cp ${filename} ${filename}_${filetime}; }
    dopeman · 2010-12-14 13:19:52 5
  • Essentially the same as funky's alias, but will not traverse filesystems and has nicer formatting. Show Sample Output


    -1
    alias dush="du -xsm * | sort -n | awk '{ printf(\"%4s MB ./\",\$1) ; for (i=1;i<=NF;i++) { if (i>1) printf(\"%s \",\$i) } ; printf(\"\n\") }' | tail"
    dopeman · 2010-07-15 10:38:27 3
  • I have come across a situation in the past where someone has unlinked a file by running an 'rm' command against it while it was still being written to by a running process. The problem manifested itself when a 'df' command showed a filesystem at 100%, but this did not match the total value of a 'du -sk *'. When this happens, the process continues to write to the file but you can no longer see the file on the filesystem. Stopping and starting the process will, more often than not, get rid of the unlinked file, however this is not always possible on a live server. When you are in this situation you can use the 'lsof' command above to get the PID of the process that owns the file (in the sample output this is 23521). Run the following command to see a sym-link to the file (marked as deleted): cd /proc/23521/fd && ls -l Truncate the sym-link to regain your disk space: > /proc/23521/fd/3 I should point out that this is pretty brutal and *could* potentially destabilise your system depending on what process the file belongs to that you are truncating. Show Sample Output


    16
    lsof +L1
    dopeman · 2010-07-14 17:21:01 4
  • This is a handy way to circumvent the "Maximum line length of 2048 exceeded" grep error. Once you have run the above command (or put it in your .bashrc), files can be searched using: lgrep search-string /file/to/search


    1
    lgrep() { string=$1; file=$2; awk -v String=${string} '$0 ~ String' ${file}; }
    dopeman · 2010-01-19 09:42:19 2
  • This does the same thing as many of the 'grep' based alternatives but allows a more finite control over the output. For example if you only wanted the process ID you could change the command: ps -ef | awk '/mingetty/ && !/awk/ {print $2}' If you wanted to kill the returned PID's: ps -ef | awk '/mingetty/ && !/awk/ {print $2}' | xargs -i kill {} Show Sample Output


    1
    ps -ef | awk '/process-name/ && !/awk/ {print}'
    dopeman · 2009-08-19 11:22:09 2
  • This command will copy files and directories from a remote machine to the local one. Ensure you are in the local directory you want to populate with the remote files before running the command. To copy a directory and it's contents, you could: ssh user@host "(cd /path/to/a/directory ; tar cvf - ./targetdir)" | tar xvf - This is especially useful on *nix'es that don't have 'scp' installed by default.


    1
    ssh user@host "(cd /path/to/remote/top/dir ; tar cvf - ./*)" | tar xvf -
    dopeman · 2009-03-31 13:08:45 6
  • This command will show the 20 processes using the most CPU time (hungriest at the bottom). You can see the 20 most memory intensive processes (hungriest at the bottom) by running: ps aux | sort +3n | tail -20 Or, run both: echo "CPU:" && ps aux | sort +2n | tail -20 && echo "Memory:" && ps aux | sort +3n | tail -20


    3
    ps aux | sort +2n | tail -20
    dopeman · 2009-03-31 12:03:34 3
  • This command will tell you the 20 biggest directories starting from your working directory and skips directories on other filesystems. Useful for resolving disk space issues.


    7
    du -xk | sort -n | tail -20
    dopeman · 2009-03-30 11:37:43 6
  • This command will replace all instances of 'foo' with 'bar' in all files in the current working directory and any sub-directories.


    -1
    perl -pi -e 's/foo/bar/g' $(grep -rl foo ./*)
    dopeman · 2009-03-27 17:21:35 8
  • This command will replace all instances of 'foo' with 'bar' in all files in the current working directory.


    -1
    perl -pi -e 's/foo/bar/g' $(grep -l foo ./*)
    dopeman · 2009-03-27 17:18:08 4

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands


Check These Out

prints the parameter you used on the previous command

An alias to select a portion of your desktop and save it as an image.

Find usb device in realtime
Using this command you can track a moment when usb device was attached.

put all lines in comment where de word DEBUG is found

Make a playlistfile for mpg321 or other CLI player
A short variant if you have only one directory whit only audio files in it.

sends your internal IP by email
This is useful if you have need to do port forwarding and your router doesn't assign static IPs, you can add it to a script in a cron job that checks if you IP as recently changed or with a trigger script. This was tested on Mac OSX.

List the Sizes of Folders and Directories

floating point operations in shell scripts
allows you to use floating point operations in shell scripts

Repeat a command until stopped
In this case it runs the command 'curl localhost:3000/site/sha' waiting the amount of time in sleep, ie: 1 second between runs, appending each run to the console. This works well for any command where the output is less than your line width This is unlike watch, because watch always clears the display.

Run TOP in Color, split 4 ways for x seconds - the ultimate ps command. Great for init scripts
One of my favorite ways to impress newbies (and old hats) to the power of the shell, is to give them an incredibly colorful and amazing version of the top command that runs once upon login, just like running fortune on login. It's pretty sweet believe me, just add this one-liner to your ~/.bash_profile -- and of course you can set the height to be anything, from 1 line to 1000! $ G=$(stty -g);stty rows $((${LINES:-50}/2));top -n1; stty $G;unset G Doesn't take more than the below toprc file I've added below, and you get all 4 top windows showing output at the same time.. each with a different color scheme, and each showing different info. Each window would normally take up 1/4th of your screen when run like that - TOP is designed as a full screen program. But here's where you might learn something new today on this great site.. By using the stty command to change the terminals internal understanding of the size of your terminal window, you force top to also think that way as well. # save the correct settings to G var. $ G=$(stty -g) # change the number of rows to half the actual amount, or 50 otherwise $ stty rows $((${LINES:-50}/2)) # run top non-interactively for 1 second, the output stays on the screen (half at least) $ top -n1 # reset the terminal back to the correct values, and clean up after yourself $ stty $G;unset G This trick from my [ http://www.askapache.com/linux-unix/bash_profile-functions-advanced-shell.html bash_profile ], though the online version will be updated soon. Just think what else you could run like this! Note 1: I had to edit the toprc file out due to this site can't handle that (uploads/including code). So you can grab it from [ http://www.askapache.com/linux-unix/bash-power-prompt.html my site ] Note 2: I had to come back and edit again because the links weren't being correctly parsed


Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: