Commands tagged filter (11)

  • Working with log files that contains variable length messages wrapped between open and close tags it may be useful to filter the messages upon a keyword. This works fine with GNU sed version 4.2 or higher, so pay attention to some unix distros (solaris, hp-ux, etc.). Linux should be ok. Show Sample Output

    cat file.txt | sed -e /<opening tag>/d -e /<closing tag>/G | sed -e '/./{H;$!d;}' -e 'x;/<string to search>/!d;'
    EBAH · 2010-11-04 10:31:15 0
  • Thanks to knoppix5 for the idea :-) Print selected lines from a file or the output of a command. Usage: every NTH MAX [FILE] Print every NTH line (from the first MAX lines) of FILE. If FILE is omitted, stdin is used. The command simply passes the input to a sed script: sed -n -e "${2}q" -e "0~${1}p" ${3:-/dev/stdin} print no output sed -n quit after this many lines (controlled by the second parameter) -e "${2}q" print every NTH line (controlled by the first parameter) -e "0~${1}p" take input from $3 (if it exists) otherwise use /dev/stdin {3:-/dev/stdin} Show Sample Output

    function every() { sed -n -e "${2}q" -e "0~${1}p" ${3:-/dev/stdin}; }
    flatcap · 2015-04-03 01:30:36 4
  • This line does not include your closing tag in the output. Show Sample Output

    sed '/'"<opening tag>"'/,/'"<closing tag>"'/{/'"<closing tag>"'/d;p};d' "<file>"
    DaveQB · 2010-11-08 21:43:00 0
  • Show's per IP of how many requests they did to the Apache webserver

    cat /var/log/apache2/access_logs | cut -d ' ' -f 1 | uniq -c | sort -n
    basvdburg · 2013-09-02 13:04:47 1
  • The first sort is necessary for ips in a list to be actually unique.

    cat /var/log/apache2/access_logs | cut -d' ' -f1 | sort | uniq -c | sort -n
    while0pass · 2013-09-07 23:57:31 0
  • Sometimes commands give you too much feedback. Perhaps 1/100th might be enough. If so, every() is for you. my_verbose_command | every 100 will print every 100th line of output. Specifically, it will print lines 100, 200, 300, etc If you use a negative argument it will print the *first* of a block, my_verbose_command | every -100 It will print lines 1, 101, 201, 301, etc The function wraps up this useful sed snippet: ... | sed -n '0~100p' don't print anything by default sed -n starting at line 0, then every hundred lines ( ~100 ) print. '0~100p' There's also some bash magic to test if the number is negative: we want character 0, length 1, of variable N. ${N:0:1} If it *is* negative, strip off the first character ${N:1} is character 1 onwards (second actual character). Show Sample Output

    function every() { N=$1; S=1; [ "${N:0:1}" = '-' ] && N="${N:1}" || S=0; sed -n "$S~${N}p"; }
    flatcap · 2015-03-21 23:44:59 4
  • Applying filter rules is what makes this a really useful command. It's usually a pain to figure out how to sync ONLY files matching a particular pattern, and often one reverts to goofy stuff like find .. -exec rsync .. The filter hides all folders from the transfer, so that only the matching folders that store the filename are left for the sync.

    rsync -avz --dry-run --include="only-include-this-filename" -f 'hide,! */' source/folder/ target/folder/
    cybertoast · 2011-03-16 16:10:42 0
  • Commandline perl filter for, using a production.log from a rails app, display on realtime the count of requests grouped by "seconds to complete" (gross round, but fair enough for an oneliner) :) Show Sample Output

    tail -f production.log | perl -ne 'if (/^\d+)/){$d = int($1/1000);print "\n";$f{$d}++;for $t (sort(keys(%f))){print $t."s: ".$f{$t}."\n"}}'
    theist · 2012-02-23 14:37:33 0
  • avoiding UUOC! cut can handle files as well. No neet for a cat.

    cut -d ' ' -f 1 /var/log/apache2/access_logs | uniq -c | sort -n
    BorneBjoern · 2013-09-17 20:05:03 0
  • by default, will output the whole line on which 'word' has been found

    grep word file.txt
    lolssl · 2015-10-02 16:01:56 0
  • Removes all lines between the lines containing "" and "", including these lines itself Backdrop: Sometimes when working with XML files without an graphical editor, large comment-/annotation-blocks taper the readability to walk through the file. I like to create a copy of such documents without these annotations. As the documentation itself is in documentation tags inside the annotation tags an therefore graphical editors tend to put the annotation tags in their own lines, this command removes all documentations within annotation-tags. Show Sample Output

    awk "/<xsd:annotation>/{h=1};!h;/<\/xsd:annotation>/{h=0}" annotatedSchema.xsd
    2chg · 2011-07-15 07:17:17 1

What's this? is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands

Check These Out

Find usb device
I often use it to find recently added ou removed device, or using find in /dev, or anything similar. Just run the command, plug the device, and wait to see him and only him rss parser
Quick and kludgy rss parser for the recent tracks rss feed from Extracts artist and track link.

Randomize lines (opposite of | sort)
random(6) - random lines from a file or random numbers

Database size
Mysql command to list the disk usage of the database

Spy audio, it only records if detects a sound or noise
It find out the mic recording level at the moment of run the command and if a noise level is higher it starts to record an mp3 file. The resulting file will have only the sounds not the silences.

Which processes are listening on a specific port (e.g. port 80)
swap out "80" for your port of interest. Can use port number or named ports e.g. "http"

Show me a histogram of the busiest minutes in a log file:
Busiest seconds: $ cat /var/log/secure.log | awk '{print substr($0,0,15)}' | uniq -c | sort -nr | awk '{printf("\n%s ",$0) ; for (i = 0; i

Convert file type to unix utf-8
converts encoding of a file to unix utf-8 useful for data files that contain what would be usable ascii text but are encoded as mpeg or some other encoding that prevents you from doing common manipulations like 'sed'

Backup all mysql databases to individual files on a remote server
It grabs all the database names granted for the $MYSQLUSER and gzip them to a remote host via SSH.

Change wallpaper

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.


Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: