Commands tagged filter (11)

  • Working with log files that contains variable length messages wrapped between open and close tags it may be useful to filter the messages upon a keyword. This works fine with GNU sed version 4.2 or higher, so pay attention to some unix distros (solaris, hp-ux, etc.). Linux should be ok. Show Sample Output


    2
    cat file.txt | sed -e /<opening tag>/d -e /<closing tag>/G | sed -e '/./{H;$!d;}' -e 'x;/<string to search>/!d;'
    EBAH · 2010-11-04 10:31:15 0
  • Thanks to knoppix5 for the idea :-) Print selected lines from a file or the output of a command. Usage: every NTH MAX [FILE] Print every NTH line (from the first MAX lines) of FILE. If FILE is omitted, stdin is used. The command simply passes the input to a sed script: sed -n -e "${2}q" -e "0~${1}p" ${3:-/dev/stdin} print no output sed -n quit after this many lines (controlled by the second parameter) -e "${2}q" print every NTH line (controlled by the first parameter) -e "0~${1}p" take input from $3 (if it exists) otherwise use /dev/stdin {3:-/dev/stdin} Show Sample Output


    2
    function every() { sed -n -e "${2}q" -e "0~${1}p" ${3:-/dev/stdin}; }
    flatcap · 2015-04-03 01:30:36 4
  • This line does not include your closing tag in the output. Show Sample Output


    1
    sed '/'"<opening tag>"'/,/'"<closing tag>"'/{/'"<closing tag>"'/d;p};d' "<file>"
    DaveQB · 2010-11-08 21:43:00 0
  • Show's per IP of how many requests they did to the Apache webserver


    1
    cat /var/log/apache2/access_logs | cut -d ' ' -f 1 | uniq -c | sort -n
    basvdburg · 2013-09-02 13:04:47 1
  • The first sort is necessary for ips in a list to be actually unique.


    1
    cat /var/log/apache2/access_logs | cut -d' ' -f1 | sort | uniq -c | sort -n
    while0pass · 2013-09-07 23:57:31 0
  • Sometimes commands give you too much feedback. Perhaps 1/100th might be enough. If so, every() is for you. my_verbose_command | every 100 will print every 100th line of output. Specifically, it will print lines 100, 200, 300, etc If you use a negative argument it will print the *first* of a block, my_verbose_command | every -100 It will print lines 1, 101, 201, 301, etc The function wraps up this useful sed snippet: ... | sed -n '0~100p' don't print anything by default sed -n starting at line 0, then every hundred lines ( ~100 ) print. '0~100p' There's also some bash magic to test if the number is negative: we want character 0, length 1, of variable N. ${N:0:1} If it *is* negative, strip off the first character ${N:1} is character 1 onwards (second actual character). Show Sample Output


    1
    function every() { N=$1; S=1; [ "${N:0:1}" = '-' ] && N="${N:1}" || S=0; sed -n "$S~${N}p"; }
    flatcap · 2015-03-21 23:44:59 4
  • Applying filter rules is what makes this a really useful command. It's usually a pain to figure out how to sync ONLY files matching a particular pattern, and often one reverts to goofy stuff like find .. -exec rsync .. The filter hides all folders from the transfer, so that only the matching folders that store the filename are left for the sync.


    0
    rsync -avz --dry-run --include="only-include-this-filename" -f 'hide,! */' source/folder/ target/folder/
    cybertoast · 2011-03-16 16:10:42 0
  • Commandline perl filter for, using a production.log from a rails app, display on realtime the count of requests grouped by "seconds to complete" (gross round, but fair enough for an oneliner) :) Show Sample Output


    0
    tail -f production.log | perl -ne 'if (/^Completed.in.(\d+)/){$d = int($1/1000);print "\n";$f{$d}++;for $t (sort(keys(%f))){print $t."s: ".$f{$t}."\n"}}'
    theist · 2012-02-23 14:37:33 0
  • avoiding UUOC! cut can handle files as well. No neet for a cat.


    0
    cut -d ' ' -f 1 /var/log/apache2/access_logs | uniq -c | sort -n
    BorneBjoern · 2013-09-17 20:05:03 0
  • by default, will output the whole line on which 'word' has been found


    0
    grep word file.txt
    lolssl · 2015-10-02 16:01:56 0
  • Removes all lines between the lines containing "" and "", including these lines itself Backdrop: Sometimes when working with XML files without an graphical editor, large comment-/annotation-blocks taper the readability to walk through the file. I like to create a copy of such documents without these annotations. As the documentation itself is in documentation tags inside the annotation tags an therefore graphical editors tend to put the annotation tags in their own lines, this command removes all documentations within annotation-tags. Show Sample Output


    -1
    awk "/<xsd:annotation>/{h=1};!h;/<\/xsd:annotation>/{h=0}" annotatedSchema.xsd
    2chg · 2011-07-15 07:17:17 1

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands


Check These Out

Killing processes with your mouse in an infinite loop
Useful for quickly cleaning your Desktop. Nice joke if launched at startup.

list files recursively by size

Virtual Console lock program
vlock command locks the current console by default. Also you can lock all the consoles on the server by using -a parameter. For details, pl man vlock. ps: Generally speaking , system will not install vlock programme . So you should use ' $ sudo apt-get install vlock ' to install vlock .

Burn CD/DVD from an iso, eject disc when finished.
cdrecord -scanbus will tell you the (x,y,z) value of your cdr (for example, mine is 3,0,0)

Erase a word
hit BACKSPACE more than once to delete more words

Get length of current playlist in xmms2

Find common lines between two files

Display which distro is installed
Works on nearly all linux distros

Which processes are listening on a specific port (e.g. port 80)
swap out "80" for your port of interest. Can use port number or named ports e.g. "http"

Check availability of Websites based on HTTP_CODE


Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: