Commands tagged filter (11)

  • Working with log files that contains variable length messages wrapped between open and close tags it may be useful to filter the messages upon a keyword. This works fine with GNU sed version 4.2 or higher, so pay attention to some unix distros (solaris, hp-ux, etc.). Linux should be ok. Show Sample Output


    2
    cat file.txt | sed -e /<opening tag>/d -e /<closing tag>/G | sed -e '/./{H;$!d;}' -e 'x;/<string to search>/!d;'
    EBAH · 2010-11-04 10:31:15 5
  • Thanks to knoppix5 for the idea :-) Print selected lines from a file or the output of a command. Usage: every NTH MAX [FILE] Print every NTH line (from the first MAX lines) of FILE. If FILE is omitted, stdin is used. The command simply passes the input to a sed script: sed -n -e "${2}q" -e "0~${1}p" ${3:-/dev/stdin} print no output sed -n quit after this many lines (controlled by the second parameter) -e "${2}q" print every NTH line (controlled by the first parameter) -e "0~${1}p" take input from $3 (if it exists) otherwise use /dev/stdin {3:-/dev/stdin} Show Sample Output


    2
    function every() { sed -n -e "${2}q" -e "0~${1}p" ${3:-/dev/stdin}; }
    flatcap · 2015-04-03 01:30:36 15
  • This line does not include your closing tag in the output. Show Sample Output


    1
    sed '/'"<opening tag>"'/,/'"<closing tag>"'/{/'"<closing tag>"'/d;p};d' "<file>"
    DaveQB · 2010-11-08 21:43:00 4
  • Show's per IP of how many requests they did to the Apache webserver


    1
    cat /var/log/apache2/access_logs | cut -d ' ' -f 1 | uniq -c | sort -n
    basvdburg · 2013-09-02 13:04:47 7
  • The first sort is necessary for ips in a list to be actually unique.


    1
    cat /var/log/apache2/access_logs | cut -d' ' -f1 | sort | uniq -c | sort -n
    while0pass · 2013-09-07 23:57:31 7
  • Sometimes commands give you too much feedback. Perhaps 1/100th might be enough. If so, every() is for you. my_verbose_command | every 100 will print every 100th line of output. Specifically, it will print lines 100, 200, 300, etc If you use a negative argument it will print the *first* of a block, my_verbose_command | every -100 It will print lines 1, 101, 201, 301, etc The function wraps up this useful sed snippet: ... | sed -n '0~100p' don't print anything by default sed -n starting at line 0, then every hundred lines ( ~100 ) print. '0~100p' There's also some bash magic to test if the number is negative: we want character 0, length 1, of variable N. ${N:0:1} If it *is* negative, strip off the first character ${N:1} is character 1 onwards (second actual character). Show Sample Output


    1
    function every() { N=$1; S=1; [ "${N:0:1}" = '-' ] && N="${N:1}" || S=0; sed -n "$S~${N}p"; }
    flatcap · 2015-03-21 23:44:59 13
  • Applying filter rules is what makes this a really useful command. It's usually a pain to figure out how to sync ONLY files matching a particular pattern, and often one reverts to goofy stuff like find .. -exec rsync .. The filter hides all folders from the transfer, so that only the matching folders that store the filename are left for the sync.


    0
    rsync -avz --dry-run --include="only-include-this-filename" -f 'hide,! */' source/folder/ target/folder/
    cybertoast · 2011-03-16 16:10:42 3
  • Commandline perl filter for, using a production.log from a rails app, display on realtime the count of requests grouped by "seconds to complete" (gross round, but fair enough for an oneliner) :) Show Sample Output


    0
    tail -f production.log | perl -ne 'if (/^Completed.in.(\d+)/){$d = int($1/1000);print "\n";$f{$d}++;for $t (sort(keys(%f))){print $t."s: ".$f{$t}."\n"}}'
    theist · 2012-02-23 14:37:33 3
  • avoiding UUOC! cut can handle files as well. No neet for a cat.


    0
    cut -d ' ' -f 1 /var/log/apache2/access_logs | uniq -c | sort -n
    BorneBjoern · 2013-09-17 20:05:03 9
  • by default, will output the whole line on which 'word' has been found


    0
    grep word file.txt
    lolssl · 2015-10-02 16:01:56 10
  • Removes all lines between the lines containing "" and "", including these lines itself Backdrop: Sometimes when working with XML files without an graphical editor, large comment-/annotation-blocks taper the readability to walk through the file. I like to create a copy of such documents without these annotations. As the documentation itself is in documentation tags inside the annotation tags an therefore graphical editors tend to put the annotation tags in their own lines, this command removes all documentations within annotation-tags. Show Sample Output


    -1
    awk "/<xsd:annotation>/{h=1};!h;/<\/xsd:annotation>/{h=0}" annotatedSchema.xsd
    2chg · 2011-07-15 07:17:17 7

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands


Check These Out

tail -f a log file over ssh into growl

Display disk partition sizes
It is the same but more faster real 0m0,007s user 0m0,011s sys 0m0,000s with my solution real 0m0,038s user 0m0,044s sys 0m0,000s with your solution :)

dump database from postgresql to a file

Keep a copy of the raw Youtube FLV,MP4,etc stored in /tmp/
Certain Flash video players (e.g. Youtube) write their video streams to disk in /tmp/ , but the files are unlinked. i.e. the player creates the file and then immediately deletes the filename (unlinking files in this way makes it hard to find them, and/or ensures their cleanup if the browser or plugin should crash etc.) But as long as the flash plugin's process runs, a file descriptor remains in its /proc/ hierarchy, from which we (and the player) still have access to the file. The method above worked nicely for me when I had 50 tabs open with Youtube videos and didn't want to have to re-download them all with some tool.

Convert seconds to [DD:][HH:]MM:SS
Converts any number of seconds into days, hours, minutes and seconds. sec2dhms() { declare -i SS="$1" D=$(( SS / 86400 )) H=$(( SS % 86400 / 3600 )) M=$(( SS % 3600 / 60 )) S=$(( SS % 60 )) [ "$D" -gt 0 ] && echo -n "${D}:" [ "$H" -gt 0 ] && printf "%02g:" "$H" printf "%02g:%02g\n" "$M" "$S" }

Change the homepage of Firefox
Pros: Works in all Windows computers, most updated and compatible command. Cons: 3 liner Replace fcisolutions.com with your site name.

Copy without overwriting

Fast, built-in pipe-based data sink
This is shorter and actually much faster than >/dev/null (see sample output for timings) Plus, it looks like a disappointed face emoticon.

bash screensaver off

return the latest kernel version from a Satellite / Spacewalk server software channel


Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: