Commands tagged filter (11)

  • Working with log files that contains variable length messages wrapped between open and close tags it may be useful to filter the messages upon a keyword. This works fine with GNU sed version 4.2 or higher, so pay attention to some unix distros (solaris, hp-ux, etc.). Linux should be ok. Show Sample Output


    2
    cat file.txt | sed -e /<opening tag>/d -e /<closing tag>/G | sed -e '/./{H;$!d;}' -e 'x;/<string to search>/!d;'
    EBAH · 2010-11-04 10:31:15 0
  • Thanks to knoppix5 for the idea :-) Print selected lines from a file or the output of a command. Usage: every NTH MAX [FILE] Print every NTH line (from the first MAX lines) of FILE. If FILE is omitted, stdin is used. The command simply passes the input to a sed script: sed -n -e "${2}q" -e "0~${1}p" ${3:-/dev/stdin} print no output sed -n quit after this many lines (controlled by the second parameter) -e "${2}q" print every NTH line (controlled by the first parameter) -e "0~${1}p" take input from $3 (if it exists) otherwise use /dev/stdin {3:-/dev/stdin} Show Sample Output


    2
    function every() { sed -n -e "${2}q" -e "0~${1}p" ${3:-/dev/stdin}; }
    flatcap · 2015-04-03 01:30:36 4
  • This line does not include your closing tag in the output. Show Sample Output


    1
    sed '/'"<opening tag>"'/,/'"<closing tag>"'/{/'"<closing tag>"'/d;p};d' "<file>"
    DaveQB · 2010-11-08 21:43:00 0
  • Show's per IP of how many requests they did to the Apache webserver


    1
    cat /var/log/apache2/access_logs | cut -d ' ' -f 1 | uniq -c | sort -n
    basvdburg · 2013-09-02 13:04:47 1
  • The first sort is necessary for ips in a list to be actually unique.


    1
    cat /var/log/apache2/access_logs | cut -d' ' -f1 | sort | uniq -c | sort -n
    while0pass · 2013-09-07 23:57:31 0
  • Sometimes commands give you too much feedback. Perhaps 1/100th might be enough. If so, every() is for you. my_verbose_command | every 100 will print every 100th line of output. Specifically, it will print lines 100, 200, 300, etc If you use a negative argument it will print the *first* of a block, my_verbose_command | every -100 It will print lines 1, 101, 201, 301, etc The function wraps up this useful sed snippet: ... | sed -n '0~100p' don't print anything by default sed -n starting at line 0, then every hundred lines ( ~100 ) print. '0~100p' There's also some bash magic to test if the number is negative: we want character 0, length 1, of variable N. ${N:0:1} If it *is* negative, strip off the first character ${N:1} is character 1 onwards (second actual character). Show Sample Output


    1
    function every() { N=$1; S=1; [ "${N:0:1}" = '-' ] && N="${N:1}" || S=0; sed -n "$S~${N}p"; }
    flatcap · 2015-03-21 23:44:59 4
  • Applying filter rules is what makes this a really useful command. It's usually a pain to figure out how to sync ONLY files matching a particular pattern, and often one reverts to goofy stuff like find .. -exec rsync .. The filter hides all folders from the transfer, so that only the matching folders that store the filename are left for the sync.


    0
    rsync -avz --dry-run --include="only-include-this-filename" -f 'hide,! */' source/folder/ target/folder/
    cybertoast · 2011-03-16 16:10:42 0
  • Commandline perl filter for, using a production.log from a rails app, display on realtime the count of requests grouped by "seconds to complete" (gross round, but fair enough for an oneliner) :) Show Sample Output


    0
    tail -f production.log | perl -ne 'if (/^Completed.in.(\d+)/){$d = int($1/1000);print "\n";$f{$d}++;for $t (sort(keys(%f))){print $t."s: ".$f{$t}."\n"}}'
    theist · 2012-02-23 14:37:33 0
  • avoiding UUOC! cut can handle files as well. No neet for a cat.


    0
    cut -d ' ' -f 1 /var/log/apache2/access_logs | uniq -c | sort -n
    BorneBjoern · 2013-09-17 20:05:03 0
  • by default, will output the whole line on which 'word' has been found


    0
    grep word file.txt
    lolssl · 2015-10-02 16:01:56 0
  • Removes all lines between the lines containing "" and "", including these lines itself Backdrop: Sometimes when working with XML files without an graphical editor, large comment-/annotation-blocks taper the readability to walk through the file. I like to create a copy of such documents without these annotations. As the documentation itself is in documentation tags inside the annotation tags an therefore graphical editors tend to put the annotation tags in their own lines, this command removes all documentations within annotation-tags. Show Sample Output


    -1
    awk "/<xsd:annotation>/{h=1};!h;/<\/xsd:annotation>/{h=0}" annotatedSchema.xsd
    2chg · 2011-07-15 07:17:17 1

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands


Check These Out

check open ports without netstat or lsof

Show current pathname in title of terminal
This helps to keep track of what is going on when you have several tabs open in your terminal. The title automatically changes when you change directories.

Poor man's nmap for a class C network from rfc1918
What do you do when nmap is not available and you want to see the hosts responding to an icmp echo request ? This one-liner will print all hosts responding with their ipv4 address.

Inserts the results of an autocompletion in the command line
Pressing ESC then * will insert in the command line the results of the autocompletion. It's hard to explain but if you look the sample output or do $ echo ESC * you will understand quickly. By the way, few reminders about ESC : - Hold ESC does the same thing as tab tab - 'ESC .' inserts the last argument of last command (can be done many times in order to get the last argument of all previous commands)

Exclude inserting a table from a sql import
Starting with a large MySQL dump file (*.sql) remove any lines that have inserts for the specified table. Sometimes one or two tables are very large and uneeded, eg. log tables. To exclude multiple tables you can get fancy with sed, or just run the command again on subsequently generated files.

Download file with multiple simultaneous connections
`aria2c` (from the aria2 project) allows. Change -s 4 to an arbitrary number of segments to control the number of concurrent connections. It is also possible to provide multiple URLs to the same content (potentially over multiple protocols) to download the file concurrently from multiple hosts.

A DESTRUCTIVE command to render a drive unbootable
THIS COMMAND IS DESTRUCTIVE. That said, lets assume you want to render your boot drive unbootable and reboot your machine. Maybe you want it to boot off the network and kickstart from a boot server for a fresh OS install. Replace /dev/fd0 with the device name of your boot drive and this DESTRUCTIVE command will render your drive unbootable. Your BIOS boot priority should be set to boot from HD first, then LAN.

Making scripts runs on backgourd and logging output
Save all output to a log.

Do one ping to a URL, I use this in a MRTG gauge graph to monitor connectivity

Create a tar of directory structure only


Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: