sed '1000000!d;q' < massive-log-file.log

Efficiently print a line deep in a huge log file

Sed stops parsing at the match and so is much more effecient than piping head into tail or similar. Grab a line range using sed '999995,1000005!d' < my_massive_file

By: root
2009-01-26 11:50:00

These Might Interest You

  • If you need to print some portion of a huge file, let's say you want to print from line 200 to 300, you can use this command to print the line from LINE1 to LINE2 of file FILE.

    Printing portion of a big file
    acirulli · 2009-03-30 11:08:38 2
  • Ever wanted to find the most recently modified files, but couldn't remember exactly where they were in a project directory with many subdirectories? The "find" command, using a combination of "-mtime -N" and "-depth -D" can be used to find those files. If your directory structure isn't very deep, just omit the "-depth -D", but if your directory structure is very deep, then you can limit the depth of the traversal using "-depth -D", where "D" is the maximum number of directory levels to descend. Show Sample Output

    find . -type f -depth -3 -mtime -5
    totoro · 2009-03-25 19:54:06 0
  • I needed to find a command to delete huge numbers of files without intensive cpu load, that command does the job :) The purpose of "+" at then end of the line, tells rm to delete multiple files at once

    find <path> -type f -exec ionice -c3 rm {} +
    tyjak · 2017-07-31 09:35:36 0
  • Problem: you want to output one line per file. you can't just 'tr -d' because you want one line per file and you don't want to use a loop. Solution: use awk to print each line without the record separator and a newline after each file. Show Sample Output

    awk 'BEGIN{ORS=""}NR!=1&&FNR==1{print "\n"}{print}END{print "\n"}' *.txt
    recursiverse · 2010-05-24 19:22:35 0
  • If you've ever wanted to change a text files contents without having to create an intermediate file. This is it. Ex is a part of vim. The command as given will delete ALL lines containing "delete this line" from the file. More Examples: print '%s/'this is a line'/'that is not the same line'/g\nwq' | ex file.txt will substitute the first string with the second string. print '3a\n'Inserted Line'\n.\n\nwq' | ex file.txt will insert the given line after line 3. CAVEAT, Some distro's like the print command, others like echo with this command. Also note there are NO error messages on failure, at least that I've ever seen. Ex can also be quite fussy as to how it takes strings, parameters, etc... I use at&t's ksh syntax may very with other shells.

    print 'g/'delete this line'/delete\nwq' | ex file.txt
    mph · 2009-07-20 19:15:29 3
  • In case you're like me and like your commands to start on clean lines, especially when you're deep into a 10-level directory tree. This can be added to .bashrc. Show Sample Output

    PS1="${PS1::-3} \n$ "
    97fyga9f4 · 2018-02-07 11:27:45 0

What Others Think

I prefer: sed -n '1000000p' massive-log-file.log
penthief · 472 weeks and 5 days ago
Have you done tests? In my examples (line 10 mio from a 19 mio lines file) head|tail was faster. sed was only slightly faster if you write: head -n 10000001 | tail -n 2 0.9 sec utime, 2 sec clock time sed -n '10000000,10000001p' 1.7 sec utime, 2.2 sec clock time sed -n '10000000,10000001p;10000002q' 0.9 sec utime, 1.2 sec clock time Otherwise sed will parse until the end, what head|tail doesn't. Overall the time differences seem irrelevant. It's more a question of beauty. :-)
weidenrinde · 131 weeks ago

What do you think?

Any thoughts on this command? Does it work on your machine? Can you do the same thing with only 14 characters?

You must be signed in to comment.

What's this? is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.


Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: