perl -ne 'if (/^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} )/ ) { $t=$1; print $_ } else { print $t . $_ }'

Copy timestamps to all lines in a log file

If you have a logfile where some lines start with timestamps like "2014-05-01 12:34:56,123" but other lines are missing the timestamp (like stack traces or object dumps), then use this script to copy the most recent timestamp to any lines that are missing it. This is useful for merging log files, since you can then safely sort by timestamp to merge the files.
Sample Output
$ cat input.txt
2014-05-01 12:34:56,123 line with ts
line without ts

$ perl -ne 'if (/^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} )/ ) { $t=$1; print $_ } else { print $t . $_ }' input.txt
2014-05-01 12:34:56,123 line with ts
2014-05-01 12:34:56,123 line without ts

0
2014-05-21 08:54:27

These Might Interest You

  • This is useful when watching a log file that does not contain timestamps itself. If the file already has content when starting the command, the first lines will have the "wrong" timestamp when the command was started and not when the lines were originally written.


    6
    tail -f file | while read line; do echo -n $(date -u -Ins); echo -e "\t$line"; done
    hfs · 2010-11-19 10:01:57 4
  • Sometimes when copying files from one place to another, the timestamps get lost. Maybe you forgot to add a flag to preserve timestamps in your copy command. You're sure the files are exactly the same in both locations, but the timestamps of the files in the new home are wrong and you need them to match the source. Using this command, you will get a shell script (/tmp/retime.sh) than you can move to the new location and just execute - it will change the timestamps on all the files and directories to their previous values. Make sure you're in the right directory when you launch it, otherwise all the touch commands will create new zero-length files with those names. Since find's output includes "." it will also change the timestamp of the current directory. Ideally rsync would be the way to handle this - since it only sends changes by default, there would be relatively little network traffic resulting. But rsync has to read the entire file contents on both sides to be sure no bytes have changed, potentially causing a huge amount of local disk I/O on each side. This could be a problem if your files are large. My approach avoids all the comparison I/O. I've seen comments that rsync with the "--size-only" and "--times" options should do this also, but it didn't seem to do what I wanted in my test. With my approach you can review/edit the output commands before running them, so you can tell exactly what will happen. The "tee" command both displays the output on the screen for your review, AND saves it to the file /tmp/retime.sh. Credit: got this idea from Stone's answer at http://serverfault.com/questions/344731/rsync-copying-over-timestamps-only?rq=1, and combined it into one line. Show Sample Output


    5
    find . -printf "touch -m -d \"%t\" '%p'\n" | tee /tmp/retime.sh
    dmmst19 · 2012-11-05 20:32:05 2
  • cp options: -p will preserve the file mode, ownership, and timestamps -r will copy files recursively also, if you want to keep symlinks in addition to the above: use the -a/--archive option


    4
    cp -pr olddirectory newdirectory
    stanishjohnd · 2009-08-22 22:11:24 1
  • Use if you have pictures all over the place and you want to copy them to a central location Synopsis: Find jpg files translate all file names to lowercase backup existing, don't overwrite, preserve mode ownership and timestamps copy to a central location


    4
    find . -iname "*.jpg" -print0 | tr '[A-Z]' '[a-z]' | xargs -0 cp --backup=numbered -dp -u --target-directory {location} &
    oracular · 2009-12-10 08:47:04 0
  • atime and mtime timestamps of $FILE2 is changed according to the ones of $FILE1. If $FILE2 doesn't exist is created. Show Sample Output


    2
    touch -r "$FILE1" "$FILE2"
    kobayashison · 2009-10-21 13:53:59 2
  • Remove blank lines from files, even if some of the lines have whitespace (spaces, tabs, etc.). Also creates a .bak copy of the original file. Works on Mac also. Thanks to those all those who left comments. Show Sample Output


    0
    sed -i.bak '/^[[:space:]]*$/d' file_name
    Highwayman · 2009-04-07 11:32:06 4

What do you think?

Any thoughts on this command? Does it work on your machine? Can you do the same thing with only 14 characters?

You must be signed in to comment.

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands



Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: