Commands tagged unicode (9)

  • [Update! Thanks to a tip from ioggstream, I've fixed both of the bugs mentioned below.] You, yes, 𝙔𝙊𝙐, can be the terror of the Internet! Why use normal, boring bullet points in your text, when you could use a ROTATED HEAVY BLACK HEART BULLET (❥)!? (Which would also be an awesome band name, by the way).  This script makes it easy to find unusual characters from the command line. You can then cut and paste them or, if you're using a GTK application, type Control+Shift+U followed by the code point number (e.g., 2765) and then a SPACE.  USAGE: Put this script in a file (I called mine "ugrep") and make it executable. Run it from the command line like so,  ugrep heart  The output will look like this,  ☙ U+2619 REVERSED ROTATED FLORAL HEART BULLET ♡ U+2661 WHITE HEART SUIT ♥ U+2665 BLACK HEART SUIT ❣ U+2763 HEAVY HEART EXCLAMATION MARK ORNAMENT ❤ U+2764 HEAVY BLACK HEART ❥ U+2765 ROTATED HEAVY BLACK HEART BULLET ❦ U+2766 FLORAL HEART ❧ U+2767 ROTATED FLORAL HEART BULLET ⺖ U+2E96 CJK RADICAL HEART ONE ⺗ U+2E97 CJK RADICAL HEART TWO ⼼ U+2F3C KANGXI RADICAL HEART  You can, of course, use regular expressions. For example, if you are looking for the "pi" symbol, you could do this:  ugrep '\bpi\b'  REQUIREMENTS: Although this is written in Bash, it assumes you have Perl installed because it greps through the Perl Unicode character name module (/usr/lib/perl5/Unicode/CharName.pm). Note that it would not have made more sense to write this in Perl, since the CharName.pm module doesn't actually include a subroutine for looking up a character based on the description. (Weird.)  BUGS: In order to fit this script in the commandlinefu limits, a couple bugs were added. ① Astral characters beyond the BMP (basic multilingual plane) are not displayed correctly, but see below. ② Perl code from the perl module being grepped is sometimes extraneously matched.  MISFEATURES: Bash's printf cannot, given a Unicode codepoint, print the resulting character to the terminal. GNU's coreutils printf (usually "/usr/bin/printf") can do so, but it is brokenly pedantic about how many hexadecimal digits follow the escape sequence and will actually die with an error if you give the wrong number. This is especially annoying since Unicode code points are usually variable length with implied leading zeros. The CharNames.pm file represents BMP characters as 4 hexits, but astral characters as 5. In the actual version of this script that I use, I've kludged around this misfeature by zero-padding to 8 hexits like so,  /usr/bin/printf "\U$(printf "%08x" 0x$hex)"  TIP 1: The author recommends "xsel" for command line cut-and-paste. For example,  ugrep biohazard | xsel  TIP 2: In Emacs, instead of running this command in a subshell, you can type Unicode code points directly by pressing Control-Q first, but you'll likely want to change the default input from octal to hexadecimal. (setq read-quoted-char-radix 16).  TIP 3: Of course, if you're using X, and you want to type one of the more common unusual characters, it's easiest of all to do it with your Compose (aka Multi) key. For example, hitting [Compose] <3 types ♥. Show Sample Output


    12
    egrep -i "^[0-9a-f]{4,} .*$*" $(locate CharName.pm) | while read h d; do /usr/bin/printf "\U$(printf "%08x" 0x$h)\tU+%s\t%s\n" $h "$d"; done
    hackerb9 · 2010-12-31 16:47:59 5
  • Generates a TV noise alike output in the terminal. Can be combined with https://www.commandlinefu.com/commands/view/9728/make-some-powerful-pink-noise


    7
    while true;do printf "$(awk -v c="$(tput cols)" -v s="$RANDOM" 'BEGIN{srand(s);while(--c>=0){printf("\xe2\x96\\%s",sprintf("%o",150+int(10*rand())));}}')";done
    ichbins · 2020-05-08 09:55:36 21
  • No need for further filedes or substitution for splitting. Simply use read a b


    1
    grep -i "$*" /usr/lib/perl5/Unicode/CharName.pm | while read a b; do /usr/bin/printf "\u$a\tU+%s\t%s\n" "$b"; done
    ioggstream · 2011-01-04 11:30:16 2
  • Converts control codes and spaces (ASCII code ≤ 32) to visible Unicode Control Pictures, U+2400 ? U+2420. Skips \n characters, which is probably a good thing. Show Sample Output


    1
    /bin/echo -e '\002Hello, Folks\t!\r' | perl -pwle 'use v5.14; s/([\N{U+0000}-\N{U+0020}])/chr(9216+ord($1))/ge;'
    scruss · 2014-06-30 01:45:40 1
  • If you tried the other Python version of Convert JSON to YAML and you end up with lines that has "!!python/unicode", this version of the command is for you.


    1
    python -c 'import sys, yaml, json; yaml.safe_dump(json.load(sys.stdin), sys.stdout, allow_unicode=True)' < foo.json > foo.yaml
    renoirb · 2015-01-20 15:58:29 0
  • This is especially useful to get crazy stuff like space characters copied to your pasteboard correctly. Source: https://github.com/mathiasbynens/dotfiles/blob/master/.functions Show Sample Output


    0
    perl -e "binmode(STDOUT, ':utf8'); print \"[email protected]\""; echo # newline
    mathias · 2012-01-02 10:34:51 0
  • Not better, but more lightweight (sed instead of perl). Above command is reduced due to length restriction of less than 256 characters and entity encoding of "Save" command on this page. This is complete command (best without entity encoding): echo -e '\x2Helo folks\t!\r' | sed "y/\x0\x1\x2\x3\x4\x5\x6\x7\x8\x9\xA\xB\xC\xD\xE\xF\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1A\x1B\x1C\x1D\x1E\x1F\x20/&#9216;&#9217;&#9218;&#9219;&#9220;&#9221;&#9222;&#9223;&#9224;&#9225;&#9226;&#9227;&#9228;&#9229;&#9230;&#9231;&#9232;&#9233;&#9234;&#9235;&#9236;&#9237;&#9238;&#9239;&#9240;&#9241;&#9242;&#9243;&#9244;&#9245;&#9246;&#9247;&#9248;/" ␂Helo␠folks␉!␍ Show Sample Output


    0
    echo -e '\x2Hello, folks\t!\r' | sed "y/\x2\x9\xD\x20/&#9218;&#9225;&#9229;&#9248;/"
    hermannsw · 2014-06-30 14:42:15 0
  • Prints a string as a sequence of hexadecimal values. Output comes in space separated pairs, regardless of ASCII or Unicode characters Show Sample Output


    0
    hexstring () { perl -p -e 's/(.)/sprintf("%02x", ord($1))/eg' << $1 }
    hellcode · 2016-05-03 22:44:24 0
  • Get 'ack' now at http://betterthangrep.com/


    -1
    ack --java '\\u.?.?.?[^0-9a-fA-F]'
    hfs · 2012-06-06 08:31:56 0

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands


Check These Out

Get AWS temporary credentials ready to export based on a MFA virtual appliance
You might want to secure your AWS operations requiring to use a MFA token. But then to use API or tools, you need to pass credentials generated with a MFA token. This commands asks you for the MFA code and retrieves these credentials using AWS Cli. To print the exports, you can use: `awk '{ print "export AWS_ACCESS_KEY_ID=\"" $1 "\"\n" "export AWS_SECRET_ACCESS_KEY=\"" $2 "\"\n" "export AWS_SESSION_TOKEN=\"" $3 "\"" }'` You must adapt the command line to include: * $MFA_IDis ARN of the virtual MFA or serial number of the physical one * TTL for the credentials

Find Duplicate Files (based on MD5 hash) -- For Mac OS X
This works on Mac OS X using the `md5` command instead of `md5sum`, which works similarly, but has a different output format. Note that this only prints the name of the duplicates, not the original file. This is handy because you can add `| xargs rm` to the end of the command to delete all the duplicates while leaving the original.

Convert seconds to [DD:][HH:]MM:SS
Converts any number of seconds into days, hours, minutes and seconds. sec2dhms() { declare -i SS="$1" D=$(( SS / 86400 )) H=$(( SS % 86400 / 3600 )) M=$(( SS % 3600 / 60 )) S=$(( SS % 60 )) [ "$D" -gt 0 ] && echo -n "${D}:" [ "$H" -gt 0 ] && printf "%02g:" "$H" printf "%02g:%02g\n" "$M" "$S" }

A one-line web server in Ruby
Here's how to serve a directory in one line of Ruby. Handy for sharing files at a conference, for example.

Rsync remote data as root using sudo
If your user has sudo on the remote box, you can rsync data as root without needing to login as root. This is very helpful if the remote box does not allow root to login over SSH (which is a common security restriction).

tar a directory and send it to netcat
tar's directory and sends to netcat listening on port 10000 On the client end: netcat [server ip] 10000 | tar xfvz - This will send it over the network and extract it on the clients machine.

Show established network connections

Split and join with split and cat.
`split -b 1k file` splits files into 1k chunks. Rejoin them with `cat x* > file`.

check open ports without netstat or lsof

Install pip with Proxy
Installs pip packages defining a proxy


Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: