Commands tagged encoding (12)

  • This is just a proof of concept: A FILE WHICH CAN AUTOMOUNT ITSELF through a SIMPLY ENCODED script. It takes advantage of the OFFSET option of mount, and uses it as a password (see that 9191? just change it to something similar, around 9k). It works fine, mounts, gets modified, updated, and can be moved by just copying it. USAGE: SEE SAMPLE OUTPUT The file is composed of three parts: a) The legible script (about 242 bytes) b) A random text fill to reach the OFFSET size (equals PASSWORD minus 242) c) The actual filesystem Logically, (a)+(b) = PASSWORD, that means OFFSET, and mount uses that option. PLEASE NOTE: THIS IS NOT AN ENCRYPTED FILESYSTEM. To improve it, it can be mounted with a better encryption script and used with encfs or cryptfs. The idea was just to test the concept... with one line :) It applies the original idea of http://www.commandlinefu.com/commands/view/7382/command-for-john-cons for encrypting the file. The embedded bash script can be grown, of course, and the offset recalculation goes fine. I have my own version with bash --init-file to startup a bashrc with a well-defined environment, aliases, variables. Show Sample Output


    6
    dd if=/dev/zero of=T bs=1024 count=10240;mkfs.ext3 -q T;E=$(echo 'read O;mount -o loop,offset=$O F /mnt;'|base64|tr -d '\n');echo "E=\$(echo $E|base64 -d);eval \$E;exit;">F;cat <(dd if=/dev/zero bs=$(echo 9191-$(stat -c%s F)|bc) count=1) <(cat T;rm T)>>F
    rodolfoap · 2013-01-31 01:38:30 14

  • 4
    iconv -f utf8 -t utf16 /path/to/file
    imsaar · 2009-12-01 21:02:58 4
  • Useful if non-ascii characters in filenames have been improperly encoded. Replace "PROBLEM" with the incorrect characters (e.g. 'é'), and "FIX" with the correct ones (e.g. '?').


    3
    for i in *;do mv "$i" "$(echo $i | sed s/PROBLEM/FIX/g)";done
    AlecSchueler · 2009-06-28 01:50:25 18
  • Get gzip compressed web page using wget. Caution: The command will fail in case website doesn't return gzip encoded content, though most of thw websites have gzip support now a days.


    3
    wget -q -O- --header\="Accept-Encoding: gzip" <url> | gunzip > out.html
    ashish_0x90 · 2010-11-27 22:14:42 8

  • 3
    find . -iname *.java -type f -exec bash -c "iconv -f WINDOWS-1252 -t UTF-8 {} > {}.tmp " \; -exec mv {}.tmp {} \;
    miguelbaldi · 2012-01-12 20:00:26 6
  • This command gives you the charset of a text file, which would be handy if you have no idea of the encoding. Show Sample Output


    2
    file -i <textfile>
    juvenn · 2009-09-08 01:33:19 4
  • http://www.mplayerhq.hu/DOCS/HTML/en/menc-feat-mpeg.html MEncoder can create MPEG (MPEG-PS) format output files. Usually, when you are using MPEG-1 or MPEG-2 video, it is because you are encoding for a constrained format such as SVCD, VCD, or DVD. To change MEncoder's output file format, use the -of mpeg option. Creating an MPEG-1 file suitable to be played on systems with minimal multimedia support, such as default Windows installs: mencoder input.avi -of mpeg -mpegopts format=mpeg1:tsaf:muxrate=2000 \ -o output.mpg -oac lavc -lavcopts acodec=mp2:abitrate=224 -ovc lavc \ -lavcopts vcodec=mpeg1video:vbitrate=1152:keyint=15:mbd=2:aspect=4/3


    1
    mencoder input.avi -of mpeg -ovc lavc -lavcopts vcodec=mpeg1video \ -oac copy other_options -o output.mpg
    slishan · 2010-01-18 13:12:03 4
  • Really helpfull when play with files having spaces an other bad name. Easy to store and access names and path in just a field while saving it in a file. This format (URL) is directly supported by nautilus and firefox (and other browsers) Show Sample Output


    1
    convert_path2uri () { echo -n 'file://'; echo -n "$1" | perl -pe 's/([^a-zA-Z0-9_\/.])/sprintf("%%%.2x", ord($1))/eg' ;} #convert2uri '/tmp/a b' ### convert file path to URI
    totti · 2013-07-01 08:54:45 8
  • Like the original command, but the -f allows this one to succeed even if the website returns uncompressed data. From gzip(1) on the -f flag: If the input data is not in a format recognized by gzip, and if the --stdout is also given, copy the input data without change to the standard output: let zcat behave as cat.


    1
    wget -q -O- --header="Accept-Encoding: gzip" <url> | gzip -cdf > out.html
    tempusername · 2014-11-29 20:42:21 9

  • 0
    uchardet <filename>
    fltman · 2014-11-28 07:47:13 8
  • I still don't know why mutt (or offlineimap or whatever in between), is borking the encoding of my files, but this fixes it. Show Sample Output


    0
    fix_saved_files() { for i in "$@" ; do mv $i $(php -r '$or = "'$i'"; mb_internal_encoding("UTF-8"); echo mb_decode_mimeheader($or) . "\n";') ; done; }
    guillaume · 2014-12-27 20:37:16 9
  • Also works with files: cat file Hello world base64 file SGVsbG8gd29ybGQK To decrypt use the -d option: echo SGVsbG8gd29ybGQK | base64 -d Hello world Show Sample Output


    -3
    echo "Hello world" | base64
    P17 · 2009-04-27 15:00:08 16

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

Share Your Commands


Check These Out

list files recursively by size

Switch 2 characters on a command line.
If you typed 'sl', put the cursor on the 'l' and hit ctrl-t to get 'ls'.

enable all bash completions in gentoo
enable each bash completion that you have installed at your system, that's very nice ;)

ttyS0 - terminal on serial connection
I actually planned to do this for quite a long time, but since I haven't had any suitable client hardware, I procrastinated this. Now, the old laptop I've got from my dad, features an RS-232 port. So, now that I had technically a client, that I could test my RS-232 connection with, I ordered a null modem cable. There is no RS-232 outlet on my desktop computer directly on the mainboard, but theres a connector on the mainbord, where a RS-232 outlet can be attached to. The outlet will then cover up a PCI slot. # Activating RS-232 Ok, once all cables were in place, I tried to access both RS-232 ports by reading from them directly. They're usually ttyS0 or ttyS1, depending what COM-Port it is. From the file /proc/tty/driver/serial, information about the serial setup can be obtained. A setserial -q /dev/ttyS0 might be usefull as well. Usually, the UART Type is 16550A, on a standard PC. Mine wasn't working though. At leas not right from the start, when I tried to read the interface with cat /dev/ttyS0 I got the following error: # cat /dev/ttyS0 cat: /dev/ttyS0: Input/output error Obviously, the driver couldn't activate the hardware. Reason was, it was deactivated in BIOS. After activating RS-232 there, it worked well. As a last action, I added myself to the uucp group, so I have user permission to the serial lines. It is not necessary for the terminal setup, but a good idea to do so, just for future projects, maybe... # Setting up a terminal Once the Serial line is configured and working properly, it's time to let a terminal run on that port. This is what I added to my /etc/inittab : s0:2345:respawn:/sbin/agetty -L 38400 ttyS0 I added it quite on the top of that file, right below the 'si' statement, mingetty cannot be used for serial connections, it cannot be run in a console, too. I tried it for testing purposes, but the cosole - along with your login program - will log you out, as soon as you log in over your serial line. '-L' means this is a local line, with no carrier signal. 38400 is the standard speed of a Linux console, it might be a bit high, I was told, but it works well. I tested that with some higher values as well (115200) and it worked too, I guess it tepends on things like cable length, etc. Last parameter, is the serial tty to listen on. The terminal type can be specified as an additional parameter at the end of the parameter list, vt102, for instance. This is sometimes required, depending on the client. After finishing editing /etc/inittab, an init q will make the system re-read /etc/inittab and apply changes. The agetty should now be listening on ttyS0. #Setting up a client It's time to establish a connection and test the serial line. I use a laptop, that has an RS-232 port, so some preliminary setup is required. I tried minicom as terminal initially, but it turned out, not to be the best client. It initializes the modem, this lasts quite long, and it doesn't convey ANSI colors. So the better option is cu, it's part of the UUCP-Package. Oh, and the serial port of that computer, has to be accessible as well, of course. Once everything was set up, I established the connection: $ cu -l ttyS0 -38400 --nostop Pretty self explanatory, I think. The --nostop option disables XON/XOFF handling. # root access over ttyS0 In order to become root over the serial terminal, the tty needs to be added to /etc/securetty I appended ttyS0 to the end of the file. It is now possible, to gain root access over the serial terminal. The agetty process needs to be restarted to apply changes. # Accessing GRUB over ttyS0 To make bootloader access possible over ttyS0, some changes to /boot/grub/menu.lst need to be done. (GRUB is the bootloader I use, I suppose LiLo has similar capabilities.) Those are the lines, I appended to the top of my menu.lst : serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1 terminal --timeout=3 serial console The serial command initiates the serial terminal option, --unit=0 defines our first serial connector, I my case, it's the only one I have on my machine. I used the standard Linux-Console speed, as well as the "8N1" connection strategy. terminal defines the terminal priorities, first terminal (serial) is the standard one, the last one is the secondary terminal (console). --timeout=3 enables a delay on both consoles, with a prompt for a keystroke. Depending on which terminal, the key is pressed, this terminal, will be used. If no key is pressed after the timeout, the standard console (in my case serial) will be used. # Relaying Kernel output on boot The Kernel accepts multiple console options, of which the last one, is the standard console, and the one that will be used in Single User mode. These are my Kernel options: title Fedora Core (2.6.20-1.2316.fc5) root (hd0,0) kernel /vmlinuz-2.6.20-1.2316.fc5 ro root=/dev/VolGroup00/LogVol00 rhgb quiet vga=795 console=tty0 console=ttyS0,38400 initrd /initrd-2.6.20-1.2316.fc5.img console=tty0 is the standard console, located on the machine, i.e. monitor and keyboard.

BIGGEST Files in a Directory
Find biggest files in a directory

HourGlass
Displays an animated hourglass for x amount of seconds

Write comments to your history.
A null operation with the name 'comment', allowing comments to be written to HISTFILE. Prepending '#' to a command will *not* write the command to the history file, although it will be available for the current session, thus '#' is not useful for keeping track of comments past the current session.

show ALL iptable rules
show your current iptable rules from every available iptable table

Convert CSV to JSON
Replace 'csv_file.csv' with your filename.

Protect your eye
Redshift will adjust the color temperature and protects eye at night -b : will adjust the brightness


Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for: