Commands tagged regex

This is a slightly modified version of http://www.commandlinefu.com/commands/view/4283/recursive-search-and-replace-old-with-new-string-inside-files (which did not work due to incorrect syntax) with the added option to sed inside only files named filename.ext

grep '^[^#]' sample.conf

\__/ |||| \_________/

| |||| |

| |||| \- Filename

| ||||

| |||\- Only character in group is '#'

| |||

| ||\- Negate character group (will match any cahracter *not* in the

| || group)

| ||

| |\- Start new character group (will match any character in the

| | group)

| |

| \- Match beginning of line

|

\- Run grep

Empty lines will also be not matched, because there has to be at least one non-hash-sign character in the line.

Runs a diff on two files ignore comments and blank lines (diff -I=RE does not work as expected). Adapted from a post found on stackexchange.

Reciprocally, we could get the node name from a give Tor IP address =>

ip2node() { curl -s -d "QueryIP=$1" http://torstatus.blutmagie.de/tor_exit_query.php | grep -oP "Server name:.*'>\K\w+" ; }

ip2node 204.8.156.142

BostonUCompSci

People are *going* to post the wrong ways to do this. It's one of the most common form-validation tasks, and also one of the most commonly messed up. Using a third party tool or library like exim means that you are future-proofing yourself against changes to the email standard, and protecting yourself against the fact that actually checking whether an email address is valid is *not possible*.

Still, perhaps your boss is insisting you really do need to check them internally. OK.

Read the RFCs. The bet before the @ is specified by RFC2821 and RFC2822. The domain name part is specified by RFC1035, RFC1101, RFC1123 and RFC2181.

Generally, when people say "email address", they mean that part of the address that the RFC terms the "addr-spec": the "blah@domain.tld" address, with no display names, comments, quotes, etc. Also "root@localhost" and "root" should be invalid, as should arbitrary addressing schemes specified by a protocol indicator, like "jimbo@myprotocol:foo^bar^baz".

So... With the smallest poetic license for readability (allowing underscores in domain names so we can use "\w" instead of "[a-z0-9]"), the RFCs give us:

^(?:"(?:[^"\\]|\\.)+"|[-^!#\$%&'*+\/=?`{|}~.\w]+)@(?=.{3,255}$)(?:[\w][\w-]{0,62}\.){1,128}[\w][\w-]{0,62}$

Not perfect, but the best I can come up with, and most compliant I've found. I'd be interested to see other people's ideas, though. It's still not going to verify you an address fersure, properly, 100% guaranteed legit, though. What else can you do? Well, you could also:

* verify that the address is either a correct dotted-decimal IP, or contains letters.

* remove reserved domains (.localhost, .example, .test, .invalid), reserved IP ranges, and so forth from the address.

* check for banned domains (whitehouse.gov, example.com...)

* check for known TLDs including alt tlds.

* see if the domain has an MX record set up: if so, connect to that host, else connect to the domain.

* see if the given address is accepted by the server as a recipient or sender (this fails for yahoo.*, which blocks after a few attempts, assuming you are a spammer, and for other domains like rediffmail.com, home.com).

But these are moving well out of the realm of generic regex checks and into the realm of application-specific stuff that should be done in code instead - especially the latter two. Hopefully, this is all you needed to point out to your boss "hey, email validation this is a dark pit with no bottom, we really just want to do a basic check, then send them an email with a link in it: it's the industry standard solution."

Of course, if you want to go nuts, here's an idea that you could do. Wouldn't like to do it myself, though: I'd rather just trust them until their mail bounces too many times. But if you want it, this (untested) code checks to see if the mail domain works. It's based on a script by John Coggeshall and Jesse Houwing that also asked the server if the specific email address existed, but I disliked that idea for several reasons. I suspect: it will get you blocked as a spambot address harvester pretty quick; a lot of servers would lie to you; it would take too much time; this way you can cache domains marked as "OK"; and I suspect it would add little to the reliability test.

Sometimes, especially when parsing HTML, you want "all text between two tags, that doesn't contain another tag".

For example, to grab only the contents of the innermost <div>s, something like:

/<div\b[^>]*>((?:(?!<div).)*)</div>/

...may be your best option to capture that text.

It's not always needed, but is a powerful arrow in your regex quiver in those cases when you do need it.

Note that, in general, regular expressions are the Wrong Choice for parsing HTML, anyway. Better approaches are solutions which let you navigate the HTML as a proper DOM. But sometimes, you just need to use the tools available to you. If you don't, then you have two problems.

With thanks to dew on Efnet's #regex, back in 2005.

This version indents subsequent lines after the first by one space, to make paragraphs visibly obvious -- remove the \3 to prevent this behavior.

Lines are only broken at spaces: long strings with no spaces will not wrap, so URLs are safe.

Replace the "75"s to make the regex linewrap to other amounts.

From the unix commandline, "fold" is likely your better choice, but this snippet is handy in editors which allow regular expressions, in scripting, and other such situations where "fold" is unavailable.

Removes special characters (colors) in '^]]Xm' and '^]]X;Ym' format from file.

Use pipe ('input | perl [...]') or stream ('perl [...]

You can use 'cat -v infile' as 'input' to show special characters instead of interpreting (there is problem with non-ASCII chars, they are replaced by M-[char]).

Uses sed with a regex to move the linenumbers to the line end. The plain regex (w/o escapes) looks like that:

^([^:]*):(.*)

# Search for an available package on Debian systems using a regex so it only matches packages starting with 'tin'.

This command is useful when you are programming, for example.

Much better alternatives - grep-alikes using perl regexps. With more options, and nicer outputs.

If you've ever tried "grep -P" you know how terrible it is. Even the man page describes it as "highly experimental". This function will let you 'grep' pipes and files using Perl syntax for regular expressions.

The first argument is the pattern, e.g. '/foo/'. The second argument is a filename (optional).

This is what I came up to generate XKCD #936 style four-word password.

Since first letter of every word is capitalized it looks a bit more readable to my eyes.

Also strips single quotes.

And yes - regex is a bit of a kludge, but that's the bes i could think of.

This works by reading in two lines of input, turning each into a list of one-character matches that are sorted and compared.

Some source package have many 'README' kind of files, among many other regular files/directories. This command could be useful when one wants to list only 'README' kind of files among jungle of other files. (e.g. I came across this situation after downloading source for module-init-tools)

Warning: This command would miss a file like => README.1 (or one with spaces in-between)

Corrections welcome.

In this way it doesn't have problems with filenames with spaces.

grep multiline in Perl regexp syntax with pcregrep

In case the line you want to join start with a char different than ", you may use \n.*"\n as regex.

----

this line ends here

but must be concatenated with this one

"this line ends here"

and should NOT be concatenated with this one

I used to do a lot of path manipulation to set up my development environment (PATH, LD_LIBRARY_PATH, etc), and one part of my environment wasn't always aware of what the rest of the environment needed in the path. Thus resetting the entire PATH variable wasn't an option; modifying it made sense.

The original version of the functions used sed, which turned out to be really slow when called many times from my bashrc, and it could take up to 10 seconds to login. Switching to parameter substitution sped things up significantly.

The commands here don't clean up the path when they are done (so e.g. the path gets cluttered with colons). But the code is easy to read for a one-liner.

The full function looks like this:

I tried using regexes to make this into a cleaner one-liner, but remove_path ended up being cryptic and not working as well:

Say you have a directory structure like "foo/, foo/data/, bar/, bar/data/". If you just want to ignore 'bar/data' and you use "ack --ignore-dir=data pattern" it will ignore both foo/data and bar/data and 'ignore-data=bar/data' etc won't work.