Hide

What's this?

commandlinefu.com is the place to record those command-line gems that you return to again and again.

Delete that bloated snippets file you've been using and share your personal repository with the world. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on, discussed and voted up or down.

If you have a new feature suggestion or find a bug, please get in touch via http://commandlinefu.uservoice.com/

Get involved!

You can sign-in using OpenID credentials, or register a traditional username and password.

OpenID
or
Sign in Register

First-time OpenID users will be automatically assigned a username which can be changed after signing in.

Hide

Stay in the loop…

Follow the Tweets.

Every new command is wrapped in a tweet and posted to Twitter. Following the stream is a great way of staying abreast of the latest commands. For the more discerning, there are Twitter accounts for commands that get a minimum of 3 and 10 votes - that way only the great commands get tweeted.

» http://twitter.com/commandlinefu
» http://twitter.com/commandlinefu3
» http://twitter.com/commandlinefu10

Subscribe to the feeds.

Use your favourite RSS aggregator to stay in touch with the latest commands. There are feeds mirroring the 3 Twitter streams as well as for virtually every other subset (users, tags, functions,…):

Subscribe to the feed for:

Hide

News

2011-03-12 - Confoo 2011 presentation
Slides are available from the commandlinefu presentation at Confoo 2011: http://presentations.codeinthehole.com/confoo2011/
2011-01-04 - Moderation now required for new commands
To try and put and end to the spamming, new commands require moderation before they will appear on the site.
2010-12-27 - Apologies for not banning the trolls sooner
Have been away from the interwebs over Christmas. Will be more vigilant henceforth.
2010-09-24 - OAuth and pagination problems fixed
Apologies for the delay in getting Twitter's OAuth supported. Annoying pagination gremlin also fixed.
Hide

Tags

Hide

Functions

Hide

Credits

Site by David Winterbottom (user root).

Facebook Email Scraper

Terminal - Facebook Email Scraper
fbemailscraper YourFBEmail Password
2010-01-31 00:44:35
User: dabom
4
Up
Down
Facebook Email Scraper

(Apparently it is too long so I put it in sample output, I hope that is OK.)

Run the long command (or put it in your .bashrc) in sample output then run:

fbemailscraper YourFBEmail Password

Voila! Your contacts' emails will appear.

Facebook seems to have gotten rid of the picture encoding of emails and replaced it with a text based version making it easy to scrape!

Needs curl to run and it was made pretty quickly so there might be bugs.

Show sample output | Add to favourites | Report as malicious

Alternatives

There is 1 alternative - vote for the best!

Terminal - Alternatives
fbemailscraper YourFBEmail Password
2010-05-26 20:52:29
User: coast
Tags: bash awk curl facebook
-15
Up
Down
Facebook Email Scraper
Comments (0) | Add to favourites | Report as malicious

Know a better way?

If you can do better, submit your command here.

What others think

Didn't work for me. All I got in return was:

rm: remove regular file `/tmp/cookies'?
Comment by bkuri 171 weeks and 6 days ago

OK, fixed it. There was a problem with "amp" being erased messing up the "id" code. Copy it and try again.

Comment by dabom 171 weeks and 6 days ago

Still a no-go, sorry...

Comment by bkuri 171 weeks and 6 days ago

Do you have curl and any of the other programs I used installed? If your password has weird symbols/spaces in it that might cause problems. Make sure you are using bash I guess, not sure how it will work with other shells.

Comment by dabom 171 weeks and 6 days ago

Here it did not work until I logged out on the website.

Script only gives you the mail addresses without anything else. Is there a way to add name/surname to the list?

Comment by TabTwo 171 weeks and 5 days ago

I've updated the command to add a timeout and retry.

As for names, here is a quick fix, it will put the profile URL above the email (this is all I will be doing, but it shouldn't be too hard to make it extract names.)

fbemailscraper () { email="$1" ; password="$2" ; id="$(curl -m 5 --retry 1 -c /tmp/cookies -A "Opera" -Ls -d "email=$email&pass=$password&persistent=1" "https://login.facebook.com/login.php?m&next=http://m.facebook.com/profile.php" | grep -o "\&am\p\;id=.*&am\p;v=feed\&am\p;refid=17" | sed -e "s/.*profile.*id=\(.*\)\&am\p\;v=feed.*/\1/g")" ; curl -m 5 --retry 1 -b /tmp/cookies -Ls -A "Opera" "http://www.facebook.com/ajax/typeahead_friends.php?u=$id&__a=1" | tr "\"" "\n" | grep facebook | tr -d "\\\\" | sed "s/www/m/g" | awk '{print $0 "?v=info&refid=17"}' | while read line ; do echo "$line" ; curl -m 5 --retry 1 -b /tmp/cookies -A "Opera" -Ls "$line" | tr \< "\n" | tr \> "\n" | grep "@.*\." | grep -v " " ; done ; rm /tmp/cookies ; }
Comment by dabom 171 weeks and 5 days ago

Found a problem with getting email from people without nicknames. The updated profile URL script is here and the one in sample output it fixed.

fbemailscraper () { email="$1" ; password="$2" ; id="$(curl -m 5 --retry 1 -c /tmp/cookies -A "Opera" -Ls -d "email=$email&pass=$password&persistent=1" "https://login.facebook.com/login.php?m&next=http://m.facebook.com/profile.php" | grep -o "\&am\p\;id=.*&am\p;v=feed\&am\p;refid=17" | sed -e "s/.*profile.*id=\(.*\)\&am\p\;v=feed.*/\1/g")" ; curl -m 5 --retry 1 -b /tmp/cookies -Ls -A "Opera" "http://www.facebook.com/ajax/typeahead_friends.php?u=$id&__a=1" | tr "\"" "\n" | grep facebook | tr -d "\\\\" | sed "s/www/m/g" | awk '{print $0 "?v=info&refid=17"}' | sed "/profile.php/s/?v=info/\&v=info/g" | while read line ; do echo "$line" ; curl -m 5 --retry 1 -b /tmp/cookies -A "Opera" -Ls "$line" | tr \< "\n" | tr \> "\n" | grep "@.*\." | grep -v " " ; done ; rm /tmp/cookies ; }
Comment by dabom 171 weeks and 4 days ago

it doesn't work for me...

i'm trying on snow leopard with curl

i'm getting this if i manually execute the second curl with my id and a correct path with the cookies:

for (;;);{"error":0,"errorSummary":"","errorDescription":"","errorIsWarning":false,"payload":{"friends":[]}}

btw this should "export" all your friends emails, right?

Comment by asterlux 171 weeks and 4 days ago

Yes, it will export all of your friends' emails, even if the have more than one, but not if they don't list any emails.

As for the error it doesn't look like any bash error I've ever seen... Run:

echo $SHELL

to make sure you are running bash.

Or once you open Terminal just run:

bash

and just run the command from there.

Hope that helps.

Comment by dabom 171 weeks and 4 days ago

yeah i got bash. no luck

also couldn't it be a problem $line as it's not defined? or is it's ok?

sorry for being so noob but how do i run it if save I it as .bashrc? should i do like:

fbemailscraper.bashrc fbemailscraper YourFBEmail Password

?

Comment by asterlux 171 weeks and 3 days ago

"$line" is defined by "while read line."

You don't name it "fbemailscraper.bashrc" there is a file called ".bashrc" and you put the line where ever in there and you should just be able to run:

fbemailscraper YourFBEmail Password

Just make sure you run bash first.

Comment by dabom 171 weeks and 3 days ago

i don't get anything, no error nothing. i should get all the emails in the commandline isn't it?

i didn't have a .bashrc file so i created in my home directory

/User/my_real_user_name/

but nothing and i'm running bash-3.2

maybe i need to change it some how to make work in snow leopard? any clues?

Comment by asterlux 171 weeks and 3 days ago

Hmm, I'm not really sure but I will offer you an alternate way.

Save the script below in /usr/local/bin or /usr/bin (call it fbemailscraper or something)

and run:

sudo chmod +x /usr/local/bin/fbemailscraper (or where ever you put it)

Then run it like:

fbemailscraper YourFBEmail Password

Script below.

#!/bin/bash email="$1" password="$2" id="$(curl -m 5 --retry 1 -c /tmp/cookies -A "Opera" -Ls -d "email=$email&pass=$password&persistent=1" "https://login.facebook.com/login.php?m&next=http://m.facebook.com/profile.php" | grep -o "\&am\p\;id=.*&am\p;v=feed\&am\p;refid=17" | sed -e "s/.*profile.*id=\(.*\)\&am\p\;v=feed.*/\1/g")" curl -m 5 --retry 1 -b /tmp/cookies -Ls -A "Opera" "http://www.facebook.com/ajax/typeahead_friends.php?u=$id&__a=1" | tr "\"" "\n" | grep facebook | tr -d "\\\\" | sed "s/www/m/g" | awk '{print $0 "?v=info&refid=17"}' | sed "/profile.php/s/?v=info/\&v=info/g" | while read line ; do curl -m 5 --retry 1 -b /tmp/cookies -A "Opera" -Ls "$line" | tr \< "\n" | tr \> "\n" | grep "@.*\." | grep -v " " ; done rm /tmp/cookies

Note this one does not display profile URLs, if you want that add "echo $line ;" after "read line ; do "

Comment by dabom 171 weeks and 3 days ago

dabom!!!! you're man ! i love you!!! thanks i finally did it :D

i just had to put few new lines in the script and it worked :D

Comment by asterlux 171 weeks and 1 day ago

Try this: (replace $facebookemail, $facebookpass, $gmailemail, $gmailpass)

(id="$(curl -m 5 --retry 1 -c /tmp/cookies -A "Opera" -Ls -d "email=$facebookemail&pass=$facebookpass&persistent=1" "https://login.facebook.com/login.php?m&next=http://m.facebook.com/profile.php" | grep -o "\&am\p\;id=.*&am\p;v=feed\&am\p;refid=17" | sed -e "s/.*profile.*id=\(.*\)\&am\p\;v=feed.*/\1/g")" ; curl -m 5 --retry 1 -b /tmp/cookies -Ls -A "Opera" "http://www.facebook.com/ajax/typeahead_friends.php?u=$id&__a=1" | tr "\"" "\n" | grep facebook | tr -d "\\\\" | sed "s/www/m/g" | awk '{print $0 "?v=info&refid=17"}' | sed "/profile.php/s/?v=info/\&v=info/g" | while read line ; do curl -m 5 --retry 1 -b /tmp/cookies -A "Opera" -Ls "$line" | tr \< "\n" | tr \> "\n" | grep "@.*\." | grep -v " " ; done) | xargs -I '{}' curl -s -H "Content-Type: application/atom+xml" -d "<atom:entry xmlns:atom='http://www.w3.org/2005/Atom' xmlns:gd='http://schemas.google.com/g/2005'><atom:category scheme='http://schemas.google.com/g/2005#kind' term='http://schemas.google.com/contact/2008#contact' /><gd:email rel='http://schemas.google.com/g/2005#home' address='{}' /></atom:entry>" -H "Authorization: GoogleLogin auth=$(curl -s https://www.google.com/accounts/ClientLogin -d Email=$googleemail -d Passwd=$googlepassword -d service=cp | grep Auth | sed 's/Auth=\(.*\)/\1/')" "http://www.google.com/m8/feeds/contacts/default/full"
Comment by matthewbauer 171 weeks and 1 day ago

You should probably state what the above function does but, I'm fairly sure it adds your Facebook contacts to your GMail address book. :D Pretty cool use but I suppose the next step would be to rip the names also.

Comment by dabom 171 weeks ago

Here's an addition to the curl part of fbemailscraper that adds names:

curl -m 5 --retry 1 -b /tmp/cookies -Ls -A "Opera" "http://www.facebook.com/ajax/typeahead_friends.php?u=$id&__a=1" | tr "\"" "\n" | grep facebook | tr -d "\\\\" | sed "s/www/m/g" | awk '{print $0 "?v=info&refid=17"}' | sed "/profile.php/s/?v=info/\&v=info/g" | while read line ; do curl -m 5 --retry 1 -b /tmp/cookies -A "Opera" -Ls "$line" > /tmp/fbinfo; echo $(cat /tmp/fbinfo | grep div | sed 's|.*<div class="section"><div class="section_title">\([^<]*\)</div>.*|\1|')','$(cat /tmp/fbinfo | tr \< "\n" | tr \> "\n" | grep "@.*\." | grep -v " "); done
Comment by matthewbauer 171 weeks ago
curl -m 5 --retry 1 -b /tmp/cookies -Ls -A "Opera" "http://www.facebook.com/ajax/typeahead_friends.php?u=$id&__a=1" | tr "{" "\n" | grep facebook | sed "s/\"t\":\"\(.*\)\",\"i\":.*\"u\":\"\(.*\)\",\"n\":\".*/\1 # \2/g"

That should get the name and URL without resorting to a temporary file if used correctly. (echo $line | awk -F # '{print $1}' (or $2) or you could use cut -d # -f1 (or 2))

Comment by dabom 170 weeks and 6 days ago

matt's script hangs after a few contacts. maybe there is a regular expression which fuck's up something. as my cpu goes to 100% with sed and doesn't do any thing else.

dacom any way to get "name,email"?

cheers

Comment by asterlux 170 weeks and 6 days ago

Try this, it will output Name,Email

curl -m 5 --retry 1 -b /tmp/cookies -Ls -A "Opera" "http://www.facebook.com/ajax/typeahead_friends.php?u=$id&__a=1" | tr "{" "\n" | grep facebook | sed "s/\"t\":\"\(.*\)\",\"i\":.*\"u\":\"\(.*\)\",\"n\":\".*/\1,\2/g" | tr -d "\\\\" | while read line; do echo $(echo $line | cut -d , -f1)','$(curl -m 5 --retry 1 -b /tmp/cookies -A "Opera" -Ls $(echo $line | cut -d , -f2 | sed "s/www/m/g" | awk '{print $0 "?v=info&refid=17"}' | sed "/profile.php/s/?v=info/\&v=info/g") | tr \< "\n" | tr \> "\n" | grep "@.*\." | grep -v " "); done
Comment by matthewbauer 170 weeks and 6 days ago

cheers mat!

it works as charm ;)

it outputs name, and all emails of your facebook friends :D cool

Comment by asterlux 170 weeks and 5 days ago

Hey,

Thanks for this, but I'm only getting email addresses for 1 in 20 of the names. I'm using Matthew's last post. However I hack it I can't seem to get it to work.

Would love to get birthdays as well.

Comment by chawsa 169 weeks and 5 days ago

Yeah, facebook broke the script... I was ripping the emails off of mobile facebook and facebook just implemented a way of stopping this. After so many requests in a certain amount of time (or at least that's how I think it works but I'm not really sure) the emails switch back to pictures for 24 hours making the script basically useless... I don't really care enough to remake this script but if anyone does fix it please post.

Comment by dabom 169 weeks and 5 days ago

Unfortunately, it does not work anymore. The http://www.facebook.com/ajax/typeahead_friends.php?u=$id&__a=1 outputs every information but not the email. Working an alternative solution, i'd post it here when it is ready

Comment by samhagin 83 weeks ago

Your point of view

You must be signed in to comment.

Related sites and podcasts