Refine bayesian spam filter for Twitter Messages

I have gotten lot of messages asking more details about how Bayesian spam filter works. When I posted Bayesian Spam Filter Trainer, i did not added reference to lot of wonderful literature that is available describing what Baye's theorem is and how is used to detect spam. So before I discuss how you can refine the process for twitter, let me give you a list of reference material.

• Bayesian spam filtering
• Baye's Theorem
• A Plan For Spam
• Better Bayesian Filtering
• A Bayesian Approach to Filtering Junk E-mail

How does it work?

I will quickly give an overview. For details I will definitely recommend reading articles that I mentioned above. Back bone of Bayesian approach is good and bad corpus to train the filter. In simple terms, we take a huge chunk of text, then split the text into individual words, remove the words that may not be of any interest and then calculate frequencies and based on that calculate probabilities of words appearing in a good or spam text.

Small context makes it harder

Using standard corpus to train filter for twitter messages works for most part. I noticed success rate of close to 90%. Well that is not bad. The issue with Twitter messages is that you only have 140 characters to establish meaningful context. And out of that 140, good 20 to 30 characters are taken by compressed URLs that spammers will add to advertise their products or trick you into going to some advertisement affiliate redirect site. So we have about 100 characters available to us to detect spam. If you notice, lot of time a good and spam message looks very close to each other. That causes messages to fall through the cracks or you get false positives.

Refine the filter

Here are some of the refinements that I added to my Spam filter service.

• The filter has been trained with live twitter messages. More data you use to train Bayesian filter, better results you will get.
• Manually go through results and classify messages as good and spam
• Make sure that sender's screen name is included in the corpus. This is based on the discussion in A Plan For Spam where it is mentioned that it is very important that you include an email's header, sender name etc. in corpus. Same applies to twitter corpus as well. You do not have elaborate meta data available. Including screen name in corpus makes sure that spammer's account gets included in spam corpus. So in edge cases where message body may be ambiguous, screen name may end up breaking the tie and correctly classify the text.
• Link included in message helps in cleaning up some corpus. The service does some post processing on the messages in the background. It checks each message for any links posted in it. And then it expands short URLs into actual URLs and then gets some meta data about the target site. It looks at information like title, keywords, description etc. to establish if the user is attempting to redirect to adult content or key loggers or other questionable sites. Based on the results, the message is re-classified.
• There is always something new to learn. Yes, the spam filter can never have enough of training. The service continuously updates the corpora and reloads new statistical data.

These are some of the techniques that I have deployed in current spam filter service. As I collect more data, these techniques will be refined further. Any suggestions are most welcome. I am learning too like my spam service.