I have gotten lot of messages asking more details about how Bayesian spam filter works. When I posted Bayesian Spam Filter Trainer, i did not added reference to lot of wonderful literature that is available describing what Baye's theorem is and how is used to detect spam. So before I discuss how you can refine the process for twitter, let me give you a list of reference material.
I will quickly give an overview. For details I will definitely recommend reading articles that I mentioned above. Back bone of Bayesian approach is good and bad corpus to train the filter. In simple terms, we take a huge chunk of text, then split the text into individual words, remove the words that may not be of any interest and then calculate frequencies and based on that calculate probabilities of words appearing in a good or spam text.
Using standard corpus to train filter for twitter messages works for most part. I noticed success rate of close to 90%. Well that is not bad. The issue with Twitter messages is that you only have 140 characters to establish meaningful context. And out of that 140, good 20 to 30 characters are taken by compressed URLs that spammers will add to advertise their products or trick you into going to some advertisement affiliate redirect site. So we have about 100 characters available to us to detect spam. If you notice, lot of time a good and spam message looks very close to each other. That causes messages to fall through the cracks or you get false positives.
Here are some of the refinements that I added to my Spam filter service.
These are some of the techniques that I have deployed in current spam filter service. As I collect more data, these techniques will be refined further. Any suggestions are most welcome. I am learning too like my spam service.