Download Source Code For Server Control (3.38 kb)

Download Assemblies For Server Control (5.26 kb)

For quite some time I have been using Tweet This button on this blog and on some of client's customized blogs. Yesterday when I tried to use that button to tweet about new posts I saw that Twitter has changed their handling of URL that used to populate the tweeter box to post a tweet. So I went to their developer support pages and noticed that Twitter has updated their APIs and provider few more options now to share the URLs on tweeter. For most part you can just put their two lines of javascript on every page and twitter will pick up the URL of that page along with title of the page and create a message for you to post as a tweet.

But if you want to have more control on the message along with the URL then you will need to be little bit work on how the HTML is constructed on your page. For example in my implementation of BlogEngine I have two sets of titles for each post. One title that is used to construct URL and second is the title that actually gets put in the TITLE tag of the page itself. Now you can see that default javascript may not work for me. Also from time to time I like to include my tweeter screen name in the tweet itself.

I have created an ASP.Net server control that you can deploy in any ASP.Net application including BlogEngine.Net. The implementation renders following HTML on the page as described in Twitter Documentation.

<script src="http://platform.twitter.com/widgets.js" type="text/javascript"></script>
<div>
  <a href="http://twitter.com/share" class="twitter-share-button"
      data-url="http://dev.twitter.com/pages/tweet_button"
      data-via="your_screen_name"
      data-text="Checking out this page about Tweet Buttons"
      data-related="anywhere:The Javascript API"
      data-count="vertical">Tweet</a>
</div>

How to include Tweet Button In BlogEngine Posts

• Compile the attached project or use the pre-compiled binaries for the control and copy them in BIN folder of your application.

• Add Register tag at the top of your page or view's control that refers to this control's assembly. For example on my blog this looks like as below.

  <%@ Register TagPrefix="ByteBlocks" Namespace="ByteBlocks.TweetThisButton" Assembly="ByteBlocks.TweetThisButton" %>
  

• Add instance(s) of control on your post or page as shown below.

  <ByteBlocks:TweetItButton ID="TweetPostButton" runat="server" CounterOrientation="Vertical" />
  

• Now set the properties of the control to specify tweet text and URL. For example I did this in code in PostView of my theme as shown below.

  public override void RenderControl(HtmlTextWriter writer)
  {
     TweetPostButton.TweetUrl = Post.AbsoluteLink.AbsoluteUri;
     TweetPostButton.TweetText = Post.MetaTitle;
     base.RenderControl(writer);
  }
  

Now you are all set with Tweet Button control in your post. Following screen shot shows how it looks. You can actually see it at bottom of this post as well and give it a try and see how it works!

I have gotten lot of messages asking more details about how Bayesian spam filter works. When I posted Bayesian Spam Filter Trainer, i did not added reference to lot of wonderful literature that is available describing what Baye's theorem is and how is used to detect spam. So before I discuss how you can refine the process for twitter, let me give you a list of reference material.

• Bayesian spam filtering
• Baye's Theorem
• A Plan For Spam
• Better Bayesian Filtering
• A Bayesian Approach to Filtering Junk E-mail

How does it work?

I will quickly give an overview. For details I will definitely recommend reading articles that I mentioned above. Back bone of Bayesian approach is good and bad corpus to train the filter. In simple terms, we take a huge chunk of text, then split the text into individual words, remove the words that may not be of any interest and then calculate frequencies and based on that calculate probabilities of words appearing in a good or spam text.

Small context makes it harder

Using standard corpus to train filter for twitter messages works for most part. I noticed success rate of close to 90%. Well that is not bad. The issue with Twitter messages is that you only have 140 characters to establish meaningful context. And out of that 140, good 20 to 30 characters are taken by compressed URLs that spammers will add to advertise their products or trick you into going to some advertisement affiliate redirect site. So we have about 100 characters available to us to detect spam. If you notice, lot of time a good and spam message looks very close to each other. That causes messages to fall through the cracks or you get false positives.

Refine the filter

Here are some of the refinements that I added to my Spam filter service.

• The filter has been trained with live twitter messages. More data you use to train Bayesian filter, better results you will get.
• Manually go through results and classify messages as good and spam
• Make sure that sender's screen name is included in the corpus. This is based on the discussion in A Plan For Spam where it is mentioned that it is very important that you include an email's header, sender name etc. in corpus. Same applies to twitter corpus as well. You do not have elaborate meta data available. Including screen name in corpus makes sure that spammer's account gets included in spam corpus. So in edge cases where message body may be ambiguous, screen name may end up breaking the tie and correctly classify the text.
• Link included in message helps in cleaning up some corpus. The service does some post processing on the messages in the background. It checks each message for any links posted in it. And then it expands short URLs into actual URLs and then gets some meta data about the target site. It looks at information like title, keywords, description etc. to establish if the user is attempting to redirect to adult content or key loggers or other questionable sites. Based on the results, the message is re-classified.
• There is always something new to learn. Yes, the spam filter can never have enough of training. The service continuously updates the corpora and reloads new statistical data.

These are some of the techniques that I have deployed in current spam filter service. As I collect more data, these techniques will be refined further. Any suggestions are most welcome. I am learning too like my spam service.

Download SpamTrainer Binaries

Download SpamTrainer Source

As more and more people are tweeting, spam is growing with it as well. Every time I search for some topic, almost half of the messages seem to fall in one of the following categories:

• Somebody is trying to sell something
• Somebody is posting links to get you affiliate web sites to make some money
• Job agencies are posting jobs
• .... and more

This week I decided to use Bayesian spam filter, that is used in most email servers to filter spam, on twitter messages. While searching around I found Bayesian Spam Filter for C#. That gave a good starting point. Without making any changes or training with any additional corpus, I was able to get very good filtering results. I observed close to 90% spam detection. I studied the messages that fell through the cracks and also studies false positives. Based on the observations I figured that issue is very limited context of 140 characters in twitter. A lot of good and spam twitter messages look pretty much the same. So the key to improving spam filtering results was to train the filter with twitter messages and not use just rely on corpus taken from emails or things like that. So I decided to build an application that I could use to generate corpus that is classified as spam and good twitter messages.

How does it work

• Start the application.

  

• Enter a search term and click on "More Data" button.
• Application will do initial classification of messages. All spam messages are displayed in Orange or light blue color.
• Double on any message to change its classification.

  

• Once you are satisfied with the results, click on "Accept" button and results are saved in appropriate good and spam files.
• You can load the new corpus results by clicking on "Reload Corpus".

Spam Filter Service

I have created a service that you can use to classify your text if you do not want to build one of your own. Following link provides more details about the service.

Spam Filter Service

In my earlier posts Daily twitter trends and Weekly twitter trends I showed how you can get daily and weekly twitter trends. That api gets you to queries that are being searched for. Next step in the process is to take those queries and get the messages posted related to those queries. Here is a simple method that combines results of trend api with search api to get popular messages on twitter.

static Dictionary<string, List<TwitterSearchStatus>> GetPopularTweets()
{
 List<TwitterSearchTrend> trends = GetDailyTrends();
 if (null == trends ||
     trends.Count == 0)
 {
  return null;
 }
 // For now only process first trend term.
 List<TwitterSearchStatus> searchResults = SearchForTerm(trends[0].Query);
 if (searchResults.Count == 0)
 {
  return null;
 }
 var resultsGrpByUser = from searchResult in searchResults
   group searchResult by searchResult.FromUserScreenName
   into userMessages
 select new { FromUser = userMessages.Key, Messages = userMessages };
 Dictionary<string, List<TwitterSearchStatus>> dict = 
  new Dictionary<string, List<TwitterSearchStatus>>();
 foreach(var userMessage in resultsGrpByUser)
 {
  Console.WriteLine(userMessage.FromUser);
  foreach(var twitterStatus in userMessage.Messages)
  {
   Console.WriteLine("\t{0}", twitterStatus.Text);
  }
  dict[userMessage.FromUser] = userMessage.Messages.ToList();
  Console.WriteLine("*************");
 }
 return dict;
}

While working on new twitter marketing tool Marketweet, I ran into a situation. I need to get count of people who a person is following. There is an API that gets you a user's profile. But if you notice the results you will find that results are not current. Some time they are cached for hours. So I was tried few APIs. Finally I found the one that does return current information. You can get IDs of friends from a user's Social Graph. There are 2 main issues with this approach as well.

• You do not know how many total results will be returned.
• What is maximum number of results per page that can be returned by this API

So to work around this issue. Here is what I ended up doing. One I set an upper limit of 100000 on number of users that an account may be following. Also grabbed user's cached profile as well and from there got number of friends. And ended up using maximum of these two numbers. So when the API runs out of more results, it returns response that translates to empty collection. And you can use that as sentinel condition to break out of iterative loop.

static List<TwitterUser> GetFollowings()
{
 IFluentTwitter ft = FluentTwitter.CreateRequest();
 ft.AuthenticateAs(LOGIN_NAME, PASSWORD);
 var profileReq = ft.Users().ShowProfileFor(LOGIN_NAME).AsJson();
 var thisUser = profileReq.Request().AsUser();
 var followersCount = thisUser.FriendsCount;
 var upLimit = (followersCount < 100000) ? 100000 : followersCount;
 var ceiling = Math.Ceiling(upLimit / 100m);
 var results = new List<TwitterUser>();
 for (var i = 1; i <= ceiling; i++)
 {
  var friendsResp = ft.Users().GetFriends().For(thisUser.Id)
	.Skip(i).AsJson().Request();
  if (null == friendsResp)
  {
   break;
  }
  IEnumerable<TwitterUser> friends = friendsResp.AsUsers();
  if (friends == null ||
   friends.Count() == 0)
  {
   break;
  }
  results.AddRange(friends);
 }
 return results;
}

This code sample will show how you can get weekly trends using REST API. The sample shows another feature of the API where you can specify a criteria to limit the results. You can specify a start date for the report. This is set by calling On method on ITwitterSearchTrends.

static List GetWeeklyTrends(DateTime ? onDate)
{
 TwitterClientInfo clientInfo = new TwitterClientInfo();
 clientInfo.ClientName = "Marketweet";
 clientInfo.ClientUrl = "http://www.byteblocks.com";
 clientInfo.ClientVersion = "1.1";
 List trendNames = new List();
 IFluentTwitter ft = FluentTwitter.CreateRequest(clientInfo);
 ft.AuthenticateAs(LOGIN_NAME, PASSWORD);
 ITwitterSearchTrends searchTrends = ft.Search().Trends().Weekly();
 if (null != onDate)
 {
  searchTrends.On(onDate.Value).AsJson();
 }
 var resp = ft.Request();
 if (null != resp)
 {
  var trends = resp.AsTrends();
  if (null != trends)
  {
   foreach (TwitterSearchTrend trend in trends.Trends)
   {
    trendNames.Add(trend.Name);
   }
  }
 }
 return trendNames;
}

This code sample will show how you can use Twitter REST API to verify if one twitter user is following a specified user or not. The use of this API is little tricky. It requires you to specify two screen names or IDs that you want to verify. There is Exists method on Friendships object in Tweetsharp API but it only takes one parameter. If you try to use the API just like that it will return error telling you that you need to specify two screen names or ids to verify. The use of this API is not very intutive. So I had to dig down into the code to figure out how it can be done. There is property VerifyScreenName on IFluentTwitterParameters. You have to set second screen name here. See how it is done in code below.

static void VerifyFriendShip(string screenName1, string screenName2)
{
 IFluentTwitter ft = FluentTwitter.CreateRequest();
 ft.AuthenticateAs(LOGIN_NAME, PASSWORD);
 ft.Friendships().Verify(screenName1);
 ft.Parameters.VerifyScreenName = screenName2;
 var resp = ft.Request();
 if (null != resp)
 {
  Console.WriteLine(resp);
 }
}

Return value from this API is true or false.

This code sample will demonstrate how you can get list of people who are following you on twitter using Twitter REST API. The reason I mentioned "following you" because this API requires authentication. So unless you have credentials of other users, you can execute this API against the authenticated user account only.

static void GetMyFollowers()
{
 IFluentTwitter ft = FluentTwitter.CreateRequest();
 ft.AuthenticateAs(LOGIN_NAME, PASSWORD);
 ft.Users().GetFollowers().AsJson();
 var resp = ft.Request();
 if (null != resp)
 {
  var users = resp.AsUsers();
  if (null != users)
  {
   foreach (TwitterUser user in users)
   {
    Console.WriteLine("User Id: {0}, Screen Name: {1}", user.Id, user.ScreenName);
   }
  }
 }
}

In this code sample I will demonstrate how you can use twitter API to add a new status message to your account. In terms of Twitter API it is termed as Update of your status.

static void AddStausMessage(string msg)
{
 IFluentTwitter ft = FluentTwitter.CreateRequest();
 ft.AuthenticateAs(LOGIN_NAME, PASSWORD);
 ft.Statuses().Update(msg).AsJson();
 var resp = ft.Request();
 if (null != resp)
 {
  var status = resp.AsStatus();
  if (null != status)
  {
   Console.WriteLine("Status Id: {0}",status.Id);
   Console.WriteLine("Message: {0}", status.Text);
  }
 }
}

When API executes successfully it returns status object in response. I tested this method using this call AddStausMessage("Publishing more code samples showing to use Twitter API");. You can see this on my profile on twitter by clicking HERE

In this code snippet I will show you how you can remove a twitter user from your follow list. Twitter API calls it destroying friendship.

static void RemoveFromFollow(string userName)
{
 IFluentTwitter ft = FluentTwitter.CreateRequest();
 ft.AuthenticateAs(LOGIN_NAME, PASSWORD);
 ft.Friendships().Destroy(userName);
 var resp = ft.Request();
 if (null != resp)
 {
  TwitterUser user = resp.AsUser();
  if (null != user)
  {}
 }
}