This project is read-only.

[Solution] Faster way to get lots of friend data

Oct 23, 2011 at 2:03 AM

I wanted to get custom data for friends. FB.User.GetFriends will return a IList<NameIDPair> but then we need to query Facebook for every friend to get specific data. To speed that up I wanted to batch requests. I believe Facebook allows a max of 20 batched queries per request. So if a user has something like 500 friends, it's better to make 25 requests of 20 than 500 requests.  So here is how  I query in batches of 20 for an app-specific group of fields.

IList<NameIDPair> friends = FB.User.GetFriends(userID , Api.AccessToken);
List<string> Fields = new List<string>
{
   "id","name","gender"
};
HashSet<Dictionary<string , string>> FriendData =
FB.User.GetFriendData(friends , Fields , Api.AccessToken); foreach ( var pair in FriendData ) { Log(pair[ "id" ] + " " + pair[ "name" ] + " " + pair[ "gender" ]); }

That query can also be made like this (contrived example to show sorting):

List<string> FriendIDs = ( from friend in friends
                           orderby friend.Name
                           select friend.ID
   ).ToList<string>();
HashSet<Dictionary<string , string>> FriendData =
FB.User.GetFriendData(FriendIDs , Fields , Api.AccessToken);

To make that happen, the following two methods need to be added to GraphApi.User:

public static HashSet<Dictionary<string , string>> GetFriendData( List<string> IDs , List<string> Fields , string AccessToken )
{
    return Helpers.ApiCaller.GetFriendData(IDs , Fields , AccessToken);
}
public static HashSet<Dictionary<string , string>> GetFriendData( IList<NameIDPair> Friends , List<string> Fields , string AccessToken )
{
    return Helpers.ApiCaller.GetFriendData(Friends , Fields , AccessToken);
}

And finally these supporting methods need to be added to Helpers.ApiCaller:

internal static HashSet<Dictionary<string , string>> GetFriendData( IList<NameIDPair> Friends , List<string> Fields , string AccessToken )
{
    List<string> FriendIDs = ( from friend in Friends
                               select friend.ID ).ToList<string>();
    return GetFriendData(FriendIDs , Fields , AccessToken);
}

internal static HashSet<Dictionary<string , string>> GetFriendData( List<string> IDs , List<string> Fields , string AccessToken )
{
    string fields = String.Join("," , Fields.ToArray());
    HashSet<Dictionary<string , string>> Data = new HashSet<Dictionary<string , string>>();
    const int SIZE = 20;
    for ( int batch = 0 ; batch < IDs.Count ; batch += SIZE )
    {
        #region Call for Facebook data in batches of SIZE items
        string[] idlist = new string[ SIZE ];
        int idct = 1;
        while ( idct + batch <= IDs.Count && idct <= SIZE )
        {
            idlist[ idct - 1 ] = IDs[ batch + ( idct - 1 ) ];
            idct++;
        }
        string ids = String.Join("," , idlist , 0 , idct - 1);
        string URL = string.Format("https://graph.facebook.com?access_token={0}&ids={1}&fields={2}" , AccessToken , ids , fields);
        string response = Helpers.WebResponseHelper.GetWebResponse(URL);
        JsonObject JO = new JsonObject(response);
        for ( int i = 0 ; i < IDs.Count ; i++ )
        {
            string id = IDs[ i ];
            JsonObject FriendJO = (JsonObject) ( JO[ id ] );
            if ( FriendJO != null )
            {
                Dictionary<string , string> friend = new Dictionary<string , string>();
                foreach ( string field in Fields )
                {
                    string data = (string) FriendJO[ field ];
                    friend.Add(field , data);
                }
                Data.Add(friend);
            }
        }
        #endregion
    }
    return Data;
}

What I don't like about that is that it's a synchronous call that will take 1-2 seconds per request to process on the server.  I think a well constructed app should spawn off a thread to do the requests while providing some sort of entertainment for the user, and check in async postbacks to see  if the result is ready.  Someone might want to re-work this to break up the query into pieces which can be completed on subsequent async callbacks.

If someone sees obvious places where that can be further optimized, please post here.  For example, I chose HashSet and Dictionary collections randomly and there might be better classes for this.

I'm hoping this code will get introduced into the core.  I think this is the first set of code that allows FGT to retrieve data in batches like this. Some variation of this code can probably be used for lots of other data types.

There's one final bit that bothers me.  If we request too many fields then the querystring might get too long.  I think it would be better to submit requests like this via HTTP POST, but I don't think FGT can do that yet.

Comments and suggestions are welcome.

Oct 24, 2011 at 4:07 PM

We can actually accomplish all this in ONE query ( ;

JsonArray JA = _Api.Fql("SELECT uid, name, pic_square FROM user WHERE uid IN (SELECT uid2 FROM friend WHERE uid1=me()) ORDER BY first_name");

foreach (JsonObject JO in JA.JsonObjects) {
   PersonDisplay PD = new PersonDisplay();
   PD.PictureUrl = JO["pic_square"].ToString();
   PD.Name = JO["name"].ToString();
   PD.UserID = JO["uid"].ToString();
}

This code is actually pasted from a page which displays all the user's friends. By using FQL query with a nested SELECT statement, we can accomplish this all in one function call.

 

 

 

Oct 25, 2011 at 1:35 AM

I like your solution and I'm sure I will use that for other purposes as I get more familiar with Facebook coding and FGT in particular.
I was just trying to copy your code style to extend the library in a consistent manner.  :)

I'm thinking the code I've provided helps to avoid some issues related to manually crafting FQL queries, and parsing Json.
That said, my code could still use a number of improvements, like an enum to replace literal field names, and overloads to allow returning a sorted list.

We can do anything we want with FQL and direct REST queries, but the whole purpose of the FGT seems to be to abstract app developers from the mechanics.
What do you think about using my User and ApiCaller methods (with some adjustments), and maybe just replacing the guts with your more effective code?  That can then be ported and enhanced over time into other methods that allow more versatility than my own more rigidly defined methods.  If nothing else this would eliminate the need for Json manipulation in app code.

I'll go with your experience on this.  At worst people can now see a couple ways to accomplish tasks, and there is code here which can be added to customized libs should someone wish to use it.  Ahh the wonders of FOSS. :)

For now I'll add your code into my project, keep refining it all, and we can take another look at this after some period of time.

Thanks!

Oct 25, 2011 at 5:07 AM

Your code is good, but I'm afraid it might make the code behind the toolkit too complex and difficult to maintain. We have to bear in mind that there are dozens of similar connections in the Facebook Graph Api structure, so if we implement it, we'll have to implement it all. What's worse is Facebook keep changing its Graph Api all the time, and their documentations are inaccurate most of the time. What I'd like to do is to balance between what is implemented in the library, and what will be implemented by developers. And I think we're doing good so far ( ;

JSON object manipulation is quite straight forward actually. It's almost the same as querying a standard database using SQL in ASP.NET. Consider the following:

SqlCommand SqlCom=new SqlCommand("SELECT [column0], [column1] FROM [tableX]", .......);
SqlDataReader dr=SqlCom.ExecuteReader();
while(dr.Read()){
    x=dr["column0"].ToString();
    y=dr["column1"].ToString();
    ......
}

We just loop through each element in the returned data set. It's the same with JSON objects above.

Oct 25, 2011 at 6:17 AM

Very well then.  I'll come up with a few more examples based on this pattern.  Thanks again!