Dynamic JSON processing in WinRT

Windows 8 is around the corner and I figured, it can’t hurt to dive into some app development for the new version of Windows. And what a version it is, a completely new development platform for developers to work on and a lot of changes to the UI and the underlying APIs. I personally love it, but I can imagine that there’s a lot of people out there that hate it right now, because their stuff is moved.

In this article I want to spend a little bit of time on something fun I wrote to make working with the Facebook API a lot easier. While not a complete sample on building Windows 8 apps, I think you will still find this information useful for writing your own apps that use a JSON API.

Meet the JSON object family

JSON is hot on the internet right now. It’s the JavaScript Object Notation, if it looks completely new to you, take a closer look. It’s just a piece of javascript describing one or more objects, arrays and their members. This serialization format has been around for quite a while as it is the standard way for Javascript to serialize objects. Today however it is mostly used in REST API’s that are used from websites. It makes things quite easy for developers as they get a format that is quite native to Javascript, thus reducing the amount of serialization problems.

For Windows 8/WinRT apps it’s quite a different story. Depending on what language you use there’s support ranging from excellent to rather funky. The support for JSON in the Javascript/HTML5 API for WinRT is excellent, .NET support is rather funky. In .NET you get the System.Data.Json namespace which contains the JsonObject, JsonArray and JsonValue components. With these you can generate and read JSON data.

What makes the JsonObject, JsonArray and JsonValue components so funky is that they are clay-like objects. For example, take the JsonObject class. Properties that were read from a JSON source are not regular properties, but can only be accessed through an indexer as the following snippet demonstrates.

var parsedValue = JsonValue.Parse("{"name":"Willem"}");
var parsedObject = parsedValue.GetObject();

var firstName = parsedObject["name"].GetString();

The value returned is not something like a String or int, but a JsonValue object. Which could be anything basically. It could be an array or another object. Or it could be a simple value like string, boolean or a number. The type of value contained in the JsonValue object is determined by its ValueType property.

Reading data from a JSON source is pretty straightforward, you can use the Parse method on the JsonValue class to get a parsed JSON value. Again, the end result depends on what you are parsing and could range from an object to a string. The JsonValue class supports reading every piece of JSON, so you don’t need to worry about the actual shape of the JSON being parsed.

Serializing a JsonObject, JsonArray or JsonValue is also straightforward. Call the Stringify method on it and you will get the typical JSON output that you can send to any JSON enabled REST service.

Why the standard API isn’t that great

As simple as it looks, the API isn’t really what I would call ergonomical to use. Initially you don’t have to worry about the shape of the JSON data you are reading, but as soon as you want to convert that parsed data into something useful for your app you have to think about how to access members on an object. Typically you are doing a lot of stuff like GetObject(), GetString() or GetArray() on a set of objects. This ends up looking something like this.

var jsonValue = JsonValue.Parse(feedResult);
var feedResultObject = jsonValue.GetObject();

if(feedResultObject.ContainsKey("error"))
{
    throw new Exception("An error ocurred while reading the contents of the feed: " + 
        feedResultObject["error"].GetString());
}

var feedData = feedResultObject["data"].GetArray();

foreach(var feedItem in feedData)
{
    var newsItem = new NewsItem();

    newsItem.Id = feedItem["id"].GetString();
    newsItem.DatePosted = DateTime.Parse(feedItem["updated_time"]);

    if(feedItem.ContainsKey("message")) 
    {
        newsItem.Message = feedItem["message"].GetString();
    }

    _newsItems.Add(newsItem);
}

That is a lot of code for something simple like getting an array of data from a response object, followed by retrieving the message property and the ID property of each item. Maintainability is very low on this kind of code and the developer frustration (ergonomy is lower) is very high on this piece of code.

Improving JSON on Windows 8

What I actually wanted to do as a developer is to write the previous example as I’ve done in the sample below.

dynamic feedResultObject = DynamicJsonValue.Parse(feedResult);

if(feedResultObject.error != null)
{
    throw new Exception("An error ocurred while reading the contents of the feed: " + 
        feedResultObject.error.message);
}

foreach(var feedItem in feedResultObject.data)
{
    var newsItem = new NewsItem();

    newsItem.Id = feedItem.id;
    newsItem.DatePosted = DateTime.Parse(feedItem.updated_time);

    if(feedItem.message != null) 
    {
        newsItem.Message = feedItem.message;
    }

    _newsItems.Add(newsItem);
}

Much cleaner and easier to write for me. And because I can still check members are there I don’t have to worry about missing members or things like that.

The way this API works is that you either create a new DynamicJsonObject/DynamicJsonArray or simply use DynamicJsonValue.Parse(input)  to get a new JSON object in your app. Make sure you store the result in a dynamic and you can access it like any other normal .NET object.

Internals of the API

The dynamic JSON API I created for my own apps is based on the use of DynamicObject. That way I enable the use of dynamic variables which make the API a lot easier to use. Basically what happens is that the original JsonValue is wrapped in a DynamicJsonObject or a DynamicJsonArray object. Each has it’s own specific implementation for handling various coding scenarios.

Handling JsonObject with DynamicJsonObject

Typical thing you want to do with an object is to get and set properties on them. The DynamicJsonObject class supports these scenarios by overriding the TryGetMember and TrySetMember methods.

public override bool TryGetMember(GetMemberBinder binder, out object result)
{
    // Javascript supports accessing non-existing members. 
    // They simply return a undefined value. So we're going to return a null here.
    // That should keep things simpler.
    if (!_original.ContainsKey(binder.Name))
    {
        result = null;
        return true;
    }

    if (TryConvertValue(binder.ReturnType, _original[binder.Name], out result))
    {
        return true;
    }

    return base.TryGetMember(binder, out result);
}

public override bool TrySetMember(SetMemberBinder binder, object value)
{
    // Process "real" JSON objects by using their value directly.
    // There's no need to process those any further, as they fit nicely to the JSON object.
    if (value is IJsonValue)
    {
        _original[binder.Name] = (IJsonValue)value;
        return true;
    }

    // Dynamic provider wrapped JSON objects can also be processed as if they were real JSON values.
    // The only thing we need to do is to grab the original object from the wrapper.
    if (value is IDynamicJsonProvider)
    {
        _original[binder.Name] = ((IDynamicJsonProvider)value).Original;
        return true;
    }

    // Use the JSON parse functionality to convert the value if any of the value processing
    // options above doesn't cut it.
    _original[binder.Name] = ConvertToJsonFormat(value);
    return true;
}

What’s important to know about the dynamic API in C# is that in each of the operations you override on DynamicObject you get a binder that contains the call data that was generated during invocation of the object. Things like the expected return type, the name of the member and other settings are contained in the binder.

For normal dynamic usage, you would need to find out what the user wanted and make sure, that the things that you return fit to that expectation. I opted for a different approach, since fitting everything to the expectation of the caller is going to be a tough job when you only got number, string, boolean, object and array. So instead of looking at the original wrapped object and check whether the value I want to return is either a string, boolean, object or array and return the appropriate thing for that. Although this will result in problems when someone is doing funky stuff, it’s the best I can do with what I got without getting into all kinds of weird serialization problems and things like that.

Another thing that is important to remember when you try to implement things like this, is that you have to return true inside a method like TryGetMember when you were able to process the request. This lets the runtime know all is well. If you do return false, the runtime will give a bind error telling the user the member does not exist. This works for all the Try… overrides you will find on DynamicObject.

A cool trick to help yourself with debugging dynamic objects is to override the GetDynamicMembers method. This method returns all the members that are dynamically generated on the object. On DynamicJsonObject this returns all the properties available to the caller.

public override IEnumerable<string> GetDynamicMemberNames()
{
    yield return "ContainsKey";

    foreach (var key in _original.Keys)
    {
        yield return key;
    }
}

There’s a few other bits of code  involved in the implementation of DynamicJsonObject, but I suggest you look those up on github if you’re interested in the dirty details.

Handling JsonArray with DynamicJsonArray

The JsonArray is a different type of JSON object I have writting dynamic support for. This object is actually an enumerable array containing a number of elements. Each element could be a different thing, an object, number, string, boolean or yet another array. The big difference compared to the DynamicJsonObject is that this time the TryGetIndex and TrySetIndex methods are overridden.

public override bool TryGetIndex(GetIndexBinder binder, object[] indexes, out object result)
{
    CheckArrayIndexers(indexes);

    if ((int)indexes[0] >= _original.Count)
    {
        throw new ArgumentException("The provided index was not found in the array");
    }

    return GetArrayValue(binder.ReturnType, _original, (int)indexes[0], out result);
}

public override bool TrySetIndex(SetIndexBinder binder, object[] indexes, object value)
{
    CheckArrayIndexers(indexes);

    if ((int)indexes[0] >= _original.Count)
    {
        throw new ArgumentException("The provided index was not found in the array");
    }

    // Process "real" JSON objects by using their value directly.
    // There's no need to process those any further, as they fit nicely to the JSON object.
    if (value is IJsonValue)
    {
        _original[(int)indexes[0]] = (IJsonValue)value;
        return true;
    }

    // Dynamic provider wrapped JSON objects can also be processed as if they were real JSON values.
    // The only thing we need to do is to grab the original object from the wrapper.
    if (value is IDynamicJsonProvider)
    {
        _original[(int)indexes[0]] = ((IDynamicJsonProvider)value).Original;
        return true;
    }

    _original[(int)indexes[0]] = ConvertToJsonFormat(value);
    return true;
}

Within these methods you get a binder and a set of indexers you can use. The set of indexers can contain anywhere from one indexer to a number of indexers. I opted to raise an exception when the user provides more than one indexer. It’s not supported by javascript, so you won’t need it here either.

Other than just accessing elements in the array by index, there’s also going to be a need for adding and removing stuff. Supporting this scenario is done by overriding the TryInvokeMember method. If the name of the method invoked matches the right name the API will pick up the call and handle it as if the method were there.

public override bool TryInvokeMember(InvokeMemberBinder binder, object[] args, out object result)
{
    if (binder.Name == "Add")
    {
        if (binder.CallInfo.ArgumentCount != 1)
        {
            throw new ArgumentException("Invalid number of arguments supplied for this operation");
        }

        object inputValue = args[0];
        _original.Add(ConvertToJsonFormat(inputValue));

        result = null;
        return true;
    }

    if (binder.Name == "Clear")
    {
        _original.Clear();

        result = null;
        return true;
    }

    if (binder.Name == "RemoveAt")
    {
        if (binder.CallInfo.ArgumentCount != 1)
        {
            throw new ArgumentException("Invalid number of arguments supplied for this operation");
        }

        _original.RemoveAt((int)args[0]);

        result = null;
        return true;
    }

    return base.TryInvokeMember(binder, args, out result);
}

Make sure you check the CallInfo.ArgumentCount instead of looking at the size of the args array. The args array will be empty when all arguments were null. Still, the user could have passed in three null pointers if he wanted to. You can only see that by looking at the CallInfo object.

The last thing I want to show about the array implementation is the support for enumeration. The trick is quite simple, but you have to know how .NET handles the foreach statement. When you write a foreach statement, the runtime will try to cast the expression after the in keyword to IEnumerable. If that doesn’t succeed it will try to find the GetEnumerator method and use the enumerator that is returned instead. So the easiest way to support enumeration is to allow the dynamic object to be converted into IEnumerable.

public override bool TryConvert(ConvertBinder binder, out object result)
    {
        // Special case for the IEnumerable. 
        // Depending on the object, this needs to result in a different kind of enumeration.
        if (binder.ReturnType == typeof(IEnumerable))
        {
            result = this.Elements;
            return true;
        }

        // Always return true when the original value is a null-pointer. 
        // This works for anything and is a valid situation for the runtime.
        if (Original.ValueType == JsonValueType.Null)
        {
            result = null;
            return true;
        }

        // Please note that the conversion of JSON values is done without actually
        // looking at the expected return type. Although that might sound like a good idea,
        // it means that this code needs a lot more cases of which most are not supported.

        if (Original.ValueType == JsonValueType.String)
        {
            result = Original.GetString();
            return true;
        }

        if (Original.ValueType == JsonValueType.Number)
        {
            return ConvertNumber(binder.ReturnType, Original, out result);
        }

        if (Original.ValueType == JsonValueType.Boolean)
        {
            result = Original.GetBoolean();
            return true;
        }

        if (Original.ValueType == JsonValueType.Array)
        {
            result = new DynamicJsonArray(Original.GetArray());
            return true;
        }

        // Return a dynamic object when the return type requested is either object or dynamic.
        // We could return a dynamic for anything else, but that would break the application in a bad way.
        // Nobody wants a dynamic object when he is asking for a string right?
        if (Original.ValueType == JsonValueType.Object)
        {
            result = new DynamicJsonObject(Original.GetObject());
            return true;
        }

        // Return false when none of the conversions above seem to work. 
        // This ensures that no invalid combination of types and actions returns anything stupid.
        result = null;
        return false;
    } 

The conversion I wrote for IEnumerable support is implemented in the DynamicJsonValue class. This class serves as the basic structure for both DynamicJsonObject and DynamicJsonArray. Within the TryConvert method (which is an override of the version that is available on DynamicObject) there’s a check for the return type. If the expected return type is IEnumerable, the Elements property is returned as a result of the operation. Both the DynamicJsonObject and the DynamicJsonArray have this property and will return an enumeration of their contents.

Conclusion and resources

With the DynamicJson API you should be able to write REST enabled apps more easily on Windows 8. If you have any feedback, please leave a comment here or just create a pull request if you think that you have a good addition to the library

You can find the sources on github: https://github.com/wmeints/FizzyLogic.DynamicJson

Enjoy!