Switching browser in CodedUI or Selenium tests based on MTM configuration part 2
More then a Year ago I wrote a blog post on how you could switch browsers based on the configuration settings in MTM. You can read more about that in this post here:http://bit.ly/1jC0Iht
So this solution unfortunately works partially and I have talked with the TFS team about this for a while but this is something that might take a while to get fixed. This week someone asked me again about this solution and I worked on a work around to get things running the way we need.
So first, what is the problem?
When you run one test case all will run fine. the Test Context will contain the right information and you can use the configuration name that is passed in the __Tfs_TestConfigurationName__ property from the TestContext.Properties collection.
But when you do a multi select of some test cases you want to run or you run a full test suite, you will run into the issue that the information being passed in the TestContext Properties collection becomes stale after the first test case is done. So if you look at the screenshot below, you can see that I picked 3 test cases to run:
When you look at the results (this test just dumps out the properties in the collection to see what is in there) you can see that for each run the configuration has the same value 
:
After reporting this back to the team, this was confirmed as a bug and was not fixed in the recent versions and updates.
How to solve the issue?
So how to solve this problem, so we can really switch the browsers based on the configuration in MTM?
the post here really helped me on my way to get this issue fixed: http://blogs.msdn.com/b/aseemb/archive/2012/08/07/code-snippets-on-test-management-apis.aspx (code snippet #7)
The solution is that we need to query the TFS server on the current test run and based on the current __Tfs_TestRunId__ that is the same for all iterations. Then we can ask for the Test Results for the current test that is running on the current machine. In the test results we can then get the Configuration information of the current running test and that will reflect the real configuration of the current iteration of the test we are running.
to make this work I needed to write up a bit more code then I anticipated, because the Test Result data is reported back to TFS once every 30 seconds and when you start a new run, you might get the data of the previous run, since TFS might lag behind a bit. So I needed to come up with a waiting pattern and only get the data when the test identifier changes, so I know we are in a new fresh run. The code I wrote makes one big assumption and that is that the tests are run in the same process, since I am using a static variable to contain the last test id. This is not pretty code, but it gets the job done.
Here is the code on querying the server and get the right configuration data:
public static string GetRealTestConfigName(TestContext tctx, string defaultBrowser)
{
// check of we are running in a MTM test context, if not backout with the default browser value
if (tctx.Properties.Contains("__Tfs_TestRunId__"))
{
// Get test run ID from current test run
int? testRunID = tctx.Properties["__Tfs_TestRunId__"] as int?;
// Get the test point so that you can find configuration etc from the test run.
string collectionName = tctx.Properties["__Tfs_TfsServerCollectionUrl__"].ToString();
string teamProjectName = tctx.Properties["__Tfs_TeamProject__"].ToString();
// we need to distinguise the machine we run on, since we can run the tests on different machines
string machineName = GetFQDNMachineName();
using (TfsTeamProjectCollection collection = new TfsTeamProjectCollection(new Uri(collectionName)))
{
// Get the ongoing run object
ITestManagementService tcmService = collection.GetService<ITestManagementService>();
ITestManagementTeamProject project = tcmService.GetTeamProject(teamProjectName);
ITestRun testRun = project.TestRuns.Find(testRunID.Value);
// Find the in-progress test. Since updates are pushed async to tfs (default interval is 30 sec)
// wait for this test to show up in the progress list, based on that we can query
// the test configuration and correct the bug in the way this data is passed to us in the
// test context, since testcontext only works for a single test
ITestCaseResultCollection testResults = null;
while (testResults == null)
{
testResults = testRun.QueryResultsByStatus(TestResultState.InProgress);
// we need to find the test result that depicts our current run on this server
if ((testResults != null) && (testResults.Count > 0))
{
var results = testResults.Where(t => t.ComputerName == machineName);
int id = results.First().TestPointId;
// if this is the first test case in a run, then I need to determine the first testPointID to see the testpoint change in consecutive runs
if (lastTestPointId == 0)
{
lastTestPointId = id;
break;
}
else
// now check on subsequent runs that we read the data that has to do with the new run
// this is indicated by a new testPointID, so if it is different from the previous run we can assume
// we have the right result and we can read that data. So break out of the wait loop
if (lastTestPointId != id)
{
lastTestPointId = id;
break;
}
}
// if we did not break, then we need to reset the while variable for next iteration
// we are going to wait for 1 second to query TFS again to see if we now get the data we are looking for
testResults = null;
Thread.Sleep(1000);
}
// so we found our testrun, now get the correct data from the results
// Find the result for current test & get test point from it. here
var testCaseResult = testResults.Where(t => t.TestPointId == lastTestPointId).FirstOrDefault();
// Now you can get more interesting data about configuration etc & use it in your test method.
Console.WriteLine("ConfigName = {0}, testPointId = {1}, MachineName = {2}", testCaseResult.TestConfigurationName, lastTestPointId, testCaseResult.ComputerName);
return testCaseResult.TestConfigurationName;
}
}
else
return defaultBrowser; // if we don't run in an agent, but e.g. local, just keep the default browser
}
private static string GetFQDNMachineName()
{
string domainName = System.Net.NetworkInformation.IPGlobalProperties.GetIPGlobalProperties().DomainName;
string hostName = Dns.GetHostName();
if (!hostName.Contains(domainName)) // if the hostname does not already include the domain name
{
hostName = hostName + "." + domainName; // add the domain name part
}
Console.WriteLine("fqdn machine name == {0}", hostName);
return hostName; // return the fully qualified domain name
}
After you have this code, you can now use this function to switch the browser for you in your test by passing it in the Testcontext.
For example in codedUI you can switch the browser now as follows:
BrowserWindow.CurrentBrowser = GetRealTestConfigName(TestContext);
If you now look at the results of the same test run you can see the code finds the right configuration. You can see in the screenshots below it will now really find the actual configuration we are running.
One issue I found with this solution is that your tests will experience a little delay, because we need to wait for the test run to show up in the TFS server, before we can query the data. But if you look at this you might not really find it a problem since it normally is only a small portion of the time spend on a test. These tests are not meant to be a fast unit test, so I guess we can live with it for now.
I sure hope this will eventually get fixed in the test infra structure. If that happens I will post back with that solution.
Hope this helps,
Marcel
Follow my new blog on http://fluentbytes.com