Windmill Reporting

Adam wrote a great post recently on some code he wrote that outputs his test results in an xml format that Hudson can digest. It serves as an example of a fairly underutilized feature in functest for reporting. It’s incredibly simple and powerful, you add a simple class to the highest level __init__.py with your report code and it will be called when all the tests have finished. Ideally you want to checkin your reporting mechanism with the source for all your tests but only run the reporter when it runs in continuous integration. Another functest feature is that any unrecognized command line = arguments are stuffed in to dictionary at functest.registry, and there is code in the windmill command line that will do the same.

$ windmill shell report=true                              [14:15]
report is not a windmill argument. Sticking in functest registry.
 
In [1]: import functest
In [2]: functest.registry
Out[2]: {'report': 'true'}

You can use this in your Report class to decide when you should actually report, you can even do different types of reporting based on what you pass. I wrote an example by refactoring Adam’s earlier Hudson example, I’ve also sped up some of the code and use ElementTree to write the xml instead of writing the string output by hand.

import functest
from functest import reports
from xml.etree import ElementTree
 
class JSUnitReporter(reports.FunctestReportInterface):
    def summary(self, test_list, totals_dict, stdout_capture):
        if functest.registry.get('report', False):
            total_sec = reduce(lambda x, y: (y.endtime - y.starttime).seconds + x, test_list, 0)
            e = ElementTree.Element('testsuite')
            e.attrib['errors'] = e.attrib['failures'] = str(totals_dict['fail'])
            e.attrib['tests'] = str(len(test_list))
            e.attrib['name'] = 'windmill.functional'
            e.attrib['time'] = str(total_sec)
            for entry in test_list:
                t = entry.endtime - entry.starttime
                test = ElementTree.Element('testcase')
                test.attrib['classname'] = test.attrib['name'] = entry.__name__
                test.attrib['time'] = str(t.seconds)+'.'+str(t.microseconds)
                if entry.result is not True:
                    failure = ElementTree.Element('failure')
                    failure.attrib['type'] = entry.tb[-1].split(':')[0]
                    failure.text = ','.join(entry.tb)
                    test.append(failure)
                e.append(test)
 
            if len(stdout_capture):
                # ElementTree doesn't support CDATA so we need to hack it a little
                replace = '#$!#$!replace#@!$'
                sysout = ElementTree.Element('system-out')
                sysout.text = replace
                e.append(sysout)
                outs = ElementTree.tostring(e).replace(replace, '<![CDATA['+stdout_capture+']]>')
            else:
                outs = ElementTree.tostring(e)
 
 
            f = open('continuous_test.log','w')
            f.write(outs) ; f.close()
 
reports.register_reporter(JSUnitReporter())

UPDATE: Mikeal has posted some additions/revisions to this code sample, here.