The white antelope or Arabian Oryx is one of the rarest animals in the world. It was extinct in the wild and is being reintroduced, but it is still vulnerable and unlike other antelopes, rarely seen. There are “rare animals” or infrequently discussed topics in testing too. In this blog I will suggest that, despite the importance of managing energy consumption to organisations and to society, it is a white antelope for testers. The active discussion or promotion of testing around energy consumption is missing from the debates and practices in the current software testing community.
Glenford Myers et al compiled in 1979 “all the knowledge in the world viewed by the eye of software tester” in the book The Art of Software Testing. In 2012, the second update of this true testers’ bible, a new topic emerged: “Mobile Testing”. For mobile devices, managing energy consumption efficiently is essential. However, in terms of software energy consumption tests, no new information was added and the only reference to this issue, as in the original version, was “programs have specific performance or efficiency objectives” (Myers, 2012, pp. 126).
In 2008 two experts in the Software Testing Area broached the subject. They were Brennan and Blatt and, in my opinion they are the exceptions – the white antelopes – up to now. They provided two different reasons for testing energy consumption: the global impact of wasted energy and the financial impact for organisations. Brennan (2008) at the Eurostar Conference 2008 said that “Ineffective / Inefficient Testing = Resource wastage / Global Impact” although the conclusions presented were mainly about the “ways of working”, i.e., about the good practices that should subsist in the development of software projects (under the macro view point), such as, for example, the testing environments being reused. Blatt, in turn, said that engineers / hardware producers just by implementing energy efficiency measures showed that they already knew the financial benefits that would result from that action, having even a specific metric: performance per watt (Blatt, 2008). Blatt has also made an essential point about cost-benefit on which all software engineers should reflect: highly efficient code will use less processing power, and therefore less energy; but is the extra development effort (and energy spent in doing it) worthwhile? (Blatt, 2008).
The existence of the international standard ISO 14756:1999 Measurement and Rating of Performance of Computer-based Software Systems, gives relevance to the topic. This standard was reviewed in 2010 (ISO Stage: 90.60 – Close of review, 2010-06-17) and considered up to date. So, in my opinion, this shows that the subject is frozen in time for testers. In contrast, hardware engineers and software developers are looking at energy efficiency as important built-in aspects of IT product and service quality, especially when we are under a mobile perspective (for example, Intel, Microsoft, Energy Aware High Performance Computing Conference).
We should be asking “what are the real reasons that explain the lack of interest within the tester’s community?” and “why hasn’t there been progress in the subject of testing the energy consumption of software?”
In my strict opinion, and as a software tester with over six years of experience, there are at least 8 arguments for this status quo of limited discussion and no progress in methods. I have outlined these with a counter-argument for each one.
- Lack of demand from the market (either by private or public sector). This lack of demand arises from two sources. First, there is no demand because customers do not feel the need to control the energy consumption used by their software. Second, testers & developers have not told their “customers” that this could be a negotiable requirement so the customers do not know that energy consumption good be tested / improved.
- We can overcome this by starting the debate within the testing community and in our organisations.
- Negative perception of environmental radicalism (particularly from the business world), so when we talk to the CEO about reducing energy consumption in their software just because it is environmentally correct they get suspicious.
- A better approach to the CEO and CFO could be something like “if your software consumes less you spend less money, so your financial investment will do more than equalize and additionally your marketing image will be improved because you may say that your software is environmentally responsible and saves the consumer money”.
- Energy consumption is seen as a hardware problem: A third argument is the fact that there is a widespread perception that the energy consumption is a hardware problem and not a software one. This argument has some merit because if we measure two different hardware devices running the same software programme we will have different consumption levels.
- We can counter this by noting that quoting ISO 14756: 1999 which demystifies the subject stating that “In order to measure the influence of software on the time behaviour of a data processing system it is necessary to measure the time behaviour of the whole system.”. This sentence in the context of ISO definition, tells us to test the energy consumption of software. You must test the platform as a whole, because for example the software under test only runs if there is an operating system (OS) below that will have an effect.
- Lack of tools makes it difficult to test: A fourth possible argument is the lack of tools dedicated to the evaluation of energy consumption, preferably open source or free use tools. This means it seems difficult to measure energy consumption.
- This argument can be countered, yet with difficulty. For example in Windows Seven (7) the Microsoft research department has developed the Joulemeter. This tool simulates consumptions of third part software’s and is free to use Also anyone can download the official website tool without any registration issue. The assurance degree of the Joulemeter can even be manually calibrated to simulate as close as possible of two types of physical measurement devices: a wattmeter (power meter) or a clamp-on meter (Matos, 2014). The first is an invasive device in the electrical supply. The second is not invasive in the electrical circuit (so does not influence, positively or negatively, the measured consumption) because the calculation is estimated by magnetic field of the calliper.
- Low cost benefit: Some may say, as another argument, that “ok, testers can assess if the energy consumption of a particular software is higher than expected”, but that it only makes sense to invest in this testing if one also invests a budget for the refactoring of the source code. So the decision to invest in an increased number of hours to develop the software is economically even more difficult to take.
- This argument is hard to counter, without a cost benefit analysis which takes into account cost savings to the organisation and the customers over time; the CEO and CFO will want to know how quickly their investment will repay them. Additionally, if code is to be refactored anyway as part of a project, refactoring for efficiency at the same time makes sense: it may not be a separate budget.
- Benefits may affect the consumer rather than the organisation: The sixth argument has to do with the benefits to an enterprise of improving the energy consumption of software. If we consider ourselves as being an enterprise that develops commercial off the shelf products (COTS) we will giving the benefit of reduced consumption to the consumer and therefore this may not be an economical improvement for us or for our internal development process.
- This can be countered by including improvements to reputation, marketing opportunities, and the opportunity to be seen as an ethical organisation appealing to an ethical market sector. Nathalie Rooseboom de Vries (2010) tell us on debate of ethics that the tester is co-responsible for the acceptance of the product so they should be go-ahead to the next step in development or production at level of all variability’s of quality.
- First world bias: This leads us to a seventh argument, which is the fact that software development mainly occurs in developed countries where electricity is supplied in an uninterrupted and abundant way (most countries have liberalized markets where there are several suppliers available). In these countries electricity is also cheap and easily accessible, in contrast to poor countries where this does not happen and normally power consumption is rationed. This means that software developers are not in the right context to realize the power rationing problems.
- As with the previous argument the ways to overcome this point come through raising awareness of the ethical and societal impact of our work. Some people in the industry already look at this like Julian Harty’s project to take kindles with solar power batteries to schools in Africa.
- It’s not fun and it’s not sexy: The eighth and final argument is perhaps the one that can explain all I have said above. In recent years the tester community has found other themes considered much sexier to debate as for example, the attempt to answer if on a new project we have in pocket we should use Agile or traditional methodologies such as the Waterfall.
- I’m tempted to suggest that maybe the counter argument here is for the industry and its leaders to be more adult in its approach and focus on what is important to the world…
My conclusion is that it is as difficult and rare to see a tester writing about software testing energy consumption as it is to see a white antelope, the Arabian Oryx, so it is in danger of extinction. But, if in the software testing industry we discuss and build on these eight arguments and their counter arguments, maybe we will be taking the first steps to change our situation. If we engage in the discussion we have a good time-line of opportunity to provide a different type of value to our organisations and customers in a way they had not anticipated, improving our practices, deliverables, and organisations, providing one type of value that is really important: helping our fainting world… and the rare white antelopes that live in it.
Paulo José Estrela Vitoriano Matos
Myers, Glenford J.; Badgett, Tom; Sandler, Corey (2012) “The Art of Software testing”, 3º Edição, Esfera John Wiley & Sons, Inc, (United States of America)
Blatt, Michael (2008), “Green Computing and Performance Testing: A New Paradigm?”, SQS Performance Testing Competency Centre [Access October 2013]. Link on URL: http://www.sqs.com/en/se/Download/SQS_RI_Green_Computing.pdf
Brennan, John (2008) [ https://www.linkedin.com/in/johnsbrennan ] “Red Hot Testing in a Green World”, The Hague, AppLabs”, [Access October 2014]. Link on URL: URL:http://www.slideshare.net/jsb1976/AppUK-Red-Hot-Testing-in-a-Green-World-V106-Animation
van Delft, Nathalie Rooseboom de Vries (2010), “The Ethics Debate”, Software Test & Performance Volume 7, Issue 4 [Access March 2015] Link on URL: http://www.funtestic.nl/Assets/Article_EthicsDebate_ST&P.pdf
Harty, Julian (2013), “Solar Powered Kindles with e-volt batteries at Nyabondo School Oyugis, Kenya” [Access March 2015] Link on URL: http://kusaidiamwalimu.org/solar-powered-kindles-with-e-volt-batteries-at-nyabondo-school-oyugis-kenya/
Matos, Paulo José Estrela Vitoriano de (2015), tutors: Carapeto, Cristina; Coelho, José Silva; “The contribution of software testing to its conception more efficient in terms of energy”, Project Master Dissertation (2013-2015), Universidade Aberta (Open University), Lisbon
ENA-HPC – Energy-Aware High Performance Computing (2010-2014) [Access March 2015] Link on URL: http://www.ena-hpc.org
Steigerwald, Bob, Chabukswar, Rajshree, Krishnan, Karthik, De Vega, Jun (2007). “Creating Energy-Efficient Software”, Working Paper. Intel Corp. Centre [Access March 2015] Link on URL: https://software.intel.com/sites/default/files/m/d/4/1/d/8/creating_energy-efficient_software.pdf
Kansal, Aman; Goraczko, Michel; Liu, Jie; Zhao, Feng (2010) “Joulemeter: Computational Energy Measurement and Optimization”, Microsoft Research, [Access March 2015] Link on URL: http://research.microsoft.com/en-us/projects/joulemeter/
Stemen, Pat (2008), “Extending Battery Life With Energy Efficient Applications” Microsoft Corporation, [Access March 2015] Link on URL: http://channel9.msdn.com/Blogs/pdc2008/PC02
Wikimedia Foundation, Inc. (2015) “Arabian oryx” [Access March 2015] Link on URL: http://en.wikipedia.org/wiki/Arabian_oryx