NatHERS Benchmark Study

127 Pages·2014·6.91 MB·English

by  Matthews, Tanya

NatHERS Benchmark Study

February 2014

Prepared for the Department of Industry by:
Floyd Energy, Wayne Floyd

With assistance from
Tony Isaacs and
Rodger Hills
Table of Contents
1 Executive Summary ......................................................................................................................... 6
1.1 Study design ............................................................................................................................ 6
1.2 Key findings ............................................................................................................................. 6
1.3 Recommendations .................................................................................................................. 8
2 Introduction .................................................................................................................................. 10
3 Background ................................................................................................................................... 10
3.1 The Industry .......................................................................................................................... 10
4 Objectives...................................................................................................................................... 12
4.1 Purpose ................................................................................................................................. 12
4.2 Deliverables ........................................................................................................................... 12
4.3 Assumptions .......................................................................................................................... 12
5 Methodology ................................................................................................................................. 13
5.1 Stage One: System design ..................................................................................................... 13
5.2 Stage Two: Participation ....................................................................................................... 15
5.3 Stage 3: Data collection and analysis .................................................................................... 17
5.4 Limitations of the methodology............................................................................................ 18
5.5 Risk Management ................................................................................................................. 19
6 Findings summary ......................................................................................................................... 21
6.1 Statistical significance of sample .......................................................................................... 21
6.2 Queries .................................................................................................................................. 24
6.3 Assessor demographics ......................................................................................................... 24
6.4 Assessor Practices ................................................................................................................. 25
6.5 Star Ratings ........................................................................................................................... 26
6.6 Software tools ....................................................................................................................... 28
6.7 AAOs ...................................................................................................................................... 29
6.8 Question scores..................................................................................................................... 30
6.9 Net Conditioned Floor Area (NCFA) ...................................................................................... 30
6.10 Zoning.................................................................................................................................... 32
6.11 Site exposure ......................................................................................................................... 33
6.12 Orientation ............................................................................................................................ 33
6.13 Windows and skylights .......................................................................................................... 33
6.14 Eaves ..................................................................................................................................... 34
6.15 Overshadowing by surrounds ............................................................................................... 34
ii | P age
6.16 Air leakage sites .................................................................................................................... 35
6.17 Ceiling penetrations .............................................................................................................. 35
6.18 Walls ...................................................................................................................................... 35
6.19 Roofs ..................................................................................................................................... 36
6.20 Floors ..................................................................................................................................... 37
7 Recommendations ........................................................................................................................ 38
7.1 Minimum training and professional development standards for NatHERS assessors ......... 38
7.2 NatHERS tool improvements ................................................................................................ 41
7.3 New resource development.................................................................................................. 42
7.4 Monitor progress and future studies .................................................................................... 43
8 Appendices .................................................................................................................................... 45
8.1 Appendix 1: Benefits to assessors of participation in the study ........................................... 45
8.2 Appendix 2: Assessor demographics .................................................................................... 46
8.3 Appendix 3: Detailed analysis of assessments ...................................................................... 54
8.4 Appendix 4: Assessor practices ............................................................................................. 80
8.5 Appendix 5: House plans ....................................................................................................... 84
8.6 Appendix 6: Questions asked for each dwelling ................................................................. 108

iii | P age
Abbreviations
Abbreviation Meaning
AAO NatHERS Assessor Accrediting Organisation
ABCB Australian Building Codes Board
ABS Australian Bureau of Statistics
ABSA Association for Building Sustainability Assessors
ACT Australian Capital Territory
AccuRate NatHERS Benchmark software tool
BASIX Building Sustainability Index – the web-based tool designed to assess the
potential performance of buildings that contain a dwelling, against thermal
comfort, water and energy criteria.
BDAV Building Designers Association of Victoria
BERS Pro NatHERS accredited software tool
Cert IV Certificate IV in NatHERS Assessment
CPD Continuing Professional Development
DI Department of Industry
FirstRate5 NatHERS accredited software tool
NatHERS Nationwide House Energy Rating Scheme
NCC National Construction Code
NCFA Net Conditioned Floor Area
NSW New South Wales
NT Northern Territory
QLD Queensland
RIS Regulatory Impact Statement
SA South Australia
SHGC Solar Heat Gain Coefficient – a property of windows that reflects how much
solar radiation is transmitted through the window
SV Sustainability Victoria
TAS Tasmania
WA Western Australia
WERS Window Energy Rating Scheme

iv | P age
Acknowledgements
The study team would like to thank the following people and organisations for their generous
assistance and support:
 ABSA
 BDAV, Kate Bell CEO
 Sustainability Victoria, FirstRate5 software, Anthony Wright
 SmartRate, BERS Pro software, Michael Plunkett
 Hearne Scientific, distributors of AccuRate, Chris Williams, Barlow Telford

Members of expert committees:
 Ted Harkness
 Peter Barlow
 Tony Butters
 Debbie Bute
 Eliza Morawska
 Katie Fallowfield
 David Canciello
 Victoria Prior
And last but not least, the study team would like to thank all participating assessors who generously
took time away from their business or leisure to participate in this study. This study represents an
important step forward for the NatHERS assessor industry and will help all assessors to produce
consistent and reliable ratings for their clients. As the first study of its kind this study was on its own
learning curve, and as a result there were inevitable glitches in its implementation. The study team
appreciates your dedication and patience for sticking with us to the end.

v | P age
1 Executive Summary
1.1 Study design
This study creates a national benchmark by measuring the accuracy of NatHERS assessments.
Assessors were randomly allocated one of four houses to rate and their assessments were compared
with a solution set prepared by a committee of expert assessors. Approximately 100 questions about
NatHERS data entry for each house were developed to specifically test how assessors applied
Technical Notes 1 and 2, whilst also testing accuracy of general data entry techniques.
A study website was developed to provide a portal for assessors to access the study, enter responses
to questions, ask questions and gather feedback. A webinar – available through YouTube – and the
study website provided instructions on participating in the study and using the data entry portal.
Assessors were initially contacted through Assessor Accrediting Organisations (AAOs) and software
providers who endorsed the study and encouraged assessors to participate. AAOs provided
Continuing Professional Development (CPD) points for participation as an incentive.
1.2 Key findings
1.2.1 Statistical significance of participants
Statistical significance depends on the sample size relative to the total population. The measure of
statistical significance used for academic publications is 95% certainty and a 5% margin of error. This
study obtained a sample of 344 assessors from an estimated sample of 1816 assessors which has an
error margin of 4.8% at 95% certainty, so the sample was statistically representative of the assessor
population as a whole. As the sample was self-selecting and not random, the nature of the bias, that
is, whether one would expect a self-selected sample to be more or less accurate than the general
population is not clear.
Sub samples by jurisdiction, AAO and software tool were also examined to determine statistical
significance. Only the sub sample of ABSA assessors fell within the 95% confidence and 5% error
margin limit. Sub samples of BDAV assessors, assessors from WA and Victoria and each of the
software tools fell within an error margin of 10% at 95% confidence. Assessors who were
unaccredited represented only 10% of participants. This does not mean that comparisons between
sub samples had no significance, but that the differences between these subsamples would need to
be greater to be able to conclude that the difference was statistically significant.
1.2.2 Star rating accuracy
Star rating accuracy is the most important element of a NatHERS assessment as this is directly
related to compliance with building regulations and the National Construction Code (NCC).
The study found that around 21% of assessments obtained the correct rating around 37% of ratings
were within 0.25 stars, 58% within 0.5 stars and 77% were within 1 star (Table 1). Alternatively, 64%
of assessors had an error greater than 0.25 of a star.

6 | P age
The high error rate in star ratings was consistent with assessors correctly answering 65% of the
questions about data inputs.
Table 1 Proportion of ratings within various star rating error bands
Error in star rating Cumulative
Within 0.25 stars 37%
Within 0.50 stars 58%
Within 0.75 stars 70%
Within 1.00 stars 77%
Within 1.50 stars 86%
Within 2.00 stars 91%
More than 2.00 stars 100%

1.2.3 NatHERS Technical Notes
NatHERS Technical Notes 1 and 2, cover rules for data entry including zone types and conditioning,
site exposure and calculation of the area of uninsulated ceiling created by ceiling penetrations. The
errors found in this study suggest that these documents are not well understood, or widely used by
the assessor industry. For example, depending on the complexity of the house plan only between 3%
and 29% of assessors correctly estimated the area of ceiling penetrations.
1.2.4 Application of Technical Note 1 Zoning rules
Arguably, the most important element of accurate modelling using NatHERS software is to correctly
zone the house plan. A large proportion of assessments were not zoned correctly according to the
zoning rules in Technical Note 1. Rooms were incorrectly combined or split, the allocation of other
day or night time-zoned rooms were not well understood and heating and cooling was incorrectly
turned off in these zones. Inspection of individual files showed that around 60% of assessors had
made errors in zoning house 1 and 4 and 85% incorrectly zoned house 2.
1.2.5 House plan complexity
Assessor errors increased significantly with the increasing complexity of design and documentation.
57% of assessors were within 0.25 stars of the correct rating for house 1, but only 41% for house 2
and 26% for house 3. Houses 2 and 3 were two storey houses (which are inherently more complex to
rate than single storey houses) and house 3 had more complex documentation than the simpler
house 1.
House 4 was an apartment on the 15th floor and only 19% of the assessors got within 0.25 stars of
the correct rating. In addition to zoning errors a significant contributing factor to the errors was
found to be selecting incorrect wind exposure and height above ground. 41% of assessors selected
the wrong exposure for house 4.
1.2.6 Net Conditioned Floor Area (NCFA)
The rating is calculated on the basis of energy loads per square metre of NCFA, so getting this right is
vital to an accurate rating. One in five ratings reported incorrect NCFA (> 5% difference), which
resulted from factors such as failing to turn on heating and cooling in other day and other night
classified zones.
7 | P age
1.2.7 Overshadowing
Overshadowing refers to the shading of the rated building by surrounding objects, typically adjacent
houses and fences. Incorrectly modelling overshadowing on a heavily overshadowed site can result
in rating errors of over 1 star.
Houses 1 to 3 each had significant overshadowing from adjacent buildings and fences. Most
assessors did not accurately model these obstructions. Errors ranged from miscalculating the height
and depth of the overshadowing obstruction to not counting the overshadowing impact of roof
ridges and fences. Over 75% of assessors that rated house 1 did not accurately model obstructions.
1.2.8 Systematic process
A wide variety of other errors were also observed that indicated the lack of a systematic approach to
the rating process by assessors. For example, 25% of assessments reported incorrect window areas,
85% reported incorrect area of ceiling penetrations, 35% had incorrect roof solar absorptance, 25%
had the incorrect R-value for waffle pod slabs and the area of different wall constructions and floor
coverings within each house showed error rates ranging from 10 to 70%.
1.2.9 Differentiation by AAO or tool
The responses of assessors were also checked to see whether the use of a particular tool or
membership of AAO was correlated with different accuracy levels. While differences were found, the
small sample sizes meant that the differences found could be explained by the error margin of the
samples.
1.2.10 Information about the assessor industry
Assessors who registered for the study were asked a number of questions which provided valuable
information on the assessor industry. Around a quarter of assessors work solely as an assessor. Two
thirds of assessors are self-employed and around half of these assessors were building designers or
architects. The average assessors had been in the job for 5.5 years and rated 90 houses per year.
Almost all assessors had some building site experience with around 63% having direct on site
experience.
1.3 Recommendations
It is important that NatHERS assessments are consistently accurate to ensure compliance with
building regulations, effectiveness in saving energy, for the reputation of the industry and to ensure
consumers get the service they pay for. The requirement that all assessors complete the Certificate
IV in NatHERS Assessment by 1 July 2015 is an important step towards improving the skills of
assessors.
This study highlighted areas where enhancements could be made to NatHERS to increase the
accuracy and consistency of NatHERS assessments. These can be grouped into four broad categories:
1.3.1 Mandatory accreditation of assessors
This study found that there was a high level of error in NatHERS ratings irrespective whether
assessors were accredited or not. The quality assurance (QA) processes of accreditation is a
mechanism that could contribute to more accurate assessments, however, there is no process
8 | P age
available to improve the accuracy of unaccredited assessors. Making accreditation mandatory for all
assessors is essential to ensure to improve the accuracy of assessments.
The error rate showed that there are benefits for further CPD training for assessors in Technical
Notes 1 and 2. This additional training should incorporate evaluation to ensure that the content is
understood by assessors. The Technical Notes could be improved by providing detailed examples to
better explain the principles.
1.3.2 NatHERS tool improvements
NatHERS Software Tools could be further improved to minimise error rates. This includes
improvements such as automatic calculation of ceiling penetration area, better guidance on the
allocation of zone occupancy types and the application of heating and cooling, automatic calculation
of waffle pod slab R values and simplifying data entry for overshadowing. The calculation of NCFA is
not consistent across tools and should be harmonised as the rating is determined on the basis of this
area.
1.3.3 New resources for assessors
A number of new resources for assessors would help to increase accuracy:
 The NatHERS assessor industry has no standard data entry and error checking procedures.
Development of such procedures would facilitate a more systematic approach to rating. This
would help to eliminate many of the errors found in this study;
 A comprehensive technical manual for assessors which explains how to model all the design
and site features which assessors may encounter in the field; and
 Work with product suppliers to develop trade literature which better meets the needs of
assessors (69% reported that they had difficulty finding the information they needed in
trade literature).
1.3.4 Improving future NatHERS assessor benchmark studies
The proposed Universal Certificate extracts data from the rating file and displays this on the
Certificate. A similar tool which extracts a larger set of data than the proposed Universal Certificate
from NatHERS rating files automatically would have significantly reduced the extent of analysis
required, facilitated analysis in greater detail and made participation in this study faster and easier
for assessors. It would also avoid issues of assessor misinterpretation of questions. The proposed
universal certificate generator could be modified to do this. It would also significantly reduce the
resources needed for QA checking by AAOs and assist in the marking of exams and tests for trainers.

9 | P age
2 Introduction
In May 2013, The Department of Industry commissioned Floyd Energy to undertake the first national
benchmark study to measure the accuracy and consistency of assessments undertaken by
Nationwide House Energy Rating Scheme (NatHERS) assessors. This study followed a study in 2012
where the Association of Building Sustainability Assessors (ABSA) was contracted by the Western
Australian Office of Energy to conduct a workshop to trial state-based interim software modelling
principles and defaults. The results of this workshop identified a number of areas where the
accuracy and consistency of NatHERS assessments could be significantly improved, such as how to
consistently treat zoning and floor covering, and how to make consistent judgements regarding
insulation and external screening.
As the results were unlikely to be unique to Western Australia, the Department of Industry has
sought to establish a national baseline of current assessor practices. The intent of the study was to
more fully understand problem areas, monitor and evaluate improvements to the Scheme, and to
work with all governments to continually improve the consistency and accuracy of NatHERS
assessments nationally.
3 Background
NatHERS provides a national framework that allows various computer software tools to rate the
potential energy efficiency of Australian houses at the design stage. NatHERS software tools fulfil a
regulatory function for assessing compliance of a building design with the thermal performance
requirements in the National Construction Code (NCC). The correct application of NatHERS software
tools is critical to the success of the Scheme. Although each state and territory government uses the
NatHERS accredited software to provide assessments, minimum standards in all other aspects of the
Scheme including application of the use of tools can vary.
Main components of the Scheme include:
 The use of accredited software tools that provide consistent and accurate modelling of
building shell performance (assessments) across a diverse range of climates to determine
heating and cooling demands;
 Minimum standards for assessors:
o Accreditation of Assessor Accrediting Organisations (AAOs) incorporating CPD and
quality assurance of assessors, Code of Conduct and mandatory Professional
Indemnity Insurance;
o Certificate IV in NatHERS Assessment qualification;
o Consistent use of NatHERS tools through nationally-applied technical notes; and
o Working toward a minimum consistent standard across all states and territories, and
 A process of continuous improvement for all aspects of the Scheme.
3.1 The Industry
The NatHERS Assessor industry is still in the early years of its development. National Regulations for
minimum NatHERS performance were introduced in 2003, and a number of states delayed the
introduction of these regulations. Prior to 2003 a number of jurisdictions had introduced minimum
requirements: NSW’s Energy Smart Homes program required a minimum 3.5 stars, the ACT required
10 | P age