{"id":1229,"date":"2016-01-14T11:47:11","date_gmt":"2016-01-14T11:47:11","guid":{"rendered":"http:\/\/blogs.sussex.ac.uk\/sussexenergygroup\/?p=1229"},"modified":"2016-04-04T13:10:38","modified_gmt":"2016-04-04T13:10:38","slug":"quality-research","status":"publish","type":"post","link":"https:\/\/blogs.sussex.ac.uk\/sussexenergygroup\/2016\/01\/14\/quality-research\/","title":{"rendered":"How good is your model? Quantifying quality research"},"content":{"rendered":"

\"a<\/a>Our latest EPSRC-funded research carried out in CIED<\/a> (and led by\u00a0SPRU, Sussex<\/a>) has something new to say about the testing of statistical models. Available now in the journal Energy Economics<\/em> and authored by Lee Stapleton<\/a>, Steve Sorrell<\/a> and Tim Schwanen<\/a>, you can access the paper here<\/a><\/p>\n

The paper estimates the so-called \u2018direct rebound effect\u2019 associated with personal car travel in Great Britain. This effect relates to the increased driving that may occur when fuel efficiency improvements make car travel cheaper. Our results suggest that the direct rebound effect has been in the region of 20% in GB over the last 40 years. Put differently, a 1% fall in average fuel prices over this period leads to a 0.2% increase in total distance travelled.<\/p>\n

To arrive at this conclusion we developed and estimated a total of 108 different regression models<\/a>. Some of these models were quite fancy, some were quite simple and together they produced a range of parameter estimates in terms of rebound and other determinants of distance travelled. So, how did we choose between them?<\/p>\n

To do this, we quantified the robustness<\/em> (strength or quality) of each model in terms of the extent to which it adhered to governing rules and assumptions about structure, stability, parsimony and the behaviour of parameter estimates and non-fitted data (residuals). \u00a0\u00a0Unfortunately, most applied research which uses these kinds of models pays insufficient attention to these rules and assumptions – many of which are well-documented in textbooks and routinely covered in courses on research methods. So why are they so often ignored? Other assumptions and rules are confined to the more technical, specialist literature which makes it easier to understand their limited diffusion in applied research.<\/p>\n

Capturing multi-dimensional concepts such as robustness<\/em> can be achieved by constructing so-called composite indicators<\/a>. In other words, uni-dimensional constituents of robustness can be combined mathematically into weighted, aggregated, multi-dimensional representations of the concept. And that\u2019s what we did. Specifically, we assessed 96 of our 108 models in terms of 13 \u2018quality indicators\u2019 to create aggregate, composite measures of robustness for each model. We took two approaches based on different weightings. The first (unequally weighted) is based on our judgement of the \u2018relative importance\u2019 of each robustness constituent. The second (equally weighted) does not differentiate in terms of importance.\u00a0 We assessed the remaining 12 models in terms of a reduced set of 6 quality indicators because the robustness of these models was trickier to assess. Here, again, we developed unequally and equally weighted robustness composites.<\/p>\n

Doing this allowed us to explore the relationship between model robustness and parameter estimates. We haven\u2019t seen this done before. Previously, this has only been approached using relatively narrow operationalisations of robustness compared to the multi-dimensional indicators developed and used here.<\/p>\n

So, the $64,000 question is Do bad methods lead to biased results?<\/em> And conversely Do good methods lead to less biased results?<\/em> We can provide answers using our robustness indicators. If there are systematic relationships between parameter estimates and robustness we should be concerned that bad methods do indeed lead to \u2018wrong\u2019 answers. But, if there is no relationship between parameter estimates and robustness we can afford to be less concerned about these bad methods.<\/p>\n

In our latest work here, we found some evidence to suggest that bad methods lead to biased results, but only some. It depends on which model parameter estimates you look at. More studies need to be done which apply comparable robustness indicators to other models to get a better handle on this. Regardless of the answer, it is a good idea to choose and use statistical models which are multi-dimensionally robust. Hence, we are currently applying the indicators developed in this work to models being developed in other projects.<\/p>\n

Lee Stapleton is a Research Fellow at the\u00a0Science Policy Research Unit and the Tyndall Centre for Climate Change Research. \u00a0Working principally\u00a0in the EPSRC-funded Centre on Innovation and Energy Demand (CIED) within the Sussex Energy Group, Lee’s\u00a0research focusses on the application of time-series (static, dynamic and co-integrating) regression\u00a0models to estimate so-called ‘rebound effects’ in the transport sector.\u00a0<\/strong><\/p>\nFollow Sussex Energy Group \"Facebook\"<\/a>\"twitter\"<\/a>\"linkedin\"<\/a><\/span>","protected":false},"excerpt":{"rendered":"

Our latest EPSRC-funded research carried out in CIED (and led by\u00a0SPRU, Sussex) has something new to say about the testing of statistical models. Available now in the journal Energy Economics and authored by Lee Stapleton, Steve Sorrell and Tim Schwanen,…<\/span><\/p>\n

Read more ›<\/a><\/div>\n

<\/p>\n","protected":false},"author":136,"featured_media":1233,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false},"categories":[96027,10265,36585,81017],"tags":[96117,90183,96118,41041,96119,96116],"yoast_head":"\n\n\n\n\n\n\n\n\n\n\n\n\n\t\n\t\n\n\n\t\n