We've fitted some models and would like to know how to use them to really understand the quality of the models. The model metrics look like this: and comparing it to ASHRAE 14 guidelines, which gives us these formulas: My questions are: 1. Is the autocorr_resid the rho (p) is B14? 2. What are the right parameters for n and m? According to an early page in ASHRAE 14, n and m are "number of observations in the baseline (or pre retrofit) and the postECM periods, respectively" If the model is a daily, should n be 365, so in this case, n' = 365 * (10.4792) / (1+0.4792) = 128.5? If the model is used to compare energy savings over a year, should m be 365? Or should m be 30 if we're comparing the energy savings on a monthly basis? 3. How many model parameters are there? In a combined heating and cooling model, should it be 5  2 betas, 2 balance points, and an intercept  or 3? Calculating all this from my example model, I get a 25.8% uncertainty for F (energy savings) of 20% at 68% confidence (t = 1) Does that seem reasonable for a daily model with this much CVRMSE? Thanks.


1. Correct  autocorr_resid is rho
2. The value of n should be 365, that is correct. It sounds like you have the right idea for m as well (i.e, if you have 30 daily predictions and want to know the uncertainty of the sum of those thirty predictions, m should be 30) with a slight caveat that CalTRACK suggests handling these calculations using a polynomial correction using experimentally derived coefficients. See section 4.3, http://docs.caltrack.org/en/latest/methods.html#section4aggregation. In that case, there is also an M (capitalized) to keep track of, which is the number of months (regardless of frequency  which is taken into account by using different coefficients for daily and monthly billing data.)
toggle quoted message
Show quoted text
On Wed, Mar 4, 2020 at 3:01 PM Si Chen < sichen@...> wrote: [Edited Message Follows]
We've fitted some models and would like to know how to use them to really understand the quality of the models. The model metrics look like this:
and comparing it to ASHRAE 14 guidelines, which gives us these formulas:
My questions are:
1. Is the autocorr_resid the rho (p) is B14? 2. What are the right parameters for n and m? According to an early page in ASHRAE 14, n and m are "number of observations in the baseline (or pre retrofit) and the postECM periods, respectively" If the model is a daily, should n be 365, so in this case, n' = 365 * (10.4792) / (1+0.4792) = 128.5? If the model is used to compare energy savings over a year, should m be 365? Or should m be 30 if we're comparing the energy savings on a monthly basis? 3. How many model parameters are there? In a combined heating and cooling model, should it be 5  2 betas, 2 balance points, and an intercept  or 3?
Calculating all this from my example model, I get a 25.8% uncertainty for F (energy savings) of 20% at 68% confidence (t = 1) Does that seem reasonable for a daily model with this much CVRMSE?
Thanks.


Steve Schmidt
A few additional comments 
 I'd call this a "bad building". Based on the CalTRACK model fit results, energy use is not very predictable. The ASHRAE Guideline 14 requirement for a good model fit is CVRMSE < 0.25; this value of 0.46 is far above that target. Perhaps you can note this to users of your system, so they don't rely too heavily on the model.
 Savings calculations using such a [poor] model will be inaccurate. I'm no statistician, but I believe an Rsquared value of 0.4 indicates some correlation, but is not considered useful for prediction. Current CalTRACK methods use any model with CVRMSE values below 1.0 to predict the counterfactual, so it's up to users to recognize when a fit is good and when it's not.
 It's odd that the cooling and heating balance points are the same; normally there are several degrees separation between the two. Maybe it's a strange building, or maybe Phil can explain this.


Is it possible for energy loads that occur at specific times of day (unrelated to CDD or HDD), due to timeofuse pricing, to negatively impact the model accuracy? If so, how can these other variables be addressed?
toggle quoted message
Show quoted text
On Thu, Mar 5, 2020 at 4:53 PM Steve Schmidt < steve@...> wrote: A few additional comments 
 I'd call this a "bad building". Based on the CalTRACK model fit results, energy use is not very predictable. The ASHRAE Guideline 14 requirement for a good model fit is CVRMSE < 0.25; this value of 0.46 is far above that target. Perhaps you can note this to users of your system, so they don't rely too heavily on the model.
 Savings calculations using such a [poor] model will be inaccurate. I'm no statistician, but I believe an Rsquared value of 0.4 indicates some correlation, but is not considered useful for prediction. Current CalTRACK methods use any model with CVRMSE values below 1.0 to predict the counterfactual, so it's up to users to recognize when a fit is good and when it's not.
 It's odd that the cooling and heating balance points are the same; normally there are several degrees separation between the two. Maybe it's a strange building, or maybe Phil can explain this.
 All the Best, M. System Smart 484.553.4570 Sent on the go. Pardon grammar/spelling.


Steve Schmidt
My belief: if the building is "well behaved" with respect to outdoor temperatures and heating and cooling loads, then other nonHVAC loads should have no impact on model fit. But I'm not an OEE expert so I'll let Phil correct this.


Thanks for pointing that out, Phil. It seems that CalTrack 4.3.2.4 has replaced ASHRAE's 1.26 "empirical coefficient" with a formula, and for M=12 (12 reporting periods) it comes out to 1.30 for billing (monthly) data and 1.39 for daily data.
Is P' calculated from P the same way here that n' is calculated from n from the ASHRAE formula, using the autocorrelation coefficient rho?
Finally how do we get the number of model parameters or "number of explanatory variables in the baseline model"?

toggle quoted message
Show quoted text
1. Correct  autocorr_resid is rho
2. The value of n should be 365, that is correct. It sounds like you have the right idea for m as well (i.e, if you have 30 daily predictions and want to know the uncertainty of the sum of those thirty predictions, m should be 30) with a slight caveat that CalTRACK suggests handling these calculations using a polynomial correction using experimentally derived coefficients. See section 4.3, http://docs.caltrack.org/en/latest/methods.html#section4aggregation. In that case, there is also an M (capitalized) to keep track of, which is the number of months (regardless of frequency  which is taken into account by using different coefficients for daily and monthly billing data.)
On Wed, Mar 4, 2020 at 3:01 PM Si Chen < sichen@...> wrote: [Edited Message Follows]
We've fitted some models and would like to know how to use them to really understand the quality of the models. The model metrics look like this:
and comparing it to ASHRAE 14 guidelines, which gives us these formulas:
My questions are:
1. Is the autocorr_resid the rho (p) is B14? 2. What are the right parameters for n and m? According to an early page in ASHRAE 14, n and m are "number of observations in the baseline (or pre retrofit) and the postECM periods, respectively" If the model is a daily, should n be 365, so in this case, n' = 365 * (10.4792) / (1+0.4792) = 128.5? If the model is used to compare energy savings over a year, should m be 365? Or should m be 30 if we're comparing the energy savings on a monthly basis? 3. How many model parameters are there? In a combined heating and cooling model, should it be 5  2 betas, 2 balance points, and an intercept  or 3?
Calculating all this from my example model, I get a 25.8% uncertainty for F (energy savings) of 20% at 68% confidence (t = 1) Does that seem reasonable for a daily model with this much CVRMSE?
Thanks.

