How to Use Model Metrics to Gauge Uncertainty
We've fitted some models and would like to know how to use them to really understand the quality of the models. The model metrics look like this:
and comparing it to ASHRAE 14 guidelines, which gives us these formulas:
My questions are:
1. Is the autocorr_resid the rho (p) is B-14?
2. What are the right parameters for n and m? According to an early page in ASHRAE 14, n and m are "number of observations in the baseline (or pre- retrofit) and the post-ECM periods, respectively" If the model is a daily, should n be 365, so in this case, n' = 365 * (1-0.4792) / (1+0.4792) = 128.5? If the model is used to compare energy savings over a year, should m be 365? Or should m be 30 if we're comparing the energy savings on a monthly basis?
3. How many model parameters are there? In a combined heating and cooling model, should it be 5 -- 2 betas, 2 balance points, and an intercept -- or 3?
Calculating all this from my example model, I get a 25.8% uncertainty for F (energy savings) of 20% at 68% confidence (t = 1) Does that seem reasonable for a daily model with this much CVRMSE?