Hi Phil,

Thanks for setting this up.

I'd like to share a little bit about what we're doing at the Hyperledger Energy Working Group.  Hyperledger is the open source blockchain project of the Linux Foundation.  I think it will have a lot of potential synergies with what you're doing.

Would you be able to give me 15 minutes to talk about this?

-----
Si Chen
Open Source Strategies, Inc.

Join our Hyperledger Open Source Carbon Accounting & Certification Working Group - Video

On Sat, Sep 19, 2020 at 10:22 AM Phil Ngo <phil@...> wrote:

Hi OpenEEmeter users!

The second OpenEEmeter meetup is scheduled for October 28, 12:30-2pm Pacific, so be sure to get it on your calendar if you haven't yet! As requested at the kickoff event, this meetup will be centered around a "show and tell", for which we will be offering a few 15-30 minute slots to discuss a project either using or adjacent to the OpenEEmeter. Please send proposals to me with a short description of what you would like to show off and how much time you think you'd need and we'll get you in if we still have room. We will send out a sign up link for attendees and presenters when we get closer to the event.

I'm looking forward to this!

Phil

--

Phil Ngo

Director of Engineering

801.244.9860 • LinkedIn

 

Recurve.com • Newsletter • LinkedIn • AngelList • Twitter

Hi OpenEEmeter users!

The second OpenEEmeter meetup is scheduled for October 28, 12:30-2pm Pacific, so be sure to get it on your calendar if you haven't yet! As requested at the kickoff event, this meetup will be centered around a "show and tell", for which we will be offering a few 15-30 minute slots to discuss a project either using or adjacent to the OpenEEmeter. Please send proposals to me with a short description of what you would like to show off and how much time you think you'd need and we'll get you in if we still have room. We will send out a sign up link for attendees and presenters when we get closer to the event.

I'm looking forward to this!

Phil

--

Phil Ngo

Director of Engineering

801.244.9860 • LinkedIn

 

Recurve.com • Newsletter • LinkedIn • AngelList • Twitter

Hi all,

My name is Nhi Ngo from Arcadia and I am working on implementing CalTRACK to evaluate one of our products. I am specifically looking at the Billing Method and want to better understand the interpretation and implementation of FSU at the residential portfolio level. I have a few questions if you don't mind.

1. Before asking other questions, I want to make sure we interpret the FSU aggregation method correctly. The FSU error band at 80% column is the direct output of the model from error_bands['FSU Error Band'] and we try to demonstrate the aggregation method for 2 sites.  As this example suggests, the portfolio FSU is smaller than each individual site's FSU (46% and 13.6%). Does this seem to be counter-intuitive and can you help confirm whether our calculation is correct?



2. Based on the statement on Recurve's website: "CalTRACK tests have shown that aggregation of individual projects, can, in most cases, deliver less than 25 percent portfolio-level savings uncertainty (at the 90 percent confidence level) with very reasonably sized portfolios", we are working on deriving expected FSU for different portfolio sizes to set confidence expectation for our pilot program. We are aiming for 1000 sites but we might have much less so it is important to set expectation if we don't have the portfolio size we want.

We currently have a sample of 1700 residential sites with average savings ~1% (some have negative savings). We are planning to use bootstrapping method to resample different portfolio sizes (100, 200...) from the 1700 and derive expected FSU. For example, we will resample 10,000 100-site portfolios and derive FSU error bands and FSU for each 100-site portfolio using the #1 calculation. However, we are unsure how to proceed to derive FSU for 10,000 portfolios. Should we take mean(FSU)? Or should we take mean(FSU error bands)/mean(sum(metered_savings)) where sum(metered_savings) is the total savings for each 100-site portfolio? If we do the later, the results seem to be more reasonable but we want to ask for your comment and suggestion if the method is statistically correct.

Also, if you can share with us the test or methodology you performed to arrive at the bolded statement above, that would be great. I think we could simulate similar test to achieve our goal.

I hope the questions are clear enough and please feel free to ask more questions to clarify. Thank you very much for your time and I apologize if the questions seem trivial.

Sincerely,
Nhi Ngo

1. Correct - autocorr_resid is rho

2. The value of n should be 365, that is correct. It sounds like you have the right idea for m as well (i.e, if you have 30 daily predictions and want to know the uncertainty of the sum of those thirty predictions, m should be 30) with a slight caveat that CalTRACK suggests handling these calculations using a polynomial correction using experimentally derived coefficients. See section 4.3, http://docs.caltrack.org/en/latest/methods.html#section-4-aggregation. In that case, there is also an M (capitalized) to keep track of, which is the number of months (regardless of frequency - which is taken into account by using different coefficients for daily and monthly billing data.)

On Wed, Mar 4, 2020 at 3:01 PM Si Chen <sichen@...> wrote:

[Edited Message Follows]

We've fitted some models and would like to know how to use them to really understand the quality of the models.  The model metrics look like this:



and comparing it to ASHRAE 14 guidelines, which gives us these formulas:



My questions are:

1. Is the autocorr_resid the rho (p) is B-14?
2.  What are the right parameters for n and m?  According to an early page in ASHRAE 14, n and m are "number of observations in the baseline (or pre- retrofit) and the post-ECM periods, respectively"   If the model is a daily, should n be 365, so in this case, n' = 365 * (1-0.4792) / (1+0.4792) = 128.5?  If the model is used to compare energy savings over a year, should m be 365?  Or should m be 30 if we're comparing the energy savings on a monthly basis?
3.  How many model parameters are there?  In a combined heating and cooling model, should it be 5 -- 2 betas, 2 balance points, and an intercept -- or 3?

Calculating all this from my example model, I get a 25.8% uncertainty for F (energy savings) of 20% at 68% confidence (t = 1)  Does that seem reasonable for a daily model with this much CVRMSE?

Thanks.

A few additional comments --

1. I'd call this a "bad building". Based on the CalTRACK model fit results, energy use is not very predictable. The ASHRAE Guideline 14 requirement for a good model fit is CVRMSE < 0.25; this value of 0.46 is far above that target. Perhaps you can note this to users of your system, so they don't rely too heavily on the model.
2. Savings calculations using such a [poor] model will be inaccurate. I'm no statistician, but I believe an R-squared value of 0.4 indicates some correlation, but is not considered useful for prediction. Current CalTRACK methods use any model with CVRMSE values below 1.0 to predict the counterfactual, so it's up to users to recognize when a fit is good and when it's not.
3. It's odd that the cooling and heating balance points are the same; normally there are several degrees separation between the two. Maybe it's a strange building, or maybe Phil can explain this.

[Edited Message Follows]

We've fitted some models and would like to know how to use them to really understand the quality of the models.  The model metrics look like this:



and comparing it to ASHRAE 14 guidelines, which gives us these formulas:



My questions are:

1. Is the autocorr_resid the rho (p) is B-14?
2.  What are the right parameters for n and m?  According to an early page in ASHRAE 14, n and m are "number of observations in the baseline (or pre- retrofit) and the post-ECM periods, respectively"   If the model is a daily, should n be 365, so in this case, n' = 365 * (1-0.4792) / (1+0.4792) = 128.5?  If the model is used to compare energy savings over a year, should m be 365?  Or should m be 30 if we're comparing the energy savings on a monthly basis?
3.  How many model parameters are there?  In a combined heating and cooling model, should it be 5 -- 2 betas, 2 balance points, and an intercept -- or 3?

Calculating all this from my example model, I get a 25.8% uncertainty for F (energy savings) of 20% at 68% confidence (t = 1)  Does that seem reasonable for a daily model with this much CVRMSE?

Thanks.

Great question - green are qualified model candidates, red are disqualified model candidates, orange is the selected model candidate. If you look carefully, you'll notice all the red ones have negative model parameters, which disqualify the models. This behavior is described in CalTRACK 3.4.3.2. (http://docs.caltrack.org/en/latest/methods.html#section-3-a-modeling-billing-and-daily-methods). It's not well documented, as you noticed, but here's the code if you're interested: https://github.com/openeemeter/eemeter/blob/c10c384e0ceb39a4eb79d4532d5516d40f9bf3be/eemeter/caltrack/usage_per_day.py#L2273-L2282.

On Thu, Feb 27, 2020 at 9:28 AM Si Chen <sichen@...> wrote:

Hello,

We just fitted a daily model, and the graph shows both red and green lines.  What do these different colored lines mean?

Hello,

We just fitted a daily model, and the graph shows both red and green lines.  What do these different colored lines mean?