OpenEEmeter Meetup
Phil Ngo
Hi OpenEEmeter users! The second OpenEEmeter meetup is scheduled for October 28, 12:302pm Pacific, so be sure to get it on your calendar if you haven't yet! As requested at the kickoff event, this meetup will be centered around a "show and tell", for which we will be offering a few 1530 minute slots to discuss a project either using or adjacent to the OpenEEmeter. Please send proposals to me with a short description of what you would like to show off and how much time you think you'd need and we'll get you in if we still have room. We will send out a sign up link for attendees and presenters when we get closer to the event. I'm looking forward to this! Phil  Phil Ngo Director of Engineering 801.244.9860 • LinkedIn
Recurve.com • Newsletter • LinkedIn • AngelList • Twitter


Event: OpenEEmeter Meetup
#calinvite
openeemeter@lists.lfenergy.org Calendar <openeemeter@...>
OpenEEmeter Meetup When: Where: Organizer: Phil Ngo phil@... An RSVP is requested. Click here to RSVP Description:


LF energy data architecture working group presentation
Sander
Hi openEEmeter community, With LF energy data architecture working group we like to have more insight in the current LF energy projects and their data architecture. The goal of the data architecture is to improve interopabilty of the LF energy projects.
We would like to get insight in the following topics. Can you guys give an 30 minute presentation around this topics during one of the office hours?
Please select a date and I will send an invite.
Data architecture working document:
Kind regards,
Data Architect
 Alliander IT Data & Insight
The content of this email, including any attachments, is personal and confidential. If this message is not intended for you, please inform the sender by return and destroy the message. Please do not use or copy the content of this email, including any attachments, or send it to third parties.


Reminder: OpenEEmeter Meetup Kickoff tomorrow at 12:30pm Pacific
Phil Ngo
I'm looking forward to getting together with some of you at the OpenEEmeter Meetup Kickoff tomorrow! Phil


Re: Implementing Fractional Savings Uncertainty (FSU)
nhi.ngo@...
Hi Phil and Steve,
Thank you very much for your responses. I really appreciate you spending time sharing your insights. Steve, your experiment seems very interesting. I would have to think more about the impact of these measurements on the portfolio FSU. Phil, your explanation is very clear and helpful. To clarify, yes, my intent is to know the expected FSU for a portfolio of 100 sites. However, with your explanation above, I think I understand and now what to do for my analysis. So thank you very much. Though there is one more issue I would like to raise. Ideally, my team is hoping to eventually move toward using hourly data. However, since uncertainty for hourly method is a tricky subject, we are planning to use daily method FSU to gauge uncertainty. My question is from your experience, do you expect FSU to decrease as we move from billing to daily method for the same site? That has been my assumption since the beginning as we have more granular data when using daily method, thus I would expect nonfractional uncertainty to decrease. However, when I conducted testing on 1 site using hourly, daily and billing method, metered savings results seem to be within acceptable range but FSU results seem to contradict my assumptions. I have significantly larger nonfractional and fractional savings uncertainty when using daily method. Do you have any suggestion or thought on this matter? Thank you very for your help! Nhi


Re: Implementing Fractional Savings Uncertainty (FSU)
Steve Schmidt
For my own edification I duplicated the FSU calculation Nhi provided in the attached spreadsheet. Hopefully I got it right.
Then for fun I "swapped" savings and FSU values between the two buildings in two different scenarios to see the impact on Portfolio FSU. Again, I hope someone will check these thought experiments to see if they make sense. If they are, it shows that the individual project saving rates, absolute saving amounts, and FSU percentage have big impacts on the portfolio FSU. Steve


Re: Implementing Fractional Savings Uncertainty (FSU)
ngo.phil@...
Hi Nhi, Thanks for reaching out. I'll do my best to answer your questions, although I will ask for some clarification on the second one.2) I'm not sure I completely understand the intent  do you want to know the expected FSU for a portfolio of 100 sites? or something else? 2a) I can point you this document from the CPUC that describes "Normalized Metered Energy Consumption Working Group Recommendations for PopulationLevel Approaches" which lists the 25% threshold. I will also ask around at Recurve if there is a publication or public dataset that can be shared to back up the statement read on the website, but I can confirm from personal experience that it is a reasonable expectation for programs with either deep savings or a very large number of projects (or both). Because the FSU is divided by the savings value, you can expect higher FSU values if you're expecting lower percent savings (this would be the case for 1% savings  you will likely find in your bootstrapping analysis that you will need a lot more projects to hit this threshold than you would with deeper savings).
On Wed, Mar 11, 2020 at 1:33 PM <nhi.ngo@...> wrote: Hi all,


Implementing Fractional Savings Uncertainty (FSU)
nhi.ngo@...
Hi all,
My name is Nhi Ngo from Arcadia and I am working on implementing CalTRACK to evaluate one of our products. I am specifically looking at the Billing Method and want to better understand the interpretation and implementation of FSU at the residential portfolio level. I have a few questions if you don't mind. 1. Before asking other questions, I want to make sure we interpret the FSU aggregation method correctly. The FSU error band at 80% column is the direct output of the model from error_bands['FSU Error Band'] and we try to demonstrate the aggregation method for 2 sites. As this example suggests, the portfolio FSU is smaller than each individual site's FSU (46% and 13.6%). Does this seem to be counterintuitive and can you help confirm whether our calculation is correct? 2. Based on the statement on Recurve's website: "CalTRACK tests have shown that aggregation of individual projects, can, in most cases, deliver less than 25 percent portfoliolevel savings uncertainty (at the 90 percent confidence level) with very reasonably sized portfolios", we are working on deriving expected FSU for different portfolio sizes to set confidence expectation for our pilot program. We are aiming for 1000 sites but we might have much less so it is important to set expectation if we don't have the portfolio size we want. We currently have a sample of 1700 residential sites with average savings ~1% (some have negative savings). We are planning to use bootstrapping method to resample different portfolio sizes (100, 200...) from the 1700 and derive expected FSU. For example, we will resample 10,000 100site portfolios and derive FSU error bands and FSU for each 100site portfolio using the #1 calculation. However, we are unsure how to proceed to derive FSU for 10,000 portfolios. Should we take mean(FSU)? Or should we take mean(FSU error bands)/mean(sum(metered_savings)) where sum(metered_savings) is the total savings for each 100site portfolio? If we do the later, the results seem to be more reasonable but we want to ask for your comment and suggestion if the method is statistically correct. Also, if you can share with us the test or methodology you performed to arrive at the bolded statement above, that would be great. I think we could simulate similar test to achieve our goal. I hope the questions are clear enough and please feel free to ask more questions to clarify. Thank you very much for your time and I apologize if the questions seem trivial. Sincerely, Nhi Ngo


Re: How to Use Model Metrics to Gauge Uncertainty
Si Chen <sichen@...>
Thanks for pointing that out, Phil. It seems that CalTrack 4.3.2.4 has replaced ASHRAE's 1.26 "empirical coefficient" with a formula, and for M=12 (12 reporting periods) it comes out to 1.30 for billing (monthly) data and 1.39 for daily data. Is P' calculated from P the same way here that n' is calculated from n from the ASHRAE formula, using the autocorrelation coefficient rho? Finally how do we get the number of model parameters or "number of explanatory variables in the baseline model"? 
On Wed, Mar 4, 2020 at 4:30 PM <ngo.phil@...> wrote:


Re: How to Use Model Metrics to Gauge Uncertainty
Steve Schmidt
My belief: if the building is "well behaved" with respect to outdoor temperatures and heating and cooling loads, then other nonHVAC loads should have no impact on model fit. But I'm not an OEE expert so I'll let Phil correct this.


Re: How to Use Model Metrics to Gauge Uncertainty
Michael S Uhl
Is it possible for energy loads that occur at specific times of day (unrelated to CDD or HDD), due to timeofuse pricing, to negatively impact the model accuracy? If so, how can these other variables be addressed?
On Thu, Mar 5, 2020 at 4:53 PM Steve Schmidt <steve@...> wrote: A few additional comments  
All the Best, M. System Smart 484.553.4570 Sent on the go. Pardon grammar/spelling.


Re: How to Use Model Metrics to Gauge Uncertainty
Steve Schmidt
A few additional comments 


Re: How to Use Model Metrics to Gauge Uncertainty
ngo.phil@...
1. Correct  autocorr_resid is rho 2. The value of n should be 365, that is correct. It sounds like you have the right idea for m as well (i.e, if you have 30 daily predictions and want to know the uncertainty of the sum of those thirty predictions, m should be 30) with a slight caveat that CalTRACK suggests handling these calculations using a polynomial correction using experimentally derived coefficients. See section 4.3, http://docs.caltrack.org/en/latest/methods.html#section4aggregation. In that case, there is also an M (capitalized) to keep track of, which is the number of months (regardless of frequency  which is taken into account by using different coefficients for daily and monthly billing data.)
On Wed, Mar 4, 2020 at 3:01 PM Si Chen <sichen@...> wrote:


How to Use Model Metrics to Gauge Uncertainty
We've fitted some models and would like to know how to use them to really understand the quality of the models. The model metrics look like this:
and comparing it to ASHRAE 14 guidelines, which gives us these formulas: My questions are: 1. Is the autocorr_resid the rho (p) is B14? 2. What are the right parameters for n and m? According to an early page in ASHRAE 14, n and m are "number of observations in the baseline (or pre retrofit) and the postECM periods, respectively" If the model is a daily, should n be 365, so in this case, n' = 365 * (10.4792) / (1+0.4792) = 128.5? If the model is used to compare energy savings over a year, should m be 365? Or should m be 30 if we're comparing the energy savings on a monthly basis? 3. How many model parameters are there? In a combined heating and cooling model, should it be 5  2 betas, 2 balance points, and an intercept  or 3? Calculating all this from my example model, I get a 25.8% uncertainty for F (energy savings) of 20% at 68% confidence (t = 1) Does that seem reasonable for a daily model with this much CVRMSE? Thanks.


Re: What do the red vs green lines in the model graph mean?
Si Chen <sichen@...>
OK, thanks for the clarification!
On Thu, Feb 27, 2020 at 9:58 AM <ngo.phil@...> wrote:


Re: What do the red vs green lines in the model graph mean?
ngo.phil@...
Great question  green are qualified model candidates, red are disqualified model candidates, orange is the selected model candidate. If you look carefully, you'll notice all the red ones have negative model parameters, which disqualify the models. This behavior is described in CalTRACK 3.4.3.2. (http://docs.caltrack.org/en/latest/methods.html#section3amodelingbillinganddailymethods). It's not well documented, as you noticed, but here's the code if you're interested: https://github.com/openeemeter/eemeter/blob/c10c384e0ceb39a4eb79d4532d5516d40f9bf3be/eemeter/caltrack/usage_per_day.py#L2273L2282.
On Thu, Feb 27, 2020 at 9:28 AM Si Chen <sichen@...> wrote: Hello,


What do the red vs green lines in the model graph mean?
Si Chen <sichen@...>
Hello,
We just fitted a daily model, and the graph shows both red and green lines. What do these different colored lines mean?


Re: gas and electric meters at the same building
Si Chen <sichen@...>
OK, thank you very much for clearing that up.
On Tue, Feb 25, 2020 at 8:25 AM <ngo.phil@...> wrote:


Re: gas and electric meters at the same building
ngo.phil@...
Definitely, for buildings with multiple meters, there should be a separate model for each meter. It is important to use different parameters for gas and electricity, as you read in the CalTRACK compliance document.
On Mon, Feb 24, 2020 at 2:55 PM Si Chen <sichen@...> wrote: Hello,


gas and electric meters at the same building
Si Chen <sichen@...>
Hello,
What do you recommend when a building has gas and electric meters? I read on https://github.com/ For natural gas meter use data, the function :any:`eemeter.fit_
cooling_balance_points= so that models using cooling degree days are not considered.3.4.3.1: :any:`eemeter.fit_
fit_cdd=True, fit_intercept_only=True, fit_cdd_only=True, fit_hdd_only=True, fit_cdd_hdd=True for electricity data, and fit_cdd=False, fit_intercept_only=True, fit_cdd_only=False, fit_hdd_only=True, fit_cdd_hdd=False for gas data.So do you recommend building separate models for gas and electric with the parameters changed?

