1. Messages
  2. Implementing Fractional Savings Uncertainty (FSU)

Re: Implementing Fractional Savings Uncertainty (FSU)

ngo.phil@...
 

Hi Nhi,

Thanks for reaching out. I'll do my best to answer your questions, although I will ask for some clarification on the second one.

1) It looks to me like you are interpreting the equation correctly. The intuition for why the combined FSU is lower than either of the individual FSUs is that more data generally leads to lower uncertainty. It may be helpful to remember that the FSU is a fractional/normalized value, quantifying the uncertainty relative to the level of savings. The non-fractional or non-normalized total savings uncertainty adds like variances do, that is, by taking the root of the sum of the squares. So the total uncertainty will be increasing, but by less than the total value of the savings, and thus the ratio of these decreases.

2) I'm not sure I completely understand the intent - do you want to know the expected FSU for a portfolio of 100 sites? or something else?

2a) I can point you this document from the CPUC that describes "Normalized Metered Energy Consumption Working Group Recommendations for Population-Level Approaches" which lists the 25% threshold. I will also ask around at Recurve if there is a publication or public dataset that can be shared to back up the statement read on the website, but I can confirm from personal experience that it is a reasonable expectation for programs with either deep savings or a very large number of projects (or both). Because the FSU is divided by the savings value, you can expect higher FSU values if you're expecting lower percent savings (this would be the case for 1% savings - you will likely find in your bootstrapping analysis that you will need a lot more projects to hit this threshold than you would with deeper savings).


On Wed, Mar 11, 2020 at 1:33 PM <nhi.ngo@...> wrote:
Hi all,

My name is Nhi Ngo from Arcadia and I am working on implementing CalTRACK to evaluate one of our products. I am specifically looking at the Billing Method and want to better understand the interpretation and implementation of FSU at the residential portfolio level. I have a few questions if you don't mind.

1. Before asking other questions, I want to make sure we interpret the FSU aggregation method correctly. The FSU error band at 80% column is the direct output of the model from error_bands['FSU Error Band'] and we try to demonstrate the aggregation method for 2 sites.  As this example suggests, the portfolio FSU is smaller than each individual site's FSU (46% and 13.6%). Does this seem to be counter-intuitive and can you help confirm whether our calculation is correct?



2. Based on the statement on Recurve's website: "CalTRACK tests have shown that aggregation of individual projects, can, in most cases, deliver less than 25 percent portfolio-level savings uncertainty (at the 90 percent confidence level) with very reasonably sized portfolios", we are working on deriving expected FSU for different portfolio sizes to set confidence expectation for our pilot program. We are aiming for 1000 sites but we might have much less so it is important to set expectation if we don't have the portfolio size we want.

We currently have a sample of 1700 residential sites with average savings ~1% (some have negative savings). We are planning to use bootstrapping method to resample different portfolio sizes (100, 200...) from the 1700 and derive expected FSU. For example, we will resample 10,000 100-site portfolios and derive FSU error bands and FSU for each 100-site portfolio using the #1 calculation. However, we are unsure how to proceed to derive FSU for 10,000 portfolios. Should we take mean(FSU)? Or should we take mean(FSU error bands)/mean(sum(metered_savings)) where sum(metered_savings) is the total savings for each 100-site portfolio? If we do the later, the results seem to be more reasonable but we want to ask for your comment and suggestion if the method is statistically correct.

Also, if you can share with us the test or methodology you performed to arrive at the bolded statement above, that would be great. I think we could simulate similar test to achieve our goal.

I hope the questions are clear enough and please feel free to ask more questions to clarify. Thank you very much for your time and I apologize if the questions seem trivial.

Sincerely,
Nhi Ngo
Join {openeemeter@lists.lfenergy.org to automatically receive all group messages.