1. Openeemeter
  2. Topics

Implementing Fractional Savings Uncertainty (FSU)

nhi.ngo@...
 

Hi all,

My name is Nhi Ngo from Arcadia and I am working on implementing CalTRACK to evaluate one of our products. I am specifically looking at the Billing Method and want to better understand the interpretation and implementation of FSU at the residential portfolio level. I have a few questions if you don't mind.

1. Before asking other questions, I want to make sure we interpret the FSU aggregation method correctly. The FSU error band at 80% column is the direct output of the model from error_bands['FSU Error Band'] and we try to demonstrate the aggregation method for 2 sites.  As this example suggests, the portfolio FSU is smaller than each individual site's FSU (46% and 13.6%). Does this seem to be counter-intuitive and can you help confirm whether our calculation is correct?



2. Based on the statement on Recurve's website: "CalTRACK tests have shown that aggregation of individual projects, can, in most cases, deliver less than 25 percent portfolio-level savings uncertainty (at the 90 percent confidence level) with very reasonably sized portfolios", we are working on deriving expected FSU for different portfolio sizes to set confidence expectation for our pilot program. We are aiming for 1000 sites but we might have much less so it is important to set expectation if we don't have the portfolio size we want.

We currently have a sample of 1700 residential sites with average savings ~1% (some have negative savings). We are planning to use bootstrapping method to resample different portfolio sizes (100, 200...) from the 1700 and derive expected FSU. For example, we will resample 10,000 100-site portfolios and derive FSU error bands and FSU for each 100-site portfolio using the #1 calculation. However, we are unsure how to proceed to derive FSU for 10,000 portfolios. Should we take mean(FSU)? Or should we take mean(FSU error bands)/mean(sum(metered_savings)) where sum(metered_savings) is the total savings for each 100-site portfolio? If we do the later, the results seem to be more reasonable but we want to ask for your comment and suggestion if the method is statistically correct.

Also, if you can share with us the test or methodology you performed to arrive at the bolded statement above, that would be great. I think we could simulate similar test to achieve our goal.

I hope the questions are clear enough and please feel free to ask more questions to clarify. Thank you very much for your time and I apologize if the questions seem trivial.

Sincerely,
Nhi Ngo


Capture.PNG

ngo.phil@...
 

Hi Nhi,

Thanks for reaching out. I'll do my best to answer your questions, although I will ask for some clarification on the second one.

1) It looks to me like you are interpreting the equation correctly. The intuition for why the combined FSU is lower than either of the individual FSUs is that more data generally leads to lower uncertainty. It may be helpful to remember that the FSU is a fractional/normalized value, quantifying the uncertainty relative to the level of savings. The non-fractional or non-normalized total savings uncertainty adds like variances do, that is, by taking the root of the sum of the squares. So the total uncertainty will be increasing, but by less than the total value of the savings, and thus the ratio of these decreases.

2) I'm not sure I completely understand the intent - do you want to know the expected FSU for a portfolio of 100 sites? or something else?

2a) I can point you this document from the CPUC that describes "Normalized Metered Energy Consumption Working Group Recommendations for Population-Level Approaches" which lists the 25% threshold. I will also ask around at Recurve if there is a publication or public dataset that can be shared to back up the statement read on the website, but I can confirm from personal experience that it is a reasonable expectation for programs with either deep savings or a very large number of projects (or both). Because the FSU is divided by the savings value, you can expect higher FSU values if you're expecting lower percent savings (this would be the case for 1% savings - you will likely find in your bootstrapping analysis that you will need a lot more projects to hit this threshold than you would with deeper savings).


On Wed, Mar 11, 2020 at 1:33 PM <nhi.ngo@...> wrote:
Hi all,

My name is Nhi Ngo from Arcadia and I am working on implementing CalTRACK to evaluate one of our products. I am specifically looking at the Billing Method and want to better understand the interpretation and implementation of FSU at the residential portfolio level. I have a few questions if you don't mind.

1. Before asking other questions, I want to make sure we interpret the FSU aggregation method correctly. The FSU error band at 80% column is the direct output of the model from error_bands['FSU Error Band'] and we try to demonstrate the aggregation method for 2 sites.  As this example suggests, the portfolio FSU is smaller than each individual site's FSU (46% and 13.6%). Does this seem to be counter-intuitive and can you help confirm whether our calculation is correct?



2. Based on the statement on Recurve's website: "CalTRACK tests have shown that aggregation of individual projects, can, in most cases, deliver less than 25 percent portfolio-level savings uncertainty (at the 90 percent confidence level) with very reasonably sized portfolios", we are working on deriving expected FSU for different portfolio sizes to set confidence expectation for our pilot program. We are aiming for 1000 sites but we might have much less so it is important to set expectation if we don't have the portfolio size we want.

We currently have a sample of 1700 residential sites with average savings ~1% (some have negative savings). We are planning to use bootstrapping method to resample different portfolio sizes (100, 200...) from the 1700 and derive expected FSU. For example, we will resample 10,000 100-site portfolios and derive FSU error bands and FSU for each 100-site portfolio using the #1 calculation. However, we are unsure how to proceed to derive FSU for 10,000 portfolios. Should we take mean(FSU)? Or should we take mean(FSU error bands)/mean(sum(metered_savings)) where sum(metered_savings) is the total savings for each 100-site portfolio? If we do the later, the results seem to be more reasonable but we want to ask for your comment and suggestion if the method is statistically correct.

Also, if you can share with us the test or methodology you performed to arrive at the bolded statement above, that would be great. I think we could simulate similar test to achieve our goal.

I hope the questions are clear enough and please feel free to ask more questions to clarify. Thank you very much for your time and I apologize if the questions seem trivial.

Sincerely,
Nhi Ngo
Steve Schmidt
 

For my own edification I duplicated the FSU calculation Nhi provided in the attached spreadsheet. Hopefully I got it right.

Then for fun I "swapped" savings and FSU values between the two buildings in two different scenarios to see the impact on Portfolio FSU. Again, I hope someone will check these thought experiments to see if they make sense.

If they are, it shows that the individual project saving rates, absolute saving amounts, and FSU percentage have big impacts on the portfolio FSU.

  -Steve

CalTrack FSU Calc.xlsx

CalTrack FSU Calc.xlsx

nhi.ngo@...
 

Hi Phil and Steve,

Thank you very much for your responses. I really appreciate you spending time sharing your insights.

Steve, your experiment seems very interesting. I would have to think more about the impact of these measurements on the portfolio FSU.

Phil, your explanation is very clear and helpful. To clarify, yes, my intent is to know the expected FSU for a portfolio of 100 sites. However, with your explanation above, I think I understand and now what to do for my analysis. So thank you very much.

Though there is one more issue I would like to raise. Ideally, my team is hoping to eventually move toward using hourly data. However, since uncertainty for hourly method is a tricky subject, we are planning to use daily method FSU to gauge uncertainty. My question is from your experience, do you expect FSU to decrease as we move from billing to daily method for the same site? That has been my assumption since the beginning as we have more granular data when using daily method, thus I would expect non-fractional uncertainty to decrease. However, when I conducted testing on 1 site using hourly, daily and billing method, metered savings results seem to be within acceptable range but FSU results seem to contradict my assumptions. I have significantly larger non-fractional and fractional savings uncertainty when using daily method. Do you have any suggestion or thought on this matter?

Thank you very for your help!

-Nhi

1 - 4 of 4