Bayesian power analysis #276 #277

cetagostini · 2023-12-05T00:06:16Z

Hey everyone!

As we discussed in issue #276, I've added a new function to CausalPy that performs power analysis. This is based on my internal work on estimating effects and conducting sensitivity analysis within Bayesian frameworks.

What will you find?

Notebook
Code example

I hope this explanation makes sense. Ideally, I'd like to have a few rounds of feedback and answer any questions you may have before moving on to unit testing.

Looking forward to hearing your thoughts!

cc: @drbenvincent

review-notebook-app · 2023-12-05T00:06:21Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

drbenvincent · 2023-12-12T11:30:03Z

FYI: I changed the filename of the notebook to snake case and I also update the pre-commits to use the --fix flag for nbqa-ruff. So this has modified the notebook a little - mainly just removing redundant imports I believe.

Codecov Report

Attention: 9 lines in your changes are missing coverage. Please review.

Comparison is base (70de921) 76.27% compared to head (c534c55) 77.40%.

Files	Patch %	Lines
causalpy/pymc_experiments.py	89.04%	8 Missing ⚠️
causalpy/utils.py	90.90%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #277      +/-   ##
==========================================
+ Coverage   76.27%   77.40%   +1.12%     
==========================================
  Files          20       20              
  Lines        1332     1425      +93     
==========================================
+ Hits         1016     1103      +87     
- Misses        316      322       +6

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

drbenvincent · 2023-12-22T14:40:17Z

Remote tests seem to be passing now :) Give me a shout whenever you manage to work on updates @cetagostini :)

cetagostini · 2024-01-02T19:33:05Z

Hey hey @drbenvincent, All the pre-commits are working on my side, except one related to ´pkg_resources´.

Let me know, if extra changes are needed.

drbenvincent

Great improvements. I didn't get time to properly review the example notebook, but I've left a bunch of change requests for the code.

If there are changes when you address the point about where in the class hierarchy the methods are, can you run make uml and that will update the UML class diagram which we have displayed in CONTRIBUTING.md

If you're ok then we'll just iterate a bit more. But like I said before, this is great so far.

drbenvincent · 2024-01-05T15:47:06Z

causalpy/pymc_experiments.py

+        """
+        return pd.DataFrame(self._power_estimation(alpha=alpha, correction=correction))
+
+    def power_plot(self, alpha: float = 0.05, correction: bool = False) -> plt.Figure:


Can I request this is called plot_power instead?

drbenvincent · 2024-01-05T15:53:26Z

causalpy/pymc_experiments.py

+
+        # Posterior Mean
+        results["posterior_estimation"] = {
+            "cumulative": round(np.mean(_mu_samples_cumulative.values), 2),


Can we change all instances of round(VALUE, 2) to round_num(VALUE, round_to) where round_to is a used-provided kwarg.

Oh, also, I think we should only do this rounding at the point we display the results. So where we calculate results, let's remove this rounding, and instead only have it when results are outputted.

If that's done by plotting pandas dataframes, then it's also possible to use pd.set_option('display.precision', 2) for example in the example notebook.

drbenvincent · 2024-01-05T16:03:30Z

causalpy/pymc_experiments.py

+        }
+        return results
+
+    def power_summary(


The user-facing methods power_summary and plot_power are currently methods of the PrePostFit class. You can see from the UML class diagram that these methods will be available to both the SyntheticControl experiment class and the Interrupted Time Series class.

Clearly from the example you are using Bayesian Power Analysis in the context of synthetic control, but can I ask... Does it make sense that these methods are also available in the interrupted time series situations? I don't know the answer, I've not thought about it. But we want to make sure that these methods are placed at the right point in the class hierarchy.

If the Bayesian Power Analysis is also appropriate for the interrupted time series context, then the best possible outcome would be to also create an example notebook for that. But I'd be fine if you instead created a separate issue to deal with that in the near future.

But if this new functionality only makes sense in the context of synthetic control, then can I ask you to either a) move these new methods into the SyntheticControl class, or b) create a Bayesian Power Analysis mix-in class instead. I don't mind which, but I think CausalPy will probably transition from the current class hierarchy to using mix-ins.

drbenvincent · 2024-01-05T16:05:24Z

causalpy/pymc_experiments.py

+        """
+        Summarize the power estimation results in a DataFrame format.
+
+        This function calls '_power_estimation' to perform power estimation calculations and then


Right now, the docs are set up to not render docstrings for private methods. At the moment I think this is fine because the API docs will focus only on what users are intended to use. If we keep it like that, then we might want to remove references to private methods in the public docstrings.

drbenvincent · 2024-01-05T16:08:30Z

causalpy/pymc_experiments.py

+        minimum detectable effects (MDE) for both cumulative and mean measures. It can apply corrections to
+        account for systematic differences in the data.
+
+        Parameters:


At the moment the parameters and returns parts of the docstrings are not rendering properly when you build the docs. Can you check the notation used in the other docstrings and just double check this is rendering ok?

drbenvincent · 2024-01-05T16:09:09Z

causalpy/pymc_experiments.py

+        This function calls '_power_estimation' to perform power estimation calculations and then
+        converts the resulting dictionary into a pandas DataFrame for easier analysis and visualization.
+
+        Parameters:


Same comment as above about the parameters and returns blocks not rendering properly in when you build and check the docs locally.

drbenvincent · 2024-01-05T16:09:37Z

causalpy/pymc_experiments.py

+        along with the confidence intervals, real mean, and posterior mean values. It allows for adjustments based on
+        systematic differences if the correction is applied.
+
+        Parameters:


Same comment as above about the parameters and returns blocks not rendering properly in when you build and check the docs locally.

drbenvincent · 2024-01-05T16:11:09Z

causalpy/utils.py

+    """
+    Calculate the probability of a given value being in a distribution defined by the posterior,
+
+    Args:


Same comment as above about the parameters and returns blocks not rendering properly in when you build and check the docs locally.

drbenvincent · 2024-01-05T16:12:25Z

causalpy/utils.py

+    if cdf_x <= 0.5:
+        probability = 2 * (cdf_x - cdf_lower) / (1 - cdf_lower - cdf_upper)
+    else:
+        probability = 2 * (1 - cdf_x + cdf_lower) / (1 - cdf_lower - cdf_upper)


I think it would be safe, and beneficial, to make sure this line is covered by tests

drbenvincent · 2024-01-05T16:13:27Z

causalpy/utils.py

+    else:
+        probability = 2 * (1 - cdf_x + cdf_lower) / (1 - cdf_lower - cdf_upper)
+
+    return abs(round(probability, 2))


My feeling is that we should not round the result at this point, only when displaying the results.

cetagostini · 2024-02-11T23:09:06Z

@drbenvincent Starting this week to add the changes requested 🚀

Bayesian power analysis pymc-labs#276

9e2bdb2

small changes on notebook

e744cfc

drbenvincent self-requested a review December 12, 2023 11:03

nbqa-ruff fix + snake case filename for notebook

e06c93c

update examples.rst so notebook shows up in readthedocs

1407bf0

drbenvincent reviewed Dec 12, 2023

View reviewed changes

drbenvincent requested changes Dec 12, 2023

View reviewed changes

drbenvincent added the enhancement New feature or request label Dec 22, 2023

Merge branch 'main' into pr/277

6fddee8

cetagostini added 2 commits December 29, 2023 00:03

Request by Ben

baef9cc

Merge branch 'main' into dev_power_analysis

6c3cf74

cetagostini requested a review from drbenvincent December 28, 2023 22:07

cetagostini added 2 commits December 29, 2023 00:12

Changes on functions

20e62d8

Small change

cb88722

small changes

c534c55

drbenvincent requested changes Jan 5, 2024

View reviewed changes

cetagostini closed this Feb 15, 2024

cetagostini mentioned this pull request Feb 15, 2024

Creating Bayesian power analysis method #292

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bayesian power analysis #276 #277

Bayesian power analysis #276 #277

cetagostini commented Dec 5, 2023 •

edited

Loading

review-notebook-app bot commented Dec 5, 2023

drbenvincent commented Dec 12, 2023

drbenvincent Dec 12, 2023 •

edited

Loading

cetagostini Dec 28, 2023

drbenvincent Jan 5, 2024

drbenvincent left a comment

drbenvincent commented Dec 12, 2023

drbenvincent commented Dec 13, 2023

codecov bot commented Dec 22, 2023 •

edited

Loading

drbenvincent commented Dec 22, 2023

cetagostini commented Jan 2, 2024

drbenvincent left a comment

drbenvincent Jan 5, 2024

drbenvincent Jan 5, 2024

drbenvincent Jan 5, 2024

drbenvincent Jan 5, 2024

drbenvincent Jan 5, 2024

drbenvincent Jan 5, 2024

drbenvincent Jan 5, 2024

drbenvincent Jan 5, 2024

drbenvincent Jan 5, 2024

drbenvincent Jan 5, 2024

drbenvincent Jan 5, 2024

cetagostini commented Feb 11, 2024

		@@ -0,0 +1,1412 @@
		{

Bayesian power analysis #276 #277

Bayesian power analysis #276 #277

Conversation

cetagostini commented Dec 5, 2023 • edited Loading

review-notebook-app bot commented Dec 5, 2023

drbenvincent commented Dec 12, 2023

drbenvincent Dec 12, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drbenvincent left a comment

Choose a reason for hiding this comment

drbenvincent commented Dec 12, 2023

drbenvincent commented Dec 13, 2023

codecov bot commented Dec 22, 2023 • edited Loading

Codecov Report

drbenvincent commented Dec 22, 2023

cetagostini commented Jan 2, 2024

drbenvincent left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cetagostini commented Feb 11, 2024

cetagostini commented Dec 5, 2023 •

edited

Loading

drbenvincent Dec 12, 2023 •

edited

Loading

codecov bot commented Dec 22, 2023 •

edited

Loading