Add footprint finder code #127

snbianco · 2024-09-18T14:38:54Z

This is a draft PR that adds the footprint finder code from TESScut. Leaving as draft since I haven't added tests yet, but I'd welcome any feedback on the actual implementation.

I tried to make things as general as possible in terms of functions and variable names, but some things are more TESS specific and will need to be pulled out into a wrapper later when we generalize. Namely, _extract_sequence_information, _create_sequence_list, and _get_cube_files_from_sequence_obs are more TESS-specific as of now. The same is true about certain parts of cube_cut_from_footprint, mainly variables.

Something I was unsure about is how to best handle multiprocessing. The cube_cut_from_footprint function takes a threads parameter to use as max_workers when using cube_cut on the gathered cube files. However, the cube_cut function also takes its own threads parameter. Should these be two separate parameters to cube_cut_from_footprint, or the same? Should threads in cube_cut just be set to 'auto'?

astrocut/cube_cut_from_footprint.py

scfleming · 2024-09-18T15:21:24Z

My initial reaction is to say let's setup a single n_threads parameter for multi-threading of any function that can use it. While on paper it might be nice to say "use 8 threads for this one but 16 for that one", that sounds like over-engineering to me at this stage of the process, and it would be much simpler to have a single n_threads parameter used globally.

astrocut/cube_cut_from_footprint.py

astrocut/__init__.py

astrocut/cube_cut_from_footprint.py

snbianco · 2024-09-24T23:04:16Z

I added unit tests for the module, but there seems to be a problem with accessing the public footprint files on S3 in the runners. From what I can find online, this is a permissions issue. The odd thing is, we have other tests that access S3 resources and work fine.

botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

fix type annotation

Use json.load() documentation, set threads to 8 for cube_cut

snbianco · 2024-09-25T14:50:57Z

I added unit tests for the module, but there seems to be a problem with accessing the public footprint files on S3 in the runners. From what I can find online, this is a permissions issue. The odd thing is, we have other tests that access S3 resources and work fine.

botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

I added a fixture to mock opening the footprint files with fsspec, and tests are passing now.

snbianco · 2024-09-26T19:34:02Z

Marking as ready as tests and documentation have been added.

falkben

There's an issue with accessing the footprint files in the bucket -- I have a ticket to work on that next week.

astrocut/footprint_cutouts.py

astrocut/tests/data/tess_ffi_footprints.json

astrocut/footprint_cutouts.py

falkben · 2024-10-02T16:30:24Z

Since we're having a delay in opening up the cached footprint files on S3, another approach could be to download the footprint directly from CAOM through vo-tap interface on the first cutout.

This query gets the SPOC footprint table from CAOM and can be run from anywhere:

https://mast.stsci.edu/vo-tap/api/v0.1/caom/sync?FORMAT=json&LANG=ADQL&QUERY=SELECT+obs_id,+t_min,+t_max,+s_region,+target_name,+sequence_number+FROM+dbo.ObsPointing+WHERE+obs_collection=%27TESS%27+AND+dataproduct_type=%27image%27+AND+target_name=%27TESS+FFI%27

And this gets the TICA footprint (takes a bit longer since it's HLSP)

https://mast.stsci.edu/vo-tap/api/v0.1/caom/sync?FORMAT=json&LANG=ADQL&QUERY=SELECT%20obs_id,%20t_min,%20t_max,%20s_region,%20target_name,%20sequence_number%20FROM%20dbo.ObsPointing%20WHERE%20obs_collection=%27HLSP%27%20AND%20dataproduct_type=%27image%27%20AND%20target_name=%27TICA%20FFI%27

We manipulate that response in tesscut to create the footprint JSON file we store in S3 with a small bit of code but we could add that into astrocut as well.

Taking advantage of the cached footprint file in S3 is likely better long term, but we could use this method initially, for this PR

falkben · 2024-10-03T14:33:59Z

Do we want to revisit any of the mocking now that we can access the footprints?

astrocut/tests/data/tess_ffi_footprints.json

astrocut/footprint_cutouts.py

falkben · 2024-10-03T16:46:06Z

astrocut/footprint_cutouts.py

+
+from .utils.utils import parse_size_input
+
+TESS_ARCSEC_PER_PX = 21  # Number of arcseconds per pixel in a TESS image


we may want to make an issue or note to come back to this and generalize this somehow so it could be used for other missions. i think it could wait for now though, as I think TESS is the only mission we'd do this for.

falkben · 2024-10-03T16:56:18Z

astrocut/footprint_cutouts.py

+
+def _s_region_to_polygon(s_region: Column):
+    """
+    Takes in a s_region string of type POLYGON or CIRCLE and returns it as


i don't think this docstring is correct -- this function as currently written only supports POLYGON, not CIRCLE

It looks like this docstring is still referring to CIRCLE but that doesn't look supported

Are you looking at the most recent commit? I do see the change on that docstring.

falkben · 2024-10-03T16:58:49Z

astrocut/footprint_cutouts.py

+@cached(cache=FFI_TTLCACHE, lock=Lock())
+def _get_s3_ffis(s3_uri, as_table: bool = False, load_polys: bool = False):
+    """
+    Fetch the S3 footprint file containing a dict of all FFIs and a polygon column
+    that holds the s_regions as polygon points and vectors.
+
+    Optional Parameters:
+        as_table: Return the footprint file as an Astropy Table
+        load_polys: Convert the s_region column to an array of SphericalPolygon objects
+    """
+    # Open footprint file with fsspec
+    with fsspec.open(s3_uri, s3={'anon': True}) as f:
+        ffis = json.load(f)
+
+    if load_polys:
+        ffis['polygon'] = _s_region_to_polygon(ffis['s_region'])
+
+    if as_table:
+        ffis = Table(ffis)
+
+    return ffis


Should this function be removed, for now?

I figured that we could leave it in since we'll need it shortly, but it's true that we don't want users trying to call it while the bucket isn't accessible. I'll remove it for now.

falkben · 2024-10-03T17:10:03Z

I was looking for the coverage results.

Doesn't need to be handled in this PR, but it looks like there's a problem with codecov upload in the github action. From Python 3.10 with numpy 1.23 and full coverage job:

[2024-10-03T16:46:10.843Z] ['info'] => Project root located at: /home/runner/work/astrocut/astrocut
[2024-10-03T16:46:10.846Z] ['info'] -> No token specified or token is empty
[2024-10-03T16:46:10.936Z] ['info'] Searching for coverage files...
[2024-10-03T16:46:10.973Z] ['info'] => Found 1 possible coverage files:
  ./coverage.xml
[2024-10-03T16:46:10.973Z] ['info'] Processing ./coverage.xml...
[2024-10-03T16:46:10.976Z] ['info'] Detected GitHub Actions as the CI provider.
[2024-10-03T16:46:11.306Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=github-action-2.1.0-uploader-0.8.0&token=*******&branch=footprint-finder&build=11166073335&build_url=https%3A%2F%2Fgithub.com%2Fspacetelescope%2Fastrocut%2Factions%2Fruns%2F11166073335&commit=9552cbb9116481980b67cbea56921b10ebb327db&job=CI&pr=127&service=github-actions&slug=spacetelescope%2Fastrocut&name=&tag=&flags=&parent=
[2024-10-03T16:46:11.478Z] ['error'] There was an error running the uploader: Error uploading to [https://codecov.io:](https://codecov.io/) Error: There was an error fetching the storage URL during POST: 429 - {"message":"Rate limit reached. Please upload with the Codecov repository upload token to resolve issue. Expected time to availability: 366s."}

[2024-10-03T16:46:11.479Z] ['info'] Codecov will exit with status code 0. If you are expecting a non-zero exit code, please pass in the `-Z` flag

Anyways, coverage looks pretty good:

                                Stmts   Miss  Cover
---------------------------------------------------
astrocut/__init__.py               14      1    93%
astrocut/asdf_cutouts.py           82      3    96%
astrocut/cube_cut.py              388      4    99%
astrocut/cutout_processing.py     247     13    95%
astrocut/cutouts.py               244     26    89%
astrocut/exceptions.py             11      0   100%
astrocut/footprint_cutouts.py     139     11    92%
astrocut/make_cube.py             427     26    94%
astrocut/utils/__init__.py          0      0   100%
astrocut/utils/utils.py            86      4    95%
astrocut/utils/wcs_fitting.py      31      5    84%
---------------------------------------------------
                                 1669     93    94%

snbianco · 2024-10-03T19:47:55Z

I was looking for the coverage results.

Doesn't need to be handled in this PR, but it looks like there's a problem with codecov upload in the github action. From Python 3.10 with numpy 1.23 and full coverage job:

[2024-10-03T16:46:10.843Z] ['info'] => Project root located at: /home/runner/work/astrocut/astrocut
[2024-10-03T16:46:10.846Z] ['info'] -> No token specified or token is empty
[2024-10-03T16:46:10.936Z] ['info'] Searching for coverage files...
[2024-10-03T16:46:10.973Z] ['info'] => Found 1 possible coverage files:
  ./coverage.xml
[2024-10-03T16:46:10.973Z] ['info'] Processing ./coverage.xml...
[2024-10-03T16:46:10.976Z] ['info'] Detected GitHub Actions as the CI provider.
[2024-10-03T16:46:11.306Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=github-action-2.1.0-uploader-0.8.0&token=*******&branch=footprint-finder&build=11166073335&build_url=https%3A%2F%2Fgithub.com%2Fspacetelescope%2Fastrocut%2Factions%2Fruns%2F11166073335&commit=9552cbb9116481980b67cbea56921b10ebb327db&job=CI&pr=127&service=github-actions&slug=spacetelescope%2Fastrocut&name=&tag=&flags=&parent=
[2024-10-03T16:46:11.478Z] ['error'] There was an error running the uploader: Error uploading to [https://codecov.io:](https://codecov.io/) Error: There was an error fetching the storage URL during POST: 429 - {"message":"Rate limit reached. Please upload with the Codecov repository upload token to resolve issue. Expected time to availability: 366s."}

[2024-10-03T16:46:11.479Z] ['info'] Codecov will exit with status code 0. If you are expecting a non-zero exit code, please pass in the `-Z` flag

Anyways, coverage looks pretty good:

                                Stmts   Miss  Cover
---------------------------------------------------
astrocut/__init__.py               14      1    93%
astrocut/asdf_cutouts.py           82      3    96%
astrocut/cube_cut.py              388      4    99%
astrocut/cutout_processing.py     247     13    95%
astrocut/cutouts.py               244     26    89%
astrocut/exceptions.py             11      0   100%
astrocut/footprint_cutouts.py     139     11    92%
astrocut/make_cube.py             427     26    94%
astrocut/utils/__init__.py          0      0   100%
astrocut/utils/utils.py            86      4    95%
astrocut/utils/wcs_fitting.py      31      5    84%
---------------------------------------------------
                                 1669     93    94%

Made an issue here: https://jira.stsci.edu/browse/ASB-29119

havok2063

This all looks reasonable to me. I only had a few small comments on some tests.

havok2063 · 2024-10-08T17:14:19Z

astrocut/tests/test_utils.py

+def test_parse_size_input():
+    """Test that different types of input are accurately parsed into cutout sizes"""
+    # With scalar as input
+    cutout_size = utils.parse_size_input(5)
+    assert np.array_equal(cutout_size, np.array((5, 5)))
+
+    # With astropy quantity as input
+    cutout_size = utils.parse_size_input(10 * u.pix)
+    assert np.array_equal(cutout_size, np.array((10, 10)) * u.pix)
+
+    # With tuple as input
+    cutout_size = utils.parse_size_input((5, 10))
+    assert np.array_equal(cutout_size, np.array((5, 10)))
+
+    # With list as input
+    cutout_size = utils.parse_size_input([10, 5])
+    assert np.array_equal(cutout_size, np.array((10, 5)))
+
+    # With array as input
+    cutout_size = utils.parse_size_input(np.array((5, 10)))
+    assert np.array_equal(cutout_size, np.array((5, 10)))


You can probably simplify this test by parametrizing over the inputs/outputs using @pytest.mark.parametrize

havok2063 · 2024-10-08T17:16:26Z

astrocut/tests/test_footprint_cutouts.py

+    """Test that FFI intersection with cutout outputs proper results"""
+    # SphericalPolygon object for cutout
+    cutout_sp = SphericalPolygon.from_radec(lon=(350, 10, 10, 350),
+                                            lat=(-10, -10, 10, 10), 
+                                            center=(0, 0))
+
+    # Intersecting object
+    intersecting = SphericalPolygon.from_radec(lon=(345, 355, 355, 345),
+                                               lat=(-15, -15, -5, -5),
+                                               center=(350, -10))
+
+    # Non-intersecting object
+    nonintersecting = SphericalPolygon.from_radec(lon=(335, 345, 345, 335),
+                                                  lat=(-15, -15, -5, -5),
+                                                  center=(340, -10))
+
+    # Edge object that intersects
+    edge_intersect = SphericalPolygon.from_radec(lon=(340, 350, 350, 340),
+                                                 lat=(-15, -15, -5, -5),
+                                                 center=(345, -10))
+
+    # Edge object that does not intersect
+    edge_nonintersect = SphericalPolygon.from_radec(lon=(340, 349, 349, 340),
+                                                    lat=(-15, -15, -5, -5),
+                                                    center=(345, -10))


You could probably parametrize this test as well.

AlexReedy

Not much exposure to this codebase yet so can't comfortably comment on the details but this looks good on the overview to me. Ran the test suite last night (after changes) and it all came back good.

astrocut/tests/test_footprint_cutouts.py

snbianco added 3 commits September 15, 2024 22:21

progress on tesscut code

2c61ca6

send warnings, return array of filepaths

c6b3e0b

More generalization, print messages

509d766

snbianco commented Sep 18, 2024

View reviewed changes

astrocut/cube_cut_from_footprint.py Outdated Show resolved Hide resolved

comment fix

ccb34ca

snbianco requested review from falkben, havok2063 and dr-rodriguez September 18, 2024 14:56

falkben reviewed Sep 18, 2024

View reviewed changes

astrocut/cube_cut_from_footprint.py Outdated Show resolved Hide resolved

falkben reviewed Sep 18, 2024

View reviewed changes

astrocut/cube_cut_from_footprint.py Outdated Show resolved Hide resolved

falkben reviewed Sep 18, 2024

View reviewed changes

astrocut/__init__.py Outdated Show resolved Hide resolved

falkben reviewed Sep 18, 2024

View reviewed changes

astrocut/cube_cut_from_footprint.py Outdated Show resolved Hide resolved

snbianco force-pushed the footprint-finder branch from b842116 to 9e51987 Compare September 24, 2024 22:52

snbianco added 4 commits September 25, 2024 08:53

style fixes

9726dc3

fix type annotation

rename module, use same thread number for cube_cut

2b29770

Use json.load() documentation, set threads to 8 for cube_cut

unit tests, don't allow cutout size of 0

3471090

fix example in docstring

735112e

snbianco force-pushed the footprint-finder branch from 9e51987 to 735112e Compare September 25, 2024 12:53

snbianco added 2 commits September 25, 2024 10:44

Use mock in unit tests

a9e0de7

rename mock

2f90a1e

documentation, fix cache position

31926a4

snbianco marked this pull request as ready for review September 26, 2024 19:32

falkben reviewed Sep 27, 2024

View reviewed changes

snbianco added 2 commits September 30, 2024 16:54

remove thread pool, filter ffi footprint files in test data

3a0a513

Add cachetools as requirement

3fb0051

work-around for footprint files using TAP

00ab68e

Remove mock from footprint tests

331148b

falkben reviewed Oct 3, 2024

View reviewed changes

astrocut/tests/data/tess_ffi_footprints.json Outdated Show resolved Hide resolved

Delete sample footprints

9552cbb

falkben reviewed Oct 3, 2024

View reviewed changes

astrocut/footprint_cutouts.py Outdated Show resolved Hide resolved

falkben reviewed Oct 3, 2024

View reviewed changes

remove cloud function, make crossmatch and ffi fetch functions public

5e2f889

snbianco requested a review from AlexReedy October 7, 2024 19:10

Remove CIRCLE from example inputs

b857594

falkben previously approved these changes Oct 7, 2024

View reviewed changes

havok2063 reviewed Oct 8, 2024

View reviewed changes

AlexReedy previously approved these changes Oct 8, 2024

View reviewed changes

parametrize tests

ceea2b5

snbianco dismissed stale reviews from AlexReedy and falkben via ceea2b5 October 8, 2024 20:52

falkben reviewed Oct 8, 2024

View reviewed changes

astrocut/tests/test_footprint_cutouts.py Show resolved Hide resolved

falkben approved these changes Oct 8, 2024

View reviewed changes

snbianco merged commit 4fb0ed4 into main Oct 8, 2024
8 checks passed

snbianco deleted the footprint-finder branch October 8, 2024 22:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add footprint finder code #127

Add footprint finder code #127

snbianco commented Sep 18, 2024 •

edited

Loading

scfleming commented Sep 18, 2024 •

edited

Loading

snbianco commented Sep 24, 2024

snbianco commented Sep 25, 2024

snbianco commented Sep 26, 2024

falkben left a comment

falkben commented Oct 2, 2024

falkben commented Oct 3, 2024

falkben Oct 3, 2024

falkben Oct 3, 2024

falkben Oct 7, 2024

snbianco Oct 7, 2024

falkben Oct 3, 2024

snbianco Oct 3, 2024

falkben commented Oct 3, 2024

snbianco commented Oct 3, 2024

havok2063 left a comment

havok2063 Oct 8, 2024

havok2063 Oct 8, 2024

AlexReedy left a comment


		from .utils.utils import parse_size_input

		TESS_ARCSEC_PER_PX = 21 # Number of arcseconds per pixel in a TESS image

Add footprint finder code #127

Add footprint finder code #127

Conversation

snbianco commented Sep 18, 2024 • edited Loading

scfleming commented Sep 18, 2024 • edited Loading

snbianco commented Sep 24, 2024

snbianco commented Sep 25, 2024

snbianco commented Sep 26, 2024

falkben left a comment

Choose a reason for hiding this comment

falkben commented Oct 2, 2024

falkben commented Oct 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

falkben commented Oct 3, 2024

snbianco commented Oct 3, 2024

havok2063 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexReedy left a comment

Choose a reason for hiding this comment

snbianco commented Sep 18, 2024 •

edited

Loading

scfleming commented Sep 18, 2024 •

edited

Loading