AWS S3 objectstorage is not working for ap-south-2 region inspite of endpoint and region is mentioned and getting default to dualstack endpoint of us-east-1

[svvsmarutii]

Thanos, Prometheus and Golang version used: 0.30.2

Object Storage Provider: AWS S3

What happened: Thanos sidecar, compact, query, storegateway components are trying to reach the S3 bucket in us-east-1 region inspite ap-south-2 region and s3 endpoint is set.

What you expected to happen: Thanos components should reach S3 bucket to the region(ap-south-2) as mentioned in the thanos helm chart

How to reproduce it (as minimally and precisely as possible): Deploy the thanos helm chart with below objectstorage.yaml configuration
objstoresecret:
type: s3
prefix: thanos
config:
bucket: "dev-dr-thanos-metrics"
region: ap-south-2
endpoint: s3.ap-south-2.amazonaws.com

Full logs to relevant components:

Anything else we need to know:
Able to see the s3 endpoint of ap-south-2 is supported by thanos https://docs.aws.amazon.com/general/latest/gr/s3.html

[harry671003]

I think this issue was raised in: #6541
You should probably use a more newer version of Thanos for the minio update to be included.

[Sahana-Nagaraja]

Hi @harry671003, I am a colleague of Maruthi. We have upgraded Thanos to 0.35.1 and now we see that Thanos is pointing to right region. But, it is trying to access dualstack endpoint of S3 bucket which is not available yet. We did not manage to find a parameter to set it to use standalone S3 endpoint in Thanos helm chart. Can you please help us here if you have a handy solution?

[douglascamata]

Sounds like this was fixed in the objstore lib (see thanos-io/objstore#112) by allowing the dualstack endpoint to be disabled. So we only need to bump the dependency here in Thanos.

[svvsmarutii]

Hi @douglascamata Thanks for update. Is it possible to provide an ETA for this change to be available in thanos ? Also, is there a quick workaround that we could do to leverage this option ?

[harry671003]

#7462 - Update objstore

[harry671003]

My PR to update the objstore version in Thanos got merged.
@svvsmarutii - could you try testing with the latest image of Thanos?
As douglascamata pointed out, setting disable_dualstack: true in the config should fix this problem.

[douglascamata]

@svvsmarutii we don't provide ETAs for changes, as it depends on the "free work" of maintainers that might be busy with their jobs. But, as the PR was already merged, it should be going out in the next Thanos release. If you would like to have a bugfix release, please reach out in #thanos-dev in the CNCF slack.