Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support lz4 framing decompression #188

Open
guptashailesh92 opened this issue Mar 14, 2024 · 1 comment
Open

Support lz4 framing decompression #188

guptashailesh92 opened this issue Mar 14, 2024 · 1 comment

Comments

@guptashailesh92
Copy link

Is there any plan to support lz4 framing support while decompression.
https://android.googlesource.com/platform/external/lz4/+/HEAD/doc/lz4_Frame_format.md

Currently Trino queries are failing if such compressed files are provided.

@dain
Copy link
Member

dain commented Mar 18, 2024

Reading the code over it doesn't look too difficult to implement is needed (we do this for Snappy)

That said, Hadoop has it's own framing format for LZ4 which isn't compatible with anything else. Does Hive or Spark support these files directly? If so, what compresson codec are you specifying? In the past Hadoop would introduce a second codec when there is a conflict between the built in framing and one provided by compression project (normally happens when the framing comes out after hadoop has created its own).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants