[geomesa-dev] [JIRA] (GEOMESA-2701) FSDS - Use S3 select for optimized p

["Emilio Lahr-Vivaz (JIRA)" <jira@xxxxxxxxxxxxxxxxxxxxx>]

Title: Message Title

Emilio Lahr-Vivaz created an issue

GeoMesa / GEOMESA-2701

FSDS - Use S3 select for optimized parquet reads from S3

Issue Type:	Story
Assignee:	Unassigned
Created:	30/Aug/19 10:58 AM
Priority:	Major
Reporter:	Emilio Lahr-Vivaz

S3 select will let us only return the columns we want, and may also support parquet predicates? It will only return data as csv or json, so we would need to run the response through a converter or something.

It's a somewhat specialized optimization, as it will only work with parquet files stored in s3, but it could save a lot of network and cost so it seems worthwhile. (technically we could also use it with 'converter' mode FSDS if the files are CSV or JSON, but that's probably not a primary use case worth pursuing.)

https://docs.aws.amazon.com/AmazonS3/latest/dev/SelectObjectContentUsingJava.html

Add Comment

Get Jira notifications on your phone! Download the Jira Cloud app for Android or iOS
This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100108-sha1:64cda0e)