Re: [technology-pmc] CQ for Spark and MetaModel

[Jim Hughes <jnh5y@xxxxxxxx>]

Thanks, Rob!  And thanks for the suggestions!

Also, is that +1 for PMC approval for those two CQs;)?

On 09/16/2015 03:40 PM, Rob Emanuele wrote:

Oh, very cool. I see from the code, you're using the CQL query to load up RDDs and then converting them to dataframes with WKT geometry packed inside the DataFrame and dates converted to TimeStamp type, etc. I'm assuming Catalyst doesn't get to push the SQL filters down to the original GeoMesa RDD's, yeah? I think that would be a good reason to target User Defined Functions and User Defined Types for geometries and geometric operations, and implement the query clauses on load as Accumulo iterators (or filters on whatever backend). But that's future work :)

Really cool stuff, congrats!

On Wed, Sep 16, 2015 at 2:36 PM, Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:

There's a PR up here for those interested:

https://github.com/locationtech/geomesa/pull/685

The approach we've taken is to try to extract CQL from the 'where' clause of the SQL statement, and use that to load the initial result sets into spark. We then pass the rest of the SQL statement off to spark SQL so that you can do things like joins, etc.

For example, it lets you create queries like:

select myAttr, count(*) as count from mySft where bbox(mySft.geom, -115, 45, -110, 50) AND mySft.dtg during 2015-03-02T10:00:00.000Z/2015-03-02T11:00:00.000Z group by myAttr

We've also created a web service front-end to facilitate kicking off spark queries.

Thanks,

Emilio

On 09/16/2015 02:22 PM, Rob Emanuele wrote:

Nice! Will it support CQL in SparkSQL via UDT's and UDF's? Would be interested in looking at the code if it's up on github.

On Wed, Sep 16, 2015 at 2:11 PM, Jim Hughes <jnh5y@xxxxxxxx> wrote:

Hi all,

GeoMesa is adding the ability to support SQL via Spark, and we've got two new CQs:

Spark 1.5.0 - For SparkSQL support:
https://dev.eclipse.org/ipzilla/show_bug.cgi?id=10154

Apache MetaModel - to parse SQL queries:
https://dev.eclipse.org/ipzilla/show_bug.cgi?id=10155

I've added Emilio (GeoMesa committer) as a cc in case there are any questions/discussion points he can help with.

Thanks in advance,

Jim

_______________________________________________
technology-pmc mailing list
technology-pmc@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://locationtech.org/mailman/listinfo/technology-pmc

_______________________________________________
technology-pmc mailing list
technology-pmc@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://locationtech.org/mailman/listinfo/technology-pmc