Re: [geotrellis-user] Area of intersection for vector data in Spark

[Ross Bernet <rbernet@xxxxxxxxxx>]

Lukasz,

Please check out this issue that Eugene wrote to help think through the solution to this problem.

If you have further questions, feel free to respond here or check out the geotrellis gitter - it is quite active and I'm sure you could bounce ideas off folks there.

Would be great to have you thinking through this problem some more.

Thanks for the inquiry!

Ross

On Tue, Apr 4, 2017 at 1:18 PM, Lukasz Tracewski <lukasz.tracewski@xxxxxxxxxxx> wrote:

Thanks Rob! My data is rather large and I have cluster at hand, which is exactly why I was looking into options what could work for me. Would you have any suggestions where to start? I am happy to contribute to the project and don't mind at all working with not-out-of-the-box solutions! I am only starting with Scala, but have good software engineering experience and know functional programming.

Cheers,
Lukasz

On 03.04.2017 22:48, Rob Emanuele wrote:

Hi Lukasz,

If you can all the features in GeoJSON, is the amount of data actually too big to fit into memory and to do on one machine? GeoTrellis would be able to help with that if it had enough memory, without spark - read in the feature collection, spatially partition the one collection, and do a bounds query using the features of the other collection to pull out intersecting geometries.

We have some functionality contained in an object called VectorJoin, which will efficiently join two vector datasets - with the caveat that those RDDs of vector data are spatially partitioned before the join. We actually don't have a good way to do this type of spatial partitioning - I've written up an issue to track this, so that next release we'd have a good solution to that problem (https://github.com/locationtech/geotrellis/issues/2116)

There's a possibility to lean on GeoTrellis for components of not-out-of-the-box solution to this problem, so if you are interested in diving deeper into this.

Thanks,

Rob

On Thu, Mar 30, 2017 at 3:10 AM, Lukasz Tracewski <lukasz.tracewski@xxxxxxxxxxx> wrote:

Hi,

I am trying to figure out how - or if at all - I can use your library to
calculate an area of intersection for two relatively large GeoJSONs.
Both are FeatureCollections that contain a number of features. Due to
the size of the data I'd like to use Spark. Can you point me in the
right direction?


Thanks,
Lucas


_______________________________________________
geotrellis-user mailing list
geotrellis-user@locationtech.org
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geotrellis-user

--

Robert Emanuele, VP of Research
Azavea |  990 Spring Garden Street, 5th Floor, Philadelphia, PA
remanuele@xxxxxxxxxx  |  @lossyrob

_______________________________________________
geotrellis-user mailing list
geotrellis-user@locationtech.org
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geotrellis-user

_______________________________________________
geotrellis-user mailing list
geotrellis-user@locationtech.org
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geotrellis-user

--

Ross Bernet, Project Manager, GeoTrellis

Azavea | 990 Spring Garden Street, 5th Floor, Philadelphia, PA

rbernet@xxxxxxxxxx | office: 215.701.7717  | mobile: 858.692.1872

Azavea is a B Corporation - we apply geospatial technology for civic and social impact

while advancing the state-of-the-art through research. Join us.