Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geomesa-users] LineString data query problem

Hello,

The issue is still due to the classloading for the custom GeoMesa filters in HBase. You need to install the geomesa-hbase-distributed-runtime jar in HBase, and then ensure that it it referenced correctly in the table configuration. There are more details here:

https://www.geomesa.org/documentation/user/hbase/install.html#deploying-the-geomesa-hbase-distributed-runtime-jar
https://www.geomesa.org/documentation/user/hbase/install.html#register-the-coprocessors

Thanks,

Emilio

On 10/22/18 9:57 AM, su sheldon wrote:

HEY Emilio, I tried it on hadoop single node cluster, the query result mentioned in the first email is still 0.

If I set the bbox range like this:

``

String bbox=“bbox(geom,180,90,-180,-90)”

``

all the data will be obtained.

But if I narrowed the scope of bbox - even if only a little ,the query results is 0:

``

String bbox=“bbox(geom,180,-89,-180,-90)”

 

``

And error log appears on HBase:

``

org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.reflect.InvocationTargetException

        at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1545)

        at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toScan(ProtobufUtil.java:1056)

        at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2578)

        at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2854)

        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:35072)

        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2349)

        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)

        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)

        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)

Caused by: java.lang.reflect.InvocationTargetException

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1541)

        ... 8 more

Caused by: org.apache.hadoop.hbase.exceptions.DeserializationException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.ClassNotFoundException: org.locationtech.geomesa.hbase.filters.CqlTransformFilter

        at org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:396)

        ... 13 more

Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.ClassNotFoundException: org.locationtech.geomesa.hbase.filters.CqlTransformFilter

        at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1545)

        at org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:393)

        ... 13 more

Caused by: java.lang.ClassNotFoundException: org.locationtech.geomesa.hbase.filters.CqlTransformFilter

        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

        at org.apache.hadoop.hbase.util.DynamicClassLoader.tryRefreshClass(DynamicClassLoader.java:173)

        at org.apache.hadoop.hbase.util.DynamicClassLoader.loadClass(DynamicClassLoader.java:140)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:348)

        at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1535)

        ... 14 more

``

I checked this class ,its in the geomesa-datastore project,it doesn’t seem to have anything with HBase,and I tried to put it on hdfs,it doesn’t work.Is there anything missing when I set up geomesa?

 

 

Sent from Mail for Windows 10

 


From: geomesa-users-bounces@xxxxxxxxxxxxxxxx <geomesa-users-bounces@xxxxxxxxxxxxxxxx> on behalf of su sheldon <susheldon@xxxxxxxxxxx>
Sent: Wednesday, October 17, 2018 9:02:35 PM
To: Geomesa User discussions
Subject: Re: [geomesa-users] 答复: LineString data query problem
 

Yeah,I ran HBase on WINDOWS,and  it might be the reason, so I’m trying to use geomesa on a complete HBase cluster.

Thanks for your reply.

 

Sent from Mail for Windows 10

 


From: geomesa-users-bounces@xxxxxxxxxxxxxxxx <geomesa-users-bounces@xxxxxxxxxxxxxxxx> on behalf of Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx>
Sent: Tuesday, October 16, 2018 7:57:54 PM
To: geomesa-users@xxxxxxxxxxxxxxxx
Subject: Re: [geomesa-users] 答复: LineString data query problem
 
Is your HBase cluster just a single node? Generally the root dir would be in hdfs, not on the local filesystem.
Is HBase running on windows? Your root dir looks like a windows path. I wouldn't be surprised if things don't work quite right on windows, I don't believe anyone's ever tried that with GeoMesa.

At any rate, I believe the issue is likely to be related to classloading the geomesa-distributed-runtime, either due to the local root dir or the windows path. In the quick start, the simple queries are returning ok, as they don't require any custom classes. But the ones that only return a subset of attributes are failing, as they require a custom GeoMesa HBase filter loaded from the distributed-runtime jar.

Thanks,

Emilio

On 10/15/18 10:34 PM, su sheldon wrote:

thanks for your reply.

I have checked data in hbase,it seems  normal ,as shown in the attachment.

 And I found another problem,When I run the Hbase QuickStart in geomesa-tutorials, Two queries did not return results

 ``

Generating test data

 

Writing test data

Wrote 2356 features

 

Running query BBOX(geom, -120.0,30.0,-75.0,55.0) AND dtg DURING 2017-12-31T00:00:00+00:00/2018-01-02T00:00:00+00:00

01 719025288=719025288|CANADA|CAN|||190|4|1|4|4|Penticton, British Columbia, Canada|CA|2018-01-01T00:00:00.000Z|POINT (-119.583 49.5)

02 719025945=719025945|DEPARTMENT OF EDUCATION||OBAMA|USA|090|20|2|20|2|North Dakota, United States|US|2018-01-01T00:00:00.000Z|POINT (-99.793 47.5362)

03 719025853=719025853|PRESIDENT||MEXICO|MEX|0874|3|3|3|2|Minnesota, United States|US|2018-01-01T00:00:00.000Z|POINT (-93.9196 45.7326)

04 719025856=719025856|PRESIDENT||MEXICO|MEX|131|3|3|3|2|Minnesota, United States|US|2018-01-01T00:00:00.000Z|POINT (-93.9196 45.7326)

05 719026960=719026960|UNITED STATES|USA|||071|6|1|6|2|Minnesota, United States|US|2018-01-01T00:00:00.000Z|POINT (-93.9196 45.7326)

06 719024925=719024925|||CORPORATION||172|2|1|2|4|Saskatoon, Saskatchewan, Canada|CA|2018-01-01T00:00:00.000Z|POINT (-106.667 52.1333)

07 719025289=719025289|CANADA|CAN|CORPORATION||115|6|1|6|4|Saskatoon, Saskatchewan, Canada|CA|2018-01-01T00:00:00.000Z|POINT (-106.667 52.1333)

08 719025290=719025290|CANADA|CAN|CORPORATION||172|8|1|8|4|Saskatoon, Saskatchewan, Canada|CA|2018-01-01T00:00:00.000Z|POINT (-106.667 52.1333)

09 719025276=719025276|SASKATCHEWAN|CAN|||020|20|1|10|4|Saskatchewan, Manitoba, Canada|CA|2018-01-01T00:00:00.000Z|POINT (-97.7167 52.15)

10 719025946=719025946|PREMIER||||015|6|1|6|4|Saskatchewan, Manitoba, Canada|CA|2018-01-01T00:00:00.000Z|POINT (-97.7167 52.15)

 

Returned 669 total features

 

Running query BBOX(geom, -120.0,30.0,-75.0,55.0) AND dtg DURING 2017-12-31T00:00:00+00:00/2018-01-02T00:00:00+00:00

Returning attributes [GLOBALEVENTID, dtg, geom]

 

Returned 0 total features

 

Running query EventCode = '051'

01 719024909=719024909|||MELBOURNE|AUS|051|10|1|10|4|Melbourne, Victoria, Australia|AS|2018-01-01T00:00:00.000Z|POINT (144.967 -37.8167)

02 719025178=719025178|AUSTRALIA|AUS|COMMUNITY||051|20|2|20|4|Sydney, New South Wales, Australia|AS|2018-01-01T00:00:00.000Z|POINT (151.217 -33.8833)

03 719025965=719025965|MIDWIFE||||051|10|1|10|4|Sydney, New South Wales, Australia|AS|2018-01-01T00:00:00.000Z|POINT (151.217 -33.8833)

04 719025509=719025509|COMMUNITY||AUSTRALIA|AUS|051|2|1|2|1|Australia|AS|2018-01-01T00:00:00.000Z|POINT (135 -25)

05 719025742=719025742|KING||||051|22|3|22|3|San Diego, California, United States|US|2018-01-01T00:00:00.000Z|POINT (-117.157 32.7153)

06 719025745=719025745|KING||||051|4|2|4|2|California, United States|US|2018-01-01T00:00:00.000Z|POINT (-119.746 36.17)

07 719025743=719025743|AUTHORITIES||||051|60|12|60|3|Wichita, Kansas, United States|US|2018-01-01T00:00:00.000Z|POINT (-97.3375 37.6922)

08 719027205=719027205|UNITED STATES|USA|SIOUX||051|4|1|4|3|Sioux City, Iowa, United States|US|2018-01-01T00:00:00.000Z|POINT (-96.4003 42.5)

09 719025111=719025111|||UNITED STATES|USA|051|2|1|2|3|Pickens County, South Carolina, United States|US|2018-01-01T00:00:00.000Z|POINT (-82.7165 34.9168)

10 719026938=719026938|PITTSBURGH|USA|||051|5|1|5|3|York County, Pennsylvania, United States|US|2018-01-01T00:00:00.000Z|POINT (-77 40.1254)

 

Returned 138 total features

 

Running query EventCode = '051' AND dtg DURING 2017-12-31T00:00:00+00:00/2018-01-02T00:00:00+00:00

Returning attributes [GLOBALEVENTID, dtg, geom]

 

Returned 0 total features

 

Done

``

It looks different from the output in https://www.geomesa.org/documentation/tutorials/geomesa-quickstart-hbase.html.

At first I thought the geomesa-hbase-distributed runtime_2.11-2.0.2.jar was misplaced, but I checked the path and it was in the ${hbase.rootdir}/lib.

This is ${hbase.rootdir} in hbase-site.xml,and there are no ${hbase.dynamic.jars.dir}

``

    <name>hbase.rootdir</name> 

    <value>file:///C:\hadoop\hadoop\tmp\hbase/root</value>

  </property> 

``

发件人: geomesa-users-bounces@xxxxxxxxxxxxxxxx <geomesa-users-bounces@xxxxxxxxxxxxxxxx> 代表 Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx>
发送时间: Monday, October 15, 2018 8:55:39 PM
收件人: geomesa-users@xxxxxxxxxxxxxxxx
主题: Re: [geomesa-users] LineString data query problem

 

Generally yes, geotools will handle that. If there is a typo or
formatting error in the WKT, then it will silently fail to null. But in
that case, the feature should fail indexing and raise an exception on write.

On 10/15/2018 08:51 AM, Anthony Fox wrote:
> Does geomesa automatically coerce the WKT string into a proper geometry?
>
> Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> writes:
>
>> Hello,
>>
>> I don't see anything obviously wrong with your code snippets, it looks
>> like it should work. Do you see data in the underlying HBase tables
>> after doing your insert? If so, you should check the logs for errors,
>> and also try enabling explain logging to see what is actually being
>> executed by your query:
>>
>> https://www.geomesa.org/documentation/user/datastores/query_planning.html#explaining-query-plans
>>
>> Thanks,
>>
>> Emilio
>>
>> On 10/15/2018 07:07 AM, su sheldon wrote:
>>> Hi,team.
>>>
>>> I am using geomesa with HBase to do some testing. In the process I
>>> found that I can't use bbox to query data when I use XZ index.
>>>
>>> This is the code I declare schema:
>>>
>>> ``
>>>
>>> attributes.append("taxiId:String,");attributes.append("*geom:LineString:srid=4326");
>>> sft = SimpleFeatureTypes./createType/(getTypeName(),
>>> attributes.toString());
>>>
>>> ``
>>>
>>> This is the code I am storing data for:
>>>
>>> ``
>>>
>>> builder.set("taxiId", key); StringBuilder pointStringBuilder =
>>> lineById.get(key);builder.set("geom", "LINESTRING (116.36919
>>> 39.90568,116.32919,39.91568
>>> ,116.33919,39.92568)");builder.featureUserData(Hints./USE_PROVIDED_FID/, Boolean./TRUE/); features.add(builder.buildFeature(key));
>>>
>>> ``
>>>
>>> This is the code I am querying for data:
>>>
>>> ``
>>>
>>> String bbox = "bbox(geom,115, 35, 120, 45)";queries.add(new
>>> Query(getTypeName(), ECQL./toFilter/(bbox)));
>>>
>>> ``
>>>
>>> The result obtained after running the query statement is empty. What
>>> is wrong with it? Is there any relevant information to check?
>>>
>>>
>>>
>>> _______________________________________________
>>> geomesa-users mailing list
>>> geomesa-users@xxxxxxxxxxxxxxxx
>>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>>> https://dev.locationtech.org/mailman/listinfo/geomesa-users
>> _______________________________________________
>> geomesa-users mailing list
>> geomesa-users@xxxxxxxxxxxxxxxx
>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>> https://dev.locationtech.org/mailman/listinfo/geomesa-users
> _______________________________________________
> geomesa-users mailing list
> geomesa-users@xxxxxxxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.locationtech.org/mailman/listinfo/geomesa-users

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users


_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users


_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users


Back to the top