Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
[geomesa-users] SchemaCopyJob is failing

Hello,

 

I’ve been attempting to convert a geomesa datastore that was generated using a 1.1.0-rc.7 build to the format of 1.1.0-rc.2, using the SchemaCopyJob (https://github.com/locationtech/geomesa/tree/master/geomesa-jobs#updating-existing-data-to-the-latest-index-format). The job is failing in org.locationtech.geomesa.features.avro.FeatureSpecificReader.readAttributes(…) – specifically on  the ‘match’ statement on serializationVersion. A bit of debugging shows that the serializationVersion being passed in to readAttributes is 0, which means that the code is not able to assign a suitable deserializer for the data in the table being converted.

 

For reference, here’s the contents of my table metadata (generated using scant –t <tablename> in the Accumulo shell):

 

~METADATA_mast_sri attributes: []    aisMessageType:Long:index=false,navStatus:Integer:index=false,rateOfTurn:Double:index=false,trueHeading:Integer:index=false,cargoDescription:String:index=false,shippingAgent:String:index=false,receiverClass:String:index=false,aidType:Long:index=false,vesselType:Long:index=false,destination:String:index=false,arrivalTime:Date:index=false,destinationVerified:Long:index=false,simonVesselType:String:index=false,unenhancedObservationId:Long:index=false,streamDataTypeId:Long:index=false,dataSourceId:Long:index=false,vesselName:String:index=false,mmsi:Integer:index=true,callSign:String:index=false,flag:Long:index=false,imo:Integer:index=false,sconum:String:index=false,breadth:Float:index=false,length:Double:index=false,netTonnage:Float:index=false,grossTonnage:Float:index=false,deadweight:Float:index=false,buildYear:Long:index=false,latitude:Double:index=false,longitude:Double:index=false,observationTime:Date:index=false,semiMajor:Float:index=false,semiMinor:Float:index=false,orientation:Float:index=false,altitude:Double:index=false,trackId:Integer:index=false,imageId:Long:index=false,observationCatCode:Long:index=false,geofeasible:Integer:index=false,displacement:Float:index=false,depth:Float:index=false,height:Float:index=false,vesselPreviousName:String:index=false,yearNameChanged:Long:index=false,draft:Float:index=false,draught:Float:index=false,satTelex:String:index=false,satPhone:String:index=false,satFax:String:index=false,mobilePhone:String:index=false,previousFlag:Long:index=false,previousCallSign:String:index=false,vesselClass:String:index=false,hullNumber:Long:index=false,currentHeading:Float:index=false,currentSpeed:Double:index=false,*geom:Point:srid=4326:index=false

~METADATA_mast_sri bounds: []    21.0593433380127:24.4429340362549:37.5012168884277:40.1512336730957

~METADATA_mast_sri dtgfield: []    observationTime

~METADATA_mast_sri featureEncoding: []    avro

~METADATA_mast_sri schema: []    %~#s%2#r%mast_sri#cstr%0,3#gh%yyyyMMdd#d::%~#s%3,2#gh::%~#s%#id

~METADATA_mast_sri tables.idx.attr.name: []    test_tail_mast_5fsri_attr_idx

~METADATA_mast_sri tables.idx.st.name: []    test_tail_mast_5fsri_st_idx

~METADATA_mast_sri tables.queries.name: []    test_tail_mast_5fsri_queries

~METADATA_mast_sri tables.record.name: []    test_tail_mast_5fsri_records

~METADATA_mast_sri visibilities: []

 

Can any of the experienced geomesans tell if this particular table format is likely to cause the issue of an unrecognized serializationVersion in FeatureSpecifiedReader?

 

Thanks!

 

Ben

 

From: Emilio Lahr-Vivaz [mailto:elahrvivaz@xxxxxxxx]
Sent: Wednesday, July 15, 2015 12:28 PM
To: Geomesa User discussions; Ben Southall
Subject: Re: [geomesa-users] Moving from geomesa-accumulo1.5-1.0.0-rc.7 to geomesa-1.1.0-rc.2

 

Hi Ben,

We don't really have a document of the breaking changes, as most users of geomesa interact with us at the geotools interface level, so they aren't affected. The only change most people would have to make is migrating data to take advantage of new index improvements - that process is outlined here:

https://github.com/locationtech/geomesa/tree/master/geomesa-jobs#updating-existing-data-to-the-latest-index-format

If you're using our serializers directly, then you can take a look at the unit tests for examples. The usage is pretty simple:

https://github.com/locationtech/geomesa/blob/master/geomesa-features/geomesa-feature-kryo/src/test/scala/org/locationtech/geomesa/features/kryo/KryoFeatureSerializerTest.scala

Kryo is the main serialization format we're supporting going forward, although we also have options to use avro or native byte buffers.

As you noticed, we changed the names from encoders to serializers to be clearer about what they are actually doing. We also moved around several packages and modules (as you also noticed).

We'll try to get the geomesa.org links sorted out, thanks bringing that to our attention!

If you have any more questions, or things still aren't clear, please let us know.

Thanks,

Emilio

On 07/15/2015 10:19 AM, Ben Southall wrote:

Hello,

 

I’m attempting to transition some code from a CCRI fork (specifically, tag geomesa-DCB-v3.1) based upon 1.0.0-rc.7 to geomesa-1.1.0-rc.2. It seems that in the interim, there have been a number of breaking changes. Some appear obvious (package name changes), some less so.

 

In particular, my code was making use of 1.5-DBC’s org.locationtech.geomesa.core.data.SimpleFeatureEncoder – has this now been broken into the SimpleFeatureSerializer/Deserializer classes? In fact, the whole feature landscape seems to be more complex than it was in earlier versions.

 

Is there a summary document of the breaking changes available?

 

Thanks very much!

 

Ben

 

p.s. The download instructions on geomesa.org are a bit wonky. The instructions to clone a branch geomesa-accumulo1.5-1.1.0-rc.2 seem to be incorrect. The clone works with the tag geomesa-1.1.0-rc.2.

 

Also, the link to the tar.gz for v1.1.0-rc.2 is broken, and the link to 1.1.0-rc.3-SNAPSHOT contains pom.xml files where the version number is 1.0.0-rc.8.




_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users

 


Back to the top