Hi,
I am trying to ingest a small CSV file into a Geomesa catalog on HBase. To do this, I am using the Geomesa command line tools, providing an existing catalog, a SimpleFeatureType schema in a .sft file, and a converter in a .convert file.
Geomesa successfully creates the SimpleFeatureType schema in the HBase catalog, it launches the MapReduce job, but it is unable to ingest any row.
This is the command line I am using:
geomesa-hbase ingest \
-c Geoloc_Praxedo_catalog \
-s /home/ec2-user/geomesa/geomesa-hbase_2.11-1.3.0/conf/sfts/geoloc_praxedo/history_feature_nodate_conf.sft \
-C /home/ec2-user/geomesa/geomesa-hbase_2.11-1.3.0/conf/sfts/geoloc_praxedo/history_feature_nodate_conf.convert \
hdfs:///datasets/Geoloc_History_1k.csv
An here are my sft and convert files:
history_feature_nodate_conf.sft:
geomesa.sfts.history_feature_nodate_conf = {
attributes = [
{ name = "ent_id", type = "Integer", index = false }
{ name = "technician_id", type = "Integer", index = false }
{ name = "vehicle_id", type = "Integer", index = false }
{ name = "geoloc_date", type = "String", index = false }
{ name = "coords", type = "Point", index = true, srid = 4326, default = true }
{ name = "event", type = "Integer", index = false }
{ name = "street", type = "String", index = false }
{ name = "zip_code", type = "String", index = false }
{ name = "city", type = "String", index = false }
{ name = "speed", type = "Double", index = false }
{ name = "counter", type = "Integer", index = false }
]
}
history_feature_nodate_conf.convert:
geomesa.converters.history-converter = {
type = "delimited-text",
format = "CSV",
id-field = "toString($fid)",
fields = [
{ name = "fid", transform = "stringToInt($1,-1)" }
{ name = "ent_id", transform = "stringToInt($2,-1)" }
{ name = "technician_id", transform = "stringToInt($3,-1)" }
{ name = "vehicle_id", transform = "stringToInt($4,-1)" }
{ name = "geoloc_date", transform = "toString($5)" }
{ name = "latitude", transform = "stringToDouble($6,-1.0)" }
{ name = "longitude", transform = "stringToDouble($7,-1.0)" }
{ name = "coords", transform = "point($latitude, $longitude)" }
{ name = "event", transform = "stringToInt($8,-1)" }
{ name = "street", transform = "toString($9)" }
{ name = "zip_code", transform = "toString($10)" }
{ name = "city", transform = "toString($11)" }
{ name = "speed", transform = "stringToDouble($12,-1.0)" }
{ name = "counter", transform = "stringToInt($13,-1)" }
]
}
Finally, here is the Geomesa log after the ingest attempt:
2017-02-22 15:00:58,290 WARN [org.apache.hadoop.util.NativeCodeLoader] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-02-22 15:01:04,869 WARN [org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory] The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
2017-02-22 15:01:06,214 INFO [org.locationtech.geomesa.tools.user] Creating schema history_feature_nodate_conf
2017-02-22 15:01:06,605 INFO [org.locationtech.geomesa.tools.user] Running ingestion in distributed mode
2017-02-22 15:01:07,241 WARN [org.locationtech.geomesa.utils.io.CloseWithLogging$] Error calling close on 'org.locationtech.geomesa.hbase.data.HBaseBackedMetadata@18d47df0':
java.lang.IllegalArgumentException: Connection is null or closed.
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:307)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getTable(ConnectionManager.java:733)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getTable(ConnectionManager.java:715)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getTable(ConnectionManager.java:549)
at org.locationtech.geomesa.hbase.data.HBaseMetadataAdapter$class.org$locationtech$geomesa$hbase$data$HBaseMetadataAdapter$$table(HBaseBackedMetadata.scala:29) at org.locationtech.geomesa.hbase.data.HBaseMetadataAdapter$class.close(HBaseBackedMetadata.scala:71)
at org.locationtech.geomesa.hbase.data.HBaseBackedMetadata.close(HBaseBackedMetadata.scala:19)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.locationtech.geomesa.utils.io.CloseWithLogging$.apply(SafeClose.scala:33)
at org.locationtech.geomesa.index.geotools.GeoMesaDataStore.dispose(GeoMesaDataStore.scala:439)
at org.locationtech.geomesa.hbase.data.HBaseDataStore.dispose(HBaseDataStore.scala:67)
at org.locationtech.geomesa.jobs.mapreduce.GeoMesaOutputFormat$.configureDataStore(GeoMesaOutputFormat.scala:42)
at org.locationtech.geomesa.tools.ingest.AbstractIngestJob.run(AbstractIngestJob.scala:65)
at org.locationtech.geomesa.tools.ingest.ConverterIngest.runDistributedJob(ConverterIngest.scala:79)
at org.locationtech.geomesa.tools.ingest.AbstractIngest.runDistributed(AbstractIngest.scala:181)
at org.locationtech.geomesa.tools.ingest.AbstractIngest.run(AbstractIngest.scala:94)
at org.locationtech.geomesa.tools.ingest.IngestCommand$class.execute(IngestCommand.scala:70)
at org.locationtech.geomesa.hbase.tools.ingest.HBaseIngestCommand.execute(HBaseIngestCommand.scala:21)
at org.locationtech.geomesa.tools.Runner$class.main(Runner.scala:23)
at org.locationtech.geomesa.hbase.tools.HBaseRunner$.main(HBaseRunner.scala:21)
at org.locationtech.geomesa.hbase.tools.HBaseRunner.main(HBaseRunner.scala)
2017-02-22 15:01:07,249 INFO [org.locationtech.geomesa.tools.user] Submitting job - please wait...
2017-02-22 15:01:12,620 WARN [org.apache.hadoop.mapreduce.JobResourceUploader] Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2017-02-22 15:02:56,778 WARN [org.apache.hadoop.hdfs.DFSClient] Slow waitForAckedSeqno took 45944ms (threshold=30000ms)
2017-02-22 15:19:17,221 ERROR [org.locationtech.geomesa.tools.user] Job failed with state FAILED due to: Task failed task_1486394655726_0034_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
2017-02-22 15:19:17,259 INFO [org.locationtech.geomesa.tools.user] Distributed ingestion complete in 00:18:10
2017-02-22 15:19:17,260 INFO [org.locationtech.geomesa.tools.user] Ingested 0 features with no failures.
I am not sure if the ingest is failling because something wroing with Geomesa or something going wrong with MapReduce.
Is maybe the warning on the method org.locationtech.geomesa.hbase.data.HBaseBackedMetadata at the oirigin of the problem ?
Thanks for your time and have a nice day!
José
