Hi,
I am trying to ingest a small CSV file into a Geomesa
catalog on HBase. To do this, I am using the Geomesa command
line tools, providing an existing catalog, a SimpleFeatureType
schema in a .sft file, and a converter in a .convert file.
Geomesa successfully creates the SimpleFeatureType schema
in the HBase catalog, it launches the MapReduce job, but it is
unable to ingest any row.
This is the command line I am using:
geomesa-hbase ingest \
-c Geoloc_Praxedo_catalog \
-s
/home/ec2-user/geomesa/geomesa-hbase_2.11-1.3.0/conf/sfts/geoloc_praxedo/history_feature_nodate_conf.sft
\
-C
/home/ec2-user/geomesa/geomesa-hbase_2.11-1.3.0/conf/sfts/geoloc_praxedo/history_feature_nodate_conf.convert
\
hdfs:///datasets/Geoloc_History_1k.csv
An here are my sft and convert files:
history_feature_nodate_conf.sft:
geomesa.sfts.history_feature_nodate_conf
= {
attributes
= [
{
name = "ent_id", type
= "Integer", index
= false }
{
name = "technician_id", type
= "Integer", index
= false }
{
name = "vehicle_id", type
= "Integer", index
= false }
{
name = "geoloc_date", type
= "String", index
= false }
{
name = "coords", type
= "Point", index
= true, srid
= 4326, default = true }
{
name = "event", type
= "Integer", index
= false }
{
name = "street", type
= "String", index
= false }
{
name = "zip_code", type
= "String", index
= false }
{
name = "city", type
= "String", index
= false }
{
name = "speed", type
= "Double", index
= false }
{
name = "counter", type
= "Integer", index
= false }
]
}
history_feature_nodate_conf.convert:
geomesa.converters.history-converter = {
type
= "delimited-text",
format
= "CSV",
id-field
= "toString($fid)",
fields
= [
{
name = "fid", transform
= "stringToInt($1,-1)" }
{
name = "ent_id", transform
= "stringToInt($2,-1)" }
{
name = "technician_id", transform
= "stringToInt($3,-1)" }
{
name = "vehicle_id", transform
= "stringToInt($4,-1)" }
{
name = "geoloc_date", transform
= "toString($5)" }
{
name = "latitude", transform
= "stringToDouble($6,-1.0)" }
{
name = "longitude", transform
= "stringToDouble($7,-1.0)" }
{
name = "coords", transform
= "point($latitude, $longitude)" }
{
name = "event", transform
= "stringToInt($8,-1)" }
{
name = "street", transform
= "toString($9)" }
{
name = "zip_code", transform
= "toString($10)" }
{
name = "city", transform
= "toString($11)" }
{
name = "speed", transform
= "stringToDouble($12,-1.0)" }
{
name = "counter", transform
= "stringToInt($13,-1)" }
]
}
Finally, here is the Geomesa log after the ingest attempt:
2017-02-22 15:00:58,290 WARN
[org.apache.hadoop.util.NativeCodeLoader] Unable to load
native-hadoop library for your platform... using
builtin-java classes where applicable
2017-02-22 15:01:04,869 WARN
[org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory]
The short-circuit local reads feature cannot be used
because libhadoop cannot be loaded.
2017-02-22 15:01:06,214 INFO
[org.locationtech.geomesa.tools.user] Creating schema
history_feature_nodate_conf
2017-02-22 15:01:06,605 INFO
[org.locationtech.geomesa.tools.user] Running ingestion
in distributed mode
2017-02-22 15:01:07,241 WARN
[org.locationtech.geomesa.utils.io.CloseWithLogging$] Error
calling close on
'org.locationtech.geomesa.hbase.data.HBaseBackedMetadata@18d47df0':
java.lang.IllegalArgumentException: Connection is
null or closed.
at
org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:307)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getTable(ConnectionManager.java:733)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getTable(ConnectionManager.java:715)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getTable(ConnectionManager.java:549)
at
org.locationtech.geomesa.hbase.data.HBaseMetadataAdapter$class.org$locationtech$geomesa$hbase$data$HBaseMetadataAdapter$$table(HBaseBackedMetadata.scala:29)
at
org.locationtech.geomesa.hbase.data.HBaseMetadataAdapter$class.close(HBaseBackedMetadata.scala:71)
at
org.locationtech.geomesa.hbase.data.HBaseBackedMetadata.close(HBaseBackedMetadata.scala:19)
at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at
java.lang.reflect.Method.invoke(Method.java:498)
at
org.locationtech.geomesa.utils.io.CloseWithLogging$.apply(SafeClose.scala:33)
at
org.locationtech.geomesa.index.geotools.GeoMesaDataStore.dispose(GeoMesaDataStore.scala:439)
at
org.locationtech.geomesa.hbase.data.HBaseDataStore.dispose(HBaseDataStore.scala:67)
at
org.locationtech.geomesa.jobs.mapreduce.GeoMesaOutputFormat$.configureDataStore(GeoMesaOutputFormat.scala:42)
at
org.locationtech.geomesa.tools.ingest.AbstractIngestJob.run(AbstractIngestJob.scala:65)
at
org.locationtech.geomesa.tools.ingest.ConverterIngest.runDistributedJob(ConverterIngest.scala:79)
at
org.locationtech.geomesa.tools.ingest.AbstractIngest.runDistributed(AbstractIngest.scala:181)
at
org.locationtech.geomesa.tools.ingest.AbstractIngest.run(AbstractIngest.scala:94)
at
org.locationtech.geomesa.tools.ingest.IngestCommand$class.execute(IngestCommand.scala:70)
at
org.locationtech.geomesa.hbase.tools.ingest.HBaseIngestCommand.execute(HBaseIngestCommand.scala:21)
at
org.locationtech.geomesa.tools.Runner$class.main(Runner.scala:23)
at
org.locationtech.geomesa.hbase.tools.HBaseRunner$.main(HBaseRunner.scala:21)
at
org.locationtech.geomesa.hbase.tools.HBaseRunner.main(HBaseRunner.scala)
2017-02-22 15:01:07,249 INFO
[org.locationtech.geomesa.tools.user] Submitting job -
please wait...
2017-02-22 15:01:12,620 WARN
[org.apache.hadoop.mapreduce.JobResourceUploader] Hadoop
command-line option parsing not performed. Implement the
Tool interface and execute your application with
ToolRunner to remedy this.
2017-02-22 15:02:56,778 WARN
[org.apache.hadoop.hdfs.DFSClient] Slow waitForAckedSeqno
took 45944ms (threshold=30000ms)
2017-02-22 15:19:17,221 ERROR
[org.locationtech.geomesa.tools.user] Job failed with
state FAILED due to: Task failed
task_1486394655726_0034_m_000000
Job failed as tasks failed. failedMaps:1
failedReduces:0
2017-02-22 15:19:17,259 INFO
[org.locationtech.geomesa.tools.user] Distributed
ingestion complete in 00:18:10
2017-02-22 15:19:17,260 INFO
[org.locationtech.geomesa.tools.user] Ingested 0 features
with no failures.
I am not sure if the ingest is failling because something
wroing with Geomesa or something going wrong with MapReduce.
Is maybe the warning on the method org.locationtech.geomesa.hbase.data.HBaseBackedMetadata
at the oirigin of the problem ?
Thanks for your time and have a nice day!
José