Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geomesa-users] Error ingesting gdelt file
Hi Raffaele,

For the classpath issues, did you have to do anything not mentioned under 'Visualize Data With GeoServer'?  I ask in case those directions list something else which may help later and also to know if you found any additional differences in your environment.  (I'm also curious about your environment if you would care to share info about your version of Hadoop, Accumulo, and ZooKeeper.)

The tutorial is confusing at this point.  The request listed will return exactly what you described: a white tile with red dots.  Hunter showed a screenshot of the GeoServer OpenLayers preview instead of that.  He additional set up a base layer, in this case, Stamen Toner (http://maps.stamen.com).

As I make a pass through the documentation, I'll add some explanation about how to add that base layer.  The great news is that you do not have to ingest OSM data or a set of tiles.  In GeoServer, you can cascade external WMS resources: http://docs.geoserver.org/latest/en/user/data/cascaded/wms.html and then form a layer group.  In a webapp, you can use Leaflet or OpenLayers to use an external WMS layer directly.  

Great questions; thanks for writing.  Let us know if you have any other questions or any suggestions for new tutorials.

Thanks,

Jim


----- Original Message -----
From:
"Geomesa User discussions" <geomesa-users@xxxxxxxxxxxxxxxx>

To:
"Geomesa User discussions" <geomesa-users@xxxxxxxxxxxxxxxx>
Cc:

Sent:
Mon, 16 Feb 2015 12:12:00 +0100
Subject:
Re: [geomesa-users] Error ingesting gdelt file


Hi Jim, after some tries and classpath fixes for geoserver, I was able to ingest reduced gdelt dataset. I also was able to publish layer for WMS request, but I didn't view any map with the first example WMS request, explained in tutorial; I saw only red squares on a white background.
Am I wrong something or have I to import also OSM data in Accumulo? Eventually is it possible pointing to Google maps?
Thanks,
Raffaele.

2015-02-14 18:02 GMT+01:00 Jim Hughes <jnh5y@xxxxxxxx>:
Hi Raffaele,

I have found our error(s).  In our ingest driver, we try and ensure the existence of a specic temp directory and push our jar to it. 

Unfortunately, this code (https://github.com/geomesa/geomesa-gdelt/blob/master/src/main/java/geomesa/gdelt/GDELTIngest.java#L136-149) seems to be creating files instead of directories.

I am going to work on fixing the project this afternoon and I'll respond with more details.  If you want a quick work-around, you can do try these steps...

1.  Remove the /tmp file and create a directory instead.  (hadoop dfs -rmr and -mkdir.)

2.  Rename the jar created by the GDELT ingest project with this command:
cp target/geomesa-gdelt-accumulo1.5-1.0-SNAPSHOT.jar target/geomesa-gdelt-1.0-SNAPSHOT.jar

I don't think it is strictly necessary, but you may also create the output directory with
hadoop dfs -mkdir /tmp/geomesa-gdelt-output

With those notes, I was able to ingest a daily update file successfully.  I grabbed it with this command:
wget data.gdeltproject.org/events/20150213.export.CSV.zip

I mention that since our project targets the 'full' rather than 'reduced' version of the GDELT dataset.

As I said, I'll write again when I've had a chance to fix the bugs in the project and to update the documentation.

Thanks,

Jim


On 02/14/2015 10:06 AM, Raffaele Palmieri wrote:
Thanks Jim,
I've done as you told and the result is the same,

Name
 Type
 Size
 Replication
 Block Size
 Modification Time
 Permission
 Owner
 Group
accumulo
 dir


2015-02-13 14:56
 rwxr-xr-x
 root
 supergroup
gdelt
 dir


2015-02-13 17:12
 rwxr-xr-x
 root
 supergroup
tmp
 file
 0 B
 1
 128 MB
 2015-02-14 16:02
 rw-r--r--
 root
 supergroup

So I think that's a bug of GDELTIngest.
Regards,
Raffaele.

2015-02-14 14:35 GMT+01:00 Jim Hughes <jnh5y@xxxxxxxx>:
Hi Raffaele,

Yes, this is strange.  Many programs assume that /tmp is a directory and that they create additional working directories under it. 

I'd suggest moving /tmp out of the way, and trying the GDELT ingest project again.  If GeoMesa code is creating the file, then that's definitely a bug and I'd love to hear more details so that we can work out a fix.

Thanks,

Jim


On 02/14/2015 02:19 AM, Raffaele Palmieri wrote:
Hi Chris,
following are the contents of hdfs root file system:

accumulo
 dir


2015-02-13 14:56
 rwxr-xr-x
 root
 supergroup
gdelt
 dir


2015-02-13 17:12
 rwxr-xr-x
 root
 supergroup
tmp
 file
 0 B
 1
 128 MB
 2015-02-13 17:43
 rw-r--r--
 root
 supergroup

It seems that tmp was created as a file, could be this the problem?
Thank you,
regards,
Raffaele.




2015-02-13 21:25 GMT+01:00 Chris Eichelberger <cne1x@xxxxxxxx>:
Raffaele,

Just to be sure, does the "/tmp" directory exist in your HDFS as a
directory with permissions that would allow the job submitter to write
to it?

Sincerely,
  -- Chris


On Fri, 2015-02-13 at 18:37 +0100, Raffaele Palmieri wrote:
> Dear geomesa's users,
> when i try to ingest gdelt giving the following command:
> # hadoop jar target/geomesa-gdelt-accumulo1.5-1.0-SNAPSHOT.jar
> geomesa.gdelt.GDELTIngest -instanceId accumulo -zookeepers
> localhost:2181 -user root -password password -tableName gdelt
> -featureName event -ingestFile hdfs:///gdelt/uncompressed/gdelt.tsv
> I obtain the following error:
>
>  org.apache.hadoop.fs.FileAlreadyExistsException: Parent path is not a
> directory: /tmp tmp
>         at
> org.apachehadoop.hdfs.server.namenode.FSDirectory.mkdirs(FSDirectory.java:1916)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addFile(FSDirectory.java:284)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2093)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2012)
>         at
> org.apache.hadoop.hdfs.servernamenode.FSNamesystem.startFile(FSNamesystem.java:1963)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:491)
>         at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:301)
>         at
> org.apache.hadoop.hdfsprotocol.proto.ClientNamenodeProtocolProtos
> $ClientNamenodeProtocol
> $2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59570)
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Server
> $ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>         at org.apache.hadoop.ipc.Server$Handler
> $1.run(Server.java:2048)
>         at org.apache.hadoop.ipc.Server$Handler
> $1.run(Server.java:2044)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoopsecurity.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
>
>
> Can you suggest a solution for the problem?
> Thanks,
> Raffaele.
> _______________________________________________
> geomesa-users mailing list
> geomesa-users@xxxxxxxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> http://www.locationtech.org/mailman/listinfo/geomesa-users


_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users



_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users


_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users



_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users


_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users

Back to the top