Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geomesa-users] Bootstrapping GeoMesa HBase on AWS S3 [SEC=UNCLASSIFIED]

Hello Emilio,

Thanks for your help.  That makes sense.  I’ll look into added the geomesa as part of an EMR step then, since doing it in the bootstrap phase is too early.

 

Cheers

Duncan

 

From: geomesa-users-bounces@xxxxxxxxxxxxxxxx [mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Emilio Lahr-Vivaz
Sent: Monday, 10 September 2018 10:51 PM
To: geomesa-users@xxxxxxxxxxxxxxxx
Subject: Re: [geomesa-users] Bootstrapping GeoMesa HBase on AWS S3 [SEC=UNCLASSIFIED]

 

Ah, I didn't realize you meant the aws bootstrap hook. Looking at their docs:

https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-bootstrap.html

Bootstrap actions are scripts that run on cluster after Amazon EMR launches the instance using the Amazon Linux Amazon Machine Image (AMI). Bootstrap actions run before Amazon EMR installs the applications that you specify when you create the cluster and before cluster nodes begin processing data.


So hbase won't be installed when the bootstrap runs, and the geomesa script expects it to be running.

Thanks,

Emilio

On 09/10/2018 02:07 AM, Gray Duncan wrote:

Hi Emilio,

Thanks for your reply.

 

Ø  So if you run the script manually, it works, but when invoked from your custom bootstrap it hangs?

Yes, that’s it

 

If I do this in the bootscript;

cat /usr/lib/hbase/conf/hbase-site.xml

I get;

cat: /usr/lib/hbase/conf/hbase-site.xml: No such file or directory

 

Is the bootstrap starting hbase before running the geomesa script?

The bootstrap isn’t starting hbase.

 

I’m not sure of the ordering of things when using bootstrapping on an aws EMR instance.

 

Cheers

Duncan

 

 

 

 

 

From: geomesa-users-bounces@xxxxxxxxxxxxxxxx [mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Emilio Lahr-Vivaz
Sent: Friday, 7 September 2018 10:35 PM
To: geomesa-users@xxxxxxxxxxxxxxxx
Subject: Re: [geomesa-users] Bootstrapping GeoMesa HBase on AWS S3 [SEC=UNCLASSIFIED]

 

So if you run the script manually, it works, but when invoked from your custom bootstrap it hangs? I'm not sure what would cause that, possibly the script assumes an actual user environment which isn't present. The line it's hanging on is checking for the hbase.rootdir in the hbase-site.xml:

ROOTDIR=`cat /usr/lib/hbase/conf/hbase-site.xml 2> /dev/null | tr '\n' ' ' | sed 's/ //g' | grep -o -P "<name>hbase.rootdir</name><value>.+?</value>" | sed 's/<name>hbase.rootdir<\/name><value>//' | sed 's/<\/value>//'`

Is the bootstrap starting hbase before running the geomesa script?

Thanks,

Emilio

On 09/07/2018 03:39 AM, Gray Duncan wrote:

Hello Emilio,

Thanks for your help.  Moving to java 8 fixed the geomesa-hbase issue.  Unfortunately the bootscript is still hanging though.

 

Cheers

Duncan

 

From: geomesa-users-bounces@xxxxxxxxxxxxxxxx [mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Emilio Lahr-Vivaz
Sent: Thursday, 6 September 2018 10:18 PM
To: geomesa-users@xxxxxxxxxxxxxxxx
Subject: Re: [geomesa-users] Bootstrapping GeoMesa HBase on AWS S3 [SEC=UNCLASSIFIED]

 

Hello,

That stack trace indicates that you're using java 7, but geomesa is compiled against and requires java 8. Likely that is causing the script to hang as well.

Thanks,

Emilio

On 09/06/2018 03:54 AM, Gray Duncan wrote:

Hi,

I’m installing GeoMesa HBase using EMR.  I’m following;

https://www.geomesa.org/documentation/tutorials/geomesa-hbase-s3-on-aws.html

 

I’m using a bootstrap script to install the geomesa-hbase-dist.

 

When the script gets to this line though;

sudo /opt/geomesa-hbase_2.11-${VERSION}/bin/bootstrap-geomesa-hbase-aws.sh

 

It hangs with the message;

Waiting for HBase to be configured.

 

Do you have any idea’s why it is hanging?

 

The EMR release I’m using is emr-5.16.0.

 

I’ve attached the terraform file and script to start an emr instance.

 

If I do this line on the instance it works;

sudo /opt/geomesa-hbase_2.11-${VERSION}/bin/bootstrap-geomesa-hbase-aws.sh

 

But then following the web instructions ‘geomesa hbase s3 on aws’ this line;

geomesa-hbase export -c geomesa.gdelt -f gdelt -m 50

 

Gives;

Exception in thread "main" java.lang.UnsupportedClassVersionError: org/geotools/data/DataStore : Unsupported major.minor version 52.0

                at java.lang.ClassLoader.defineClass1(Native Method)

                at java.lang.ClassLoader.defineClass(ClassLoader.java:808)

                at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

                at java.net.URLClassLoader.defineClass(URLClassLoader.java:442)

                at java.net.URLClassLoader.access$100(URLClassLoader.java:64)

                at java.net.URLClassLoader$1.run(URLClassLoader.java:354)

                at java.net.URLClassLoader$1.run(URLClassLoader.java:348)

                at java.security.AccessController.doPrivileged(Native Method)

                at java.net.URLClassLoader.findClass(URLClassLoader.java:347)

                at java.lang.ClassLoader.loadClass(ClassLoader.java:430)

                at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:323)

                at java.lang.ClassLoader.loadClass(ClassLoader.java:363)

                at java.lang.ClassLoader.defineClass1(Native Method)

                at java.lang.ClassLoader.defineClass(ClassLoader.java:808)

                at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

                at java.net.URLClassLoader.defineClass(URLClassLoader.java:442)

                at java.net.URLClassLoader.access$100(URLClassLoader.java:64)

                at java.net.URLClassLoader$1.run(URLClassLoader.java:354)

                at java.net.URLClassLoader$1.run(URLClassLoader.java:348)

                at java.security.AccessController.doPrivileged(Native Method)

                at java.net.URLClassLoader.findClass(URLClassLoader.java:347)

                at java.lang.ClassLoader.loadClass(ClassLoader.java:430)

                at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:323)

                at java.lang.ClassLoader.loadClass(ClassLoader.java:363)

                at java.lang.ClassLoader.defineClass1(Native Method)

                at java.lang.ClassLoader.defineClass(ClassLoader.java:808)

                at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

                at java.net.URLClassLoader.defineClass(URLClassLoader.java:442)

                at java.net.URLClassLoader.access$100(URLClassLoader.java:64)

                at java.net.URLClassLoader$1.run(URLClassLoader.java:354)

                at java.net.URLClassLoader$1.run(URLClassLoader.java:348)

                at java.security.AccessController.doPrivileged(Native Method)

                at java.net.URLClassLoader.findClass(URLClassLoader.java:347)

                at java.lang.ClassLoader.loadClass(ClassLoader.java:430)

                at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:323)

                at java.lang.ClassLoader.loadClass(ClassLoader.java:363)

                at org.locationtech.geomesa.hbase.tools.HBaseRunner$.createCommands(HBaseRunner.scala:27)

                at org.locationtech.geomesa.tools.Runner$class.parseCommand(Runner.scala:47)

                at org.locationtech.geomesa.hbase.tools.HBaseRunner$.parseCommand(HBaseRunner.scala:21)

                at org.locationtech.geomesa.tools.Runner$class.main(Runner.scala:28)

                at org.locationtech.geomesa.hbase.tools.HBaseRunner$.main(HBaseRunner.scala:21)

                at org.locationtech.geomesa.hbase.tools.HBaseRunner.main(HBaseRunner.scala)

 

Any idea’s as to what is going wrong?

 

Cheers

Duncan

 

 

 

Geoscience Australia Disclaimer: This e-mail (and files transmitted with it) is intended only for the person or entity to which it is addressed. If you are not the intended recipient, then you have received this e-mail by mistake and any use, dissemination, forwarding, printing or copying of this e-mail and its file attachments is prohibited. The security of emails transmitted cannot be guaranteed; by forwarding or replying to this email, you acknowledge and accept these risks.
-------------------------------------------------------------------------------------------------------------------------






_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users

 

Geoscience Australia Disclaimer: This e-mail (and files transmitted with it) is intended only for the person or entity to which it is addressed. If you are not the intended recipient, then you have received this e-mail by mistake and any use, dissemination, forwarding, printing or copying of this e-mail and its file attachments is prohibited. The security of emails transmitted cannot be guaranteed; by forwarding or replying to this email, you acknowledge and accept these risks.
-------------------------------------------------------------------------------------------------------------------------





_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users

 

Geoscience Australia Disclaimer: This e-mail (and files transmitted with it) is intended only for the person or entity to which it is addressed. If you are not the intended recipient, then you have received this e-mail by mistake and any use, dissemination, forwarding, printing or copying of this e-mail and its file attachments is prohibited. The security of emails transmitted cannot be guaranteed; by forwarding or replying to this email, you acknowledge and accept these risks.
-------------------------------------------------------------------------------------------------------------------------




_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users

 

Geoscience Australia Disclaimer: This e-mail (and files transmitted with it) is intended only for the person or entity to which it is addressed. If you are not the intended recipient, then you have received this e-mail by mistake and any use, dissemination, forwarding, printing or copying of this e-mail and its file attachments is prohibited. The security of emails transmitted cannot be guaranteed; by forwarding or replying to this email, you acknowledge and accept these risks.
-------------------------------------------------------------------------------------------------------------------------

Back to the top