Re: [geomesa-users] Bootstrapping GeoMesa Accumulo and Spark on AWS

[Anthony Fox <anthony.fox@xxxxxxxx>]

You may have to configure IAM roles for your instances to have the
appropriate rights on S3.


Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> writes:

> I believe that if you're in AWS, your instances will already be
> configured to connect to s3. But regardless, if you connect to the
> accumulo master docker, you should be able to see your classpath with
> the following command:
>
> geomesa classpath
>
> The hadoop core-site.xml file should be in one of the folders on your
> classpath. If you filter out the jars, you should see the folders:
>
> geomesa classpath | grep -v 'jar$'
>
> Check the hadoop core-site.xml file that you find in one of those
> folders for the aws credentials - follow the directions in the link I
> sent if they aren't already present.
>
> Thanks,
>
> Emilio
>
> On 09/26/2017 01:00 PM, Brooks Isoldi wrote:
>> Hi Emilio,
>>
>> The tutorial you shared seems to be geared towards installing each
>> module on a separate system, whereas I am trying to get the Bootstrap
>> GeoMesa, Accumulo and Spark on EMR tutorial working.
>>
>> Can you provide any guidance on configuring Accumulo for S3 in this setup?
>>
>> Thank you!
>>
>>
>>
>> -Brooks
>>
>>
>> On 09/25/2017 02:07 PM, Emilio Lahr-Vivaz wrote:
>>> Hello,
>>>
>>> I believe that we don't support direct ingestion using the 's3'
>>> prefix. You can ingest using 's3a' or 's3n' (which are supported
>>> through hadoop's API), but you have to set up your AWS credentials as
>>> outlined here:
>>>
>>> http://www.geomesa.org/documentation/user/accumulo/commandline_tools.html#enabling-s3-ingest
>>>
>>> You can alternatively download the files to a local filesystem and
>>> ingest them from there.
>>>
>>> I'll open a ticket to fix that tutorial, sorry about that.
>>>
>>> Thanks,
>>>
>>> Emilio
>>>
>>> On 09/25/2017 01:34 PM, Brooks Isoldi wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I'm attempting to follow the Boostrapping Geomesa/Accumulo/Spark
>>>> tutorial located at
>>>> http://www.geomesa.org/documentation/tutorials/geodocker-geomesa-spark-on-aws.html
>>>> <http://www.geomesa.org/documentation/tutorials/geodocker-geomesa-spark-on-aws.html>
>>>> and am unable to ingest the GDELT data. It's saying it "Ingested 0
>>>> features" when I run the ingest command.
>>>>
>>>> Here are the commands I ran (correcting for a couple typo's I found
>>>> in the tutorial).
>>>>
>>>> $ FILES=$(seq 7 -1 1 | xargs -n 1 -I{} sh -c "date -d'{} days ago'
>>>> +%Y%m%d" | xargs -n 1 -I{} echo
>>>> s3://gdelt-open-data/events/{}.export.csv | tr '\n' ' ')
>>>> $ sudo docker exec accumulo-master geomesa ingest -c geomesa.gdelt
>>>> -C gdelt -f gdelt -s gdelt -u root -p secret $FILES
>>>> $ sudo docker exec accumulo-master geomesa export -c geomesa.gdelt
>>>> -f gdelt -u root -p secret -m 100
>>>>
>>>>
>>>> The first command seems to work just fine...echo'ing $FILES returns
>>>> a list of valid s3 paths.
>>>>
>>>> The ingest command seems to work successfully (meaning, no errors),
>>>> but I get the following and it does not appear to actually ingest
>>>> anything.
>>>>
>>>> INFO Creating schema gdelt
>>>> INFO Running ingestion in local mode
>>>> INFO Ingesting 7 files with 1 thread
>>>> [ ] 0% complete 0 ingested 0 failed in 00:00:02
>>>> INFO Local ingestion complete in 00:00:02
>>>> INFO Ingested 0 features with no failures.
>>>>
>>>> The export command simply spits out all of the column headers
>>>> without any rows (features).
>>>>
>>>>
>>>> If anyone can point to what I'm doing wrong here, I would greatly
>>>> appreciate it.
>>>>
>>>> Thank you!
>>>>
>>>>
>>>>
>>>> --
>>>> Brooks Isoldi, Software Developer
>>>>
>>>> Traversed
>>>> 7164 Columbia Gateway Drive, Suite 120A
>>>> Columbia, MD 21046
>>>>
>>>>
>>>> _______________________________________________
>>>> geomesa-users mailing list
>>>> geomesa-users@xxxxxxxxxxxxxxxx
>>>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>>>> https://dev.locationtech.org/mailman/listinfo/geomesa-users
>>>
>>>
>>>
>>> _______________________________________________
>>> geomesa-users mailing list
>>> geomesa-users@xxxxxxxxxxxxxxxx
>>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>>> https://dev.locationtech.org/mailman/listinfo/geomesa-users
>>
>>
>>
>> _______________________________________________
>> geomesa-users mailing list
>> geomesa-users@xxxxxxxxxxxxxxxx
>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>> https://dev.locationtech.org/mailman/listinfo/geomesa-users
>
> _______________________________________________
> geomesa-users mailing list
> geomesa-users@xxxxxxxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.locationtech.org/mailman/listinfo/geomesa-users