Re: [geowave-dev] Understanding BasicMapReduceIT.testIngestAndQueryGener

[Rich Fecher <rfecher@xxxxxxxxx>]

The warnings you are seeing are expected.  Take a look at a Travis CI log and compare: https://travis-ci.org/ngageoint/geowave/jobs/65616149#L983

Travis CI is expected to pass and should run the same as your environment.

BasicMapReduceIT.testIngestAndQueryGeneralGpx first ingests GPX files using the command line ingest tool.  You can see how it builds the command line arguments and programmatically calls the main method: https://github.com/ngageoint/geowave/blob/master/test/src/main/java/mil/nga/giat/geowave/test/mapreduce/MapReduceTestEnvironment.java#L45

Then after the data is ingested, it uses the GeoWaveInputFormat to query the data.  Packaged with the tests are shapefiles with filter criteria and shapefiles with the expected results that match the ingested files with the filter criteria.  At this line, you can see there is expected results and filter criteria passed into the test job: https://github.com/ngageoint/geowave/blob/master/test/src/test/java/mil/nga/giat/geowave/test/mapreduce/BasicMapReduceIT.java#L136

Then finally, in setting up the job, the parameters of the GeoWaveInputFormat get set using the TestJobRunner in the runTestJob method:

https://github.com/ngageoint/geowave/blob/master/test/src/test/java/mil/nga/giat/geowave/test/mapreduce/BasicMapReduceIT.java#L249

TestJobRunner extends from GeoWaveJobRunner so these parameters are in fact passed into the GeoWaveInputFormat which sets the job context appropriately:

https://github.com/ngageoint/geowave/blob/master/extensions/datastores/accumulo/src/main/java/mil/nga/giat/geowave/datastore/accumulo/mapreduce/GeoWaveJobRunner.java#L87

The GeoWaveInputFormat then uses those parameters to perform the filtering in the mapreduce job and the expected results are compared to the mappers input.

You can see that the job runner uses counters that are set by the VerifyExpectedResultsMapper to verify that the mapper received the same filtered GPX points as the expected results shapefile.  Here is where it verifies the results:

https://github.com/ngageoint/geowave/blob/master/test/src/test/java/mil/nga/giat/geowave/test/mapreduce/BasicMapReduceIT.java#L338

Let us know if this makes sense or if you have further confusion.

On Sun, Jun 7, 2015 at 4:50 PM, Vignesh Prabhu <me@xxxxxxxx> wrote:

Hi,

I am trying to understand BasicMapReduceIT.testIngestAndQueryGeneralGpx
to know how geowave supports basic MapReduce job with Accumulo. When I
run the test locally on Eclipse, it passes for me successfully but I am
not able to understand what it is actually doing in the background. I
get the following output in the console. Can someone please explain what
is the test doing.
On debugging, what I find is that AbstractMapReduceIngest class is
throwing the warning "No job jar file set.  User classes may not be
found. See Job or Job#setJar(String)." when job.waitForCompletion(true)
 is called in line 143. What am I missing here?

08 Jun 01:54:03 WARN [mapreduce.MapReduceTestEnvironment] - Ingesting
'data/general_gpx_test_case/input_gpx/' - this may take several minutes...
08 Jun 01:57:23 WARN [util.NativeCodeLoader] - Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable
Jun 07, 2015 8:30:22 PM org.geoserver.platform.GeoServerExtensions
checkContext
WARNING: Extension lookup 'GeoServerResourceLoader', but
ApplicationContext is unset.
Jun 07, 2015 8:30:22 PM org.geoserver.platform.GeoServerExtensions
checkContext
WARNING: Extension lookup 'GeoServerResourceLoader', but
ApplicationContext is unset.
Jun 07, 2015 8:30:22 PM org.geoserver.platform.GeoServerExtensions
checkContext
WARNING: Extension lookup 'GeoServerResourceLoader', but
ApplicationContext is unset.
Jun 07, 2015 8:30:22 PM org.geoserver.platform.GeoServerExtensions
checkContext
WARNING: Extension lookup 'GeoServerResourceLoader', but
ApplicationContext is unset.
08 Jun 02:00:22 WARN [mapreduce.JobSubmitter] - No job jar file set.
User classes may not be found. See Job or Job#setJar(String).
08 Jun 02:02:39 WARN [mapreduce.JobSubmitter] - No job jar file set.
User classes may not be found. See Job or Job#setJar(String).
08 Jun 02:03:06 WARN [mapreduce.JobSubmitter] - Hadoop command-line
option parsing not performed. Implement the Tool interface and execute
your application with ToolRunner to remedy this.
08 Jun 02:03:06 WARN [mapreduce.JobSubmitter] - No job jar file set.
User classes may not be found. See Job or Job#setJar(String).

Regards,

Vignesh Prabhu
_______________________________________________
geowave-dev mailing list
geowave-dev@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geowave-dev