Anthony,
Thanks for the detailed explanation and putting my worries at ease (and for fixing the chattiness).
Beau
From: geomesa-users-bounces@xxxxxxxxxxxxxxxx [mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx]
On Behalf Of Anthony Fox
Sent: Thursday, August 07, 2014 4:42 PM
To: Geomesa User discussions
Subject: Re: [geomesa-users] Is GeoMesa-triggered Accumulo "TabletServerBatchReader not shutdown" warning something I can safely ignore?
I think we just fixed this warning, though I'm not sure it's made it into master just yet. Some more details: the query planner receives the user's query and decides which table will optimally satisfy the query. In some cases, the attribute
index table is the best candidate. In those cases, we open two scanners, one to retrieve the ids from the reverse index in the attribute table and one to resolve the data entries in the records table. This process is run in two threads for performance but
has a blocking queue so as not to overwhelm memory in cases where the size of the result is huge or the user code does not need to seek to the end of the result set. We weren't quietly shutting down the threads and closing the internal scanners. It won't
cause any issues other than being overly chatty. Too chatty - I think we resolved it using IOUtils.closeQuietly from commons-io.
Thanks,
Anthony
On Thu, Aug 7, 2014 at 4:33 PM, Beau Lalonde <blalonde@xxxxxxxx> wrote:
Hi,
I am using a version of GeoMesa from early last week (late July 2014) and I am frequently witnessing the following, presumably accumulo, log message:
WARN TabletServerBatchReader:82 - TabletServerBatchReader not shutdown; did you forget to call close()?
I have done some investigating and cannot find out how to reproduce the warning 100% of the time, but I should emphasize that in our production code we see the warning a lot. We have yet to see any bad behavior result from the warning (except, perhaps slower
processing?); however, we are nonetheless concerned about the warning.
Through unit testing of GeoMesa, I only see the warning popup (semi-consistently) in some of my tests (not all of them). The nature of the tests where I see the warning indicates to me that the warning is caused by the feature modification code and/or the feature
removal code (featureStore.modifyFeatures(), and/or featureStore.removeFeatures()), because I usually do not see the warning until after the modifyFeatures() or removeFeatures() methods are first invoked.
My questions are:
1) What within GeoMesa is causing the warning?
2) Is this a GeoMesa bug?
3) Is the warning anything that I need to be concerned about?
4) Is there a way to prevent the warning?
Some more background, I seem to see the warning almost all of the time when I run the following test (which passes):
@Test
public void testRemoveFeatureByID_AddRemoveAdd()
{
// add three features, then remove one, verify its removal, add two features (one of which has the same ID as the one removed), verify addition
String id1 = "ID1";
assertTrue(addFeature(id1, 45.1d, 123.4, 15L));
String id2 = "ID2";
assertTrue(addFeature(id2, 45.0d, 123.3, 16L));
String id3 = "ID3";
assertTrue(addFeature(id3, 45.2d, 123.5, 17L));
this.geoMesaHelper.removeFeatureByID(GeoMesaHelperTest.FEATURE_NAME, id1);
Set<String> ids = this.geoMesaHelper.getIDsFromFeatureCollection(this.geoMesaHelper.query(this.dataStore, GeoMesaHelperTest.FEATURE_NAME, "")); // unbounded query
assertNotNull(ids);
assertEquals(2, ids.size());
assertTrue(ids.contains(id2));
assertTrue(ids.contains(id3));
// add back feature with id1 with different SKU
System.out.println("adding feature with id: " + id1);
assertTrue(addFeature(id1, 45.1d, 123.4, 25L));
// add a uniquely new feature that hasn't been added before
String id4 = "ID4";
System.out.println("adding feature with id: " + id4);
assertTrue(addFeature(id4, 45.25d, 123.55, 18L));
System.out.println("another query");
ids = this.geoMesaHelper.getIDsFromFeatureCollection(this.geoMesaHelper.query(this.dataStore, GeoMesaHelperTest.FEATURE_NAME, "")); // unbounded query
assertNotNull(ids);
assertTrue(ids.contains(id4));
assertTrue(ids.contains(id2));
assertTrue(ids.contains(id3));
assertTrue(ids.contains(id1));
assertEquals(4, ids.size());
}
When I run the above test, I see the following at the console:
15:47:09,035 WARN IndexQueryPlanner:356 - Querying Accumulo without ST filter.
15:47:09,356 WARN IndexQueryPlanner:356 - Querying Accumulo without ST filter.
adding feature with id: ID1
adding feature with id: ID4
15:47:10,990 WARN TabletServerBatchReader:82 - TabletServerBatchReader not shutdown; did you forget to call close()?
another query
15:47:12,237 WARN IndexQueryPlanner:356 - Querying Accumulo without ST filter.
Further notes:
1) The TabletServerBatchReader warning, when it appears, seems to consistently get triggered by the second addFeature() method; however, if I add Thread.sleep(5000) statements before each addFeature() method, I do not witness the TabletServerBatchReader
warning. I usually don't see the TabletServerBatchReader warning unless I have recently performed a feature modification or removal. I have, however, on at least one occurrence, seen the warning when it was nowhere near a feature modification/removal.
2) Both of my calls to this.geoMesaHelper.getIDsFromFeatureCollection() are unbounded queries and thus trigger two of the "WARN IndexQueryPlanner:356 - Querying Accumulo without ST filter." warnings.
3) The other "WARN IndexQueryPlanner:356 - Querying Accumulo without ST filter." warning is triggered by the this.geoMesaHelper.removeFeatureByID() method.
Thanks in advance,
Beau
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users
|
