Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geomesa-users] Is GeoMesa-triggered Accumulo "TabletServerBatchReader not shutdown" warning something I can safely ignore?

Beau,

I think we just fixed this warning, though I'm not sure it's made it into master just yet.  Some more details: the query planner receives the user's query and decides which table will optimally satisfy the query.  In some cases, the attribute index table is the best candidate.  In those cases, we open two scanners, one to retrieve the ids from the reverse index in the attribute table and one to resolve the data entries in the records table.  This process is run in two threads for performance but has a blocking queue so as not to overwhelm memory in cases where the size of the result is huge or the user code does not need to seek to the end of the result set.  We weren't quietly shutting down the threads and closing the internal scanners.  It won't cause any issues other than being overly chatty.  Too chatty - I think we resolved it using IOUtils.closeQuietly from commons-io.

Thanks,
Anthony


On Thu, Aug 7, 2014 at 4:33 PM, Beau Lalonde <blalonde@xxxxxxxx> wrote:
Hi,

I am using a version of GeoMesa from early last week (late July 2014) and I am frequently witnessing the following, presumably accumulo, log message:
    WARN TabletServerBatchReader:82 - TabletServerBatchReader not shutdown; did you forget to call close()?

I have done some investigating and cannot find out how to reproduce the warning 100% of the time, but I should emphasize that in our production code we see the warning a lot.  We have yet to see any bad behavior result from the warning (except, perhaps slower processing?); however, we are nonetheless concerned about the warning.

Through unit testing of GeoMesa, I only see the warning popup (semi-consistently) in some of my tests (not all of them). The nature of the tests where I see the warning indicates to me that the warning is caused by the feature modification code and/or the feature removal code (featureStore.modifyFeatures(), and/or featureStore.removeFeatures()), because I usually do not see the warning until after the modifyFeatures() or removeFeatures() methods are first invoked.

My questions are:
    1) What within GeoMesa is causing the warning?
    2) Is this a GeoMesa bug?
    3) Is the warning anything that I need to be concerned about?
    4) Is there a way to prevent the warning?

Some more background, I seem to see the warning almost all of the time when I run the following test (which passes):
        @Test
        public void testRemoveFeatureByID_AddRemoveAdd()
        {
                // add three features, then remove one, verify its removal, add two features (one of which has the same ID as the one removed), verify addition

                String id1 = "ID1";
                assertTrue(addFeature(id1, 45.1d, 123.4, 15L));
                String id2 = "ID2";
                assertTrue(addFeature(id2, 45.0d, 123.3, 16L));
                String id3 = "ID3";
                assertTrue(addFeature(id3, 45.2d, 123.5, 17L));

                this.geoMesaHelper.removeFeatureByID(GeoMesaHelperTest.FEATURE_NAME, id1);

                Set<String> ids = this.geoMesaHelper.getIDsFromFeatureCollection(this.geoMesaHelper.query(this.dataStore, GeoMesaHelperTest.FEATURE_NAME, "")); // unbounded query

                assertNotNull(ids);
                assertEquals(2, ids.size());
                assertTrue(ids.contains(id2));
                assertTrue(ids.contains(id3));

                // add back feature with id1 with different SKU
                System.out.println("adding feature with id: " + id1);
                assertTrue(addFeature(id1, 45.1d, 123.4, 25L));

                // add a uniquely new feature that hasn't been added before
                String id4 = "ID4";
                System.out.println("adding feature with id: " + id4);
                assertTrue(addFeature(id4, 45.25d, 123.55, 18L));

                System.out.println("another query");
                ids = this.geoMesaHelper.getIDsFromFeatureCollection(this.geoMesaHelper.query(this.dataStore, GeoMesaHelperTest.FEATURE_NAME, "")); // unbounded query

                assertNotNull(ids);
                assertTrue(ids.contains(id4));
                assertTrue(ids.contains(id2));
                assertTrue(ids.contains(id3));
                assertTrue(ids.contains(id1));
                assertEquals(4, ids.size());
        }

When I run the above test, I see the following at the console:
    15:47:09,035  WARN IndexQueryPlanner:356 - Querying Accumulo without ST filter.
    15:47:09,356  WARN IndexQueryPlanner:356 - Querying Accumulo without ST filter.
    adding feature with id: ID1
    adding feature with id: ID4
    15:47:10,990  WARN TabletServerBatchReader:82 - TabletServerBatchReader not shutdown; did you forget to call close()?
    another query
    15:47:12,237  WARN IndexQueryPlanner:356 - Querying Accumulo without ST filter.

Further notes:
    1) The TabletServerBatchReader warning, when it appears, seems to consistently get triggered by the second addFeature() method; however, if I add Thread.sleep(5000) statements before each addFeature() method, I do not witness the TabletServerBatchReader warning.  I usually don't see the TabletServerBatchReader warning unless I have recently performed a feature modification or removal.  I have, however, on at least one occurrence, seen the warning when it was nowhere near a feature modification/removal.
    2) Both of my calls to this.geoMesaHelper.getIDsFromFeatureCollection() are unbounded queries and thus trigger two of the "WARN IndexQueryPlanner:356 - Querying Accumulo without ST filter." warnings.
    3) The other "WARN IndexQueryPlanner:356 - Querying Accumulo without ST filter." warning is triggered by the this.geoMesaHelper.removeFeatureByID() method.


Thanks in advance,
Beau

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users


Back to the top