Beau,
I'm looking into the warning for closing BatchScanners...could you
send me your helper classes so I can replicate the bug?
Andrew
On 08/08/2014 08:12 AM, Beau Lalonde
wrote:
Anthony,
Thanks
for the detailed explanation and putting my worries at ease
(and for fixing the chattiness).
Beau
From:
geomesa-users-bounces@xxxxxxxxxxxxxxxx
[mailto:geomesa-users-bounces@xxxxxxxxxxxxxxxx]
On Behalf Of Anthony Fox
Sent: Thursday, August 07, 2014 4:42 PM
To: Geomesa User discussions
Subject: Re: [geomesa-users] Is GeoMesa-triggered
Accumulo "TabletServerBatchReader not shutdown" warning
something I can safely ignore?
I think we just fixed this warning,
though I'm not sure it's made it into master just yet. Some
more details: the query planner receives the user's query
and decides which table will optimally satisfy the query.
In some cases, the attribute index table is the best
candidate. In those cases, we open two scanners, one to
retrieve the ids from the reverse index in the attribute
table and one to resolve the data entries in the records
table. This process is run in two threads for performance
but has a blocking queue so as not to overwhelm memory in
cases where the size of the result is huge or the user code
does not need to seek to the end of the result set. We
weren't quietly shutting down the threads and closing the
internal scanners. It won't cause any issues other than
being overly chatty. Too chatty - I think we resolved it
using IOUtils.closeQuietly from commons-io.
Thanks,
Anthony
On Thu, Aug 7, 2014 at 4:33 PM, Beau
Lalonde <blalonde@xxxxxxxx>
wrote:
Hi,
I am using a version of GeoMesa from early last week
(late July 2014) and I am frequently witnessing the
following, presumably accumulo, log message:
WARN TabletServerBatchReader:82 -
TabletServerBatchReader not shutdown; did you forget to
call close()?
I have done some investigating and cannot find out how
to reproduce the warning 100% of the time, but I should
emphasize that in our production code we see the warning
a lot. We have yet to see any bad behavior result from
the warning (except, perhaps slower processing?);
however, we are nonetheless concerned about the warning.
Through unit testing of GeoMesa, I only see the warning
popup (semi-consistently) in some of my tests (not all
of them). The nature of the tests where I see the
warning indicates to me that the warning is caused by
the feature modification code and/or the feature removal
code (featureStore.modifyFeatures(), and/or
featureStore.removeFeatures()), because I usually do not
see the warning until after the modifyFeatures() or
removeFeatures() methods are first invoked.
My questions are:
1) What within GeoMesa is causing the warning?
2) Is this a GeoMesa bug?
3) Is the warning anything that I need to be
concerned about?
4) Is there a way to prevent the warning?
Some more background, I seem to see the warning almost
all of the time when I run the following test (which
passes):
@Test
public void testRemoveFeatureByID_AddRemoveAdd()
{
// add three features, then remove one,
verify its removal, add two features (one of which has
the same ID as the one removed), verify addition
String id1 = "ID1";
assertTrue(addFeature(id1, 45.1d, 123.4,
15L));
String id2 = "ID2";
assertTrue(addFeature(id2, 45.0d, 123.3,
16L));
String id3 = "ID3";
assertTrue(addFeature(id3, 45.2d, 123.5,
17L));
this.geoMesaHelper.removeFeatureByID(GeoMesaHelperTest.FEATURE_NAME,
id1);
Set<String> ids =
this.geoMesaHelper.getIDsFromFeatureCollection(this.geoMesaHelper.query(this.dataStore,
GeoMesaHelperTest.FEATURE_NAME, "")); // unbounded query
assertNotNull(ids);
assertEquals(2, ids.size());
assertTrue(ids.contains(id2));
assertTrue(ids.contains(id3));
// add back feature with id1 with
different SKU
System.out.println("adding feature with
id: " + id1);
assertTrue(addFeature(id1, 45.1d, 123.4,
25L));
// add a uniquely new feature that
hasn't been added before
String id4 = "ID4";
System.out.println("adding feature with
id: " + id4);
assertTrue(addFeature(id4, 45.25d,
123.55, 18L));
System.out.println("another query");
ids =
this.geoMesaHelper.getIDsFromFeatureCollection(this.geoMesaHelper.query(this.dataStore,
GeoMesaHelperTest.FEATURE_NAME, "")); // unbounded query
assertNotNull(ids);
assertTrue(ids.contains(id4));
assertTrue(ids.contains(id2));
assertTrue(ids.contains(id3));
assertTrue(ids.contains(id1));
assertEquals(4, ids.size());
}
When I run the above test, I see the following at the
console:
15:47:09,035 WARN IndexQueryPlanner:356 - Querying
Accumulo without ST filter.
15:47:09,356 WARN IndexQueryPlanner:356 - Querying
Accumulo without ST filter.
adding feature with id: ID1
adding feature with id: ID4
15:47:10,990 WARN TabletServerBatchReader:82 -
TabletServerBatchReader not shutdown; did you forget to
call close()?
another query
15:47:12,237 WARN IndexQueryPlanner:356 - Querying
Accumulo without ST filter.
Further notes:
1) The TabletServerBatchReader warning, when it
appears, seems to consistently get triggered by the
second addFeature() method; however, if I add
Thread.sleep(5000) statements before each addFeature()
method, I do not witness the TabletServerBatchReader
warning. I usually don't see the
TabletServerBatchReader warning unless I have recently
performed a feature modification or removal. I have,
however, on at least one occurrence, seen the warning
when it was nowhere near a feature modification/removal.
2) Both of my calls to
this.geoMesaHelper.getIDsFromFeatureCollection() are
unbounded queries and thus trigger two of the "WARN
IndexQueryPlanner:356 - Querying Accumulo without ST
filter." warnings.
3) The other "WARN IndexQueryPlanner:356 - Querying
Accumulo without ST filter." warning is triggered by the
this.geoMesaHelper.removeFeatureByID() method.
Thanks in advance,
Beau
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password,
or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users
|