|
Emilio:
Great catch. Should have thought about the multiple
iterators.
And thanks for the efficiency.
On 4/25/17 10:55 AM, Emilio Lahr-Vivaz
wrote:
Hi David,
I think the issue you are having is calling collection.features()
repeatedly. This method returns a new iterator each time you call
it, which then must be closed. So calling
`collection.features().hasNext(); collection.features().close();`
will open two iterators, one of which is closed immediately and
the other which is never closed. Since you're only ever checking
the first result from a feature iterator, it's behaving correctly,
but executing your query multiple times (once per iterator) and
leaving open connections.
There's another improvement that you could do (although it won't
change the underlying result, it will be much more efficient)
which is to use a modifying feature writer instead of doing a
query and then modify (pseudo code example):
SimpleFeatureWriter writer = dataStore.getFeatureWriter(typeName,
filter, Transaction.AUTO_COMMIT);
while (writer.hasNext()) {
SimpleFeature toModify = writer.next();
// set the attributes you want to update in the toModify feature
writer.write();
}
writer.close();
hth,
Emilio
On 04/24/2017 05:43 PM, David Boyd
wrote:
All: I have been fighting these issues for some time and
have narrowed it down to the below chunk of code.
Based on the log file attached the error occurs randomly at
the lines of code below labeled Error here in trace
You will see a number of calls to
collection.features().close() commented out. If I enable
those, I get a ton
of errors about the socket already being closed. I have
tried enabling different combinations of the call to close
and get basically the same results.
The near random nature of this occurring if you look at the
trace appears to be some sort of connection or
thread pool timing issue based on reuse.
Has anyone of the list seen this error. I am using Accumulo
1.7.2 and Geomesa 1.3.1.
private boolean
updateFeatureIfExists(String featuresetname, CoalesceRecord
record)
throws IOException, CQLException,
CoalesceDataFormatException
{
boolean updated = false;
SimpleFeatureStore store = (SimpleFeatureStore)
connect.getGeoDataStore().getFeatureSource(featuresetname);
FilterFactory2 ff =
CommonFactoryFinder.getFilterFactory2();
FeatureId fid =ff.featureId(record.getKey());
// TODO Need to add compare of modified time to see if
there was an update
Filter filter = ff.id(Collections.singleton(fid));
// Need to escape the fully qualified column in the
feature set for filters
//Filter filter =
CQL.toFilter("\""+featuresetname+".recordKey\" =" + "'" +
record.getKey() + "'");
SimpleFeatureCollection collection =
store.getFeatures(filter);
int size = collection.size();
boolean hasNext =
collection.features().hasNext(); <<<<<
Error here in trace
LOGGER.info("Entity Collection size: {} hasNext:
{}",size,hasNext);
if (hasNext )
{
SimpleFeature thefeature =
collection.features().next(); <<< Error here in
trace
//LOGGER.info("Features found:
{}",collection.size());
//collection.features().close();
// Transactions not supported by GEOMESA
Transaction transaction = new
DefaultTransaction();
store.setTransaction(transaction);
List<String> fieldNames = new
ArrayList<String>();
List<Object> fieldVals = new ArrayList();
String liststring;
for (CoalesceField<?> field :
record.getFields())
{
// update
//store.modifyFeatures(field.getName(),
field.getValue(), filter);
fieldNames.add(field.getName());
Object fieldvalue = field.getValue();
switch (field.getDataType()) {
// These types should be able to be
handled directly
case BOOLEAN_TYPE:
case DOUBLE_TYPE:
case FLOAT_TYPE:
case INTEGER_TYPE:
case LONG_TYPE:
case STRING_TYPE:
case URI_TYPE:
case LINE_STRING_TYPE:
case POLYGON_TYPE:
fieldVals.add(fieldvalue);
break;
case STRING_LIST_TYPE:
liststring =
Arrays.toString((String[]) fieldvalue);
fieldVals.add(liststring);
break;
case DOUBLE_LIST_TYPE:
liststring =
Arrays.toString((double[]) fieldvalue);
fieldVals.add(liststring);
break;
case INTEGER_LIST_TYPE:
liststring = Arrays.toString((int[])
fieldvalue);
fieldVals.add(liststring);
break;
case LONG_LIST_TYPE:
liststring = Arrays.toString((long[])
fieldvalue);
fieldVals.add(liststring);
break;
case FLOAT_LIST_TYPE:
liststring = Arrays.toString((float[])
fieldvalue);
fieldVals.add(liststring);
break;
case GUID_LIST_TYPE:
liststring = Arrays.toString((UUID[])
fieldvalue);
fieldVals.add(liststring);
break;
case BOOLEAN_LIST_TYPE:
liststring =
Arrays.toString((boolean[]) fieldvalue);
fieldVals.add(liststring);
break;
case GUID_TYPE:
String guid = ((UUID)
fieldvalue).toString();
fieldVals.add(guid);
break;
case GEOCOORDINATE_LIST_TYPE:
MultiPoint points = new
GeometryFactory().createMultiPoint((Coordinate[]) fieldvalue);
fieldVals.add(points);
break;
case GEOCOORDINATE_TYPE:
Point point = new
GeometryFactory().createPoint((Coordinate) fieldvalue);
fieldVals.add(point);
break;
// Circles will be converted to polygons
case CIRCLE_TYPE:
// Create Polygon
CoalesceCircle circle =
(CoalesceCircle) fieldvalue;
GeometricShapeFactory factory = new
GeometricShapeFactory();
factory.setSize(circle.getRadius());
factory.setNumPoints(360); // 1 degree
points
factory.setCentre(circle.getCenter());
Polygon shape =
factory.createCircle();
fieldVals.add(shape);
break;
case DATE_TIME_TYPE:
fieldVals.add (((DateTime)
fieldvalue).toDate());
break;
case FILE_TYPE:
case BINARY_TYPE:
default:
break;
}
}
try {
store.modifyFeatures( fieldNames.toArray(new
String[fieldNames.size()]), fieldVals.toArray(), filter);
} catch (IllegalArgumentException e)
{
LOGGER.error(e.getMessage(), e);
updated = false;
return updated;
}
updated = true;
transaction.commit();
transaction.close();
} //else {
// collection.features().close();
// }
return updated;
}
private boolean updateLinkIfExists(String featuresetname,
CoalesceLinkage link)
throws IOException, CQLException,
CoalesceDataFormatException
{
boolean updated = false;
SimpleFeatureStore store = (SimpleFeatureStore)
connect.getGeoDataStore().getFeatureSource(featuresetname);
FilterFactory2 ff =
CommonFactoryFinder.getFilterFactory2();
FeatureId fid =ff.featureId(link.getKey());
Filter filter = ff.id(Collections.singleton(fid));
// Need to escape the fully qualified column in the
feature set for filters
//Filter filter =
CQL.toFilter("\""+featuresetname+".recordKey\" =" + "'" +
record.getKey() + "'");
SimpleFeatureCollection collection =
store.getFeatures(filter);
int size = collection.size();
boolean hasNext =
collection.features().hasNext(); <<< Error here in trace
LOGGER.info("Linkage Collection size: {} hasNext:
{}",size,hasNext);
if (hasNext)
{
//LOGGER.info("Found Linkages:
{}",collection.size());
SimpleFeature thefeature =
collection.features().next(); <<<
Error here in trace
// if (collection.features().hasNext()) {
// collection.features().close();
// }
// Transactions not supported by GEOMESA
Transaction transaction = new
DefaultTransaction();
store.setTransaction(transaction);
//SimpleFeature thefeature =
collection.features().next();
final String linkfields[] = {
LINKAGE_ENTITY1_KEY_COLUMN_NAME,
LINKAGE_ENTITY1_NAME_COLUMN_NAME,
LINKAGE_ENTITY1_SOURCE_COLUMN_NAME,
LINKAGE_ENTITY1_VERSION_COLUMN_NAME,
LINKAGE_ENTITY2_KEY_COLUMN_NAME,
LINKAGE_ENTITY2_NAME_COLUMN_NAME,
LINKAGE_ENTITY2_SOURCE_COLUMN_NAME,
LINKAGE_ENTITY2_VERSION_COLUMN_NAME,
LINKAGE_LAST_MODIFIED_COLUMN_NAME,
LINKAGE_LABEL_COLUMN_NAME,
LINKAGE_LINK_TYPE_COLUMN_NAME
};
Object[] values = new Object[] {
link.getEntity1Key(),
link.getEntity1Name(),
link.getEntity1Source(),
link.getEntity1Version(),
link.getEntity2Key(),
link.getEntity2Name(),
link.getEntity2Source(),
link.getEntity2Version(),
link.getLastModified(),
link.getLabel(),
link.getLinkType()
};
store.modifyFeatures(linkfields, values, filter);
updated = true;
transaction.commit();
transaction.close();
} //else {
// collection.features().close();
// }
return updated;
}
--
========= mailto:dboyd@xxxxxxxxxxxxxxxxx ============
David W. Boyd
VP, Data Solutions
10432 Balls Ford, Suite 240
Manassas, VA 20109
office: +1-703-552-2862
cell: +1-703-402-7908
============== http://www.incadencecorp.com/ ============
ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture
Chair ANSI/INCITS TC Big Data
Co-chair NIST Big Data Public Working Group Reference Architecture
First Robotic Mentor - FRC, FTC - www.iliterobotics.org
Board Member- USSTEM Foundation - www.usstem.org
The information contained in this message may be privileged
and/or confidential and protected from disclosure.
If the reader of this message is not the intended recipient
or an employee or agent responsible for delivering this message
to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication
is strictly prohibited. If you have received this communication
in error, please notify the sender immediately by replying to
this message and deleting the material from any computer.
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users
--
========= mailto:dboyd@xxxxxxxxxxxxxxxxx ============
David W. Boyd
VP, Data Solutions
10432 Balls Ford, Suite 240
Manassas, VA 20109
office: +1-703-552-2862
cell: +1-703-402-7908
============== http://www.incadencecorp.com/ ============
ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture
Chair ANSI/INCITS TC Big Data
Co-chair NIST Big Data Public Working Group Reference Architecture
First Robotic Mentor - FRC, FTC - www.iliterobotics.org
Board Member- USSTEM Foundation - www.usstem.org
The information contained in this message may be privileged
and/or confidential and protected from disclosure.
If the reader of this message is not the intended recipient
or an employee or agent responsible for delivering this message
to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication
is strictly prohibited. If you have received this communication
in error, please notify the sender immediately by replying to
this message and deleting the material from any computer.
|
