| Re: [geomesa-users] Ingestion speed |
|
Mario, To add a little more to what Emilo has mentioned there are a few accumulo considerations that control ingest. Write Ahead Logs (WAL) are a large factor in limiting ingest. With WALs on, the data must be sync'd to disk as well as placed in an in memory buffer before being minor compacted to disk. Each sync to disk (usually an hsync by hadoop which causes an underlying filesystem fsync) is very expensive. Thus, minimizing these costs are important. As Emilio mentioned make sure to enable native memory maps with "tserver.memory.maps.native.enabled" (which is the default in 1.6) They key to ingest is getting enough clients writing data, have enough network to handle it, and have enough memory on the tservers to handle lots of ingest. On the tserver you can play with: 1. increasing the in memory buffer...This parameter is "tserver.memory.maps.max" and requires a tserver restart 2. increasing the number of WALS associated with a tablet...this parameter is table.compaction.minor.logs.threshold but is only important if you are seeing this as the reason for minc in the logs. It's OK to increase it to like 5-6 it just makes recovery slower. On a long running ingest if you see spikes that take the rate almost down to zero that's generally what is happening. It gets worse as the number of tablets increases. 3. increase the queue of mutations to sync to the WALS: tserver.mutation.queue.max If you have lots of waiting minor or major compactions on individual tservers (noted in parenthesis in the monitor): 4. increase tserver.compaction.major.concurrent.max 5. increase tserver.compaction.minor.concurrent.max When increasing all of these parameters you must note that the server load can increase in terms of CPU, memory, and network bandwidth at the expense of query if your boxes aren't large. In terms of memory, give accumulo lots...regularly we run with 30-40 gigs. For hadoop use 2.4 or higher, install native libraries for hadoop, and enable short circuiting. The goal of quick ingest is to fill up the in memory buffers as fast as we can and then compact them out and to do this on every tserver. Some stats for a 20 node Amazon cluster with network-mounted disk and WAL on: averages ~250k simple features (with 22 attributes each) per second which means when you throw an index in you'll end up getter 1M entries/s which 50k per node. If you want to head higher than that you'll have to consider using Accumulo Bulk Ingest which requires building rfiles with some process such as MapReduce, Storm, NiFi, or other framework and then loading them directly into accumulo. Generally you're getting to a much larger cluster before you need to do things like that. I like to set my BatchWriter mem higher with low latencies and make lots of batch writers. Lastly, note that ingesting streams of data in time order is slow than a map-reduce approach with batchwriters. I'll run an ingest at some point with some smaller features around 7-8 attrs and let you know what the rate was. Andrew On 11/12/2015 12:25 PM, Emilio
Lahr-Vivaz wrote:
|