Re: [orion-dev] Orion Server stress test results

[Simon Kaegi <Simon_Kaegi@xxxxxxxxxx>]

The scripts were not running on the box and in terms of location were a mixture of both inside and outside the foundation -- (I see Denis posted an hour ago that said it better). I think our "write" numbers were actually artificially high and would expect a dramatically higher proportion of reads so you're right that we probably will hit network I/O bottlenecks before either disk or cpu come into play.

-Simon

John Arthorne---02/09/2011 05:05:22 PM---Great results, thanks for doing this Denis. Just to be clear, were the 12 "fake user" processes als

From:	John Arthorne/Ottawa/IBM@IBMCA
To:	Orion developer discussions <orion-dev@xxxxxxxxxxx>
Date:	02/09/2011 05:05 PM
Subject:	Re: [orion-dev] Orion Server stress test results
Sent by:	orion-dev-bounces@xxxxxxxxxxx

---

Great results, thanks for doing this Denis. Just to be clear, were the 12 "fake user" processes also running on the same box? I.e., is the CPU load the cumulative load for the server plus the 12 clients, or just the server. In a real deployment I would expect network I/O to be the main bottleneck here. 

John

Denis Roy <denis.roy@xxxxxxxxxxx>
Sent by: orion-dev-bounces@xxxxxxxxxxx

02/09/2011 01:15 PM

Please respond to
Orion developer discussions <orion-dev@xxxxxxxxxxx>

To	orion-dev@xxxxxxxxxxx
cc
Subject	[orion-dev] Orion Server stress test results

Apologies if this is a bit exhaustive.  This morning Simon and I stress-tested a test Orion server.  The test only lasted about 10 minutes since it became clear we could not kill it.  Herein is my attempt at offering some sysadmin insight.

Test method: a shell script that logs in, and loops indefinitely, performing any one of the six actions below.  Unlike in the real world, each action has an equal, random chance of being executed.

- Get a random file, including a "zip-on-the-fly" export link
- Open a random directory
- Get the 'export' version of a directory or project (zip)
- Create a file in a random directory
- Edit an existing file (download a file, add 6KB of random content, then save it)
- Create a new folder somewhere in the tree

A single instance of the script was launched, then another, until we eventually had 12 instances running.


Server Details:
Intel SR1600UR with 16 CPU cores (Intel E5540 @ 2.53 GHz)
SuSE Linux Entreprise Server 11 running under Xen virtualization
8G RAM @ 1066MHz
Disk I/O: two regular SATA drives connected to an LSI raid controller (RAID 1, mirroring used)


By the numbers:
Total user accounts on server: over 20,000
Runtime: a few minutes
Total requests: 21261
GETs: 10650
POSTs: 9001
PUTs: 1609

For 2 minutes the system handled 25+ req/sec, peaking at 50+ req/sec for over 10 seconds.

Peak CPU load was 4 cores @ 100% with 12 idle cores
Peak Disk Read load was absolute zero.  The entire workspace was kept in RAM.
Peak Disk Write load was 21 MB/sec, with an average of about 12 MB/sec during peak period.  Disk writes were quite high since the odds of a new login (creating a workspace) and adding 6KB to a file were high.

Memory usage at start:         Free: 6.9G Write Buffers: 181M File Cache: 660M
Memory usage at end:           Free: 6.3G Write Buffers:195M File Cache 936M    (server workspace grew about 330M and was entirely contained in RAM)

Conclusion:
A single server has the potential of serving hundreds of concurrent, active users.  Scalability could be achieved by placing the serverworkspace on a shared filesystem and load-balancing the Orion servers, and perhaps adding a caching http server between the client and the Orion server. The two first bottlenecks would be Disk I/O and CPU.  I suspect CPU usage is partly due to the zip-on-the-fly nature of the export links.

I hope some of this is helpful.  Let me know if you have any questions.

Denis_______________________________________________
orion-dev mailing list
orion-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/orion-dev
_______________________________________________
orion-dev mailing list
orion-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/orion-dev