| Re: [cdt-dev] Improved reliability of Index tests |
|
Hi Lukas,
I can reproduce the intermittent failures. I created bug 431684 to track this so we can take the discussion there. Marc-Andre On 3/31/2014 3:06 PM, Marc-André Laperle wrote: Hi Lukas, Thank you for the information. I've had intermittent failures on my fastest PC before so I'll try your scenario a bit later to see if I can reproduce it. I suggested increasing the timeout because on VERY slow machines I've seen it go above 10 secs but given that it happens on fast pcs then it must be a different issue.Calling the refresh alone does not improve the situation as it does with the workspace builds. File contents in Eclipse get cached. Since these caches might only be updated with resource changes I can see a good chance that rebuilding effectively enforces some update of contents which will then get into the indexer.I'm not clear on which cache might be still be outdated after a refresh but perhaps that's the case. Hopefully reproducing it on my end can narrow it down a bit. Marc-Andre On 14-03-31 01:21 PM, Lukas Felber wrote:Hi Marc-Andre The effect of failing tests is completely random. The correlation I see is, that the more the indexer is used, the more it happens and also that if it runs on slower pc's, the failure go away (on our one-core vm-build-server test do never fail, on my quad-core pc they do a lot). Here at the HSR, I'm also seeing many others with the same problems (lots of student projects, mostly cdt-refactoring stuff often heavily making use of the indexer) at least until I show them what you see now in my gerrit patch. Basically any test involving the indexer/bindings fails sometimes. There is no real pattern its just completely random. To me its pretty obvious that cdt-indexing has a synchronization issue. Failing happens not in waitForIndexer or waitUntilFileIsIndexed but rather while asserting results produced by the indexer which are just wrong. Meaning I see stuff like "expected:<2> but was:<1>" meaning the indexer did not yield 2 but wrongly only one IIndexName. I can understand your theory about the doing some random stuff might make enough delay to help. But since test execution (i did many runs of the full cdt-core-tests suite and tonns of 100s of executions of IndexBugsTests) do not take longer (difference is marginal) I do not think that your theory is right. Also, the removing of Thread.sleep calls in a loop speaks for my solution. Calling the refresh alone does not improve the situation as it does with the workspace builds. File contents in Eclipse get cached. Since these caches might only be updated with resource changes I can see a good chance that rebuilding effectively enforces some update of contents which will then get into the indexer. Here some stats (average over several 650 IndexBugsTest runs): * Original: Failures 6-7, Time 72-81sec * With -Dindexer.timeout=60: Failures 5-7, Time 75-81sec * My solution: Failures 0-0, Time 76-80sec Can you reproduce anything like this on your computer? Basically, I also must say if adding/increasing some timeout somewhere helps, then the solution is broken. To fix such issues, some synchronisation/joining is missing. And thats the point where a fix should come from. I hope that you see it that way too! Lukas On 31.03.2014 16:42, Marc-André Laperle wrote:Hi Lukas, I've seen tests fail before because of waitForIndexer timing out. Which tests are failing for you? Do they fail with an asserting failed in waitForIndexer or waitUntilFileIsIndexed? I think the reason why adding the workspace.build calls might be helping is that it adds an additional delay and gives the indexer a better chance to finish. But this is just a theory. I didn't think the index tests depended on the build and if they do, I'd be curious to know why. Perhaps it's only the refresh that's necessary? Then I wonder why that would be the case because I believe all the files are created in the workspace through the eclipse.resources APIs. Could you try increasing the indexer timeout and see if that also improves for you? You can add -Dindexer.timeout=60 in your VM arguments (that's if you launch from inside Eclipse, for maven you have to add it in the pom.xml files). Marc-Andre ________________________________________ From: cdt-dev-bounces@xxxxxxxxxxx [cdt-dev-bounces@xxxxxxxxxxx] on behalf of Lukas Felber [lfelber@xxxxxx] Sent: Monday, 31 March 2014 8:08 AM To: CDT General developers list. Subject: Re: [cdt-dev] Improved reliability of Index tests Hi Marc-André Please see https://git.eclipse.org/r/#/c/24080/ again. It is now rebased to master. The new build even got +1 from hudson. I actually do not exactly know at what point the workspace.build(IncrementalProjectBuilder.FULL_BUILD) method helps the cdt index tests. It seems to refresh resources and hence get rid of outdated cache stuff. This then makes the index more up to date than without rebuilding. Lukas On 28.03.2014 17:59, Marc-André Laperle wrote:Hi Lukas, Can you explain what is the purpose of the build? Also, if you rebase your patch, I made changed to CDescriptorTests and codan so that they run more reliably. Marc-André On 14-03-28 12:58 PM, Lukas Felber wrote:Hello Everyone I created a small patch ( https://git.eclipse.org/r/24080 ) that improves the reliability of index tests. When running the 65 IndexBugsTests 50 times (total 3250 test runs) from a test suite, I got 90 to 100 random test failures. After my adaptions, failures are reduced to 0-5 failures (sadly not to 0). Meaning a failure reduction of ~95%. When running all core tests, I never saw an index test fail (in 10+ runs). Those CDescriptorTests do still fail (I did not touch these). The adaptions do not have any impact on run-time (+/- 3%). And I was able to remove the very ugly part that called Thread.sleep() in a loop. The better result is achieved by having two additional calls to workspace.build(IncrementalProjectBuilder.FULL_BUILD) which basically is similar to a "clean..." call from the "Project" eclipse ui-menu. I hope that some of you guys can try the patch and check if it does not have any negative effects somewhere (I only checked the cdt-core and codan tests). Cheers Lukas PS: I will most likely not be active in CDT development in the near future since I'm leaving HSR and moving on to IBM. _______________________________________________ cdt-dev mailing list cdt-dev@xxxxxxxxxxx https://dev.eclipse.org/mailman/listinfo/cdt-dev_______________________________________________ cdt-dev mailing list cdt-dev@xxxxxxxxxxx https://dev.eclipse.org/mailman/listinfo/cdt-dev_______________________________________________ cdt-dev mailing list cdt-dev@xxxxxxxxxxx https://dev.eclipse.org/mailman/listinfo/cdt-dev _______________________________________________ cdt-dev mailing list cdt-dev@xxxxxxxxxxx https://dev.eclipse.org/mailman/listinfo/cdt-dev_______________________________________________ cdt-dev mailing list cdt-dev@xxxxxxxxxxx https://dev.eclipse.org/mailman/listinfo/cdt-dev_______________________________________________ cdt-dev mailing list cdt-dev@xxxxxxxxxxx https://dev.eclipse.org/mailman/listinfo/cdt-dev |