| Re: [cdt-dev] Handling huge C/C++ project in eclipse |
|
From: Bill White
Sent: Wednesday, June 25, 2014 2:35 PM
To: CDT General developers list.
Reply To: CDT General developers list.
Cc: Parallel Tools Platform general developers
Subject: Re: [cdt-dev] Handling huge C/C++ project in eclipse
|
Thanks Ronald and Doug .
After reading your comments i am feeling like I am stuck again with my huge code base on which i wan't to use Eclipse in it's original form.
Bringing my huge code base on my local machine or on my remote server is not the problem. The problem arises when the indexing start, even if i assign 4GB of RAM for eclipse still it throws out of memory while indexing (both on my windows and Linux box). I did try all possible permutation and combinations of allocating memory to eclipse, but no success.
So i thought of doing some weird changes in eclipse.
Main component in eclipse which i want to use is CDT, so my scope was first limited to that.
The two things which i identified is the bottle neck in eclipse for huge code base are1. Eclipse recursive refresh. Which itself takes long time for a huge code base. And once it completes, it notifies all the resource change listeners. This is when the hell break lose with CDT.2. CDT indexer which is the most memory intensive, that too when it tries to index the huge code base in one go, it throws out of memory most of the time. And till the time it's running, the eclipse is not usable at all. Even if it does a complete indexing there is no guarantee that the Eclipse will not throw out of memory (My personal feeling is that CDT maintains a lot of information in memory).
So i put my hack in eclipse code.
1. I stopped the INFINITE_DEPTH refresh in eclipse.2. Created a custom recursive refresh call "On demand" (via command). And allowed the Depth One refresh call on Tree expansion from Explorer.3. Stopped the recursive refresh calls from CDT components.4. Plugged in a pre-build index file which i created by running cdt indexer in headless mode on my server via CDT extension point. The pre-build index file size is 4GB.5. To make call hierarchy and open deceleration usable, i just patched a refresh call for the Bindings found from pre-build index file.
Cool now i can create a C/C++ project without worrying about the code base size. Project gets created in no time. Eclipse don't give out of memory since it doesn't deal with complete source tree. i can refresh only those resources recursively which i want to look into or want to modify in my code base. And still i can utilize most of the functionality of CDT require during coding.
This is quite an ugly approach. But works for me since the Refresh and Indexing happens on demand.The major drawback is that i have modified the Eclipse Refresh mechanism.
So i was looking into Remote project, but again it has its own problem as mentioned in the mail thread below. Synchronize Project will again bring back the complete source tree indexing (in one go) back to my local machine and for sure Eclipse will give me out of memory.
I have changed the subject of the mail and have put cdt-dev mailing list also in the "To" list if anyone can suggest me something cleaner.
Thanks,Vishal
On Wed, Jun 25, 2014 at 2:20 AM, Doug Schaefer <dschaefer@xxxxxxx> wrote:
This has been a very interesting discussion. It certainly reminds us why the remote project support was added in the first place. I personally think that it's such a hard problem, there probably isn't a good solution for hooking up an IDE to such an extremely large code base located on a remote machine, as much as we want there to be.
But one take away I have that might make synchronized projects work out is that you're desktop probably has as much power as the server you're running the indexer on, especially since the indexer is single threaded. It'll probably take the same amount of time to index on either and then you're left with the remote tooling doing expensive look ups on the server and the latency in passing results back to the workstation. I'd be excited to hear if there was much success using it that way and kill my doubt.
Also, Gigs of source isn't such a big thing any more when we have terabyte drives in our laptops. It may be an expensive first time configuration to get all that copied over to the workstation, but if the synchronizer has done it's job well, things should be very fast after that.
Anyway, just another opinion for the pile. You get to choose your path, but I'm afraid the remote project one isn't well travelled and I'm not as confident on where it leads.
Doug.
From: ptp-dev-bounces@xxxxxxxxxxx [ptp-dev-bounces@xxxxxxxxxxx] on behalf of Roland Schulz [roland@xxxxxxx]
Sent: Tuesday, June 24, 2014 4:41 PM
To: Parallel Tools Platform general developers
Subject: Re: [ptp-dev] Remote Project
I recommend to use synchronized project and tune the local indexer. See e.g. http://stackoverflow.com/questions/9565125/whats-the-recommended-eclipse-cdt-configuration-for-big-c-project-indexer-ta for tips of how to do that.
Roland
On Tue, Jun 24, 2014 at 11:54 AM, Vishal Gupta <vishal.vit@xxxxxxxxx> wrote:
Ideally telling Yes i want to get the Indexing done on the server.And i am hopefull that it will ease the load on the local machine.
Yes i want to get everything index, but currently even my smaller project is not getting indexed
On Tue, Jun 24, 2014 at 9:11 PM, Roland Schulz <roland@xxxxxxx> wrote:
Hi,
do you want to use the index? Do you need everything indexed?
Roland
On Tue, Jun 24, 2014 at 8:22 AM, Vishal Gupta <vishal.vit@xxxxxxxxx> wrote:
Hello Greg,
Thanks for the feedback.
Let me give you a background why i thought of giving a try to the remote project.
1. I have a very huge code base ( in GBs)2. Using Eclipse to import the complete code base on the local machine is crazy. Eclipse can't withstand such a huge code base and the indexer makes things non-usable.3. So the solution was to checkout the code base on server and I tried to use the Remote Project to access it. The benefit i thought was that the indexing (which is the most memory intensive operation will happen on the server).
Now as you are suggesting to use the Synchronized project, my biggest nightmare "indexing" will again be happening on the local machine. Which will again make eclipse non-usable for my code base. Another thing is to sync the code base initially, my personal feeling is that it will take ages to do initial syncing between geographic different location.
I am very much open to contribute code if you people can guide me to a solution.
Thanks,Vishal
On Tue, Jun 24, 2014 at 5:16 PM, Greg Watson <g.watson@xxxxxxxxxxxx> wrote:
Hi Vishal,
Many of the problems you describe are also apparent even when accessing a remote project in the same geographic location, or even the same local network, and tend to be inherent in the nature of remote projects. It may be possible to speed up directory listings as you suggest, but this will not do much to speed up the overall performance of remote projects.
As a result of these issues, we have transitioned to another type of project called synchronized projects. These projects maintain both a local and remote copy of the source files. Apart from the time it takes to do the initial synchronization, these projects tend to work much faster with Eclipse, since the indexer and other expensive operations only happen on the local copy of the files.
Synchronized projects are available in both the Kepler and soon-to-be-released Luna versions of Eclipse for Parallel Application Developers package.
Ongoing development of remote projects has pretty much ceased, so it is unlikely there will be any future updates unless you would like to contribute something.
Regards,
Greg
On Jun 24, 2014, at 6:59 AM, Vishal Gupta wrote:
> Hello ,
>
> I have just started using Remote Project.
> I am accessing my project in a geographically different location. (Remote machine the server is in USA and local machine is in India)
> My initial impressing is that it's very slow to access even a small
> project with 1000 files (around few MB). I am using the "remote tool" configuration
> to access my project on remote machine.
> I tried looking into it and my initial finding is :
>
> Project Creation itself take a long time as the .project file and .setting folder is created and updated on the remote machine. Could it be possible to create the .project on the local machine ?
> I think the same happens with RSE connections where a hidden project is created on the local system for the connections and the configuration files are stored in it.There may be better solution for sure.
>
> For me the refresh of the files and folder is also very slow, my initial feeling is that the "ls" operation via sftp connection is taking more time. Any possibility to speed it up?
>
> The Indexing for my CProject is not working. The PDOM files is getting created on the remote system but the indexing is not happening. Do i need to do anything extra ?
> I checked the status of the Remote Indexing operation and it only had info and no error.
> I am still digging into the problem. Just wanted to check if i have missed anything ?
>
>
> Thanks,
> Vishal
> _______________________________________________
> ptp-dev mailing list
> ptp-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/ptp-dev
_______________________________________________
ptp-dev mailing list
ptp-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/ptp-dev
--
ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
865-241-1537, ORNL PO BOX 2008 MS6309
_______________________________________________
ptp-dev mailing list
ptp-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/ptp-dev
--
ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
865-241-1537, ORNL PO BOX 2008 MS6309
_______________________________________________
ptp-dev mailing list
ptp-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/ptp-dev
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev