Re: [cdt-dev] Remote C/C++ Project Support Prototype

[Chris Recoskie <recoskie@xxxxxxxxxx>]

Hi Jeff,

Cross posting to ptp-dev...

Interestingly enough, it seems like most of what you did is done by PTP/RDT or CDT already in other ways. For instance, PTP has a framework for remote file access and launching remote commands which we use to target both RSE and a lightweight SSH provider called Remote Tools.

Your IRemoteFile proxy does a lot of things that we have been using CDT's EFSExtentions for (i.e. path extraction/mapping), although it does have the nice addition that it tells you something about the OS targeted, which EFS does not, and which we hadn't added to the EFSExtensions but probably could/should have/still ought to. Unfortunately the problem with IPath is that it's really designed to only deal with paths on the local filesystem, and IPaths tend to get passed down deep into the bowels of CDT, at which point you've sort of lost any way of getting back at the EFS objects they came from. We sort of rely on the fact that IPath.toString() uses UNIX path separators and so because we pretty much only care about targeting UNIX/Linux on the back end, things work out, but a current issue with this is that targeting Windows as a remote system is pretty broken.

For determining how to launch the commands, we have gone about this two ways. For makefile builds, we have our own RemoteMakeBuilder that asks the PTP service model for the build service provider, and then uses it to execute the remote builds. For managed build we use the ICommandLauncher coupled with a remote version of the toolchain so that when MBS launches the commands they execute on the remote machine, once again using the build service provider. The downside to your approach as you have done it so far is you don't have the flexibility to build on a different machine than you are editing on, or to build on multiple machines as a part of one build, which the RDT model allows for. Our model also allows for having files reside on your local machine for editing and source control integration but building on the remote machine.

One thing that it looks like you haven't really covered (at least from your email, I haven't looked at your code) is environment variables. If you are using the same local toolchains that CDT provides, then the environment variables that MBS uses to execute the build will be those from the local machine, and that can really mess up builds sometimes. Part of why we have remote toolchains in RDT is so that we can provide environment suppliers that get the environment from the remote machine.

The last thing... if you are using the indexer directly on either a mount or on EFS resources it's going to be significantly slower than if all the files were local or if you used the RDT remote indexer which does all the indexing on the server. There is a significant performance hit to pulling down all the source files over the wire. On a toy project you might not notice much, but on a "real" project and especially a large project, it's very significant. Also along these lines, doing binary parsing over the wire is quite possibly the slowest thing I have ever seen, to the point where in our remote toolchains we remove the specifications for the binary parsers. Even on relatively small projects the binary parser gets way out of hand.

The fact that I have not looked at your code brings up an interesting point... I'm not sure I *CAN* look at it unless it is already EPL licensed. Is it? Of course I know you intend it to be so if you are thinking of submitting it to CDT but are all the license files there, etc.? Normally we don't look at code until there's been a submission in Bugzilla, which implies it's being submitted under the EPL. Kind of a weird artifact of all this distributed source control stuff with Git... I can't look at it till I know it's properly licensed, but I don't know it's licensed properly unless I go look... cyclic!

At any rate, I'm interested in continuing the discussion... it might be worth having a call at some point in the near future.

===========================
Chris Recoskie
Team Lead, IBM CDT and RDT
IBM Toronto

Jeff Johnston ---01/12/2012 06:51:53 PM---I have recently been working on Remote Project support for Autotools projects and wanted to start a

From:	Jeff Johnston <jjohnstn@xxxxxxxxxx>
To:	"CDT General developers list." <cdt-dev@xxxxxxxxxxx>
Date:	01/12/2012 06:51 PM
Subject:	[cdt-dev] Remote C/C++ Project Support Prototype
Sent by:	cdt-dev-bounces@xxxxxxxxxxx

---

I have recently been working on Remote Project support for Autotools 
projects and wanted to start a discussion on what changes I made in the 
CDT and Autotools and whether this would be something the CDT would want 
to pursue.

To start with, my goal was to eliminate the need for separate Project 
types or wizards which IMO there are lots to confuse the user as-is.  To 
create the remote project, I simply wanted to specify the location of a 
C or C++ project as being remote.  To do this, in the C/C++  Project 
Wizards, one can specify a non-default location outside of the workspace 
and there is a file-system pull-down where one can choose the rse 
filesystem.  From there, there is a browser to look at existing remote 
connections and select an existing remote directory.  Trying this out 
was relatively easy, but as it turns out the RSE adds the connection 
name at the end of the URI it creates for the locaion, but does not add 
the project name.  So, to create the project name directory, one has to 
add the name of new project ahead of the connection query portion of the 
URI.  This obviously could be improved in the future so that the editing 
step would not be needed.

For my design, I chose to use RDT.  The beauty of RDT is that it allows 
conversion between paths and URIs.  This is an important feature in that 
it is possible to leave existing UI elements as-is and there is no real 
learning curve for the end-user.  The context of paths can be implied by 
the project location.  If the project is remote, the paths are on the 
same remote machine and no need to specify the full URI all over the 
place.  The user doesn't really need to see the remote machine URI all 
over the place and by supporting paths, the data formats do not require 
changing (e.g. paths occur in many classes in the CDT and there would be 
no need to switch to URIs).

So, the first thing was for me to create the framework.

In my particular experiment, I created a RemoteProxyManager in 
org.eclipse.cdt.core.internal.remoteproxy that implemented 
IRemoteProxyManager:

public interface IRemoteProxyManager {
         String EXTENSION_POINT_ID = "RemoteProxyManager"; //$NON-NLS-1$
         String MANAGER_NAME = "manager"; //$NON-NLS-1$
         String SCHEME_ID = "scheme"; //$NON-NLS-1$
         public IRemoteFileProxy getFileProxy(IProject project) throws 
CoreException;
         public IRemoteCommandLauncher getLauncher(IProject project) 
throws CoreException;
         public String getOS(IProject project) throws CoreException;
}

The job of the proxy manager was to fish out proxies for file and 
command access as well as to identify the remote OS.

For file access, there is the IRemoteFileProxy interface:

  public interface IRemoteFileProxy {

         public URI toURI(IPath path);
         public URI toURI(String path);
         public String toPath(URI uri);
         public String getDirectorySeparator();
         public IFileStore getResource(String path);

}

When accessing a path, one uses the project to get the IRemoteFileProxy 
and then paths can be converted to URIs or IFileStore objects directly. 
  A path for storing or displaying in the UI can be obtained from the 
URI via the API.

The IRemoteCommandLauncher interface allows one to run commands on the 
same system as the project:
public interface IRemoteCommandLauncher {

         public final static int OK = 0;
         public final static int COMMAND_CANCELED = 1;
         public final static int ILLEGAL_COMMAND = -1;

         public Process execute(String commandPath, String[] args, 
String[] env, String changeToDirectory, IProgressMonitor monitor) throws 
CoreException;
         public Process execute(String commandPath, String[] args, 
String[] env, String changeToDirectory, boolean usePTY, IProgressMonitor 
monitor) throws CoreException;
         public int waitAndRead(OutputStream output, OutputStream err, 
IProgressMonitor monitor);
         public String getErrorMessage();
         public String getCommandLine();

}

In my implementation, I also created a plug-in extension that would 
allow specification of an IRemoteProxyManager.  The remote proxy manager 
would register itself to handle certain URI schema.  The default 
RemoteProxyManager class only handles local references and then if not 
local, looks for an extension to handle the URI schema.  It then punts 
what ever method call was requested to the IRemoteProxyManager extension 
class.  For example, I created a remote proxy manager that uses RDT and 
handles URIs starting with rse:// and remotetools:// which handles 
projects set up via RSE or RDT.

I created the separate extension because my experiment started by 
altering the Autotools plug-ins in Linux Tools.  I did not want to make 
Autotools force RDT to be required so I used the extension.  The plug-in 
that implements the extension for RSE and RDT connections using RDT is 
not required for local project users.  For the CDT, I copied over the 
code from Linux Tools for the most part so I also copied over the 
extension.  The extension has the benefit that other systems could be 
used to plug in and handle other remote system types (e.g. IIRC, RDT 
doesn't support Windows as a target).

The design could easily be altered to use RDT directly and the CDT would 
then have a dependency on RDT.

Once the framework was in place, I started converting the Autotools and 
CDT code to use the proxies and found a few things that needed changing 
as a rule:

1. All path accesses that are project located can not use java.io.File.
    Instead, the project is used to get the IRemoteFileProxy and then
    a path can be used to get an IFileStore (EFS file).

2. The getLocation() method can not be used for a resource.  This
    returns null for a remote object.  Instead, one must use
    getLocationURI(), then use the IRemoteFileProxy to convert to a path
    which is what getLocation() returns.

3. Checking for existence requires getting the IFileStore as in 1), then
    using the fetchInfo() method to get the IFileInfo exists()
    method.

4. Running a command requires using a IRemoteCommandLauncher grabbed by
    using the project rather than a ICommandLauncher.

Once my changes were in place, I was able to create a remote Autotools 
Hello world project using the C Project Wizard (all I did was specify a 
remote location).  The remote project configures and builds on the 
remote machine.  I made a change so that the Binaries folder shows up 
and I am able to execute the binary on the remote system and have the 
results show up in the Console.  In addition, I was able to invoke 
Autotools commands remotely.

I did not change all code in the CDT as this was a proof of concept.  I 
added support to the TranslationUnit so the indexer appears to work but 
while it uses the correct remote source file, I did not make it use 
external headers so that would require further changes.  There is a 
remote indexer and if the indexer changes are not desired, the remote 
indexer could be defaulted for the end-user by the C Project wizard.

I didn't actually have to change too much in the Autotools plug-ins nor 
the CDT to get all of this to work.

If someone would like to see the changes I made, I have a fork of the 
CDT with a branch called "rdt" which you can access read-only here:

git://github.com/jjohnstn/cdt.git

The Linux Tools changes for Autotools can be found in the RDT branch of 
the org.eclipse.linuxtools git repository.

The Autotool changes are not in the master branch and are not part of 
the code move currently under way.

Would there be interest in pursuing this further?  I'm sure there are 
things I haven't thought of.

I could also look into posting a screen-cast somewhere so you could see 
how it all works.

-- Jeff J.
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev