Re: [epsilon-dev] Proposal to merge parallel-epsilon into master

[Dimitris Kolovos <dimitris.kolovos@xxxxxxxxxx>]

Hi Sina,

Thanks for the comprehensive summary and for all your hard work on
this! Please find some comments and questions below

- I would recommend renaming .launch projects to .cli to make their
purpose clearer.
- I'm all up for the use of reflection if it means more concise and
future-proof code.
- I'm not too worried about the breaking change in EVL.
- I may have missed this in your email, but has there been an
extension to the Eclipse-based EVL development tools that allows the
user to select the EvlModule implementation they wish to run their
constraints with?
- Do any of the changes clash with changes made by Horacio in the
context of his work on incremental execution?

Cheers,
Dimitris

On 13 April 2018 at 14:53, arcanefoam@xxxxxxxxx <arcanefoam@xxxxxxxxx> wrote:
> Hi Sina,
>
> The only comments I can make right now (without looking into the code) is
> that org.eclipse.epsilon.erl.engine.launch should be decoupled from the EMC.
> For this, a new interface could be provided (e.g. IModelCli) that adds a
> couple of methods that EMC drivers can decide to implement to be usable form
> the cli interface.
>
> The second one is that from what you showed me the use of reflection in the
> cli interface seems a bit of a stretch (again, I would need to look with
> more detail at what you did). Perhaps we can reuse the work I did on the
> standalone execution to use it as a backbone for the cli interface.
>
> Drop me an email so we can schedule a meeting to look further into these.
>
> Cheers,
>
>
> On Fri, Apr 13, 2018 at 1:05 PM, Sina Madani <sinadoom@xxxxxxxxxxxxxx>
> wrote:
>>
>> Hi all,
>>
>>
>>
>> As some of you may know, I have been working on parallelising EVL for many
>> months now. After extensive development, testing, refinement and
>> benchmarking, I believe the project is of production-ready maturity. The
>> changes made in this branch are extensive, so I will try to summarise them
>> below. Please let me know if you have any suggestions and/or objections to
>> this request.
>>
>>
>>
>> New Projects
>>
>> org.eclipse.epsilon.launch - allows for standalone launching of Epsilon
>> programs from command line. Uses org.apache.commons.cli_1.4, but this is not
>> on Orbit, so not included in the parallel-epsilon branch. This provides a
>> command-line argument parser as well as infrastructure for running programs,
>> getting results and profiling.
>> org.eclipse.epsilon.erl.engine.launch - an extension of the base
>> org.eclipse.epsilon.launch project which allows to run ERL programs
>> standalone (i.e. without Eclipse) through command-line (i.e. an executable
>> JAR file). The required inputs are path to script, model and metamodel
>> (currently only EMF models are supported). The module can be chosen by
>> appending the package name after "org.eclipse.epsilon." in the "-module"
>> option, so for example to run ETL one would add "-module etl.EtlModule"
>> (note that ETL must be on classpath for this to work). Additional arguments
>> to the module can also be passed and will be inferred through reflection.
>> For example, to pass an int argument of value 4 to the constructor of the
>> module, simply add "int=4". To profile execution time, use "-profile"
>> option. To show results, use "-results".
>> org.eclipse.epsilon.evl.engine.launch - extends
>> org.eclipse.epsilon.erl.engine.launch for EVL.
>> org.eclipse.epsilon.profiling.test - test for new classes added to
>> profiling (see below).
>> org.eclipse.epsilon.erl.engine.test - Infrastructure for thorough
>> equivalence testing. The "ErlEquivalenceTests" base class is particularly
>> complex but greatly reduces the repetitive code required to perform
>> equivalence tests between different modules. The class has been documented
>> and is mainly there for testing parallel ERL extensions.
>>
>>
>> Additions
>>
>> org.eclipse.epsilon.common - Extensions of commonly used
>> java.util.function interfaces which allow throwing of checked exceptions.
>> Concurrency utilities. Additional methods in CollectionUtil. Ability to get
>> CPU model for Linux and Windows systems in OperatingSystem. "BaseDelegate"
>> allows for a class to have a parent, and to look up items in the parent and
>> also merge items from or into parent (see FrameStack example below).
>> org.eclipse.epsilon.profiling - two new classes: "ProfileDiagnostic" and
>> "BenchmarkUtils" for standalone command-line profiling. These measure
>> execution time and memory usage, with adjustable units for memory (i.e. can
>> convert between bytes, MB, GB etc.). Facilities are provided for
>> benchmarking any type of code passed as a Runnable, Function, Supplier,
>> Consumer and their checked variants.
>> org.eclipse.epsilon.eol.engine - functional interfaces which can throw
>> EolRuntimeException.
>> org.eclipse.epsilon.erl.engine - Concurrency utilities for parallel
>> execution. This includes an interface "IErlContextParallel" along with
>> skeletal implementation, as well as a custom "ErlExecutorService" interface
>> (and a ThreadPoolExecutor implementation).
>> org.eclipse.epsilon.evl.engine - Parallel implementations of EVL, building
>> on the concurrency framework introduced in ERL. These execution strategies
>> are implemented as extensions of the abstract "EvlModuleParallel" and reside
>> in the org.eclipse.epsilon.evl.concurrent package. Currently the best two
>> strategies are "EvlModuleParallelElements" and "EvlModuleParallelStaged".
>> Also a shallow static analysis class "EvlGraph" can order constraints in an
>> EvlModule based on satisfies calls, so that those depended on are executed
>> first.
>> org.eclipse.epsilon.evl.engine.test.acceptance  - Add many new tests to
>> exercise all features of EVL, as well as new scripts, models and metamodels.
>> Added equivalence tests between original EvlModule and the parallel ones.
>> Infrastructure built on top of org.eclipse.epsilon.erl.engine.test.
>>
>>
>> Changes
>>
>> Many collections in EOL (namely ArrayLists) are now sized to avoid wasting
>> memory.
>> Cached operations map assigned in build if applicable, more efficient
>> checks, implemented as a synchronized WeakHashMap.
>> AbstractModuleElement has equals and hashCode.
>> StackTraceManager uses ArrayDeque instead of Stack (which is a legacy
>> class and is extremely slow because it is synchronized). Performance of all
>> Epsilon programs should be noticeably better as a result.
>> FrameStack (and FrameStackRegion) can now be toggled as concurrent (and
>> have a parent). FrameStacks (and regions) can now be merged - typically used
>> for merging concurrent FrameStack into the single-threaded parent.
>> EolModule asyncStatementQueue is now a Queue instead of ArrayList, leading
>> to simpler code.
>> MultiMap has concurrent version (toggled with boolean).
>> CachedModel has concurrent version (again toggled with boolean, default is
>> true).
>> Parallel getAllOfKindFromModel/getAllOfTypeFromModel for large EMF models.
>> Concurrent AbstractEmfModel (toggled with boolean).
>> OperationContributorRegistry has concurrent version (toggled with boolean)
>> and like with FrameStack, can have parents and be merged.
>> ExecutorFactory has concurrent version (toggled with boolean). As with
>> FrameStack, can have parents and ExecutionListeners can be merged.
>> EolRuntimeException's message can now be cached if desired, to capture the
>> AST stack trace at a particular point in time for later reporting (useful
>> for parallel execution exception handling semantics). A recursive method to
>> find an EolRuntimeException from a Throwable is also added.
>> IEvlContext has new methods for grouping unsatisfied constraints, sorting
>> based on number of failures. The UnsatisfiedConstraints is now a Set instead
>> of a List as it should have been, since it makes no sense to have duplicate
>> unsatisfied constraints.
>> Constraints are only added to the ConstraintTrace if they are the target
>> of a dependency (works in conjunction with new Set<Constraint>
>> constraintDependedOn).
>> equals and hashCode for EVL data structures - used for equivalence
>> testing.
>> EvlModule.executeImpl returns Set<UnsatisfiedConstraint> instead of null.
>> Moved Execution logic for checking all constraints out of
>> ConstraintContext and into EvlModule.
>> Concurrent data structures in EVL (with serial thread confinement pattern
>> where appropriate). Includes ConstraintTrace and Set<UnsatisfiedConstraint>.
>> dt support for Parallel EVL. Currently implemented as check box in
>> "Advanced" tab.Used EvlModuleParallelElements with
>> Runtime.getRuntime.availableProcessors() number of threads.
>> Various other small optimisations and minor refactoring in EOL.
>>
>>
>> I am also aware that running Epsilon standalone (i.e. without Eclipse/GUI)
>> is something that a few people are interested in. I would be happy to
>> collaborate with anyone on this, perhaps by enhancing the functionality of
>> the new CLI/reflection-based org.eclipse.epsilon.erl.engine.launch project.
>>
>> Finally I'd like to note that despite the number of changes, every attempt
>> has been made to ensure backwards compatibility, so for example in places
>> where a concurrent data structure is used, with the exception of CachedModel
>> the default is the original non-concurrent structure. The only "breaking"
>> change is that EVL's results are now a Set<UnsatisfiedConstraint> instead of
>> a List<UnsatisfiedConstraint>; though it is difficult to think of a scenario
>> where index-based access and/or ordering would be required for EVL results.
>> Furthermore all tests are passing (with the exception of EGL-related tests
>> on Windows, which presumably was never used as a test platform).
>>
>>
>> Thanks,
>>
>> Sina Madani
>>
>>
>>
>> _______________________________________________
>> epsilon-dev mailing list
>> epsilon-dev@xxxxxxxxxxx
>> To change your delivery options, retrieve your password, or unsubscribe
>> from this list, visit
>> https://dev.eclipse.org/mailman/listinfo/epsilon-dev
>
>
>
> _______________________________________________
> epsilon-dev mailing list
> epsilon-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from
> this list, visit
> https://dev.eclipse.org/mailman/listinfo/epsilon-dev



-- 
Dimitris Kolovos
Professor of Software Engineering
Department of Computer Science
University of York
http://www.cs.york.ac.uk/~dkolovos

EMAIL DISCLAIMER http://www.york.ac.uk/docs/disclaimer/email.htm