Re: [january-dev] IDataset <-> python

[<Matt.Gerring@xxxxxxxxxxxxx>]

The dataset flattener probably does most of what you need:
https://github.com/DawnScience/scisoft-core/blob/master/uk.ac.diamond.scisoft.analysis.xmlrpc/src/uk/ac/diamond/scisoft/analysis/rpc/flattening/helpers/DatasetHelper.java

On the python side we have the equivalent:
https://github.com/DawnScience/scisoft-core/blob/master/uk.ac.diamond.scisoft.python/src/scisoftpy/python/pyflatten.py


-----Original Message-----
From: january-dev-bounces@xxxxxxxxxxx [mailto:january-dev-bounces@xxxxxxxxxxx] On Behalf Of Matt.Gerring@xxxxxxxxxxxxx
Sent: 30 January 2017 09:34
To: january-dev@xxxxxxxxxxx; erwin.de.ley@xxxxxxxxxx; cxh@xxxxxxxxxxxx
Subject: Re: [january-dev] IDataset <-> python

Hi Scott,

It is exciting that you are using it for that! This is a great new usage, exactly the kind of thing we were hoping for in making the work open source, thanks for picking it up.

I introduced py4j to dawnsci and in that project we send around ndarry<->IDataset from a py4j connection to do things like plotting; so you are not alone in doing this kind of thing. There is something called AnalysisRCP which flattens ndarray and unflattens in Java as IDataset. This was commissioned by Diamond Light Source as a work package with Kichwa Coders and Jonah from Kichwa is one of the project leads of January...

To expand on what Peter said, the flattening service, which is the heart of integrating ndarry with IDataset is currently here:
https://github.com/DawnScience/scisoft-core/tree/master/uk.ac.diamond.scisoft.analysis.xmlrpc

We have spoken often and done little (AFAIK) about making a separately compiling/building/testing github project for this work. However my current understanding is that this code needs to be reworked to make it usable by any project. I did some work to put it into its current low dependency form but it would need more I think to be useful to you. There are still a few dependencies there: 
https://github.com/DawnScience/scisoft-core/blob/master/uk.ac.diamond.scisoft.analysis.xmlrpc/META-INF/MANIFEST.MF

Eclipse Triquetrum also looked at some of this stuff and may have another version. I have copied in Erwin and Christopher just in case.

In terms of releases, the plan is to follow eclipse release trains but as Peter indicates there may be need for more. This is because Diamond Light Source funds changes in it that sometimes require something at short notice (well it could be anyone doing the funding but DLS tends to so far...)

Cheerio,

Matt

-----Original Message-----
From: january-dev-bounces@xxxxxxxxxxx [mailto:january-dev-bounces@xxxxxxxxxxx] On Behalf Of Peter.Chang@xxxxxxxxxxxxx
Sent: 30 January 2017 09:28
To: january-dev@xxxxxxxxxxx
Subject: Re: [january-dev] IDataset <-> python


Hi Scott,

We do have and use two mechanisms for communicating with Python: xml-rpc and Py4J. At the moment, they are part of Dawn Science (https://github.com/DawnScience/scisoft-core) and are based on saving the data to a filesystem that is accessible to both sides.

The next version of January will be a major one though there has not been any discussion I am looking at a March release.

Best regards,
 Peter



 

-----Original Message-----
From: january-dev-bounces@xxxxxxxxxxx [mailto:january-dev-bounces@xxxxxxxxxxx] On Behalf Of Scott Lewis
Sent: 27 January 2017 19:46
To: january-dev@xxxxxxxxxxx
Subject: [january-dev] IDataset <-> python

Hi Folks,

Is there any existing work on going between an IDataset instance (in 
java) and a Dataset instance in Python using Py4j?   We are using 
January to build an IoT data analytics framework that is built on 
java/OSGi/kura/our stuff.   It includes a way to dynamically 
create/add/remove osgi services that pass IDataset instances between components to perform various analytics.

One thing that we might be interested in doing is having these osgi services implemented via Python...which would necessitate exchanging IDataset <-> Python dataset instances.

Another question:   When is the next expected release of January?    
Will it be a major, minor, maintenance?

Thanksinadvance for any information.

Scott


_______________________________________________
january-dev mailing list
january-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/january-dev

--
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________
january-dev mailing list
january-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/january-dev
_______________________________________________
january-dev mailing list
january-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/january-dev