SlideShare a Scribd company logo

Eclipse Con Europe 2014 How to use DAWN Science Project

•Download as PPTX, PDF•
•
M
Matthew Gerring

This document summarizes the DawnScience Eclipse project, which is an open source not-for-profit project on GitHub. It aims to provide APIs and reference implementations for loading, describing, slicing, transforming, and plotting multidimensional scientific data. Phase 1 from 2014-2015 defined long-term APIs and a reference implementation for HDF5 loading, data description, plotting, and slicing interfaces. Phase 2 in 2016 will release concrete implementations. The project utilizes Eclipse technologies and collaborates with scientific facilities.

Related slideshows

PyTorch for Deep Learning Practitioners
PyTorch for Deep Learning PractitionersPyTorch for Deep Learning Practitioners
PyTorch for Deep Learning Practitioners
 
Networkx tutorial
Networkx tutorialNetworkx tutorial
Networkx tutorial
 
Hadoop map reduce concepts
Hadoop map reduce conceptsHadoop map reduce concepts
Hadoop map reduce concepts
 
1 of 30
-An open source not for profit project -On GitHub ‘DawnScience’ - Diamond Light Source Ltd. and the ESRF are largely publically funded research facilities
Who – Diamond Light Source – ORNL – IBM – IFP Energies Nouvelles – TECH'Advantage – MARINTEK / Itema – Clemson University – Lablicate – Uppsala University – Kichwa Coders – The Facility for Rare Isotope Beams at Michigan State University • Charter including a vision for the future • Contributions from DAWN and NiCE being made – we will look at DAWNSci • Presentations in the US (2012) and last year, Germany (2013) • Eclipse Foundation investing in the group • Git / Jenkins / Marketing Collaborations Science Working Group science.eclipse.org
DAWNSCI Eclipse Project Phased Delivery • Phase 1 2014/2015 - Definition of long term APIs and Reference Implementation – HDF5 Loading / Saving API – Description of data and some mathematical operations – Plotting interface (only) based on data description – Slicing interface description – Examples of how to use API and reference implementation (binary) • Phase 2 2016 – Release of concrete implementation(s) – TBD …What we said/promised last time… So this presentation will show you how to make use of that so far released...
Diamond Light Source - RCP Based Projects • Generic Data Acquisition • Client in RCP server CORBA • Data Analysis Workbench • That you already know and love Actively (each ~10s developers) developed at DLS • Control Systems Studio • Community active on many sites • DESY, NSLS2, KEK, ITER DLS Information
Diamond Light Source - Data Analysis Group Team Mostly Doing DAWN DAWN - RCP front end for Data Analysis. Support of third party code for Data Analysis. Help/support with python scripts. DAWN includes Numpy. DIALS Working in the crystallography area producing new software and running automatic pipelines (cluster) for processing data produced during data collections. Tomography Working with scientists to support visualization software. Responsible for reconstruction pipelines running on GPU clusters Web ISPyB, ICAT, SyncLink and SyncWeb productions allowing users to interact with experimental data remotely. Spectrosopy / 1D tools Developing tools for 1D analysis like peak fitting. Python interactivity a key requirement. DLS Information
The DAWN Product DAWN Overview See previous presentations at Eclipsecon... ...Well ok then here is a little bit
Eclipse Con Europe 2014 How to use DAWN Science Project
– For images • Line, Box, Sector integration • Diffraction image interpretation, line profile for ‘D-spacing’ • Color mapping / Histogramming • Pixel Information and region control – For XY Graphs • Peak Fitting and Line Fitting • Derivative and other functions, including user defined • Scientific tools – XAFS Analysis Tool – SAXS – Use of eclipse architecture, extension points and pages inside PageBookView. DAWN Lots of Visual Tools DAWN Overview
Demonstration – Visual Tools Example showing various visual tools DAWN Overview
• Cutting through N-dimensional data – With an XY plot – As an image – As a 3D iso-surface – Hyper 3D • Important to run everything concurrently – Use of Jobs – Use of ordinary threads – Use of blocking queues Slicing data DAWN Overview
Demonstration – Slicing and dicing Example opening a tomography file and slicing it DAWN Overview
Introducing The DAWNSci Eclipse Project... • Data • Loading (inc. HDF5) • Description • Slicing • Transformation • Mathematic • Metadata • User Interface • Plotting 1/2/3 D • Slicing nD Phase 1 is API with a reference implementation via a p2 site http://www.dawnsci.org/eclipse/getting-started-with-dawnsci
Data in DAWNSci Example : Loading and Manipulating Data This example • Columns of ASCII data • Separated by tabs • Read as 1D arrays Other format with a loader: srs, dat, flt, gff, mca, csv, xy, xye, txt, tif, tiff, cbf, img, ciff, mccd, edf, pgm, cor, bruker, jpg, jpeg, png, f2d, msk, mib, mar3450, pck3450, raw, mrc, gz, bz2, zip, h5, hd5, hdf5, nxs, nexus, hdf, mat Extension point to add more...
DAWNSCI - ILoaderService Purpose – load any numerical file format using a low dependency service. Keeps code modular and it is easy to use. How – Loaders can be contributed by extension point for custom file formats. Many loaders come pre-registered with the service. Example - Reading and slicing data ILoaderService service = (ILoaderService)Activator.getService(ILoaderService.class IDataHolder holder = service.getData(<file path>, ...); IDataset full = holder.getDataset(“Column_1”); // full is the full array of data loaded into memory represented by // an object called IDataset. Mathematics and plotting can be done // using this object as if it were an array. Show LoaderExamples... Data in DAWNSci
DAWNSCI - ILazyDataset and IDataset Purpose – represent data without physically loading it into memory, delaying this action until it is required. How – The LazyDatasetImpl might use HDF5 or directories of files to represent stacks of data. Example - Slicing data ILazyDataset lz = holder.getLazyDataset(“Big”); // Not all loaded, too big System.out.println(“Your data shape is ”+ lz.getShape()); IDataset slice = lz.getSlice(...); // Now it’s loaded IDataset all = holder.getDataset(“myData”); // Load everything!! IDataset view = slice.getSliceView(...); // Data without new array // Real slices of the data can be taken from the ILazyDataset and // expressions written which result value is only evaluated when the // slice is visualized. Show NumpyExamples and LazyExamples Data in DAWNSci
DAWNSCI - IMetaData Purpose – To hold metadata about the conditions in which the data were taken How – Metadata holds scalars and axes. The axes will be sliced with the data automatically. Example - Metadata IMetaData meta = holder.getMetadata(); System.out.println(“The temperature was ”+ meta.getMetaValue(“T”); IDataset data = holder.getDataset(“myData”); meta = holder.getMetadata(); System.out.println(“The temperature was ”+ meta.getMetaValue(“T”); // The meta data can exist for the whole file or for individual datasets. Data in DAWNSci
DAWNSCI - Python/Numpy Purpose – to make available data in the GUI to python scripting How – A flattening service transfers data to and from python. IDataset appears as a numpy array in a seemless way. Data in DAWNSci
Demonstration – Examples Plotting System Plotting in DAWNSci
DAWNSCI - IPlottingService Purpose – to provide access to plotting without making references to the individual plotting technology in your code. How – Different plotting systems can be retrieved from the plotting service and are contributed by extension point. DAWNSCI phase 2 will contribute a draw2d / JavaFX based plotting system impl. Example – Getting a plotting system IPlottingService ps = (IPlottingService)Activator.getService(IPlottingService.clas IPlottingSystem sys = ps.createPlottingSystem(); // This example gets the users current plotting system choice. // There are preferences set up by DAWNSci based on each plotting // system available / contributed. createPlottingSystem() instatiates // and returns the current users preference. Plotting in DAWNSci
DAWNSCI - IPlottingSystem Purpose – Ability to plot data in 1D, 2D and 3D. Ability to manage selection regions, axes, annotations, printing etc How – The interface IPlottingSystem is implemented similar to a view part in e3 with a createPlotPart(...) method to add plotting to any SWT composite. Example – Making a plotting system system.createPlotPart(parent, "Image Example", getViewSite().getActionBars(), PlotType.IMAGE, this); IImageTrace image = system.createPlot2D(slice, null, new NullProgressMonitor()); // Plotting system here plots the slice which we previously retrieved. Plotting in DAWNSci
DAWNSCI - IRegion and IRegionSystem Purpose – Allow selections on the data for algorithms and plotting to use, for instance sectors. How – The implementation of IPlottingSystem is also IRegionSystem. This allows creation of regions and their configuration/listening. Implemented using draw2d. final IRegion region = sys.createRegion(“myRegion”, RegionType.BOX); region.setRegionColor(ColorConstants.orange); region.setROI(new RectangularROI(0,0,100,100); region.setMobile(false); region.setUserObject(FittedFunction.class); sys.addRegion(region); region.addROIListener(new IROIListener() { // methods for listening to the user moving regions. } Plotting in DAWNSci
Demonstration – Examples Tools Look at the plotting examples Plotting in DAWNSci
DAWNSCI - IToolSystem Purpose – Encapsulate a set of operations for mathematical interaction with plotting. How – Generally when using AbstractPlottingSystem a tool system comes for free. Simply return it with the getAdapter(...) method to Example – Connecting a tool system to the outside world // In your parts public Object getAdapter(final Class clazz) { if (clazz == IToolPageSystem.class) { return sys.getAdapter(clazz); } } // Now the external world can see the tool system and tool pages // registered by extension point will be available. Plotting in DAWNSci
DAWNSCI - IToolPage Purpose – Use of IPageView pages to provide helper user interface for a tool linking to a plotting system. How – Declare with extension point and plot dimensionality and page will appear on plotting systems with registered IToolSystems. Plotting in DAWNSci
Eclipse Con Europe 2014 How to use DAWN Science Project
public class ExampleTool extends AbstractToolPage { private Composite control; public ExampleTool() { // Create your listeners to the main plotting // Perhaps create a plotting system here from the PlottingFactory which is your side plot. } @Override public ToolPageRole getToolPageRole() { return ToolPageRole.ROLE_1D; } @Override public void createControl(Composite parent) { this.control = new Composite(parent, SWT.NONE); control.setBackground(Display.getDefault().getSystemColor(SWT.COLOR_WHITE)); control.setLayout(new GridLayout(1, false)); GridUtils.removeMargins(control); // User interface shown in a page to the side of the plot. // ... For instance: a side plot, a Viewer part } @Override public Control getControl() { return control; } @Override public void setFocus() { // If you have a table or tree in your tool, set focus here. } @Override public void activate() { super.activate(); // Now add any listeners to the plotting providing getPlottingSystem()!=null } @Override public void deactivate() { super.deactivate(); // Now remove any listeners to the plotting providing getPlottingSystem()!=null } @Override public void dispose() { super.dispose(); // Anything to kill off? This page is part of a view which is now disposed and will not be used again. } }
DAWNSCI - ISliceSystem Purpose – Provide a standardized user interface design to slicing multi-dimenional data. How – Contributed by extension point, when multi-dimensional data is opened the current active slice system is used to slice the data. Example – Making a slicesystem public class SliceSystemImpl extends AbstractSliceSystem { public Control createPartControl(Composite parent){ //... } // Rather a lot of other methods to go into here currently. Plotting in DAWNSci
DAWNSCI - ISlicingTool Purpose – An extension point registered tool for slicing multi- dimensional data How – Generally extend AbstractSliceTool and then implement code for doing a slice you need. Example public class ExampleSlicingTool extends AbstractSlicingTool { public void militarize() { //... User chosen the tool } public void demilitarize() { //... } } // Several examples in the code base Plotting in DAWNSci
Conclusion DAWNSCI - The Future  More collaboration! More pull requests!  Github move or mirror dawn-eclipse to eclipse/dawnsci  Plan for phase 2 – the reference implementation release  Cannot do all at once as ~1 million lines  Release Isosurface implementation in 2015 (backed by JavaFX in Java8)  Prototype already created summer 2014 and is functional - if you want to try it, get in touch   A processing pipeline implementation in 2015.  To be used for data reduction at Diamond but in principle any mathematical pipeline can be created and run over stacks
Conclusion DAWNSCI - Conclusion  Thanks for listening, more questions...  Email us on science-iwg@eclipse.org  science.eclipse.com is available for new members join now, see Andrew Ross...  Beverage later?

More Related Content

Eclipse Con Europe 2014 How to use DAWN Science Project

  • 1. -An open source not for profit project -On GitHub ‘DawnScience’ - Diamond Light Source Ltd. and the ESRF are largely publically funded research facilities
  • 2. Who – Diamond Light Source – ORNL – IBM – IFP Energies Nouvelles – TECH'Advantage – MARINTEK / Itema – Clemson University – Lablicate – Uppsala University – Kichwa Coders – The Facility for Rare Isotope Beams at Michigan State University • Charter including a vision for the future • Contributions from DAWN and NiCE being made – we will look at DAWNSci • Presentations in the US (2012) and last year, Germany (2013) • Eclipse Foundation investing in the group • Git / Jenkins / Marketing Collaborations Science Working Group science.eclipse.org
  • 3. DAWNSCI Eclipse Project Phased Delivery • Phase 1 2014/2015 - Definition of long term APIs and Reference Implementation – HDF5 Loading / Saving API – Description of data and some mathematical operations – Plotting interface (only) based on data description – Slicing interface description – Examples of how to use API and reference implementation (binary) • Phase 2 2016 – Release of concrete implementation(s) – TBD …What we said/promised last time… So this presentation will show you how to make use of that so far released...
  • 4. Diamond Light Source - RCP Based Projects • Generic Data Acquisition • Client in RCP server CORBA • Data Analysis Workbench • That you already know and love Actively (each ~10s developers) developed at DLS • Control Systems Studio • Community active on many sites • DESY, NSLS2, KEK, ITER DLS Information
  • 5. Diamond Light Source - Data Analysis Group Team Mostly Doing DAWN DAWN - RCP front end for Data Analysis. Support of third party code for Data Analysis. Help/support with python scripts. DAWN includes Numpy. DIALS Working in the crystallography area producing new software and running automatic pipelines (cluster) for processing data produced during data collections. Tomography Working with scientists to support visualization software. Responsible for reconstruction pipelines running on GPU clusters Web ISPyB, ICAT, SyncLink and SyncWeb productions allowing users to interact with experimental data remotely. Spectrosopy / 1D tools Developing tools for 1D analysis like peak fitting. Python interactivity a key requirement. DLS Information
  • 6. The DAWN Product DAWN Overview See previous presentations at Eclipsecon... ...Well ok then here is a little bit
  • 8. – For images • Line, Box, Sector integration • Diffraction image interpretation, line profile for ‘D-spacing’ • Color mapping / Histogramming • Pixel Information and region control – For XY Graphs • Peak Fitting and Line Fitting • Derivative and other functions, including user defined • Scientific tools – XAFS Analysis Tool – SAXS – Use of eclipse architecture, extension points and pages inside PageBookView. DAWN Lots of Visual Tools DAWN Overview
  • 9. Demonstration – Visual Tools Example showing various visual tools DAWN Overview
  • 10. • Cutting through N-dimensional data – With an XY plot – As an image – As a 3D iso-surface – Hyper 3D • Important to run everything concurrently – Use of Jobs – Use of ordinary threads – Use of blocking queues Slicing data DAWN Overview
  • 11. Demonstration – Slicing and dicing Example opening a tomography file and slicing it DAWN Overview
  • 12. Introducing The DAWNSci Eclipse Project... • Data • Loading (inc. HDF5) • Description • Slicing • Transformation • Mathematic • Metadata • User Interface • Plotting 1/2/3 D • Slicing nD Phase 1 is API with a reference implementation via a p2 site http://www.dawnsci.org/eclipse/getting-started-with-dawnsci
  • 13. Data in DAWNSci Example : Loading and Manipulating Data This example • Columns of ASCII data • Separated by tabs • Read as 1D arrays Other format with a loader: srs, dat, flt, gff, mca, csv, xy, xye, txt, tif, tiff, cbf, img, ciff, mccd, edf, pgm, cor, bruker, jpg, jpeg, png, f2d, msk, mib, mar3450, pck3450, raw, mrc, gz, bz2, zip, h5, hd5, hdf5, nxs, nexus, hdf, mat Extension point to add more...
  • 14. DAWNSCI - ILoaderService Purpose – load any numerical file format using a low dependency service. Keeps code modular and it is easy to use. How – Loaders can be contributed by extension point for custom file formats. Many loaders come pre-registered with the service. Example - Reading and slicing data ILoaderService service = (ILoaderService)Activator.getService(ILoaderService.class IDataHolder holder = service.getData(<file path>, ...); IDataset full = holder.getDataset(“Column_1”); // full is the full array of data loaded into memory represented by // an object called IDataset. Mathematics and plotting can be done // using this object as if it were an array. Show LoaderExamples... Data in DAWNSci
  • 15. DAWNSCI - ILazyDataset and IDataset Purpose – represent data without physically loading it into memory, delaying this action until it is required. How – The LazyDatasetImpl might use HDF5 or directories of files to represent stacks of data. Example - Slicing data ILazyDataset lz = holder.getLazyDataset(“Big”); // Not all loaded, too big System.out.println(“Your data shape is ”+ lz.getShape()); IDataset slice = lz.getSlice(...); // Now it’s loaded IDataset all = holder.getDataset(“myData”); // Load everything!! IDataset view = slice.getSliceView(...); // Data without new array // Real slices of the data can be taken from the ILazyDataset and // expressions written which result value is only evaluated when the // slice is visualized. Show NumpyExamples and LazyExamples Data in DAWNSci
  • 16. DAWNSCI - IMetaData Purpose – To hold metadata about the conditions in which the data were taken How – Metadata holds scalars and axes. The axes will be sliced with the data automatically. Example - Metadata IMetaData meta = holder.getMetadata(); System.out.println(“The temperature was ”+ meta.getMetaValue(“T”); IDataset data = holder.getDataset(“myData”); meta = holder.getMetadata(); System.out.println(“The temperature was ”+ meta.getMetaValue(“T”); // The meta data can exist for the whole file or for individual datasets. Data in DAWNSci
  • 17. DAWNSCI - Python/Numpy Purpose – to make available data in the GUI to python scripting How – A flattening service transfers data to and from python. IDataset appears as a numpy array in a seemless way. Data in DAWNSci
  • 18. Demonstration – Examples Plotting System Plotting in DAWNSci
  • 19. DAWNSCI - IPlottingService Purpose – to provide access to plotting without making references to the individual plotting technology in your code. How – Different plotting systems can be retrieved from the plotting service and are contributed by extension point. DAWNSCI phase 2 will contribute a draw2d / JavaFX based plotting system impl. Example – Getting a plotting system IPlottingService ps = (IPlottingService)Activator.getService(IPlottingService.clas IPlottingSystem sys = ps.createPlottingSystem(); // This example gets the users current plotting system choice. // There are preferences set up by DAWNSci based on each plotting // system available / contributed. createPlottingSystem() instatiates // and returns the current users preference. Plotting in DAWNSci
  • 20. DAWNSCI - IPlottingSystem Purpose – Ability to plot data in 1D, 2D and 3D. Ability to manage selection regions, axes, annotations, printing etc How – The interface IPlottingSystem is implemented similar to a view part in e3 with a createPlotPart(...) method to add plotting to any SWT composite. Example – Making a plotting system system.createPlotPart(parent, "Image Example", getViewSite().getActionBars(), PlotType.IMAGE, this); IImageTrace image = system.createPlot2D(slice, null, new NullProgressMonitor()); // Plotting system here plots the slice which we previously retrieved. Plotting in DAWNSci
  • 21. DAWNSCI - IRegion and IRegionSystem Purpose – Allow selections on the data for algorithms and plotting to use, for instance sectors. How – The implementation of IPlottingSystem is also IRegionSystem. This allows creation of regions and their configuration/listening. Implemented using draw2d. final IRegion region = sys.createRegion(“myRegion”, RegionType.BOX); region.setRegionColor(ColorConstants.orange); region.setROI(new RectangularROI(0,0,100,100); region.setMobile(false); region.setUserObject(FittedFunction.class); sys.addRegion(region); region.addROIListener(new IROIListener() { // methods for listening to the user moving regions. } Plotting in DAWNSci
  • 22. Demonstration – Examples Tools Look at the plotting examples Plotting in DAWNSci
  • 23. DAWNSCI - IToolSystem Purpose – Encapsulate a set of operations for mathematical interaction with plotting. How – Generally when using AbstractPlottingSystem a tool system comes for free. Simply return it with the getAdapter(...) method to Example – Connecting a tool system to the outside world // In your parts public Object getAdapter(final Class clazz) { if (clazz == IToolPageSystem.class) { return sys.getAdapter(clazz); } } // Now the external world can see the tool system and tool pages // registered by extension point will be available. Plotting in DAWNSci
  • 24. DAWNSCI - IToolPage Purpose – Use of IPageView pages to provide helper user interface for a tool linking to a plotting system. How – Declare with extension point and plot dimensionality and page will appear on plotting systems with registered IToolSystems. Plotting in DAWNSci
  • 26. public class ExampleTool extends AbstractToolPage { private Composite control; public ExampleTool() { // Create your listeners to the main plotting // Perhaps create a plotting system here from the PlottingFactory which is your side plot. } @Override public ToolPageRole getToolPageRole() { return ToolPageRole.ROLE_1D; } @Override public void createControl(Composite parent) { this.control = new Composite(parent, SWT.NONE); control.setBackground(Display.getDefault().getSystemColor(SWT.COLOR_WHITE)); control.setLayout(new GridLayout(1, false)); GridUtils.removeMargins(control); // User interface shown in a page to the side of the plot. // ... For instance: a side plot, a Viewer part } @Override public Control getControl() { return control; } @Override public void setFocus() { // If you have a table or tree in your tool, set focus here. } @Override public void activate() { super.activate(); // Now add any listeners to the plotting providing getPlottingSystem()!=null } @Override public void deactivate() { super.deactivate(); // Now remove any listeners to the plotting providing getPlottingSystem()!=null } @Override public void dispose() { super.dispose(); // Anything to kill off? This page is part of a view which is now disposed and will not be used again. } }
  • 27. DAWNSCI - ISliceSystem Purpose – Provide a standardized user interface design to slicing multi-dimenional data. How – Contributed by extension point, when multi-dimensional data is opened the current active slice system is used to slice the data. Example – Making a slicesystem public class SliceSystemImpl extends AbstractSliceSystem { public Control createPartControl(Composite parent){ //... } // Rather a lot of other methods to go into here currently. Plotting in DAWNSci
  • 28. DAWNSCI - ISlicingTool Purpose – An extension point registered tool for slicing multi- dimensional data How – Generally extend AbstractSliceTool and then implement code for doing a slice you need. Example public class ExampleSlicingTool extends AbstractSlicingTool { public void militarize() { //... User chosen the tool } public void demilitarize() { //... } } // Several examples in the code base Plotting in DAWNSci
  • 29. Conclusion DAWNSCI - The Future  More collaboration! More pull requests!  Github move or mirror dawn-eclipse to eclipse/dawnsci  Plan for phase 2 – the reference implementation release  Cannot do all at once as ~1 million lines  Release Isosurface implementation in 2015 (backed by JavaFX in Java8)  Prototype already created summer 2014 and is functional - if you want to try it, get in touch   A processing pipeline implementation in 2015.  To be used for data reduction at Diamond but in principle any mathematical pipeline can be created and run over stacks
  • 30. Conclusion DAWNSCI - Conclusion  Thanks for listening, more questions...  Email us on science-iwg@eclipse.org  science.eclipse.com is available for new members join now, see Andrew Ross...  Beverage later?