Εντυπώσεις από το EGI User Forum 2011

Την προηγούμενη εβδομάδα διεξήχθει στο Vilnius της Λιθουανίας το EGI User Forum παράλληλα με το EMI Technical Forum. Το EGI (European Grid Infrastructure) είναι η αρχή διαχείρισης της πανευρωπαϊκής υπολογιστικής και αποθηκευτικής υποδομής ενώ το EMI (European Middleware Initiave) είναι η αρχή ανάπτυξης του μεσολογισμικού για τη χρήση κατανεμημένων υποδομών τύπου Grid, Cloud και HPC στην Ευρώπη. Το συνέδριο έδωσε τη δυνατότητα σε Ευρωπαίους ερευνητές να μοιραστούν την εμπειρία τους από τη χρήση κατανεμημένων υποδομών, να δημοσιεύσουν τα αποτελέσματα της έρευνάς τους και να ενημερωθούν για τις τελευταίες εξελίξεις σε θέματα υπολογιστικών υποδομών, κυρίως τύπων Grid και Cloud, και του ενδιάμεσου μεσολογισμικού.

Σε σχέση με το τελευταίο η κοινότητα αναμένει με ανυπομονησία τo EMI-1.0 release που θα περιλαμβάνει υπάρχοντα και νέα εργαλεία για την υποβολή και διαχείριση εργασιών και για τη διαχείριση δεδομένων σε κατανεμημένους αποθηκευτικούς πόρους. Η φιλοδοξία και η προοπτική που δίνει το ίδιο το EMI είναι ότι η αναμενόμενη έκδοση 1.0 θα μπορεί να εφαρμοστεί παράλληλα σε Grid, Cloud και HPC υποδομές έτσι ώστε οι χρήστες να έχουν τη δυνατότητα ανάλογα με τα δικαιώματα πρόσβασης που διαθέτουν στις διάφορες υποδομές απο ένα εννιαίο περιβάλλον να μπορούν να διαχειρίζονται εργασίες και δεδομένα ανεξάρτητα από το άν αυτά βρίσκονται σε Grid, Cloud ή πιο “κλασσικές” HPC πλατφόρμες.

Η ομάδα του Κέντρου Επιστημονικών Υπολογιστικών Υπηρεσιών παρουσίασε στα πλαίσια του EGI User Forum το morph, ένα νέο εργαλείο που αναπτύσσει για την υποβολή παραμετρικών εργασιών. Χρησιμοποιώντας το morph έναντι των εργαλείων του gLite για την υποβολή παραμετρικών εργασιών οι επι μέρους εργασίες που αποτυνχάνουν επαναυποβάλλονται αυτομάτως προς εκτέλεση χωρίς ο χρήστης να χρειάζεται να τις ξεχωρίσει μία προς μία και να τις υποβάλλει εκ νέου. Η χρήση της υπηρεσίας morph για την υποβολή παραμετρικών εργασιών αποτελεί λοιπόν έναν εγγυημένο τρόπο ώστε όλες οι επιμέρους παραμετρικές εργασίες να ολοκληρωθούν επιτυχώς. Περισσότερες λεπτομέρειες σχετικά με την υπηρεσία, που βρίσκεται στο στάδιο της ανάπτυξης, μπορούν οι ενδιαφερόμενοι να βρουν εδώ.

Νεότερα σχετικά με το μηχανισμό MPI-Start

Η πολυπληθία εκδόσεων της βιβλιοθήκης MPI, καθώς επίσης και οι διάφορες υλοποίησεις που λειτουργούν σε υπολογιστικές συστοιχίες ή υπερυπολογιστές στην Ευρώπη έχει προκαλέσει έναν εύλογο προβληματισμό στην ομάδα ανάπτυξης μεσολογισμικού του EMI (European Middleware Initiative) ως προς το ποιά υλοποίηση είναι πιο απλή για τους χρήστες, στην εκτέλεση παράλληλων εφαρμογών βασισμένες στο πρωτόκολλο MPI.

Στα πλαίσια του EGI User Forum 2011 η ομάδα εργασίας εργαλείων παράλληλου προγραμματισμού του EMI παρουσίασε την προοπτική της αποδεχόμενη το μηχανισμό MPI-Start, που ήδη υλοποιείται και χρησιμοποιείται από το μεσολογισμικό gLite ως ένα εννιαίο εργαλείο για όλους τους Ευρωπαίους χρήστες των μεσολογισμικών ARC, gLite και UNICORE. Πρακτικά, αυτό σημαίνει ότι χωρίς σημαντικές αλλαγές στον τρόπο υποβολής των MPI εργασιών, οι Έλληνες χρήστες πλεγματικών υποδομών (όπου χρησιμοποιείται το μεσολογισμικό gLite) με ελάχιστες ή και καθόλου αλλαγές στα script υποβολής θα μπορούν να εκτελούν με τον ίδιο μηχανισμό τις παράλληλες εφαρμογές τους και σε επιπλέον υπολογιστικούς πόρους (είτε τύπου Grid είτε τύπου HPC) μέσω των μεσολογισμικών ARC και UNICORE. Μάλιστα η ομάδα εργασίας του EMI πρόκειται άμεσα να επεκτείνει το μηχανισμό MPI-Start ώστε να συμπεριλαμβάνονται σε αυτόν οι δυνατότητες υποβολής OpenMP ή υβριδικών MPI+OpenMP εργασιών.

Περισσότερες πληροφορίες για το περιβάλλον υποβολής παράλληλων εργασιών MPI με το μηχανισμό MPI-Start μπορείτε να δείτε παρακολουθώντας το σχετικό screencast.

Technical Forum 2011 to be held in September in Lyon

As a successful User Forum draws to a close, the next meeting is already becoming a reality.

Steven Newhouse, Director of EGI.eu, opened the call for participation for the EGI 2011 Technical Forum in Lyon (19-23 September 2011), during the closing plenary session of the User Forum in Vilnius. The Technical Forum will be organised by EGI.eu and the Institut des Grilles du CNRS, the French National Grid Initiative.

While the User Forum held this week focused on user communities, with an emphasis on new tools, new applications and user support services, the Technical Forum will review the community's plans and progress towards the adoption of a federated virtualised infrastructure for European researchers.

The Technical Forum, to be co-located with the OGF33 meeting and Grid 2011, ‘will be a great opportunity to learn more about the state of the art of e-Infrastructure in Europe, including standards development and the community’s long-term plans for offering cloud interfaces to our users,’ says Newhouse.

The call for participation is now officially open. The Technical Forum welcomes submissions on a wide variety of EGI-related topics.

The event will also feature an exhibition area open for organisations active within the EGI community (projects, NGIs, companies) to present their work. In addition, end-users, application and tool developers, operations staff and others are encouraged to participate by submitting abstracts for technical sessions, posters, demonstrations, training sessions and workshops.
 

Technical Forum 2011 to be held in September in Lyon

As a successful User Forum draws to a close, the next meeting is already becoming a reality.

Steven Newhouse, Director of EGI.eu, opened the call for participation for the EGI 2011 Technical Forum in Lyon (19-23 September 2011), during the closing plenary session of the User Forum in Vilnius. The Technical Forum will be organised by EGI.eu and the Institut des Grilles du CNRS, the French National Grid Initiative.

While the User Forum held this week focused on user communities, with an emphasis on new tools, new applications and user support services, the Technical Forum will review the community's plans and progress towards the adoption of a federated virtualised infrastructure for European researchers.

The Technical Forum, to be co-located with the OGF33 meeting and Grid 2011, ‘will be a great opportunity to learn more about the state of the art of e-Infrastructure in Europe, including standards development and the community’s long-term plans for offering cloud interfaces to our users,’ says Newhouse.

The call for participation is now officially open. The Technical Forum welcomes submissions on a wide variety of EGI-related topics.

The event will also feature an exhibition area open for organisations active within the EGI community (projects, NGIs, companies) to present their work. In addition, end-users, application and tool developers, operations staff and others are encouraged to participate by submitting abstracts for technical sessions, posters, demonstrations, training sessions and workshops.
 

User Forum kicks off in Vilnius

The EGI User Forum has started yesterday in Vilnius, Lithuania.

If you want to know what is going on, the GridCast blog is following the event with an international team of bloggers and you can see pictures of the event and Vilnius in Flickr.

First day highlights:

  • Steven Newhouse, EGI.eu's director, signed four Memorandums of Understanding with the WeNMR project, SAGA, the GISELA initiative and StratusLab. More information and pictures coming soon!

  • Rimantas Žylius, Lithuania's Minister of Economy, spoke at the opening plenary session and found time to have a chat with the GridCast team (video).

  • Last year in the 2010 Technical Forum, Kostas Glinos, Head of the European Commission's GÉANT & e-Infrastructures Unit, presented Steven Newhouse with the EGI-InSPIRE project's grant agreement. Yesterday, Steven thanked Kostas with an overview of the year's project achievements and an EGI pen!

User Forum kicks off in Vilnius

The EGI User Forum has started yesterday in Vilnius, Lithuania.

If you want to know what is going on, the GridCast blog is following the event with an international team of bloggers and you can see pictures of the event and Vilnius in Flickr.

First day highlights:

  • Steven Newhouse, EGI.eu's director, signed four Memorandums of Understanding with the WeNMR project, SAGA, the GISELA initiative and StratusLab. More information and pictures coming soon!

  • Rimantas Žylius, Lithuania's Minister of Economy, spoke at the opening plenary session and found time to have a chat with the GridCast team (video).

  • Last year in the 2010 Technical Forum, Kostas Glinos, Head of the European Commission's GÉANT & e-Infrastructures Unit, presented Steven Newhouse with the EGI-InSPIRE project's grant agreement. Yesterday, Steven thanked Kostas with an overview of the year's project achievements and an EGI pen!

User training workshops: Heavy User Communities leading the way

The EGI User Forum 2011 in Vilnius has a packed programme offering a diverse range of workshops, demonstrations, presentations and tutorials with a focus on end-user training. With more than twenty three workshops and tutorials on offer over four days, the EGI-InSPIRE Heavy User Communities (HUCs) shall be playing their part.

Here is an overview of the HUC training workshops:

SHIWA platform

The Life Sciences community will introduce the SHIWA platform, a multi-system workflow execution platform and interoperability solution, supporting Askalon, MOTEUR, P-GRADE and Triana workflows. Workflow environments shield the end-users from the details of the grid infrastructure. The examples used at this training event are taken from the Life Sciences domain, but the tools presented here have much wider applications and should be of interest to all user communities.

  • Location: Theta; Time 11-Apr-2011 @14:00; Duration 03h30'

Using a StratusLab cloud infrastructure

StratusLab open-source cloud distribution allows resource centres to expose their computing resources as an ‘Infrastructure as a Service’ (IaaS) type cloud.

This tutorial presents the main StratusLab features and how they can be used by system administrators and scientists alike. They will learn how StratusLab-based infrastructures can be integrated with the EGI, and how the cloud services complement grid services. Practical exercises will teach the participants how to launch virtual machines, customise their computing environment, share those environments, manage virtual disks, and define complete services.

Participants will be provided credentials to access a StratusLab cloud infrastructure and must bring a laptop with python (2.6+), java (1.6+), and an ssh client installed.

  • Location: Gamma; Time 12-Apr-2011 @11:00; Duration 01h30'

Earth Science Data Processing Tools and Applications

The Earth Science community has a rich and extensive repository of data stored outside EGI and access to this data during job execution is a mandatory requirement. The Earth Sciences community is also interested in using the OPeNDAP protocol and Hyrax Data Server. Hyrax offers many features that go beyond high-performance access to distributed datasets, such as an extensible component-based architecture, multiple data representations, static or dynamic THREDDS catalogues. Due to the many different technologies, data-centres, standards and pseudo-standards, however, it seems that no general solution can be found. This talk should be of interest to anyone interested in these or similar issues.

  • Location: Zeta; Time 12-Apr-2011 @16:00; Duration 00h30'

Shared Services Tools based on the Ganga job definition

Ganga is a user-targeted job management tool designed to provide a homogeneous environment for processing data on a variety of technology ‘back-ends’. Initially developed within the high-energy physics (HEP) domain, Ganga has been adopted by a wide variety of other user communities as their default analysis and task-management system. The modular nature of Ganga means that communities can easily, if desired, develop their own suite of tools independent of both the core code and those of other communities. This presentation will use case-studies to illustrate the ease with which non-LHC communities (for example medical research), have adopted Ganga as their chief job-submission tool.

  • Location: Zeta; Time: 13-Apr-2011 @12:00; Duration: 30'

Experiment Dashboard

The Experiment Dashboard applications for infrastructure monitoring are widely used by the LHC virtual organisations for the computing shifts and site commissioning activities. The LHC Experiment Dashboard consists of:

  • Site Usability Dashboard, which uses tailored VO tests within the existing Site Availability Monitoring (SAM) system;

  • Site Status Board, which allows VOs to construct customised monitoring views;

  • SiteView, a single point of entry for site administrators, to understand how their site is used by the LHC VOs and to detect potential problems and ensure effective site performance.

The Dashboard applications are essential LHC computing operations tools. However, they are generic and can be adapted for other community’s needs. The talk will give an overview the Dashboard applications, highlighting the possibility of exploiting these applications outside the LHC domain.

  • Location: Lambda; Time: 13-Apr-2011 @16:00; Duration: 01h30'

MPI Hands on training

The Message Passing Interface (MPI) standards and their implementations are currently the most prevalent frameworks on which parallel applications are built. A significant problem in exploiting MPI applications on the grid is the inherent nature of its heterogeneous environments. Different MPI implementations, system interconnects or job managers can be found at different resource centres. In order to run an application, the end-user needs some ‘a priori’ knowledge about the resources. MPI-Start offers a unique and stable interface to execute parallel applications at the gLite based grid sites. It aims to hide the differences and complexities of the heterogeneous systems that compose a grid infrastructure by providing a high-level abstract layer. This presentation introduces the basic concepts of MPI, together with a detailed description of MPI-Start and how to use it.

The tutorial is suitable to all users who wish to adapt their MPI-based applications for use on the grid using gLite.

  • Location: Iota; Time: 14-Apr-2011 @11:00; Duration: 01h30'

Kepler

Kepler is a free and open source workflow engine used extensively by the FUSION community. It is designed to help scientists and developers to easily create, execute, share and reuse their models across the scientific and engineering domains. In particular, Kepler includes components that integrate with different middleware stacks (e.g. gLite or UNICORE). Kepler workflows can be decomposed into smaller parts, thus allowing complex tasks to be divided into much simpler ones. This feature provides workflow designers with ability to build re-usable, modular sub-workflows. These can be saved and applied to other workflow. This introductory tutorial should be of interest to all users keen to explore Kepler's powerful capabilities. It will begin by showing how to use Kepler to build basic workflows; use relation paths and synchronisation; and to create control structures such as "if-else" and loops. Finally, job submission, monitoring, and data management shall complete the tutorial.

  • Location: Zeta; Time: 14-Apr-2011 @14:00; Duration: 03h30'


John Walsh, Grid-Ireland Operations Centre, Trinity College Dublin

User training workshops: Heavy User Communities leading the way

The EGI User Forum 2011 in Vilnius has a packed programme offering a diverse range of workshops, demonstrations, presentations and tutorials with a focus on end-user training. With more than twenty three workshops and tutorials on offer over four days, the EGI-InSPIRE Heavy User Communities (HUCs) shall be playing their part.

Here is an overview of the HUC training workshops:

SHIWA platform

The Life Sciences community will introduce the SHIWA platform, a multi-system workflow execution platform and interoperability solution, supporting Askalon, MOTEUR, P-GRADE and Triana workflows. Workflow environments shield the end-users from the details of the grid infrastructure. The examples used at this training event are taken from the Life Sciences domain, but the tools presented here have much wider applications and should be of interest to all user communities.

  • Location: Theta; Time 11-Apr-2011 @14:00; Duration 03h30'

Using a StratusLab cloud infrastructure

StratusLab open-source cloud distribution allows resource centres to expose their computing resources as an ‘Infrastructure as a Service’ (IaaS) type cloud.

This tutorial presents the main StratusLab features and how they can be used by system administrators and scientists alike. They will learn how StratusLab-based infrastructures can be integrated with the EGI, and how the cloud services complement grid services. Practical exercises will teach the participants how to launch virtual machines, customise their computing environment, share those environments, manage virtual disks, and define complete services.

Participants will be provided credentials to access a StratusLab cloud infrastructure and must bring a laptop with python (2.6+), java (1.6+), and an ssh client installed.

  • Location: Gamma; Time 12-Apr-2011 @11:00; Duration 01h30'

Earth Science Data Processing Tools and Applications

The Earth Science community has a rich and extensive repository of data stored outside EGI and access to this data during job execution is a mandatory requirement. The Earth Sciences community is also interested in using the OPeNDAP protocol and Hyrax Data Server. Hyrax offers many features that go beyond high-performance access to distributed datasets, such as an extensible component-based architecture, multiple data representations, static or dynamic THREDDS catalogues. Due to the many different technologies, data-centres, standards and pseudo-standards, however, it seems that no general solution can be found. This talk should be of interest to anyone interested in these or similar issues.

  • Location: Zeta; Time 12-Apr-2011 @16:00; Duration 00h30'

Shared Services Tools based on the Ganga job definition

Ganga is a user-targeted job management tool designed to provide a homogeneous environment for processing data on a variety of technology ‘back-ends’. Initially developed within the high-energy physics (HEP) domain, Ganga has been adopted by a wide variety of other user communities as their default analysis and task-management system. The modular nature of Ganga means that communities can easily, if desired, develop their own suite of tools independent of both the core code and those of other communities. This presentation will use case-studies to illustrate the ease with which non-LHC communities (for example medical research), have adopted Ganga as their chief job-submission tool.

  • Location: Zeta; Time: 13-Apr-2011 @12:00; Duration: 30'

Experiment Dashboard

The Experiment Dashboard applications for infrastructure monitoring are widely used by the LHC virtual organisations for the computing shifts and site commissioning activities. The LHC Experiment Dashboard consists of:

  • Site Usability Dashboard, which uses tailored VO tests within the existing Site Availability Monitoring (SAM) system;

  • Site Status Board, which allows VOs to construct customised monitoring views;

  • SiteView, a single point of entry for site administrators, to understand how their site is used by the LHC VOs and to detect potential problems and ensure effective site performance.

The Dashboard applications are essential LHC computing operations tools. However, they are generic and can be adapted for other community’s needs. The talk will give an overview the Dashboard applications, highlighting the possibility of exploiting these applications outside the LHC domain.

  • Location: Lambda; Time: 13-Apr-2011 @16:00; Duration: 01h30'

MPI Hands on training

The Message Passing Interface (MPI) standards and their implementations are currently the most prevalent frameworks on which parallel applications are built. A significant problem in exploiting MPI applications on the grid is the inherent nature of its heterogeneous environments. Different MPI implementations, system interconnects or job managers can be found at different resource centres. In order to run an application, the end-user needs some ‘a priori’ knowledge about the resources. MPI-Start offers a unique and stable interface to execute parallel applications at the gLite based grid sites. It aims to hide the differences and complexities of the heterogeneous systems that compose a grid infrastructure by providing a high-level abstract layer. This presentation introduces the basic concepts of MPI, together with a detailed description of MPI-Start and how to use it.

The tutorial is suitable to all users who wish to adapt their MPI-based applications for use on the grid using gLite.

  • Location: Iota; Time: 14-Apr-2011 @11:00; Duration: 01h30'

Kepler

Kepler is a free and open source workflow engine used extensively by the FUSION community. It is designed to help scientists and developers to easily create, execute, share and reuse their models across the scientific and engineering domains. In particular, Kepler includes components that integrate with different middleware stacks (e.g. gLite or UNICORE). Kepler workflows can be decomposed into smaller parts, thus allowing complex tasks to be divided into much simpler ones. This feature provides workflow designers with ability to build re-usable, modular sub-workflows. These can be saved and applied to other workflow. This introductory tutorial should be of interest to all users keen to explore Kepler's powerful capabilities. It will begin by showing how to use Kepler to build basic workflows; use relation paths and synchronisation; and to create control structures such as "if-else" and loops. Finally, job submission, monitoring, and data management shall complete the tutorial.

  • Location: Zeta; Time: 14-Apr-2011 @14:00; Duration: 03h30'


John Walsh, Grid-Ireland Operations Centre, Trinity College Dublin

The countdown for the EGI User Forum has started

With less than a week to go before the start of the User Forum in Vilnius (11-14 April), the organisers are pleased to announce another three keynote speakers invited to the event.

  • Ian Fisk, from FermiLab, will describe WLCG experiences, based on the early results from using their global Tiered data distribution structure (Tier 0-3). These experiences have motivated the proposed move to a cached data model, and he will indicate how this knowledge can help other communities to use distributed infrastructures;

  • Tommi Nyrönen, development manager at CSC –the Finnish IT Centre for Science – and the Finnish contact for the ELIXIR project, will introduce the collaborative ‘Hub and Nodes’ structure of this large distributed European project, and the IT strategies and infrastructure it is exploring to support its research community;

  • Nick Barcet, from Canonical, will talk about how Ubuntu can be used to create Infrastructure as a Service cloud data centres, and is now becoming a popular guest operating system to deploy in public clouds.

They will join the other two keynote speakers, Steve Rawlings from the University of Oxford and Ruth Pordes from the Open Science Grid, announced in March.

Online registration for the User Forum is now closed, but it will be possible to register for the event at the conference site.

See you all in Vilnius!

 

The countdown for the EGI User Forum has started

With less than a week to go before the start of the User Forum in Vilnius (11-14 April), the organisers are pleased to announce another three keynote speakers invited to the event.

  • Ian Fisk, from FermiLab, will describe WLCG experiences, based on the early results from using their global Tiered data distribution structure (Tier 0-3). These experiences have motivated the proposed move to a cached data model, and he will indicate how this knowledge can help other communities to use distributed infrastructures;

  • Tommi Nyrönen, development manager at CSC –the Finnish IT Centre for Science – and the Finnish contact for the ELIXIR project, will introduce the collaborative ‘Hub and Nodes’ structure of this large distributed European project, and the IT strategies and infrastructure it is exploring to support its research community;

  • Nick Barcet, from Canonical, will talk about how Ubuntu can be used to create Infrastructure as a Service cloud data centres, and is now becoming a popular guest operating system to deploy in public clouds.

They will join the other two keynote speakers, Steve Rawlings from the University of Oxford and Ruth Pordes from the Open Science Grid, announced in March.

Online registration for the User Forum is now closed, but it will be possible to register for the event at the conference site.

See you all in Vilnius!