EDGI introduces new tools to submit thousands of jobs to the grid

At the EGI User Forum, held in Vilnius, Lithuania, the EDGI project successfully demonstrated the use of the new MetaJob feature, which enables user to submit jobs to the EDGI Desktop Grid through gLite using one single command. The new feature can be used to wrap or describe thousands of jobs into one single MetaJob.

“Desktop Grids are especially useful for applications like parameter sweeps where you have thousands and thousands of similar jobs,” explains Ad Emmen, from AlmereGrid. “MetaJob is a command line ‘script’ that helps an EGI user to create these jobs for his/her application, and submit them automatically through standard gLite interfaces.”

The efficiency of MetaJob was demonstrated with great success at the EDGI booth. During the day, the EDGI team submitted 10,000 jobs several times to show the EGI User Forum visitors how this can easily be processed by BOINC, without disrupting the gLite infrastructure.

For Jozsef Kovacs, from the MTA SZTAKI (the Computer and Automation Research Institute, Hungarian Academy of Sciences), MetaJob makes it easier to handle a high number of jobs. MetaJob also allows “to increase the scalability of the service grid to desktop grid bridging services deployed by the EDGI project by decreasing the load of the service grid components – for example gLite wms,” he adds.

From a technical point of view the solution is based on the well-known Service Cluster Grid to Desktop Grid bridge, where an extra MetaJob description file - containing the definition of a huge number of jobs - is attached to the job as input file. This MetaJob input file is recognised by the 3GBridge component, which then creates the jobs and inserts them into BOINC. This solution also works with XtremWeb on the Desktop Grid side or ARC on the Service/Cluster Grid side without any modification.

This work was performed in the framework of "Task JRA1.5: Solve Service/Cluster Grid to Desktop Grid bridge scalability issues" in the EDGI project.

"This successful EDGI demonstration clearly shows that Desktop Grids can be used effortlessly for scientific day-to-day calculations,” said Peter Kacsuk from the EDGI project. “The relevance of Desktop Grids for e-Science has been proven once and for all.”