This is a follow-up to my earlier post:
Question: How to Launch an MPI Parallel Program Programmatically
An Iceberg Problem
I've been stuck on my MPI-Java integration experiment for more than a week and so far haven't found a way out of this problem. My problem is quite similar to this posting on the comp.parallel.mpi newsgroup. Unfortunately, the thread was first posted in the year 2001 and, almost five years later, no one has answered the question.
Let me explain my situation: I want to convert my parallel code, which is written in C and is based on the MPICH and PETSc parallel-computing libraries, into a Web service. From my point of view, three solutions are available: The first one is to use Apache Axis C++ Web Services library to extend the existing code into a Web service. The second one is to leave the parallel code as is and then develop a Java Web service that acts a proxy to execute the parallel code via a Java Runtime.exec() method call. The third one is to use the parallel code in the same way the old-fashioned Web applications were developed, thru some Common Gateway Interface (CGI) mechanisms.
On my first attempt, I went the Java Runtime.exec() plus mpiexec way. It worked well for small Hello World code with simple matrix computation and MPI function calls but it doesn't work when I told Runtime.exec() to launch my parallel structural analysis code. An excerpt of the error is attached at the end of this post.
I also looked at the Axis C++ way, which, for me, is the most direct (but painful) way to create a Web service from existing C code. From preliminary study, if I choose to avoid mpiexec to make a self-contained Web service, it seems like I will have to use the MPI_Comm_spawn function to initialize the master and the worker nodes. Looking at the sample codes, my feeling is that this is a bit much for me as I will have to learn how to use another toolkit -- the Axis C++ -- and to invest considerable time to be familiar enough to master it. I still remember how I spent almost one month in 2004 learning undocumented features of Apache Axis to create a custom serializer-deserializer module for my Java codes.
I also had a glance at CGI and again don't feel like investing my time in learning yet another technology.
I think I will have to stick with the "Java Runtime.exec() plus mpiexec" way. I will post further development about this on this blog.
References:
Question: How to Launch an MPI Parallel Program Programmatically
Google Groups : comp.parallel.mpi
Problems with Runtime.getRuntime().exec
Keywords: mpi, parallel-computing, java, problems
Update: Jan 18, 06
The poster of the comp.parallel.mpi thread also cross-posted to comp.lang.java.programmer and there are 5 messages on this thread. Here is the link: Problems with Runtime.getRuntime().exec. Unfortunately, this thread leads to no solution.
Update: Jan 18, 06
A friend of mine suggested another solution: using socket communication -- by letting the Java proxy Web service communicate with the master of the cluster, which is implemented in C, thru socket communication. He also introduced me the class ProcessBuilder, which is a new Java class introduced in JDK 5.0, as an alternative to the Runtime.exec() solution.
Attachment:
Error from Java Runtime.exec() plus mpiexec:
Question: How to Launch an MPI Parallel Program Programmatically
An Iceberg Problem
I've been stuck on my MPI-Java integration experiment for more than a week and so far haven't found a way out of this problem. My problem is quite similar to this posting on the comp.parallel.mpi newsgroup. Unfortunately, the thread was first posted in the year 2001 and, almost five years later, no one has answered the question.
Let me explain my situation: I want to convert my parallel code, which is written in C and is based on the MPICH and PETSc parallel-computing libraries, into a Web service. From my point of view, three solutions are available: The first one is to use Apache Axis C++ Web Services library to extend the existing code into a Web service. The second one is to leave the parallel code as is and then develop a Java Web service that acts a proxy to execute the parallel code via a Java Runtime.exec() method call. The third one is to use the parallel code in the same way the old-fashioned Web applications were developed, thru some Common Gateway Interface (CGI) mechanisms.
On my first attempt, I went the Java Runtime.exec() plus mpiexec way. It worked well for small Hello World code with simple matrix computation and MPI function calls but it doesn't work when I told Runtime.exec() to launch my parallel structural analysis code. An excerpt of the error is attached at the end of this post.
I also looked at the Axis C++ way, which, for me, is the most direct (but painful) way to create a Web service from existing C code. From preliminary study, if I choose to avoid mpiexec to make a self-contained Web service, it seems like I will have to use the MPI_Comm_spawn function to initialize the master and the worker nodes. Looking at the sample codes, my feeling is that this is a bit much for me as I will have to learn how to use another toolkit -- the Axis C++ -- and to invest considerable time to be familiar enough to master it. I still remember how I spent almost one month in 2004 learning undocumented features of Apache Axis to create a custom serializer-deserializer module for my Java codes.
I also had a glance at CGI and again don't feel like investing my time in learning yet another technology.
I think I will have to stick with the "Java Runtime.exec() plus mpiexec" way. I will post further development about this on this blog.
References:
Question: How to Launch an MPI Parallel Program Programmatically
Google Groups : comp.parallel.mpi
Problems with Runtime.getRuntime().exec
Keywords: mpi, parallel-computing, java, problems
Update: Jan 18, 06
The poster of the comp.parallel.mpi thread also cross-posted to comp.lang.java.programmer and there are 5 messages on this thread. Here is the link: Problems with Runtime.getRuntime().exec. Unfortunately, this thread leads to no solution.
Update: Jan 18, 06
A friend of mine suggested another solution: using socket communication -- by letting the Java proxy Web service communicate with the master of the cluster, which is implemented in C, thru socket communication. He also introduced me the class ProcessBuilder, which is a new Java class introduced in JDK 5.0, as an alternative to the Runtime.exec() solution.
Attachment:
Error from Java Runtime.exec() plus mpiexec:
Executing: mpiexec -n 2 /home/shared/thitiv/parefg-20060107_1451/parefg /home/shared/thitiv/parefg-20060107_1451/tension.in /home/shared/thitiv/parefg-20060107_1451/tension.out
OUTPUT>Petsc Release Version 2.3.0, Patch 43, April, 26, 2005
OUTPUT>See docs/changes/index.html for recent updates.
OUTPUT>See docs/faq.html for hints about trouble shooting.
OUTPUT>See docs/index.html for manual pages.
OUTPUT>-----------------------------------------------------------------------
OUTPUT>/home/shared/thitiv/parefg-20060107_1451/parefg on a linux-gnu named ubuntu by thitiv Wed Jan 18 14:50:13 2006
OUTPUT>Libraries linked from /usr/local/lib/petsc-2.3.0/lib/linux-gnu
OUTPUT>Configure run at Tue Jan 3 09:36:34 2006
OUTPUT>Configure options --with-mpi-dir=/usr/local/bin/mpich2 --with-shared=0
OUTPUT>-----------------------------------------------------------------------
OUTPUT>[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
OUTPUT>[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
OUTPUT>[0]PETSC ERROR: likely location of problem given in stack below
OUTPUT>[0]PETSC ERROR: --------------- Stack Frames ---------------
OUTPUT>[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
OUTPUT>[0]PETSC ERROR: INSTEAD the line number of the start of the function
OUTPUT>[0]PETSC ERROR: is given.
OUTPUT>[0]PETSC ERROR: --------------------------------------------
OUTPUT>[0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
OUTPUT>[0]PETSC ERROR: Signal received!
OUTPUT>[0]PETSC ERROR: !
ERROR>[cli_0]: aborting job:
ERROR>application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
OUTPUT>rank 0 in job 3 ubuntu_32773 caused collective abort of all ranks
OUTPUT> exit status of rank 0: killed by signal 9
ExitValue: 137
Comments