User: Password:
Subscribe / Log in / New account

Checkpoint/restart: it's complicated

Checkpoint/restart: it's complicated

Posted Nov 12, 2010 20:39 UTC (Fri) by dlang (subscriber, #313)
In reply to: Checkpoint/restart: it's complicated by Np237
Parent article: Checkpoint/restart: it's complicated

the problem is that just dong c/r on the process isn't good enough, it also has to do c/r on every other thing that the process talks to.

for example,

saving tcp connection info does you no good unless the other end of the tcp connection gets checkpointed at the same instant.

saving pending disk writes does no good if the file you are writing to is off on some other system and will contain writes after the checkpoint

system level c/r is useful for planned outages, but when you are in HPC environments, you have enough nodes that this is really not good enough, you will have unplanned outages, and unless your c/r can back out all these other side effects, it's not going to be able to be used for these outages.

(Log in to post comments)

Checkpoint/restart: it's complicated

Posted Nov 12, 2010 21:52 UTC (Fri) by Np237 (subscriber, #69585) [Link]

This is precisely why you need the help of the MPI library and the resource manager: so that all processes related to a given job can be handled at the same time.

Most of the things you describe are already handled by BLCR, although still in an imperfect way.

Checkpoint/restart: it's complicated

Posted Nov 13, 2010 6:37 UTC (Sat) by dlang (subscriber, #313) [Link]

you miss my point.

doing checkpointing of the apps on any one system isn't good enough.

you need to checkpoint the app and everything that it talks to on _every_ system at once (and make sure that you do it at the same instant so that there's no chance of data being in flight between systems to make it inconsistant)

Checkpoint/restart: it's complicated

Posted Nov 13, 2010 8:44 UTC (Sat) by Np237 (subscriber, #69585) [Link]

No, you are missing my point. When I wrote “all processes related to a given job”, I really mean all processes, on all cluster nodes.

Yes, it’s complicated. But with the help of the MPI library you can close all connections (since all inter-nodes connections are supposed to go through MPI) in a synchronized way. This is what BLCR + OpenMPI already do.

Checkpoint/restart: it's complicated

Posted Nov 15, 2010 17:51 UTC (Mon) by orenl (guest, #21050) [Link]

There are two types of checkpoints here:

1) Coordinated checkpoint of all participating processes (across the cluster) so that the entire jobs can be restarted later from this checkpoint. This is useful for fault-tolerance.

2) Checkpoint of processes on one (or more) nodes and then restart on a different node (or set of nodes). This is useful for load-balancing, but also for maintenance, e.g. by vacating an over-heating node.

The former is usually done combining a C/R mechanism with a C/R-aware implementation of e.g. MPI. The latter is more tricky if one would like to do seamless live migration.

Linux-CR supports both.

Copyright © 2017, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds