Taken from page posted by Andy Hinck, Nov. 15, 2001 and annotated.  7/2/02 - SCH

1.  Reboot the SGI computer (if it has shutdown; this is not usually the case the SGIs are usually held during the outage by the UPSs to which they are attached)

2.  Next, go the SGI.  Login as the user that was logged in  when the machine went down.  In a shell, start the program by typing xwinnmr -r.  This will kill any xwinnmr processes that might still be running.

3.  The first thing you should do after starting xwinnmr is to configure.  To do this, type cf on the command line.  The software will then prompt you for the nmr superuser password.  Respond to all queries by clicking OK (i.e. accept all default settings).  You will almost certainly see some error messages and warnings.  Just click OK and ignore.  Once cf is complete, run through it again.  Usually on the second pass you will not get any error messages and/or warnings (If you do, this may in fact indicate that there is a problem).

4.  After readjusting the lock field to the approximate value for D2O and reading a shim file, the spectrometer should now be ready to use. [This appears to concern the situation where the sample is still in the bore from an acquisition during which the power failed, and the object is to restart the same acquisition.  lock d2o will reestablish the deuterium lock.  I believe that the reference to lock field just means that lock phase, gain and power won't need readjusted, although lock d2o will automatically reset all of the lock parameters.  The reference to loading a shim file assumes that you have saved your final shim configuration after shimming this particular sample.  If not, you will have to reshim.  However, in some cases the saved shim file doesn't restore a good shim, so you should always check the shim.  Recalibrating pulses shouldn't be necessary.  Unless you have ejected the sample, retuning shouldn't be necessary. Do check the temperature control (edte), since gas flow failures may occur during protracted power failures.]

5.  If after the cf operation, lock d2o and other operations indicate communications failures, try ii to reinitialize the interface.  If that gives a communications error, try cfbsms to reinitialize the BSMS.  In principle the BSMS initialization parameters should be stored in CPROMs onboard, but we have seen it need reinitialized after power failure.  Again, use the superuser password, and agree with all defaults.

As a last resort:

1.  Open the console cabinet doors and turn off the power switches to the individual components.  Most units have toggle switches.  Exceptions to this are the PTS (frequency synthesizers) and on the 600 MHz spectrometer the Shomandl frequency unit.  Neither have accessible power switches.  Other exceptions are the cpu units and on the 500 MHz spectrometer and the frequency controller unit (lower left) which are turned on/off by turning the key by 90 degrees.

2.  At this point, you can now restore the main power to the console by pressing the large green power switch on the front of the console. On the 600, this will restore power to the cooling bath.  On the 500, to turn the cooling unit back on you will have to hit the "on" button (on the cooling unit itself).

3.  Next, one by one power up the individual components.  The 'cpu' unit should be booted last (the order by which the other components are turned on is not important). [The cpu on the Avance600 is controlled by the key at the top of the middle section].

Repeat the cf initialization steps above.

Comments:

I've found the SGI is a strange state a couple of times, and rebooting it alone fixed the problem.  One such case was invoked by trying to do a 2D processing while another dataset was being acquired.  That locked the SGI up so tight that the power button wouldn't even turn it off.  This is now understood to be caused by a bug in xwinnmr version 3.0 that should be repaired in version 3.1  The acquisition was left to finish.  Then the power plug on the SGI had to be pulled to force a reboot.  The newly acquired data was OK.  Trying to reprocess the data that it hung up trying to process resulted in mush.

The spectrometers themselves are running on a large scale non interruptable power source that should survive even prolonged power failures.  The circuit breakers for the noninterruptable supply are in the box in the back of the utility closet in the Avance500/600 room.  These shouldn't be touched, and shouldn't be affected by power outages.  Once we tripped one of these thinking that we were correcting another power failure problem and that's the only occassion I know of that created a true spectrometer power failure requiring a reboot.

The SGI's are currently on small independent uninterruptable supplies.  They will only last an hour or so, so the SGI's won't survive a prolonged power failure.  When the SGI goes down, the acquisition computer in the spectrometer will keep sending data, but it is not being saved on the SGI.  If upon rebooting the SGI, there is a continuing acquisition that xwinnmr can not cancel with ii, then use the reset button on the CCU unit on the acquisition unit itself.  There is a plan to run the spectrometer power through a transformer and to the SGIs to secure them against protracted power failures.  If that is implemented, then you'll notice that the power cords of the SGIs will go to a transformer unit and then to the back of an acquisition unit intead of to a wall plug.

Note that the air dryer is not secured against power failure.  So it is possible for the computers and the spectrometer to survive a power failure, but for the refrigeration units to freeze up with loss of gas flow and temperature control.  Hence, you've got to check the temperature control (edte) after any acquisition that appears to have finished normally when there may have been power failures.  There is a plan to wire the air dryer to noninterruptable power in the future to eliminate this vulnerability.  See the gas flow failure page.

One incident occurred after the system had survived several protracted power failures.  Then apparently a brief power glitch put it in a state where communications between the SGI and the spectrometer were disrupted.  The first indication of a problem was that invoking edte gave a message about a communication problem and to check the ethernet cable.  Other ethernet connections worked.  The dedicated cable to the spectrometer could have been partially dislodged from the back of the SGI, but pushing it in didn't resolve the problem.  Rebooting the SGI did not resolve the problem.  One pass through rebooting the Avance600 failed to resolve it, although we missed powering off two of the units.  Second time around the large red power off button was used to take the spectrometer off line, then all the units were set to off, the green button was pushed, and the units were each brought on line, cpu last.  This fixed the problem.