Setting up a new HP P2000 G3 FC MSA with dual controllers over the last couple of days for a small staging environment, I ran into issues from the word go. The device in question was loaded with 24 SFF disks and two Controllers (Controller A and B).
On the very first boot we noticed a fault (amber) LED on the front panel. Inspecting the back of the unit, I noticed that Controller A and B were both still flashing their green “FRU OK” LEDs, (which according to the manual means that the controllers were still booting up), even after waiting a number of hours. On Controller A, I could see a blinking amber “Fault/Service Required” LED. Following through the troubleshooting steps in the manuals lead nowhere as the end synopsis was to check the event logs. Even the Web interface was acting up – I could not see the controller’s listed, could not see any disks and the event logs were completely empty. Obviously there was a larger issue at hand preventing the MSA and even the Web interface from functioning properly. To further confuse matters, after shutting down and restarting the device, controller B starting blinking the amber LED instead of A this time, both still stuck in their “Booting up” state. Refer to the linked LED diagram below and you’ll see that the LED flashing green is labelled as 6, and the amber blinking LED is the one labelled as 7 on the top controller in the diagram.
After powering the unit down completely, and then powering back up again, the MSA was still stuck in the same state. Powering down the unit once more, removing and reseating both controllers did not help either. Lastly, I powered it all off again, removed controller A completely, then powered up the device with just Controller B installed. Surprisingly the MSA booted up perfectly, and LED number 6 (FRU OK) went a nice solid green after a minute or so of booting up. No amber LEDs were to be seen. Good news then! Hot plugging controller A back in at this stage with the device powered on resulted in both controllers reporting a healthy status and all the disks and hardware being detected. A final test was done by powering off everything and powering it back up again as it should be from a cold start. Everything worked this time.
Here is a photo of the rear of the device once all was resolved showing the solid green FRU OK LEDs on both controllers.
Bit of an odd one, but it would seem that controllers together were preventing each other from starting up. Removing one then booting up with this seemed to solve the problem, and at the end of the day all hardware was indeed healthy. After this the 24 disks were assigned and carved up into some vdisks to be presented to our ESXi hosts!
Really pleased to hear it helped you! Thanks for dropping by.
Hi Sheen,
Can’t express my gratitude on your valuable article.
It solved a big problem as it helped me to return back & see my Hard drives with the data available
MY God always bless you
Hi Sean!
Just wanted to thank you for sharing this, it helped us today. 🙂
Good stuff! Great to hear it helped 🙂
Hi Sean,
Thank you so much for your post, I was really stuck on this one. I am not a storage expert but I was the only one available to help bring back a customer online. HP manuals and troubleshooting all failed to work.
Cheers!
Hi Kumar,
Are you able to shutdown the entire enclosure and try a power cycle? Anything more than this I won’t really be able to help you with – this sounds to me like a hardware issue, so a complete power cycle might be the best way if you can afford the down time. I recently had an issue with our P2000 at work and contacted HP support. Even though it was out of warranty, they still helped me troubleshoot and found the issue to be a faulty controller.
Cheers
Sean
Hi Sean,
One of our MSA2000 controllers A and B doesn’t show any responce but enclouser got powered ON Fans, power units also working perfectley. Suddenly it got stopped and no amber no light. I have removed controller node and replace it again but still it doesn’t showing any response. No lights at back side of controllers..Could you please help me in this..
Don’t forget the serial port access to these controllers.It gives valuable info on what state the controllers are in.Check the msa cli guide on HP for info. Seen a few apparent lockups caused by firmware updates(controllers can copy between themselves). Use a USB mini cable with the HP driver and putty to hook up.
Excellent, glad it worked for you. Good to see another South African IT dude around here 🙂
Thanks for the post. Saved my buttocks during a DR invocation, due to an extended power failure, where the SAN wouldn’t boot up properly. I was thinking of trying this but doubted myself just before reading this. Maybe us IT guys aren’t that crazy after all. Thanks a million.
I have another issue regarding HP MSA2000, which may possibly be linked up with this one. As I was working on the server with whom my RAID is configured. The RAID drives suddenly disappeared. I checked the RAID enclosure physically and it was giving all 3 LEDs on front panel i.e., UID, Fault and Heartbeat LEDs as permanently ON but all the LEDs on all of the hard drives were OFF. It has only one controller in it and there is no light ON on the controller. Both the power supplies are giving solid green lights and their fans are running. Please see if any one of you can relate a possible solution to it.
Regards,
Thanks for the feedback Brian – glad this post helped you, and cheers for the status command – very useful!
Hi Sean,
I have an identical setup I am trying to get going as well. Thanks for the solution even got my web browser to work after trying this! Just in case anybody else is stuck I found the command show shutdown-status really helpful in trying to find out the states of the controllers.
Hi Corey,
Ouch! Glad it helped you sort the issue out though 🙂 Thanks for the feedback.
Cheers
Sean
Hi Sean,
I Thank you as well for posting this solution. We had the exact same problem on HP P2000 G3 MSA Array and were able to get both controllers working again with this solution. Our entire University campus Datacenter was offline until i came across this
KUDOS!!!
Hi Joel,
Good to hear, and glad it helped! Thanks for leaving some feedback.
Cheers,
Sean
Hi Sean,
Just adding my thanks. I’m configuring a P2000 G3, iSCSI not FC. It was shipped to me with both controllers installed and I had the same issue as you. Booting off one controller has fixed the problem. Adding the second controller back in, it continues to function without issue.
Appreciate you sharing your solution. Take care.
Hi Andy, excellent! Glad to hear it someone out 🙂 Thanks for dropping the feedback here.
Sean
Hi Sean,
Thanks for posting this resolution, I had the exactly same issue and tried all sort of HP user guide recommendations without success. I came across this article during the night and it worked when I tried it the next day. Never was I so eager to go work before. Lol.
This is my first time setting a SAN and FC switch, for a moment I almost give up, doubting my own tech capability. Your generosity has once boosted my confidence.
Will definitely back to visit this site again.
Cheers