Houston, Hold On To Your Butts

On Saturday, May 24th, at approximately 2230h, the HI-SEAS habitat went completely dark. The entire power system suffered a critical failure… due to my own action. The entire story is reminiscent of a mixture of Apollo 13 and Jurassic Park, so Houston, hold on to your butts!

The HI-SEAS habitat is controlled by dozens of web-enabled sensors and relays. The sensors give us the condition of the habitat: the charge of our battery stacks, the output of our AC inverters, the ambient temperature in several locations, the concentration of CO2, and more.  The relays switch on fans, the hydronic heating system, and regulate the power generation in response to the sensor array. They’ll even isolate the battery stacks in the event of a power failure to protect the batteries from damage…

One of the sensor array's responsible for monitoring the HI-SEAS habitat's power systems.
One of the sensor arrays responsible for monitoring the HI-SEAS habitat’s power systems.

I was assigned to a task by one of our Systems Ground Controllers at  1658h: to reset the sensor array connected to our fuel cell backup system. It hadn’t been reporting the condition of the fuel cell for several weeks, and on a recent robotic resupply, our hydrogen source had been replenished, so we needed that data from the fuel cell.

Speaking of sensors, can you reset all of the CBW devices in the container/workshop? Go to the units and pull the terminal block off. Actually if you rock the top back, it will pull out the power pins (24v) and reboot it. — Systems Ground Controller

I received the message as I checked my e-mail prior to bed, around 2200h. It sounded simple enough… “Go to the units and pull the terminal block off.” The instructions couldn’t have been more clear. I hemmed and hawed about whether or not to go ahead and do it that night, or wait until morning.

I should have waited until morning.

The sensor array is located above the battery stacks in the shipping container module, and consists of several rails of electronics: power supplies, fuses, and the sensors themselves. All the sensors are daisy-chained to the same power supply, so pulling the terminal block would disrupt the flow of electricity to all of the sensors. Unfortunately, one of the sensors daisy-chained was controlling a relay that would isolate the battery stacks in the event of a power failure.

I rocked the terminal block off the first sensor. The habitat went dark.

I rocked the terminal block back on. The habitat stayed dark. The only glow came from the battery stacks, which were happily reporting that they were online and in good condition, only they weren’t in discharge mode.

Houston, we have a problem.

The emergency reboot procedure is simple and straight forward. Connect a set of backup batteries to the system by clicking a single fuse on. The backup batteries, in theory, would power on the sensor array, which would then detect that the batteries had been isolated, and close the relay to restore power to the habitat.

Click… Nothing.

The battery backups were dead. The sensor array couldn’t draw power, and the battery stacks weren’t supplying power to the habitat. Cue the backup backup strategy, since the backup battery and the fuel cell couldn’t turn on the sensor array, maybe powering the gasoline generator would.

The gasoline generator has a control box inside the shipping container that allows us to turn on the generator and crank the engine without leaving the habitat.

Click… Crank… Nothing.

The habitat was still completely dark, except for the lights of the battery stacks telling us that they were happily willing to restore power to the entire system, if only they were allowed to talk to the sensor array.

HI-SEAS battery stack's impression of HAL's "I can't do that, Dave..."
HI-SEAS battery stack’s impression of HAL’s “I can’t do that, Dave…”

As an exercise in futility, we donned EVA suits and went outside to start the gasoline generator manually. That, finally, succeeded. But it didn’t succeed in powering the sensor array, no. The gasoline generator was running into the same brick wall as the battery stacks, it was ready and willing to supply power, if only the sensor array was turned on.

Time for a backup backup backup plan. The sensor array wasn’t getting power from the backup battery, it wasn’t getting power from the battery stacks or the fuel cells, and it wasn’t getting power from the gasoline generator. Time for a last-ditch effort.

The two MX-C suits have lithium iron phosphate (LiFePO4) batteries that each supply 12 volts fully charged. The sensor array needs 24 volts to start, and since we have two of the batteries we can connect them in series to get the necessary voltage. Remember, all the while we are doing this, the EVA, the wiring, the testing, the swearing, with nothing more than a couple of flashlights.

Well, it was finally time.

Hold on to your butts. No, not really! It’s a quote from Jurassic Park. — Me

An arc of light jumped as I touched the wires from the MX-C batteries to the terminals of the sensor array. The sensor array clicked, its lights flashed. The habitat stayed dark. For. Five. Breathless. Seconds.

The fans on the battery array were the first indication that something had worked. The started whirring. Next, a series of subsystems flashed to life: the inverter controllers flickered and booted, then the inverters themselves started clicking and whirring. Finally, the habitat lights blinded our dark-adapted eyes. The power was back on.

It took another hour before we had all systems up and running again. Most notably, the generator was still on, and the battery stacks  were having a bit of an argument with them over whose job it was to run the habitat power. That kept our communication system offline, but once we shut down the generator, as Ellie Sattler would say, “Mr. Hammond, I think we’re back in business.”

Right before being attacked by a velociraptor. But that’s a story for another day…

(PS, if you look carefully in the featured image, you can see the yellow lasers being fired by the Keck Observatory on Mauna Kea)

Presentation: Real Life on Fake Mars – LogiCON 2014

This past Saturday LogiCON 2014  played out successfully. My own presentation had a very warm reception, and the video feedback that I received from the crowd exceeded my expectations!

If you’d like to view my presentation, it’s now available on YouTube:

We had time to address three questions, summaries of which I’ve transcribed:

Q1: What am I finding the most difficult on sMars, and is it what I expected?

A1: The isolation at the HI-SEAS habitat is tremendous. The nearest human activity are the distant lights at the top of Mauna Kea. There are no plants or animals at the HI-SEAS habitat, and therefore no sound. The only sound, other than the sounds we generate, is the wind and the occasional rain storm.

Also, stop posting pictures of food on social media. Dehydrated food has a list of pros and cons, and the biggest con is the blandness!

Q2: How do we resolve interpersonal conflict at HI-SEAS?

A2: To be honest, there isn’t a lot of conflict in the habitat. There are two reasons for that, the first is that we address conflict early and often, before things can spiral out of control. The second is that we involve all of the crew in conflict, and rationalize solutions using a consensus model.

Q3: How is our water managed?

A3: We have “robotic” resupplies when out water storage runs low, and we have “robotic” grey water recovery when our waste water tanks are nearly full. These are human operations, and during these times we shut the window and turn on music to minimize the effect of the “robotic” activity has on the crew.

My HI-SEAS Reading List

With limited internet capabilities, I brought along quite the stack* of books to entertain me while I’m living inside the HI-SEAS habitat. Outside of my Martian duties, my PhD responsibilities and time spent exercising I spend my spare time reading books, playing video games and watching shows and movies.

The books that I’ve chosen are a good mix of my interests and do not, by any means, represent a complete list of books that I’ve read (you can find a more comprehensive list over on goodreads). I don’t intend to read all these books, but as of today I’m setting a goal to read one book per week, so really we are only talking about 10 more books, in addition to the 4 I’ve read and the 4 I’m finishing. I’m happy to take recommendations, but as far as the HI-SEAS mission is concerned, this list is pretty much written in stone.

*The Kindle Paperwhite that I’m using measures a hair over 10 mm in thickness.

Books in Progress

The Book On The Taboo Against Knowing Who You Are – Alan Watts
Mindfulness in Plain English – Henepola Gunaratana
Red Mars – Kim Stanley Robinson
Outliers – Malcolm Gladwell

Books to Read

Starship Troopers – Robert A. Heinlein
Personal MBA – Josh Kaufman
The Jasons – Ann Finkbeiner
Tao Te Ching – Lao Tzu
The Moon is a Harsh Mistress – Robert A. Heinlein
The New Alchemy – Alan Watts
The Fault in Our Stars – John Greene
Fleet of Worlds – Larry Niven
Uncertainty – Jonathan Fields
Pushing Ice – Alastair Reynolds
Redshirts – John Scalzi
The Innovator’s Dilemma – Clayton M. Christensen
Blue Mars – Kim Stanley Robinson
Pragmatism – William James
Autobiography of a Yogi – Paramhansa Yoganada
Cutting Through Spiritual Materialism – Chögyam Trungpa
The Innovator’s Solution – Clayton M. Christensen
Man’s Search for Meaning – Viktor E. Frankl
The Startup Owner’s Manual – Steve Blank and Bob Dorf
Zen Mind, Beginner’s Mind – Shunryū Suzuki
The Lean Startup – Eric Ries
Tunnel in the Sky – Robert A. Heinlein
Time Enough for Love – Robert A. Heinlein
The Puppet Masters – Robert A. Heinlein
Friday – Robert A. Heinlein
Green Mars – Kim Stanley Robinson
Tao of Pooh – Benjamin Hoff
Te of Piglet – Benjamin Hoff

Books Read

An Astronaut’s Guide to Life on Earth – Chris Hadfield (4/5)
The Martian – Andy Weir (5/5)
The Joyous Cosmology – Alan Watts (3/5)
Spin – Robert Charles Wilson (5/5)

LogiCON 2014: Real Life on Simulated Mars

LogiCON is a yearly conference put on by a group of excellent individuals, celebrating logic, critical thinking, and the scientific method. This year’s conference will take place in Edmonton on May 24th, and plays host to an excellent array of speakers. The conference is geared towards general interest public topics. I highly recommend anyone in Edmonton and the surrounding areas go, tickets are on sale now.

Ross takes a GigaPan image of the HI-SEAS environment. Photo by Annie Caraccio.
Ross takes a GigaPan image of the HI-SEAS environment. Photo by Annie Caraccio.

Since I’m way out here on sMars, I’ll be uploading my presentation in remotely. After the conference, it will be available on YouTube. Here is the abstract for my talk:

Real Life on Simulated Mars

The first trip to Mars may occur as early 2030, but there is a lot of work to be done to prepare for a successful mission to the red planet. Analog simulations like the Hawaii Space Exploration Analog & Simulation are used to study everything from life-support systems to the psychological impact on future astronauts. These low-cost analogs help researchers plan long-duration missions that closely replicate many of the conditions that exist on Mars. For the last two months, I have been living in a habitat built on the desolate lava fields of Mauna Loa in Hawaii, simulating the life of a Martian in the Tharsis region of Mars.

 

Featured image by Casey Stedman.

Episode 0 – 36 Days on sMars

This is the first episode of the Spincrisis podcast!

10255129_10154167243085565_2725944072305184938_n

In this episode, I talk give a brief introduction to who I am, and why I applied for the HI-SEAS program (Hawaii Space Exploration Analog and Simulation). Since we are now on day 36 of the HI-SEAS program, I answer a number of questions that have been sent to me over the last 5 weeks.

I answer the question, why the title ‘Spincrisis’?

I also talk about the future of this podcast, weekly crew interviews and recaps during the HI-SEAS program, and more general interest opinions on science, technology and space exploration after the HI-SEAS program has come to an end.

Questions and comments about HI-SEAS? E-mail me at my delayed e-mail address: ross@delaysnrf.ksc.nasa.gov.