Category: HI-SEAS

On Saturday, May 24th, at approximately 2230h, the HI-SEAS habitat went completely dark. The entire power system suffered a critical failure… due to my own action. The entire story is reminiscent of a mixture of Apollo 13 and Jurassic Park, so Houston, hold on to your butts!

The HI-SEAS habitat is controlled by dozens of web-enabled sensors and relays. The sensors give us the condition of the habitat: the charge of our battery stacks, the output of our AC inverters, the ambient temperature in several locations, the concentration of CO2, and more.  The relays switch on fans, the hydronic heating system, and regulate the power generation in response to the sensor array. They’ll even isolate the battery stacks in the event of a power failure to protect the batteries from damage…

I was assigned to a task by one of our Systems Ground Controllers at  1658h: to reset the sensor array connected to our fuel cell backup system. It hadn’t been reporting the condition of the fuel cell for several weeks, and on a recent robotic resupply, our hydrogen source had been replenished, so we needed that data from the fuel cell.

Speaking of sensors, can you reset all of the CBW devices in the container/workshop? Go to the units and pull the terminal block off. Actually if you rock the top back, it will pull out the power pins (24v) and reboot it. — Systems Ground Controller

I received the message as I checked my e-mail prior to bed, around 2200h. It sounded simple enough… “Go to the units and pull the terminal block off.” The instructions couldn’t have been more clear. I hemmed and hawed about whether or not to go ahead and do it that night, or wait until morning.

I should have waited until morning.

The sensor array is located above the battery stacks in the shipping container module, and consists of several rails of electronics: power supplies, fuses, and the sensors themselves. All the sensors are daisy-chained to the same power supply, so pulling the terminal block would disrupt the flow of electricity to all of the sensors. Unfortunately, one of the sensors daisy-chained was controlling a relay that would isolate the battery stacks in the event of a power failure.

I rocked the terminal block off the first sensor. The habitat went dark.

I rocked the terminal block back on. The habitat stayed dark. The only glow came from the battery stacks, which were happily reporting that they were online and in good condition, only they weren’t in discharge mode.

Houston, we have a problem.

The emergency reboot procedure is simple and straight forward. Connect a set of backup batteries to the system by clicking a single fuse on. The backup batteries, in theory, would power on the sensor array, which would then detect that the batteries had been isolated, and close the relay to restore power to the habitat.

Click… Nothing.

The battery backups were dead. The sensor array couldn’t draw power, and the battery stacks weren’t supplying power to the habitat. Cue the backup backup strategy, since the backup battery and the fuel cell couldn’t turn on the sensor array, maybe powering the gasoline generator would.

The gasoline generator has a control box inside the shipping container that allows us to turn on the generator and crank the engine without leaving the habitat.

Click… Crank… Nothing.

The habitat was still completely dark, except for the lights of the battery stacks telling us that they were happily willing to restore power to the entire system, if only they were allowed to talk to the sensor array.

As an exercise in futility, we donned EVA suits and went outside to start the gasoline generator manually. That, finally, succeeded. But it didn’t succeed in powering the sensor array, no. The gasoline generator was running into the same brick wall as the battery stacks, it was ready and willing to supply power, if only the sensor array was turned on.

Time for a backup backup backup plan. The sensor array wasn’t getting power from the backup battery, it wasn’t getting power from the battery stacks or the fuel cells, and it wasn’t getting power from the gasoline generator. Time for a last-ditch effort.

The two MX-C suits have lithium iron phosphate (LiFePO4) batteries that each supply 12 volts fully charged. The sensor array needs 24 volts to start, and since we have two of the batteries we can connect them in series to get the necessary voltage. Remember, all the while we are doing this, the EVA, the wiring, the testing, the swearing, with nothing more than a couple of flashlights.

Well, it was finally time.

Hold on to your butts. No, not really! It’s a quote from Jurassic Park. — Me

An arc of light jumped as I touched the wires from the MX-C batteries to the terminals of the sensor array. The sensor array clicked, its lights flashed. The habitat stayed dark. For. Five. Breathless. Seconds.

The fans on the battery array were the first indication that something had worked. The started whirring. Next, a series of subsystems flashed to life: the inverter controllers flickered and booted, then the inverters themselves started clicking and whirring. Finally, the habitat lights blinded our dark-adapted eyes. The power was back on.

It took another hour before we had all systems up and running again. Most notably, the generator was still on, and the battery stacks  were having a bit of an argument with them over whose job it was to run the habitat power. That kept our communication system offline, but once we shut down the generator, as Ellie Sattler would say, “Mr. Hammond, I think we’re back in business.”

Right before being attacked by a velociraptor. But that’s a story for another day…

(PS, if you look carefully in the featured image, you can see the yellow lasers being fired by the Keck Observatory on Mauna Kea)

This past Saturday LogiCON 2014  played out successfully. My own presentation had a very warm reception, and the video feedback that I received from the crowd exceeded my expectations!

If you’d like to view my presentation, it’s now available on YouTube:

We had time to address three questions, summaries of which I’ve transcribed:

Q1: What am I finding the most difficult on sMars, and is it what I expected?

A1: The isolation at the HI-SEAS habitat is tremendous. The nearest human activity are the distant lights at the top of Mauna Kea. There are no plants or animals at the HI-SEAS habitat, and therefore no sound. The only sound, other than the sounds we generate, is the wind and the occasional rain storm.

Also, stop posting pictures of food on social media. Dehydrated food has a list of pros and cons, and the biggest con is the blandness!

Q2: How do we resolve interpersonal conflict at HI-SEAS?

A2: To be honest, there isn’t a lot of conflict in the habitat. There are two reasons for that, the first is that we address conflict early and often, before things can spiral out of control. The second is that we involve all of the crew in conflict, and rationalize solutions using a consensus model.

Q3: How is our water managed?

A3: We have “robotic” resupplies when out water storage runs low, and we have “robotic” grey water recovery when our waste water tanks are nearly full. These are human operations, and during these times we shut the window and turn on music to minimize the effect of the “robotic” activity has on the crew.

With limited internet capabilities, I brought along quite the stack* of books to entertain me while I’m living inside the HI-SEAS habitat. Outside of my Martian duties, my PhD responsibilities and time spent exercising I spend my spare time reading books, playing video games and watching shows and movies.

The books that I’ve chosen are a good mix of my interests and do not, by any means, represent a complete list of books that I’ve read (you can find a more comprehensive list over on goodreads). I don’t intend to read all these books, but as of today I’m setting a goal to read one book per week, so really we are only talking about 10 more books, in addition to the 4 I’ve read and the 4 I’m finishing. I’m happy to take recommendations, but as far as the HI-SEAS mission is concerned, this list is pretty much written in stone.

*The Kindle Paperwhite that I’m using measures a hair over 10 mm in thickness.

Books in Progress

The Book On The Taboo Against Knowing Who You Are – Alan Watts
Mindfulness in Plain English – Henepola Gunaratana
Red Mars – Kim Stanley Robinson
Outliers – Malcolm Gladwell

Books to Read

Starship Troopers – Robert A. Heinlein
Personal MBA – Josh Kaufman
The Jasons – Ann Finkbeiner
Tao Te Ching – Lao Tzu
The Moon is a Harsh Mistress – Robert A. Heinlein
The New Alchemy – Alan Watts
The Fault in Our Stars – John Greene
Fleet of Worlds – Larry Niven
Uncertainty – Jonathan Fields
Pushing Ice – Alastair Reynolds
Redshirts – John Scalzi
The Innovator’s Dilemma – Clayton M. Christensen
Blue Mars – Kim Stanley Robinson
Pragmatism – William James
Autobiography of a Yogi – Paramhansa Yoganada
Cutting Through Spiritual Materialism – Chögyam Trungpa
The Innovator’s Solution – Clayton M. Christensen
Man’s Search for Meaning – Viktor E. Frankl
The Startup Owner’s Manual – Steve Blank and Bob Dorf
Zen Mind, Beginner’s Mind – Shunryū Suzuki
The Lean Startup – Eric Ries
Tunnel in the Sky – Robert A. Heinlein
Time Enough for Love – Robert A. Heinlein
The Puppet Masters – Robert A. Heinlein
Friday – Robert A. Heinlein
Green Mars – Kim Stanley Robinson
Tao of Pooh – Benjamin Hoff
Te of Piglet – Benjamin Hoff

Books Read

An Astronaut’s Guide to Life on Earth – Chris Hadfield (4/5)
The Martian – Andy Weir (5/5)
The Joyous Cosmology – Alan Watts (3/5)
Spin – Robert Charles Wilson (5/5)

LogiCON is a yearly conference put on by a group of excellent individuals, celebrating logic, critical thinking, and the scientific method. This year’s conference will take place in Edmonton on May 24th, and plays host to an excellent array of speakers. The conference is geared towards general interest public topics. I highly recommend anyone in Edmonton and the surrounding areas go, tickets are on sale now.

Since I’m way out here on sMars, I’ll be uploading my presentation in remotely. After the conference, it will be available on YouTube. Here is the abstract for my talk:

Real Life on Simulated Mars

Earlier this week we ran into a little problem. Then that little problem turned into a big problem. The solution of the big problem then turned into a little problem, which today, I think we can finally put to rest.

At the beginning of last week, our water consumption was a record-breaking low, or so we thought. According to the sensor in our water thank, there was over 41% of our water remaining on Saturday night.

At first we didn’t notice that the sensor reading wasn’t moving at all, in fact, we didn’t catch it for the first 24 hours. 41%, all day long. The first indication of troubles was that there were gas bubbles in the hot water lines of the Habitat. During dishes, the water-spout would intermittently spew air with the rinsing water. That’s when we first noticed the problem.

Of course, we can’t just stick our heads outside and peer into the tanks, we require our EVAs be planned and approved before we open the door. So we submitted a plan for the next morning to go and check the physical levels in the water tanks.

As a precaution, we filled some big pots from the cold water line, and after we had about 5 gallons stored, the cold water line started spewing air as well. Just to make me scratch my head a little more, there was hot water coming out of the cold water line with the air!

The next morning revealed the true nature of our problem: we had drained the water tanks dry, even with the water sensor reporting 41%. Not only did we empty the water tanks, we also emptied the water lines running to the Habitat’s pump. The hot water coming out of the cold water tap was from the solar heater draining hot water through the cold water pipes. We were officially out of water.

Our first step was to request an emergency robotic water resupply truck to bring us more water (it’s not robots, of course, but we cover our only window so we don’t know who or what is filling our tanks). Then, with some of the water we had saved, we had to go out and re-prime the water lines.

Re-priming the water lines would be fairly straightforward: disconnect the water lines, pour water into them, reconnect them and turn on the pump. The water has to fill the lines for the pump to work, otherwise the pump is just pulling on air and can’t generate enough force to pull the water column at the other end. But we had to do it while wearing our spacesuits.

It took three of us in spacesuits to re-prime the lines, and we accomplished it in record time: under 20 minutes. While we were jubilant that we had accomplished the EVA objectives, we neglected to notice that the water tank’s sensor wire runs along the water pipe as well, and when we disconnected the water pipe, we tore the sensor’s wire as well.

So, here we are with water pumping normally in the Habitat, but now our sensor is reading -15% water levels, a true head scratcher. We’d have to go outside AGAIN to find the problem and repair it.

As an aside, the crew is currently reading ‘The Martian’ for our book club. There’s a passage (above) in here that I thought I’d share to put the next task into some context, “Ever done electronics while wearing a spacesuit? Pain in the ass.” Well, that’s enough context. We submitted an EVA request to go out and find the broken wires and solder them back together.

Delicate electronics are one thing when you have a well-lit work bench, but quite another when you are kneeling on lava rock with a 20 kilogram suit straining your back! Needless to say, the sensor wire was properly repaired and weatherproofed, and we can now read the water sensor data from the comfort of the Habitat! Yay!