10 yrs - still running: Martian rover Opportunity

It is amazing Toyota did not build it!

xkcd: Spirit

 

I think this is an amazing technological achievement. Hopefully our wisdom will grow as much as our technology has.

 

It's because the Americans have a secret cloaked satellite orbiting Mars which shoots down Russian, Japanese and European rockets as they approach.

Because they don't want the rest of us to get access to the Martian technology.

Obviously.

 

The US has it's share of failures going to Mars. It's persistence that is the difference here, not necessarily superiority.

 

Funny you should mention this mission. I was the network engineer who got the assignment to connect the mission control facility in San Diego to the NASA network. At the time, we had several design standards:

1. mission critical - requires redundant, diverse circuits, redundant, failover equipment, and realtime failover.
2. mission - redundant, specific equipment, automatic failover
3. premium - eight hour service restore with minimum response time
4. standard - next business day

Normally, we would follow at least mission design rules but the rovers were only expected to last 90 days and cost was an important (primary) criteria. So we first had to custom build to the real requirements:

• ~30 mbs bandwidth
  • mission command and status with NASA center for spacecraft communications (very sensitive!)
    • independent and redundant circuit and equipment
  • mission data sharing with the internet
• three, mission critical, 4-wire, voice channels

After looking at several approaches ranging, the final design was:

• customer site
  • top of range, Cisco 26xx router with DS-3 and built-in ethernet
  • top of range, Cisco 17xx router with three, 4-wire phone WICs and T1 interface
    • six, 'hoot-and-hollar' lines into the facility mission control room
  • small Cisco 3xxx series switch connecting the two routers to the local LAN
• NASA facility
  • center Internet router
    • add DS-3 network interfaces
    • add T1 network interface
    • add ethernet to mission communications network, isolated network
    • add ethernet to center voice communications building
  • small Cisco 17xx router with three, 4-wire phone WICs at center voice communications building
    • six, 'hoot-and-hollar' lines into voice communications network

Routing was interesting:

• Parallel routing, DS-3 and T1 which based on bandwidth, preferred 45 mbs DS-3
• QOS for T1 and DS-3
  • highest - network router and switch management
  • next - Voice-over-IP for the 'hoot-and-hollar'
  • next - mission command and status traffic
  • last - everything else

So we built this out to last 90 days of the mission . . . only it didn't end. The two rovers keep running thanks to Martian dust-devils that cleaned the dust off of the solar arrays.

Over time, technologies changed and other engineers got to upgrade and build out replacements as technology aged, circuits changed, and new requirements came down the pike. Eventually, the mission voice network went all voice-over-IP and the 4-wire voice went away (THANKFULLY!) But I got to design and build the first and remember it fondly even though it exists no longer.

Bob Wilson

ps. QOS - Quality Of Service is a way packets going out an interface are ordered so the most important get to the front of the line. It is a little more involved as there are bandwidth budgets but still fun stuff.

 

I didn't want to risk alerting them. But yes. Lots of them are in NASA (I bet bwilson4web is Illuminati), so they're happy for NASA to turn up. But they don't want the Europeans, the Japanese or the Russians to find out about their secret base on Mars.

This isn't just the reason that the Russian, European and Japanese missions never make it; it's the reason Opportunity has lasted so long. It pops into the Illuminati secret base for a quick service and polish every few months.

 

That must have been an incredibly exciting thing to have worked on.

 

Yes! I found "tail site" network connections were always interesting:

1. price/performance ratio - remote sites often had (have!) significant budget and performance challenges. So 'thinking outside the box' and 'negotiating with local support staff' becomes an art . . . sadly not always appreciated by folks with their eye on the circuits and routers costing an order of magnitude more.
2. dealing with remote staff - often treated like poorer relations, they are in isolate circumstances, away from network technology but very close to some fundamental science . . . what I call "the real stuff." For example, NASA goes to some pretty isolated spaces like a valley in Alaska that is exceptionally free from EMI for sensitive, deep-space telescopes and receivers and about 50 miles from a sounding rocket launch site . . . I supported network interfaces to both and remoteness offers some interesting challenges.
3. 'things that go bump' - a call from a remote site that we had been doing some ISDN testing revealed someone keep calling them . . . from a non-NASA number. We traced it to a non-NASA building about half a mile from my home. So I contacted the building manager and we found someone who had just vacated a rented office had 'tapped' an ISDN phone line in another office. Everything was reported to our IT Security folks . . . hummmm.
4. low serial numbers - NASA is often at the 'bleeding edge' of technology and technologies age. That leads to having to keep equipment working whose original company had been bought out and the second company had its phones disconnected. So I would bring my home VOMs and scopes to try and resurrect equipment that should have been shipped to the recycler's a decade ago. Imagine trying to make a B-36 fly without access to the documentation and nothing beyond yellowing paper, user's manual . . . been there, done that!
5. price/performance (again) - ordinarily a second circuit is put in for failover and sits there idle. But I did some early work on load-balancing across primary and fail-over circuits. To do it right, you have to address traffic management when a failure leads to circuit saturation and 'less important' traffic has to stand aside for the priority traffic.

Many of my co-workers dived into big iron, circuits, routers and switches and certainly that is important. There is serious money in that part of the network and the challenges are not trivial. But it also led to 'layers of architects' . . . like layers of paint . . . that often left some curious patch works of engineering styles. Tag-team engineering, well it is as if one cook planned a seven course meal; a second cook came in 15 minutes later only to find the equipment budget already spend; a third cook came in 20 minutes later only to find a warehouse of equipment never seen before, and; finally the newly hired short-order cook from 'Waffle House' shows up to make it all work. Sometimes I would look at what was done in the 'big iron' network side, knowing and friends will all the cooks, and after a day or so, walk down the hallway 'muttering to myself'.

NASA gets an amazing amount of new science and technology done for not a whole lot of money. Often at the leading edge, two shuttle disasters and two Apollo incidents, one fatal, also means NASA is sometimes at the bleeding edge. What we, NASA civil servants and supporting contractors, accomplish is like the poster of the duck on water whose feet are paddling 'like h*ll.'

Of course nobody else could possibly have a similar job . . . NOT! Having worked civilian, military and NASA tasks, I can't remember when I didn't do 44-48 hours each week, unpaid overtime, which my wife has never understood . . . but she still likes me.

Bob Wilson

 

It is all a hoax, just like the moon landings, simply filmed at area 51!

Icarus

 

No. Area 51 is a hoax, all staged on the moon.

 

No Mr. Anderson, not the moon.

Bob Wilson

 

"I really wish you wouldn't do that Dave!"

Icarus

 

Something I mentioned to Chuck, network technology changes rapidly so my work from 10 years ago has all been replaced:

• vendor dropped support on original routers - a router that last 5 years is 'long on the tooth'. The equipment has been replaced.
• changed requirements - the NASA IT Security folks eventually decided the 'sensitive' information had to be on a different network. So instead of a direct connection, another 'hop' was added to separate the Internet known from the 'sensitive' interface.
• Voice-over-IP replaced 4-wire audio network - all of those old NASA mission controllers were using analog, voice bridges with a separate pair for listening and talking. Otherwise, someone who got 'excited' could monopolize the audio bridge when the project team needed an more focused sharing of what is going on. But 4-wire audio has all but disappeared except for some older PBX systems that hopefully no one in their right mind continues to use . . . unless they want to pay more and get less.

Networks and equipment has the life-span of mice, Douglas Adams.

Bob Wilson

 

so now we know wy the missions did not fail..

BOB!!!

 

I have been through 3 generations of Routers and Switches in that time, but you know, the Fiber Optic and Cat 5 I ran in 1992 is still good enough. 10 gig ethernet may make that untrue soon.