[HN Gopher] NASA can't figure out what's causing computer issues...
___________________________________________________________________
NASA can't figure out what's causing computer issues on the Hubble
telescope
Author : fortran77
Score : 282 points
Date : 2021-06-24 13:43 UTC (9 hours ago)
(HTM) web link (www.npr.org)
(TXT) w3m dump (www.npr.org)
| prog17analyst1 wrote:
| I know people say this a lot, but in this case I really think a
| (at least partial) rewrite in Rust of the Hubble software would
| be very beneficial. We could gather some of the most
| distinguished coders here in hacker-news and create a task force
| to show them the benefits of rust's memory safety.
| jandrese wrote:
| I'm pretty sure Hubble didn't crash due to a memory overflow.
| It is almost certainly a hardware failure somewhere, and Rust's
| memory safety won't help you if a failed bus or flaky memory
| chip is corrupting your data.
| nashashmi wrote:
| > Most of Hubble's components have redundant back-ups, so once
| scientists figure out the specific component that's causing the
| computer problem, they can remotely switch over to its back-up
| part.
|
| Of course they do! I wonder if they ever had to put another part
| out of service. I also wonder whether the twin of the part could
| also suffer the same failure at the same time without being used.
| mikeytown2 wrote:
| Gyroscopes are in short supply on Hubble currently
| wongarsu wrote:
| And that despite regular servicing back when we still had the
| Space Shuttle.
|
| Hubble started out with 6 gyroscopes, in 1996 they replaced
| four of them, by 1999 four had failed so they replaced all
| six, by 2009 three had failed again, so they replaced all
| six. Now they are again down to three, and one of the
| remaining ones has a defect that required some workarounds.
| The last three gyros are at least a new design that should
| last a bit longer.
| pbhjpbhj wrote:
| It sounds like a specific failure of gyros, what
| characteristic causes that failure? Are they more
| susceptible to cosmic rays or something? Do you know how
| they've mitigated that failure?
| dangrossman wrote:
| This webpage describes Hubble's gyros and the reason some
| of the earlier ones failed:
| https://esahubble.org/about/general/gyroscopes/
| Koshkin wrote:
| They should have moved to the cloud. /ducks
| tomrod wrote:
| This is a truly stellar wit.
|
| But in all honesty, how would an internet across the solar
| system work?
| hughrr wrote:
| UUCP via tightbeam.
| throw_away wrote:
| https://en.wikipedia.org/wiki/Interplanetary_Internet
| dekhn wrote:
| store and forward
| selimnairb wrote:
| UUCP
| dekhn wrote:
| I wasn't joking: https://www.quantamagazine.org/vint-
| cerfs-plan-for-building-...
| IgorPartola wrote:
| I'd it can work over carrier pigeon [1], it can work over
| long distance radio.
|
| [1] https://en.m.wikipedia.org/wiki/IP_over_Avian_Carriers
| JumpCrisscross wrote:
| > _how would an internet across the solar system work?_
|
| Realistically, each ~150 ms sphere would have its own cloud
| infrastructure. Those systems would then bridge with one
| another. So idk AWS on Earth and DogeNet on Mars.
|
| I would love for a distributed model as much as the next guy,
| but it's unlikely to happen for the same reasons that it
| isn't happening today.
| bluGill wrote:
| More than 150ms. There are currently satellite internet
| services that you can get that use geosynchronous orbits
| outside of that 150ms zone.
|
| The idea is sound, but the zone needs to be a bit bigger in
| reality. I think the moon is close enough to earth to be in
| the same zone (assuming antennas on "both sides", and
| special routing protocols to deal with day/night cycles)
| tomrod wrote:
| But if each node communicated within its 150ms sphere, and
| you place nodes 10ms (3,000 km?), could this serve as a
| mesh network?
| skylanh wrote:
| I think you have an interesting idea, but might not be
| thinking of the physics involved.
|
| > The minimum distance from the Earth to Mars is about
| 54.6 million kilometers. The farthest apart they can be
| is about 401 million km. The average distance is about
| 225 million km.
|
| Loosely, the speed of light is ~300,000km/s. So 182s,
| 1333s, and 750s as an absolute minimum length of time
| from end to end.
|
| So, there are varying orbits, that's one problem. The
| other problem is getting items into solar orbits.
|
| I didn't think of this, but now you have an even bigger
| problem of trying to keep those items in some sort of
| array that is in a direct line between Earth<->Mars.
|
| If we hand-wave away that problem, the next problem is
| that each hop is adding latency, so, the direct answer
| is: no, it makes things slower, and it's a significantly
| harder problem than just communicating across that
| distance.
| tomrod wrote:
| Array? Nah, just a very dense distribution :)
|
| But I appreciate the thought you put into this. Thank
| you!
| falcolas wrote:
| Slowly.
|
| There's open efforts to work on a protocol that would work
| with the extreme latency and packet loss. They're really
| quite fascinating.
| zahrc wrote:
| Can't wait to have gigabit speed on the moon while my home
| broadband still dies when I open a Netflix stream
| detaro wrote:
| Comms to rovers etc use store-and-forward protocols with pre-
| planning of when which node will be able to see which other
| node. (E.g. it's calculated when which dish on earth can send
| a signal that will be seen by a probe around e.g. Mars, and
| then when the probe can downlink to a rover on the surface,
| and when the replies can be transmitted)
|
| Look into "Delay-tolerant networking" for more details.
| dylan604 wrote:
| Nah, they went for heavenly computing instead
| IgorPartola wrote:
| AWS Lens: giant satellite telescope control as a service!
| BlewisJS wrote:
| This kinda already exists: https://aws.amazon.com/ground-
| station/
| marcus_holmes wrote:
| Is this becoming the tech equivalent of "not The Onion"?
| Pick an implausible service and guess if it's actually
| available on AWS...
| [deleted]
| ianbooker wrote:
| They did:
|
| https://en.wikipedia.org/wiki/Stratospheric_Observatory_for_...
| prox wrote:
| Didn't they also get a few older (Hubble type) spy satellites
| to use? I remember a story about that.
| atommclain wrote:
| My hazy memory is that the NRO offered NASA a few satellite
| bodies that they could populate with optic systems. Since
| they were originally designed for monitoring earth they
| aren't well suited for capturing data at much longer
| distances. If I recall correctly I think someone explained
| that it's like trying to peer through a straw, it works,
| but ideally you'd want a much wider field of view. And
| apparently the James Web telescope handles this much
| better.
| privong wrote:
| The NRO satellites actually have an optical design with a
| larger field of view than Hubble does. One of the NRO
| spacecraft busses is being use for the WFIRST / Nancy
| Grace Roman space telescope because of the wider field of
| view.
| atommclain wrote:
| Good info, thanks for the corrections!
| Y_Y wrote:
| They're launching one, the other is waiting for a use-
| case.
| chias wrote:
| I can't figure out what's causing computer issues _on my desk._ I
| 'm not even in space.
| ruined wrote:
| have you tried turning it off and then on again
| chias wrote:
| Yup! Nevertheless, it remains a macbook ;)
| mcc1ane wrote:
| everything's in space
| sharkweek wrote:
| "It's all in your head, but so is everything."
| slver wrote:
| Technically the universe is projected into our brain and we
| only perceive that projection. The problem is that it's a
| very shitty projection.
| Koshkin wrote:
| Good enough not to miss the bowl.
| snowwrestler wrote:
| "Everything in space is trying to kill you. And everything is
| in space."
| ssully wrote:
| Responding to this comment while debugging a CI/CD pipeline
| failure for the last hour. I'll toast to the NASA engineers
| with my cup of coffee.
| testingcodehere wrote:
| Have you tried turning it off and turning it on again?
| ortusdux wrote:
| Time to bust out the tinkertoys!
|
| https://www.nasa.gov/feature/goddard/hubble-memorable-moment...
| desktopninja wrote:
| I wonder if NASA would welcome a live worldwide collaboration to
| try solve this problem.
| lamontcg wrote:
| The probably 5 people in the world who are domain experts in
| the Hubble's control system don't really need a hundred
| backseat drivers, that won't help any.
|
| And as someone who has been invited into "war rooms" by
| managers who do the "you're smart so of course you can help
| these other smart people stuck on a hard problem" there's a
| real skill to being able to read the room and know when to back
| the fuck out of it or just shut the fuck up -- which most
| intellectuals don't have. Sit and listen for awhile and take it
| in. Then maybe take your best idea and ask a very toned down
| question. If the person who seems to be leading the
| troubleshooting instructs you on why that's wrong, throws in 3
| neighboring ideas that also don't work, with 5 reasons you
| haven't considered for why that's the entirely wrong path, then
| just nod in agreement and be quiet and see what you can learn
| from the domain experts.
|
| Peppering that team with a dozen outside "experts" is going to
| be useless because they'll just start getting really defensive
| after awhile, and even if someone winds up throwing out the
| right solution they'll probably reflexively reject it.
|
| OTOH if that team ASKS for someone who has expertise the team
| lacks and needs, then go assemble a team skilled at the use of
| cellphones and the internet to hunt that person down and drag
| them into the conversation.
| bluGill wrote:
| Give me 5-10 years at NASA working on Hubble and I can be a
| domain expert useful in the room. Until then I'm a C++ expert
| who needs to keep his mouth shut unless asked a difficult C++
| question (I wouldn't be surprised if Hubble is written in Ada
| and they can't possibly have a difficult C++ question).
| desktopninja wrote:
| I doubt we (humans) can be respectful enough to each other in
| truly a global event where an individual partakes in the war
| room. What I'm thinking here is, and I admit a rather
| simplistic view ... here is the problem; here is how to
| observe/debug, submit what you think would be the solution.
| This would be reviewed/vetted.
|
| Most likely school/college/university teams knowledgeable in
| the subject matter would be the "individuals".
| [deleted]
| beprogrammed wrote:
| Too many cooks in the kitchen.
|
| Despite the article trying to phrase it as if they have no idea
| what to do, they know there computer incredibly well, it's a
| matter of going through the steps and isolating the problem.
| desktopninja wrote:
| RE: Too many cooks in the kitchen. Very right. But sometimes
| I think humans can behave well.
| spamizbad wrote:
| Might be a hardware issue. Tin whiskers? Electromigration?
| qzw wrote:
| _pushes up glasses_ I would watch the heck out of a Twitch stream
| of their debugging /brainstorming sessions. I always loved the
| movie _Apollo 13_ , especially the technical troubleshooting
| parts.
| macksd wrote:
| Honestly that movie is a lot of why I got into math and
| engineering.
|
| Jim Lovell was undeniably a badass but I watched that movie and
| thought the heroes were the ones reading telemetry off a
| computer screen and using their slide rule to figure out what
| to do. I hope Hidden Figures does that for another generation.
| nonameiguess wrote:
| Not was. _Is_!
|
| I noticed on the last season of the Expanse that Luna
| headquarters was named after him and bothered to look him up.
| Dude's 93 and still kicking!
|
| It's amazing how well the astronaut medical screening worked.
| Unless they get killed in the line of duty, these guys are
| all living incredibly long.
| seanc wrote:
| Like John Aaron, the steely eyed missile man!
| geoduck14 wrote:
| Me, earlier this week:
|
| Did that work... no. Well, what about... THIS... still no. 3
| hours later... clear the cache?!? Aww crap
| dehrmann wrote:
| I have no idea if it would be more or less exciting than a tech
| company warroom.
| falcrist wrote:
| It's worth noting that the people in that movie were WAY more
| loud and emotional than the real NASA engineers and operators.
|
| You can see how NASA people react to tough situations by
| watching the videos of mission control during the Challenger
| and Columbia disasters. No shouting. No arguments. Just cool
| professionalism and restrained emotions.
|
| They have a job to do, and they do it well even under stress.
| "Steely-eyed missile men/women" indeed.
| bumby wrote:
| > _the people in that movie were WAY more loud and emotional
| than the real NASA engineers and operators._
|
| There are plenty of NASA engineers and leaders who lose their
| cool. I'm only saying that so people don't overly lionize
| them in a way that prevents them from pursuing a similar job
| because they feel they are somehow cut from a different
| cloth.
| erosenbe0 wrote:
| Everybody knows that when presented with the irrefutable
| evidence that the Challenger o-rings would fail, they more
| or less just let the astronauts die. Definitely cut from
| same cloth as any other org.
| bumby wrote:
| That's not quite accurate. It wasn't that there was
| "irrefutable" evidence that the o-rings would fail, it
| was there wasn't data that they would, or wouldn't, fail.
|
| "The O-rings were never tested in extreme cold."[1]
|
| There wasn't data which led to discussions about
| uncertainty, but that shouldn't be conflated with
| irrefutable evidence of failure.
|
| The obviousness of it (like many engineering failures)
| was only apparent in hindsight.
|
| "Evidence, in retrospect, points to a long period of
| time, especially based on post-flight inspections when
| the joint design weakness was 'sending a message' and the
| true potential of this message was not perceived and
| reacted to."[2]
|
| "Not perceived" isn't compatible with "irrefutable
| evidence that it would fail".
|
| [1] https://www.space.com/31732-space-shuttle-challenger-
| disaste...
|
| [2]https://www.govinfo.gov/content/pkg/GPO-
| CRPT-99hrpt1016/pdf/...
| kabdib wrote:
| Henry S F Cooper Jr.'s book _The Evening Star_ describes some
| of the remote debugging and other problem solving that was
| necessary when the Magellan probe experienced computer problems
| while orbiting Venus. It 's been a few decades since I read it,
| but it was pretty detailed and rather exciting.
| kevmo wrote:
| This sort of comment is why I still read HN.
| barkingcat wrote:
| And the people downvoting this comment is why I will stop
| reading HN.
| [deleted]
| NeutronStar wrote:
| What did that comment actually bring to the conversation?
| eitland wrote:
| It encourages others to post more these comments.
|
| Since comment scores was removed this is the only way to
| signal this to others besides the original commenter.
|
| That said it should not be overused. If it annoys someone
| I guess they should downvote it but I don't think there
| is a need to reflexively downvote every time someone adds
| a friendly meta comment.
|
| (And if people start gaming it for karma farming I guess
| it should be downvoted relentlessly until that stops :-)
| RHSeeger wrote:
| It didn't necessarily bring anything to this one
| conversation. It did, however, communicate that "this is
| the type of information that that person finds valuable
| on Hacker News". And knowing what other people in your
| social group like to hear/discuss is an important part of
| keeping that group vibrant and wonderful.
|
| So no, it's probably not as useful as the comment it was
| referring too, but it was useful (to some of us) as it
| pertains to the community as a whole.
| gentleman11 wrote:
| Occasionally, very occasionally, it's nice to read
| somebody just expressing enthusiasm instead of just
| posting a clever counter argument. It's like a spice that
| you only want a little of but that's still nice
| jorvi wrote:
| Eh, I can understand both sides of the fence. 'this is
| why I read HN' is nothing but a slightly more verbal
| '+1', but as you stated it does humanize HN and makes it
| feel more social.
|
| In terms of downvotes, what really irks me and what I
| often see is people posting factually correct
| information, but still being sent into faded oblivion
| because some sect of the community's worldview doesn't
| agree with the facts.
| ben0x539 wrote:
| I don't understand this viewpoint. Information being
| factually correct is a low bar. I have a lot of factually
| correct information that is irrelevant or misleading, or
| that I could state in a way that drags down the level of
| discourse more than it illuminates truth. Factually
| correct information is usually involved in tu quoque
| fallacies, or used to goad people into drawing false,
| non-sequitur conclusions. The Hacker News guidelines lay
| out a list of expectations for comments that go beyond
| factual correctness.
|
| If someone uses factually correct information to make a
| comment thread worse, I can see how downvotes could be
| justified.
| Teever wrote:
| "this"
| r_police wrote:
| I mean, you could watch the same news on r/news if you
| want to see that kind of comments. We don't do that here,
| and tourists always try to emulate their customes but it
| is still wrong.
|
| Just commenting as a person with many years on this
| platform.
| Dylan16807 wrote:
| Does that spice go bad if it turns gray?
| kbelder wrote:
| Conversation.
| pc86 wrote:
| It's arguably even worse than just commenting "This." At
| least that is small enough you can scan over it and
| barely even register its existence. But this fedora-
| tipping "Thank you kind sir this is the type of Internet
| Content I enjoy!" doesn't even afford you that luxury.
| omikun wrote:
| How could it have been reworded to avoid the "fedora-
| tipping" connotation?
|
| I'm being sincere here since I also appreciate book
| recommendations and I get probably half my book
| recommendations from HN.
| Freestyler_3 wrote:
| I think the point is to instead of using the keyboard,
| use the mouse to click the up arrow, and leave it at
| that. (I know how tempting it is to reply quickly to
| something, I have the urge to just post whats going on my
| mind right away unfiltered. So I am very forgiving, but
| not everyone is)
| pbhjpbhj wrote:
| That only tells the person who owns the comment that you
| appreciate it, with comment scores you're correct that
| almost all "I like this" comments are wrong, without
| comment scores then they become useful again.
| amalcon wrote:
| Worth considering that comment scores were hidden for a
| reason. Exposing that information to everyone, as opposed
| to just the comment author, does not necessarily improve
| the discussion.
| bee_rider wrote:
| This sort of comment is why I still read HN!
| jbuhbjlnjbn wrote:
| I really dislike the downvote function because it
| reinforces self-censoring. And I completely loathe the
| implementation of it, you need xxxx upvotes to downvote
| posts....I have no words.
|
| Well, in opposing it I especially read the faded comments
| and upvote any of those that are not completely
| abhorrent.
|
| Take that, ycombinator.
| beerandt wrote:
| >I especially read the faded comments and upvote
|
| This seems to happen a lot more frequently here than
| anywhere else.
|
| I'm not really sure what that says, other than people
| still read comments that are faded. Also that people
| shouldn't worry about self-censoring.
|
| I don't have a problem with downvotes or the karma needed
| to do it, but
|
| I do sometimes wish it were possible to reply to a dead
| comment, especially if you vouch for it and it's still
| dead.
|
| Sometimes they're worth defending, or is relevant in a
| non-obvious way, and sometimes the comment itself is
| discussion worthy, as it relates to the topic, even if
| it's wrong or seems trollish.
| ArcticCelt wrote:
| This youtube series of video, follow a group that restored an
| Apollo Guidance Computer that a collector basically pulled from
| the trash.
|
| https://www.youtube.com/watch?v=2KSahAoOLdU&list=PL-_93BVApb...
|
| My favorite part is when they needed the version of the
| software that was used for the moon landing but they only had
| the source code for a previous version (scanned from giant
| binder) and the hash value of the version of the landing. By a
| series of educated guesses, by reading memos and by analysis of
| the source code they modified the old code the exact way so it
| gave them the correct hash, confirming that they correctly and
| exactly recreated the original code.
|
| It's being a while and I go from memory, I might have some
| details wrong. See this video for this story.
| https://www.youtube.com/watch?v=-JTa1RQxU04
| Koshkin wrote:
| Debugging on a computer that is down is not a very exciting
| process.
| mkarr wrote:
| _pg down_
|
| Sigh.
|
| _pg down_
|
| Sigh
|
| ...
|
| Repeat for hours.
| qzw wrote:
| In the movie _Hero_ [0], two kung fu masters fight a battle
| purely in their minds. And when the mental fight was over,
| they only execute one physical move to finish the battle.
|
| Think of this as the computer equivalent of that scene.
|
| [0] https://www.imdb.com/title/tt0299977/?ref_=fn_al_tt_1
| ISL wrote:
| What a wonderful visual representation of the notion that,
| "a battle is won before it begins."
| pbhjpbhj wrote:
| Which is a Tsun-tzu reference presumably, he says don't
| enter a battle unless you have 'already' won (through
| preparation, numerical supremacy, etc.).
| avaldes wrote:
| Like the battle between Sherlock and Moriarty in Sherlock
| Holmes: A Game of Shadows?
| the_af wrote:
| Yes. A lot of Western action movies owe their inspiration
| to Chinese movies (and I suppose, viceversa). In this
| case Hero (2002) -- or a similar movie, since I doubt it
| invented this trope -- is likely an inspiration for A
| Game of Shadows (2011).
| jasonwatkinspdx wrote:
| HK movies clearly inspired the action movies of the 90s.
| barbazoo wrote:
| Both: Dance around for 10 minutes trying to out-physics
| each other
|
| Guy 1: Why don't I just poke him with the pointy bit
|
| Great scene though, makes me want to watch the whole movie.
| qzw wrote:
| Yeah, this is strictly artistic kung fu, which is
| basically high-mortality ballet. There are also many
| "realistic" martial arts films, if that's your thing. I
| enjoy both styles, depending on mood.
| the_af wrote:
| Agreed. Wuxia to be specific, which is kinda like fantasy
| kung fu and has a long tradition.
|
| It includes powers like becoming weightless, killing with
| a single movement, flying, etc.
| NetOpWibby wrote:
| This sounds amazing!
| bshep wrote:
| In case anyone wants to watch a clip of the fight:
|
| https://youtu.be/AeeoEpmyb2Y
| fishtoaster wrote:
| _Hero_'s always been one of my favorites. A lot of kung
| fu movies try to strike a balance between aesthetics and
| realism - I really enjoy a movie that picks one (in this
| case the former) and goes all in on it. It's got a fight
| that takes place entirely on the surface of a lake, and
| another that takes places in a forest of falling leaves
| that change color several times throughout the scene.
| It's an incredibly beautiful movie.
| gautamcgoel wrote:
| Ugh, such a good movie... If you haven't seen it, do
| yourself a favor and go see it.
| meepmorp wrote:
| No, but "Debugging on a computer that is down... in space,"
| does sound more interesting, right?
|
| You have a computer that you can only interact with over a
| radio link, and need to make it start working again with only
| what you know about how the system is built and a limited set
| of remote commands. Sounds like something I'd get obsessed
| with solving.
| amelius wrote:
| There is a game in here, somewhere, somehow.
| zomglings wrote:
| Paging Zachtronics (https://www.zachtronics.com/).
| only_as_i_fall wrote:
| I'll settle for the post-mortem
| pbourke wrote:
| Speak for yourself
| zepearl wrote:
| I did something like that when trying to boot my brand new
| root server in Finland a few weeks ago (tried ~50 times while
| having UEFI enabled plus mdadm raid1 on GPT partitions, never
| worked, asked support to disable UEFI, worked).
|
| Confirming that not being able to ping/connect to it during
| the failed attempts was absolutely not exciting :)
| smileysteve wrote:
| especially with extremely long response times.
| afterburner wrote:
| I think what you're really saying is you'd watch the movie
| version of this.
|
| And considering there are no life or death stakes, it still
| wouldn't be as exciting as Apollo 13.
| hungryforcodes wrote:
| That might not be true though.
|
| This guy took about 30 hours of video of him porting an 80s
| version of unix to the ESP8266. Warts and all -- live!
|
| I've started to watch it and it's fascinating!
|
| https://www.youtube.com/watch?v=cDHcGY7EzUM&t=62s
|
| You could have a whole channel with different teams debugging
| satellite technology and if you're bored, it would probably
| be quite interesting. The bigger problem is most likely
| concerns about IP and secret protocols and so on.
|
| "And now Bob will log into TeleSat123 via SSH." <We see bob
| type in root / password123>
|
| "Oups, uh..gosh we'll just go to a commercial break!"
| stackbutterflow wrote:
| I guess the last thing you want when you're debugging
| something during your work is for the whole word to watch
| over your shoulder.
| keanebean86 wrote:
| This would be cool for earth satellites.
|
| On the other hand watching a stream involving something on
| Mars, let alone voyager, would be pretty boring!
|
| Send: ls
|
| Ok let's take a 20 minute break.
| sfink wrote:
| I've worked on something like this, just a lot more mundane.
| We had Linux PCs strapped to the ceiling of various
| locations, mostly malls, together with a camera and projector
| to produce an interactive display on the floor. I had a
| couple of times where somebody would be onsite and the
| projector would be off or the display would be mangled. And
| it takes quite a while to get a lift to get up to the box (if
| it would even be allowed at that time of day), there was no
| network at that time, and all they had was a wireless IR
| keyboard that occasionally dropped keypresses.
|
| Imagine dictating shell commands, over the phone, to a
| salesperson who has no idea what half the characters are that
| you're asking him to type, and the only output signal I could
| come up with was ejecting the CD tray, which was just visible
| from the ground...
|
| (Note that the goal wasn't usually to fix things on the spot,
| it was more to triage things like whether we needed to have a
| replacement projector on hand, which was a big deal.)
| abnry wrote:
| Job Posting: NASA programmer, needs at least 1 wpm typing
| speed and experience with compiling large projects.
| gundul wrote:
| Best programmer. -1 wpm.
| Koshkin wrote:
| Only needs a keyboard with one key.
| mywittyname wrote:
| I'm so fast that I can do -127wpm. Only in certain
| software though.
| mkr-hn wrote:
| Time to bring back flowchart templates.
| diamondo25 wrote:
| The amount of preparation is much, much more than every
| other "accessible" installation. Typos are the worst to
| recover from, backspace usually doesnt exist. As I've sent
| commands to our Linux-running satellites, its usually
| prepending your commands with ctrl-c characters and a
| couple newlines at the end, just to make sure it runs and
| nothing is left in the buffer. There is also a possibility
| that commands get executed multiple times, and there are
| usually limits in transmission speed, processing speed, and
| frame length. Sending a lot of characters over a terminal
| can cause characters to be eaten, creating typos you can't
| see, affecting the commanding immensely.
| 3pt14159 wrote:
| I'm surprised theres no error correction in your uplink.
| Crazy.
| diamondo25 wrote:
| It depends on the API. If your API is "put this data over
| uart to the TTY", and the uart of the device is
| overloaded and drops characters... Or maybe mangles
| characters due to bitflips. Or what have you. Its all
| possible!
| bentcorner wrote:
| Isn't there some way to ensure that what you typed is
| what is being executed? Dropping characters from the
| terminal sounds terrifying.
|
| I don't know enough about ssh and terminals to know if
| it's possible to type "12345" and see "12345" echoed back
| to me but really what the remote session sees is "1245".
| diamondo25 wrote:
| Yes, terminals usually echo back the characters. In our
| case this would be buffered and we could request the
| buffer. But that would still take some operations. Best
| way, usually, is to send a bunch of commands in a way you
| ensure proper order of execution (eg write a file, check
| checksum of file, execute file), and make sure you can
| pull the logs afterwards.
|
| Nowadays, links and systems get easier to work with, and
| you can sometimes have a literal TTY open to the system,
| like Reactor Hello World has (
| https://reaktorspace.com/reaktor-hello-world/ ). However,
| this is over S-band, which is a 2Mbit/s link, so overhead
| for a stable TTY (or ethernet connection) is a lot less
| than using UHF/VHF.
| abnry wrote:
| Very fascinating! You haven't happened to written a blog
| post or something on this, have you? I am sure HN would
| love reading about it.
| diamondo25 wrote:
| Sorry, I did not. There are plenty of stories on the
| internet about cubesats, they get launched by
| universities even :)
| diamondo25 wrote:
| From my work experience its like this: 1. Assemble commands
| to run 2. Run the commands and see results in the 15 minute
| window 3. See if you can do more commanding in the minutes
| you have left 4. Make a new plan, wait for next pass, and
| goto 1
|
| For LEO satellites, that usually means you have 2 blocks of 3
| 13 minute passes, when the groundstation is in The
| Netherlands. For a Svalbard groundstation, you get a lot
| more, but still 13 minute or less passess.
| throwawayboise wrote:
| If you ever worked on a busy mainframe your compile jobs
| could easily be queued for 20-30 minutes. Made you much more
| careful to check for typos and do test runs of the code "in
| your head" before submitting.
| idreyn wrote:
| In case you haven't seen it: https://apolloinrealtime.org/13/
| 1911z wrote:
| Thank you for sharing, this is amazing
| belter wrote:
| Fantastic site. Thanks for sharing.
| Izkata wrote:
| Fun weirdness of even limited multilingualness: For some
| reason my brain first parsed this as "a _pollo_ in real time
| " - or, from Spanish, "a chicken in real time".
| eschneider wrote:
| I dunno. That sort of thing is exactly my job, except the
| remote devices are still on earth somewheres. What you'd see is
| me sitting in a library drinking coffee and looking at source
| code and schematics until I had an answer that matched the
| evidence.
|
| Satisfying, but not exactly must watch tv.
| moocowtruck wrote:
| you just killed any future dramatic space troubleshooting
| film scenes for me
| Izkata wrote:
| > Satisfying, but not exactly must watch tv.
|
| What's in your head could be though. That's my pet theory on
| the movie _Hackers_ , what we're seeing on the computer
| screens isn't what's actually there, it's the characters'
| mental constructs visualized.
| trothamel wrote:
| If you want to see debugging a computer in space, check out
| Apollo 13's sequel, Apollo 14. The moon landing is being held
| up by shorted-out switch that's causing the LM to abort the
| landing, and it's up to the programmers back home to figure out
| how to work around it in time to allow the landing.
|
| Apollo 13 was the story of a 'successful failure', while Apollo
| 14 shows how hard work and creative thinking can turn failure
| into success.
| cameldrv wrote:
| Don Eyles' book Sunburst and Luminary has a chapter on this,
| and Don was primarily responsible for the Apollo 14
| workaround. The book is also generally just a fantastic
| account of what it was like to develop software for the
| Apollo Guidance Computer.
| trothamel wrote:
| Also about living through the sexual revolution. It's a
| really interesting book, but as much of a memoir as a
| technical book.
| bumby wrote:
| Was this the scenario where there was a false positive
| warning light but they had no way to test if it was truly a
| false alarm? I remember attending a talk by an Apollo
| engineer who convinced the control room that the switch
| design had a propensity to a short and it really came down to
| a probability-based judgement call
| NikolaNovak wrote:
| Is there actually an Apollo 14 movie? I can find Apollo 18
| but not a Apollo 14 feature movie.
|
| I saw this but it's a short documentary and may not be what
| you meant: https://www.amazon.com/Apollo-14-Complete-
| Downlink-Edition/d...
| trothamel wrote:
| No, or at least not that I know of. I was having a bit of
| fun by declaring Apollo 14 (the mission) the sequel to
| Apollo 13 (the mission).
| cratermoon wrote:
| There's the HBO series "From the Earth to the Moon", which
| covers this with the Apollo 14 mission. Highly recommended.
| [deleted]
| NikolaNovak wrote:
| Ahh, yes; I've seen the series multiple times - agreed
| that it's great, especially for those who enjoyed Apollo
| 13. I wasn't sure if there was a different Apollo-14
| movie OP/Trothamel was referring to...
| geocrasher wrote:
| Scott Manley to the rescue: "The Computer Hack That Saved
| Apollo 14" https://www.youtube.com/watch?v=wSSmNUl9Snw
| caycep wrote:
| Someone is going to suggest unplugging it and plugging it in
| again, i'm sure
| mehphp wrote:
| See, remote work doesn't work!
| belter wrote:
| They have two computers:
|
| - First they had a DF-224 flight computer and a - Science
| Instrument Control and Data Handling (SI C&DH)
|
| Initially DF-224 between missions got installed a coprocessor:
| https://asd.gsfc.nasa.gov/archive/hubble/a_pdf/news/facts/Co...
|
| During another servicing mission they replaced it with something
| called the Advanced Computer with Intel 80486:
| https://asd.gsfc.nasa.gov/archive/hubble/a_pdf/news/facts/FS...
|
| It looks like its about 50,000 lines of code in the C and
| Assembly programming languages.
|
| https://www.nasa.gov/pdf/327688main_09_SM4_Media_Guide_rev1....
|
| Fig 5-10 is the Data Management Subsystem
|
| https://asd.gsfc.nasa.gov/archive/sm3a/downloads/sm3a_media_...
| belter wrote:
| There is also a Help Desk but its probably busy right now...
|
| "Welcome to the Hubble Space Telescope Help Desk"
|
| https://stsci.service-now.com/hst?id=hst_index
| mikeytown2 wrote:
| The cause of the failure is most likely tin whiskers [1] or
| radiation.
|
| [1] https://nepp.nasa.gov/WHISKER/background/index.htm.
| ComputerGuru wrote:
| I thought NASA no longer used lead-free solder to avoid this.
| kube-system wrote:
| I think a lot of safety-critical systems even here on earth
| have exemptions from lead-free regulations because of this...
| but even lead can form whiskers.
| bumby wrote:
| Is there evidence of either yet? I'm not sure how you get from
| "potential" failure mode to "most likely"
| mikeytown2 wrote:
| They're trying to find the broken component; what caused the
| issue is usually a handful of things in space. Those are
| usually the top causes of component failure in that
| environment.
| bumby wrote:
| Although a relatively small part of my career, I spent some
| time working within the quality arm of an aerospace org. A
| lot more on propulsion systems, but both those and
| satellites are usually required to meet J-STD specs for
| electronic builds. After a few failure investigations, you
| become acutely aware of the dangers of prematurely jumping
| to conclusions.
| NelsonMinar wrote:
| So no evidence then.
| IAmMaulik wrote:
| >"They're very primitive computers compared to what's in your
| cell phone,"
|
| I do not understand why they can't just swap out the computer for
| a better and more modern one. Am I missing something here?
| McGlockenshire wrote:
| How do you propose they perform the physical swap?
|
| The space vehicles we used for this purpose have been retired
| and we have no replacements.
| woodruffw wrote:
| > Am I missing something here?
|
| When dealing with high-latency, high-radiation environments,
| more modern isn't necessarily better: denser ICs mean greater
| susceptibility to radiation (and consequently more expensive
| hardening). They also can't exactly fly up there and swap out
| random bits on short notice -- I'm not sure if the US even has
| a the current capability to perform physical maintenance on the
| Hubble.
| LadyCailin wrote:
| They don't.
|
| > Hertz said that because Hubble was designed to be serviced
| by the space shuttle and the space shuttle fleet has since
| been retired, there are no future plans to service the outer
| space observatory.
| ellisv wrote:
| It's in outer space
| [deleted]
| 0xFFFE wrote:
| They should have gone serverless.
| lbriner wrote:
| I am no space expert but maybe they forgot to disable Android
| system updates, that's what seems to have caused my Samsung Tab
| S2 to slow to a crawl ;-)
| guilhas wrote:
| Google pushing unwanted apps
| deeviant wrote:
| Da, it's the ALIENS.
| stevespang wrote:
| Russian hackers ordered by Putin to embarrass the United States
| sabujp wrote:
| obviously cosmic rays
| stakkur wrote:
| _" If this computer were in the lab, we'd be hooking up monitors
| and testing the inputs and outputs all over the place, and would
| be really quick to diagnose it," he said. "All we can do is send
| a command from our limited set of commands and then see what data
| comes out of the computer and then send that data down and try to
| analyze it."_
|
| They've just mostly described my career.
| bencollier49 wrote:
| Still probably faster than deploying to AWS.
| belter wrote:
| "According to NASA, the 3 computers aboard the Hubble Space
| Telescope contain over 50,000 lines of code in the C and
| Assembly programming languages."
| https://www.leeholmes.com/writing/hubble.pdf
|
| I am going to go out on a limb here and post my diagnostic:
| There is some global counter that overflow as the system was
| not rebooted for a while...NASA...take it from here :-)
| thrower123 wrote:
| Maybe the James Webb will go up some day...
| tiborsaas wrote:
| https://webbcountdown.com/
| occamschainsaw wrote:
| When we power our quantum computers with room temperature
| superconductors and sustainable fusion.
|
| (all 20 years away ofcourse)
| peter303 wrote:
| Scheduled for Halliween this year, but could slip.
| jpeter wrote:
| Sounds like the plot of the three body problem
| hacker_homie wrote:
| Tri-Solaris hacked the telescope and their coving it up?
| pavlov wrote:
| Microprobes dropped by 'Oumuamua are making sure humans don't
| look where they're not supposed to.
|
| (It's UFO season! Everything is aliens again.)
| dane-pgp wrote:
| Perhaps it's time to coin the term Tsoukalos's Law of
| Headlines.
| teclordphrack2 wrote:
| Could it be a windows update?
| fouric wrote:
| I know that, at different times, NASA has used Forth[1] and
| Lisp[2] in some of their space applications. Both of these
| languages offer REPLs that generally accelerate the debugging
| process, and while your "average" Lisp might be unsuitable for
| hard real-time applications (due to the presence of a garbage
| collector, usually without the hard real-time constraints that
| you _can_ get out of garbage collectors with extreme effort), I
| wonder if they have _some_ equivalently interactive system on-
| board the Hubble.
|
| > Most of Hubble's components have redundant back-ups, so once
| scientists figure out the specific component that's causing the
| computer problem, they can remotely switch over to its back-up
| part.
|
| Wait, then why don't they just switch over each component in
| turn? The "divide and conquer" debugging strategy.
|
| [1] https://www.forth.com/resources/space-applications/
|
| [2] https://flownet.com/gat/jpl-lisp.html
| beerandt wrote:
| >"The rule of thumb is when something is working you don't
| change it," Hertz said. "We'd like to change as few things as
| possible when we bring Hubble back into service."
| AnIdiotOnTheNet wrote:
| A lesson not taught to any modern software developer. Instead
| they change things all the time for no real reason other than
| that they want to change things.
| 35fbe7d3d5b9 wrote:
| One of the best senior engineers I worked with taught me
| how to run an outage. The most important thing? _Stop_ what
| you are doing, take charge, and get everyone else to stop
| what they are doing.
|
| The best case scenario of a bunch of engineers flailing
| about on a bridge turning knobs is that you luck into a fix
| but don't know how you got there. But you're more likely to
| make things worse.
| boardwaalk wrote:
| Sounds like "locking the doors" (Space Shuttle
| disasters). Although, there was really not much to
| recover from there.
| etskinner wrote:
| I hadn't heard of this before, chilling but cool: https:/
| /www.theguardian.com/world/2003/feb/13/columbia.space...
| londons_explore wrote:
| In my experience, the best strategy depends a lot on the
| severity of the outage.
|
| If all the alarms are going off because of a loss of
| redundancy, then currently there is no outage. The
| correct move should be carefully considered, and maybe
| tested in the sandbox environment.
|
| If there is currently a 100% outage, it's best to go all
| out on trying every possible fix, because typically
| you'll restore service quicker that way. Sure,
| occasionally you dig yourself a deeper hole, but
| _usually_ it 's the best strategy.
| ComputerGuru wrote:
| > If there is currently a 100% outage, it's best to go
| all out on trying every possible fix, because typically
| you'll restore service quicker that way. Sure,
| occasionally you dig yourself a deeper hole, but usually
| it's the best strategy.
|
| Maybe. _Once an outage has happened_ , an additional 30
| minutes or hour of outage, depending 100% on the service
| in question, might be a bearable cost if it means
| preventing a domino effect of future issues caused by
| measures taken to restore the outage in a hurry.
| CGamesPlay wrote:
| > If there is currently a 100% outage, it's best to go
| all out on trying every possible fix, because typically
| you'll restore service quicker that way. Sure,
| occasionally you dig yourself a deeper hole, but usually
| it's the best strategy.
|
| Almost assuredly not. If a system hits a 100% outage,
| there are about to be a series of cascading failures by
| dependent systems. If you don't even understand which
| system is the root cause, all you're doing is testing a
| bunch of never-before-tried combinations in production
| and hoping something works.
| grumple wrote:
| This is not true. If something's working, you don't change
| it for no reason.
|
| Business requirements and requests change _all the time_.
| 90% of our work is done in response to that. The other 10%
| is fixing up technical problems due to increased scaling or
| bugs found, and then basically never do we upgrade a system
| to keep up with security updates or change to a more modern
| tech.
| AnIdiotOnTheNet wrote:
| > This is not true. If something's working, you don't
| change it for no reason
|
| Sure, there's often a reason like "we wanted to write it
| in a different language" or "we've overhauled the UI to
| be slower and more cumbersome".
| Nextgrid wrote:
| "we need to justify our front-end developers' and
| designers' jobs"
| skylanh wrote:
| Another comment keyed onto a concrete example of why not, so
| I'll go with a presumptive reason:
|
| Some of those elements will be part of the major service
| windows, and have expected operational and standby lifetimes.
|
| So if a component with two elements has a service window of 10
| years, and each element contributes to meeting that service
| window, then you've bumped your major service window from 10
| years to a significant factor less than that.
|
| e.g. the expected use profile might be: use element 1 for 6
| years or 60% of service, switch to element 2 for 4 years,
| replace both during 10 year maintenance window. Interrupting
| that by bringing element 2 up reduces that window and
| contingency plans if the service window cannot be met.
|
| I don't know, and I'm just talking out my you-know-what.
| edgeform wrote:
| > Wait, then why don't they just switch over each component in
| turn? The "divide and conquer" debugging strategy.
|
| Let's say the CPU is the actual issue, but the problem
| manifests itself in the memory module. You swap over to the
| backup memory module, and suddenly the problem vanishes!
|
| Two months later, the problem manifests again. Identical
| presentation. This time, there is no backup to switch over to
| test.
|
| You fly a Very Expensive Mission to the telescope only to find
| out the CPU was the issue, and if you had figured that out
| originally you wouldn't be up here with four memory modules.
| nucleardog wrote:
| Let's say the memory is the actual issue, but it's manifested
| itself by data being corrupted and triggering undesired
| behaviour. Unfortunately, the corrupted state has already
| been written back to persistent storage.
|
| So you swap in the backup storage module and all your
| problems go away. Until it happens again and corrupts _that_
| too.
| rurban wrote:
| Forth yes, lisp not. Lisp was only used on ground to simulate
| the rover.
|
| Also a repl in space only makes sense in earth orbit, but not
| farther away, with 8-20min waiting time for a packet roundtrip
| to Mars. Those machines really need proper and faster decision
| making (AI, think lots of `if` statements and proper modeling)
| on board, eg to perform landing or docking maneuvers. Or to
| detect and workaround radiation damage in its own circuits.
| dylan604 wrote:
| What if you backup is failing in the exact same way?
| Koshkin wrote:
| That would probably mean that the whole thing just isn't
| there anymore.
| dylan604 wrote:
| Or much more likely the same component was used as a back
| up, and is failing in a similar fashion. It's obvious the
| thing is still there.
| etskinner wrote:
| > Wait, then why don't they just switch over each component in
| turn? The "divide and conquer" debugging strategy.
|
| My guess would be that they want to try that method only if
| this debugging doesn't work. Imagine that there's an electrical
| issue in item 1 that fries item 2. If you switch over to item
| 2b, then you fry item 2b too!
|
| This is exactly what happened with the Soviet Salyut 7 station.
| They tripped an over-current protection, didn't fix the root
| issue, and remotely turned the circuit back on. A series of
| electrical shorts then rendered the entire station without
| power, resulting in the need for one of the most daring station
| rescue stories of all time:
|
| https://arstechnica.com/science/2014/09/the-little-known-sov...
| voldacar wrote:
| Wow, that's an amazing story. Thanks for posting, i had no
| idea something like that ever took place
| dmckeon wrote:
| > Mission controllers, very tired now that the end of their
| 24-hour shift was approaching
|
| Are shifts this long still common practice in US or RU space
| programs?
| baryphonic wrote:
| > while your "average" Lisp might be unsuitable for hard real-
| time applications (due to the presence of a garbage collector,
| usually without the hard real-time constraints that you can get
| out of garbage collectors with extreme effort), I wonder if
| they have some equivalently interactive system on-board the
| while your "average" Lisp might be unsuitable for hard real-
| time applications (due to the presence of a garbage collector,
| usually without the hard real-time constraints that you can get
| out of garbage collectors with extreme effort), I wonder if
| they have some equivalently interactive system on-board the
| Hubble.
|
| This is fascinating to me. Do you have any pointers to
| information/research/projects focused on hard real-time garbage
| collection? A Lisp with hard real-time garbage collection (even
| if Herculean to implement) would be fantastic.
| retrac wrote:
| I've seen at least one implementation that uses explicit
| regions; before doing something with say a bunch of cons
| operations, you allocate a region, all the evaluation is done
| in the region, it returns a result to the parent region, and
| the region is then manually dropped, freeing the space used.
| Set up another region for the next large evaluation and so
| on.
|
| Almost C style, and I'm sure just as error prone, but it
| seems like it could work.
|
| https://github.com/wolfgangj/bone-lisp
| mietek wrote:
| How about a Lisp without the need for garbage collection at
| all?
|
| http://web.archive.org/web/20020331165324/http://home.pipeli.
| ..
| retrac wrote:
| Ah, Rust's grandparent!
| baryphonic wrote:
| This is also fascinating. Thank you! :)
| astrange wrote:
| As long as there aren't cycles of course you can do
| deterministic GC, it's just reference counting. It also helps
| if the program is single-threaded since otherwise any memory
| allocation/freeing can be unpredictable (since it probably
| locks.)
| guenthert wrote:
| Not exactly real time (as in proven bounded response time),
| but a noteworthy effort: https://franz.com/services/conferenc
| es_seminars/jlugm00/conf...
| rtkwe wrote:
| > Wait, then why don't they just switch over each component in
| turn? The "divide and conquer" debugging strategy.
|
| It's better to understand the problem than to just start
| changing stuff hoping you find the right thing even
| systematically. There's not a huge rush to fix this since it's
| the payload computer and the telescope is still being
| maintained by other systems. A lot bad could happen, if the
| switching system is flakey you could maybe get stuck in a bad
| system, or if there's a number of faults you might damage one
| of the backups. Without the shuttle there's not a plan to
| service it any more so why take the risk rushing through to the
| most simplistic debugging method?
| lalalandland wrote:
| Could computer issue be related to increased solar activity/
| solar storms ?
|
| https://www.youtube.com/watch?v=YuGY8foAiHc&t=636s
| dave_sid wrote:
| This thing?
| https://static.wikia.nocookie.net/villains/images/0/07/Tumbl...
| desktopninja wrote:
| For pure comedic value: Someone deployed a K8s cluster on hubble
| and now its lost DNS because the master nodes connected to a
| starlink satellite
| Koshkin wrote:
| This is good: Starlink or such would be probably an ideal way
| to connect a space telescope (or any spacecraft) to the
| internet.
| Denvercoder9 wrote:
| Actually, it wouldn't be. Starlink orbits at about the same
| altitude, but the satellites have their radio antenna pointed
| downwards to Earth, so they can't connect with each other.
| beerandt wrote:
| Starlink has lateral uplink/downlink lasers, but yes that
| would be a complicated solution to a problem that doesn't
| exist.
| NortySpock wrote:
| Last I heard laser links were in testing, and was
| currently only being used for communicating in the same
| orbit (a single, linear string of satellites all orbiting
| in the same plane and at the same altitude)
| nonameiguess wrote:
| They actually use the NRO's Quasar relay satellites for this.
| They don't connect to the "internet," but rather to NRO
| mission ground stations, but they need that single point of
| ingress to Earth anyway because the hardware decryption
| modules, algorithms, and key-loading mechanisms only exist in
| military comms equipment, not IP routers. From there,
| provided the data itself is unclassified or can be
| downgraded, it can get to the Internet.
|
| It's arguably an interesting question whether the government
| would consider using commercial relay satellites instead of
| just the Quasar constellation, though. The data stream
| doesn't need to be decrypted on the satellite, just
| forwarded. Obviously, you can't prevent radio from being
| intercepted, so throwing in a hop you don't own doesn't
| actually add any risk. You're totally reliant on the strength
| of your encryption either way.
| dylan604 wrote:
| More likely they got it to run Doom.
| desktopninja wrote:
| Hehe: https://opensource.com/article/21/6/kube-doom
| optimalsolver wrote:
| https://www.smbc-comics.com/comic/2011-02-17
| Animats wrote:
| NASA has a Satellite Servicing Capabilities Office developing on-
| site robotic servicing capabilities for the Hubble and other
| large satellites. This is connected to the On Orbit Servicing,
| Assembly, and Manufacturing program at NASA Goddard. They've been
| working toward this for a decade.[1]
|
| No part of that effort has actually repaired anything in space.
|
| [1] https://nexis.gsfc.nasa.gov/osam/index.html
| Iwan-Zotow wrote:
| Putin?
| unnouinceput wrote:
| Quote: "Most of Hubble's components have redundant back-ups, so
| once scientists figure out the specific component that's causing
| the computer problem, they can remotely switch over to its back-
| up part."
|
| So it's time to do what every gamer does when the rig fails.
| Switch parts and see who's the culprit. And yes, I do understand
| the next quote: <"The rule of thumb is when something is working
| you don't change it," Hertz said. "We'd like to change as few
| things as possible when we bring Hubble back into service.">
|
| But between having nothing anymore, since Hubble had its last
| maintenance in 2009 (per quote: "The last time astronauts visited
| Hubble was in 2009 for its fifth and final servicing mission.")
| and have something now that definitely would fail later, I'd
| choose the latter.
| scoutt wrote:
| > At first NASA scientists wondered if a "degrading memory
| module" on Hubble was to blame.
|
| Funny enough, nobody posted the link to the article that says
| "70% of bugs are memory issues" (or something like that) yet.
| Black101 wrote:
| There's no way that's true... maybe 70% of bugs that crash your
| computer though.
| TwoBit wrote:
| "memory issue" seems overly broad or vague.
| lamontcg wrote:
| 70% of all security fixes Microsoft releases are memory safety
| bugs.
|
| https://news.hitb.org/content/microsoft-70-percent-all-secur...
|
| This isn't a security issue, NASA isn't Microsoft, and
| physically degraded memory isn't the same as a memory safety
| programming bug.
|
| I'll certainly bet that article is super popular with the rust
| crowd though.
| steveklabnik wrote:
| That article specifically is popular with the Rust crowd,
| yes, but moreover, that roughly that number (70%-80%) has
| been replicated by a multiple big tech companies, not just
| Microsoft. Chrome was another large name.
|
| (And yes you're right none of this has to do with this stuff,
| for sure.)
| behnamoh wrote:
| Off-topic but it reminded me of that story about using LISP for
| debugging a spacecraft remotely from the Earth:
|
| https://baltazaar.wordpress.com/2009/07/20/a-story-about-lis...
| bguberfain wrote:
| Extreme remote debug
| tyingq wrote:
| From a NASA post 2 days ago[1]:
|
| "The operations team is investigating whether the Standard
| _Interface (STINT) hardware, which bridges communications between
| the computer's Central Processing Module (CPM) and other
| components, or the CPM itself is responsible for the issue. The
| team is currently designing tests that will be run in the next
| few days to attempt to further isolate the problem and identify a
| potential solution. "_
|
| So "can't figure out" sounds more like "haven't yet figured out",
| but they have remaining ideas to play through.
|
| [1] https://www.nasa.gov/feature/goddard/2021/operations-
| underwa...
| jsrcout wrote:
| A better headline for the article:
|
| NASA hasn't yet figured out what's causing computer issues on the
| Hubble telescope
| literallyaduck wrote:
| I heard from an insider that it was a popup from an unlicensed
| copy of winrar, I have notified my neighbors and the braintrust
| of grandchildren, nephews, nieces and a corgi are working out a
| solution, unfortunately they are having problems getting
| minecraft, roboblox and fortnight to work with the payload
| software. The experts who originally wrote the code and then
| retired and are unable to help since the suffered from covid
| related 5g headaches. The management then outsourced the problem
| but cannot understand the contractors not because of a language
| barrier but because on Zoom debugging calls they are required to
| wear masks even though an ocean separates them. Never fear I'm
| dual booting Arch (BTW) and Windows 11 and have written a
| preliminary AI, Blockchain, ML application in Visual Basic and am
| on the case, it now routes through an Android app on the Amazon
| app store that can communicate to a Ham tower through a TNC and a
| Baofeng radio but I am waiting on a part from an overseas
| shipment. FedEx says it is still in transit on the "Ever given"
| which was routed through Ireland and was blocked by a creature
| called the waterhorse, but I gave it tree fiddy.
| programmer_an wrote:
| Programmer/analyst here and ready to help you NASA, just ask me
| and I'll clear a few hours of my agenda for you. I know people
| say this a lot, but in this case I really think a (at least
| partial) rewrite in Rust of the Hubble software would be very
| beneficial. We could gather some of the most distinguished coders
| here in hacker-news and create a task force to show them the
| benefits of rust's memory safety.
| rurban wrote:
| Then maybe lookup "stack overflow" in rust, and you will be
| delighted. https://github.com/rust-
| lang/rust/issues?q=is%3Aissue+is%3Ao...
|
| 583 closed, 156 open. That much to "memory safety" in rust.
| qayxc wrote:
| I'm not sure you understand what "memory safety" means.
| ignoranceprior wrote:
| Not sure if serious or joking. Poe's law.
| xf1cf wrote:
| From what I gathered the problem is in the _memory module_ and
| not memory itself. It would be like your RAM failing. No amount
| of rust or memory safety can help with that. Moreover you don't
| exactly re-upload the entire Hubble system from earth. You may
| be interested in looking into NASA's actual software integrity
| requirements as they are quite stringent and one of the reasons
| they use some antiquated languages. Rust is far too new, young,
| and buggy to even consider.
|
| Appreciate the seemingly ever present optimism of rustaceans to
| golden hammer the language though.
| acuozzo wrote:
| Can you clear a few hours on your agenda for me?
|
| I'd like to learn how to use Rust to work around memory
| corruption resulting from irradiating the hell out of the RAM
| in my PC at home.
|
| I'm talking enough radiation to flip more bits than ECC is
| capable of detecting & fixing.
|
| I want to do all of my programming remotely... right next to
| the the Elephant's Foot.
| bilekas wrote:
| Have they tried turning it off and on again ?
| nemacol wrote:
| Time for some percussive maintenance.
| dylan604 wrote:
| Last I heard, it's not running any version of Windows.
| Koshkin wrote:
| Looks like they may have but the thing didn't come back on.
| rajandatta wrote:
| Anyone else look at the headline and feel this is one of the
| dumbest headlines ever. It makes it sound like NASA's
| incompetent. 'Why haven't you fixed Hubble yet?'.
|
| There doesn't seem to be any nuance or respect that they're
| trying to repair an orbiting telescope that was launched 30 years
| ago and designed 40 years ago - and that people are patiently
| trying to sort through a fully autonomous system 400 miles above
| the surface of the Earth with a very large set of failure
| options.
|
| For me - huge props to NASA and other organizations that do this
| kind of work and keep these systems running for decades. I need
| to reboot Windows every 2-3 days
| bamboozled wrote:
| No, I just read it like they must have a really hard problem
| and hope they find a solution soon.
| asymptosis wrote:
| I thought they were riffing on recent UFO hype.
| spockz wrote:
| To me this title comes across as just factual and not
| diminutive in any way.
| foxpurple wrote:
| Can't work out gives a sense that they have tried everything
| and failed. "Haven't worked out yet" is still factual and
| implies that they are still working on it.
| davesque wrote:
| Yeah, I hear you on that. Another possibility is that NPR is
| trying to manufacture a sense of mystery or surprise as
| journalists often do with science stories. A bit less nefarious
| but also sort of annoying in its own way.
| JJMcJ wrote:
| Either mystery or bad puns based on the science.
|
| Extra points if you can throw "Einstein" in the headline.
| mancerayder wrote:
| > trying to manufacture a sense of mystery or surprise
|
| That's a positive spin on clickbait.
| kevin_thibedeau wrote:
| It's more than annoying. It comes part and parcel with the
| dumbing down of society.
| geoduck14 wrote:
| Most of my work could benefit from a journalist
|
| > Overworked Engineer misses semicolon. All night review
| session finds it, data gets loaded!
|
| > Management insists Jira stories be routed to new Epic.
| Team Lead spends hours learning Jira API before giving up
| and doing it "the hard way"
| tharkun__ wrote:
| Completely not related to the topic at hand but ;)
|
| I love this "spend hours figuring out the API" then
| giving up.
|
| Maybe it's just me but I have noticed that a lot (and it
| mean a lot) of devs will overestimate greatly on anything
| manual they need to do that they don't like. And spend
| hours if not days trying to automate it. Which is fine if
| it's gonna be needed again soon or over and over. But
| really, what is so bad in spending literally 5 minutes
| doing the above manually with a Jira filter and bulk
| edit? And by extension sometimes there's not even a bulk
| edit and you need to do something by clicking the same 5
| steps 50 times to acgieve something. Again 5 minutes of
| actual work. Just put on a nice fast song from whatever
| music genre you happen to love and do it. Done.
|
| Is it just me?
| jcun4128 wrote:
| I find a lot of content on YT is like that. The actual work
| is skipped hand-waved then it's every other second cut
| scene and some music on top... Idk. Started unsubscribing
| from channels lately. Still some good ones.
| mhh__ wrote:
| > Anyone else look at the headline and feel this is one of the
| dumbest headlines ever. It makes it sound like NASA's
| incompetent. 'Why haven't you fixed Hubble yet?'.
|
| I didn't get that impression.
|
| This does intrigue me - I like browsing hackernews, but I often
| get the impression that some people (not the PC specifically)
| here are either ridiculously anal about English or genuinely do
| not parse sentences the way I do.
| skissane wrote:
| > I often get the impression that some people (not the PC
| specifically) here are either ridiculously anal about English
| or genuinely do not parse sentences the way I do.
|
| People with autistic traits sometimes parse sentences in an
| overly literal or precise manner. I rarely do this anymore,
| but when I was a child and teenager I did it more often.
|
| When I'm speaking of "autistic traits", I'm not speaking just
| of people diagnosed with autism/ASD (who are of course
| represented here), but also people with broad autism
| phenotype (BAP), the subclinical manifestation of ASD. BAP is
| when you have more of the symptoms of ASD than the average
| person does, but not enough to justify an actual diagnosis of
| ASD. BAP is quite common in software engineers, and STEM
| professionals more generally, so I think there are likely a
| lot of people on this site with BAP (albeit most of them have
| probably never heard of it.) The people you are talking about
| quite possibly do have some degree of BAP, and this behaviour
| is quite possibly a manifestation of their BAP.
| jacobwilliamroy wrote:
| I think you may be projecting some self esteem issues here.
| 2Gkashmiri wrote:
| i once read an article on most probably arstechnica about a nasa
| enthusiast who found some space probe documentation in someones
| garage. he goes on to actually use that to communicate with the
| probe and issue commands. something like the booster had emptied
| or leaked or something. i am not sure what exactly it was. that
| was a fascinating read
| NortySpock wrote:
| http://spacecollege.org/isee3/
|
| Solar probe sent to comet, rebooted years later to try to put
| it back into it's solar mission.
| DudeInBasement wrote:
| Probably someone doesn't understand cache and flushing/invalidate
| SniperOwl wrote:
| "Nobody uses a computer over 20 years old" - Apple Excitives
| dangerface wrote:
| Did they try turning it off and on?
| programmer_an wrote:
| I know people say this a lot, but in this case I really think a
| (at least partial) rewrite in Rust of the Hubble software would
| be very beneficial. We could gather some of the most
| distinguished coders here in hacker-news and create a task force
| to show them the benefits of rust's memory safety.
| dang wrote:
| This is the third time you've posted this. Please stop.
| setug wrote:
| Wouldn't it be a good idea to open the source code for community
| inspection? Of course NASA would panic with Russia and China
| inspecting for "hostile" actions, but hey, if they don't know
| what to do, why not calling the expert reverse engineers of the
| world?
| beerandt wrote:
| Idk about the payload computer, but I've got to think guidance
| and operation controls would have at least some remnants of
| legacy keyhole technology, or would expose hardware details
| that might still be sensitive information, even if the software
| was a total rewrite.
| setug wrote:
| It's a 30/40-yr old code and hardware. Now that I think about
| it, the problem could be related to some "exotic" decisions
| made by that time...
| beerandt wrote:
| Lots of 30/40 year old hardware is still classified,
| especially spy technology.
| setug wrote:
| Then hiding it when it's on Earth could make sense,
| actually attacking Hubble is much more complicated that
| anything on earth, given that you can actually put your
| hands on it, connecting through a JTAG and understanding
| what's wrong (besides spying).
| _joel wrote:
| Hubble is basically a US spy satellite, pointing outwards
| instead of inwards. I'm sure there will be similar classes of
| hardware still in operation, so could be some sensitivity
| there.
| beerandt wrote:
| Yeah- that's what keyhole is: a codename for a series of
| nro/ nga spy sats.
|
| Edit: and Hubble was built from a surplus skeleton of one,
| through a government transfer.
|
| https://en.m.wikipedia.org/wiki/Key_Hole
| recov wrote:
| Interesting thought. I would say something like the hubble
| transcends politcal/governmental boundaries... although I do
| wonder how much if its software is used in other secretive
| satellites.
| jandrese wrote:
| Hubble is basically a repurposed spy satellite so it may
| still be sensitive. Although I doubt any of its sisters are
| still flying.
| hindsightbias wrote:
| They are believed to have lifespans as long as Hubble. The
| last block 4 was launched in 2013.
|
| https://en.wikipedia.org/wiki/USA-245
|
| USA-314, launched this year, is allegedly a KH derivative.
| jandrese wrote:
| Hubble has only lasted as long as it has because it was
| serviced on 4 separate occasions--although one of those
| was to fix a manufacturing defect. I don't think any of
| the spysats had service missions.
| jasonwatkinspdx wrote:
| No one knows for sure publicly, but NRO offered two more
| KeyHole bodies to NASA some years back. That's may be a
| hint they're considered obsolete.
| rurban wrote:
| On the contrary I doubt that the US military would give up
| advanced imaging technology, like reading car plates from
| space, for nothing. That's the Hubble. Nothing else comes
| close.
| jonegan wrote:
| I volunteer as rubber duck!
| datalus wrote:
| How specific is the Hubble that you can only repair it with the
| Space Shuttle? o_O
| Denvercoder9 wrote:
| It doesn't have anything to do with specificity, the Space
| Shuttle was just the only manned spacecraft powerful enough to
| get out to Hubble's orbit and back, and that had an airlock so
| you could actually access Hubble.
| rtkwe wrote:
| All the operating human launch systems are just capsules meant
| to either free fly or dock at a station, they don't have
| airlocks to let people out so using them for a Hubble repair
| would require a lot of modification and danger to use the whole
| capsule as an airlock. [0]
|
| [0] Except Soyuz I guess their orbital module would allow you
| to keep the descent module pressurized but it's still way
| outside the design so there's no telling if the module would
| remain operational.
| jccooper wrote:
| Soyuz has been used for spacewalks before, and the cabin is
| tolerant of vacuum. They haven't done that in ages, so it's
| possible that's been optimized out, but I'd suspect that
| requirement's been respected over the years.
| jandrese wrote:
| The Space Shuttle is the only vehicle ever built that can do
| in-orbit service. It's not that the Hubble is special in that
| regard, it is that the Shuttle was special.
| datalus wrote:
| So does this also mean that the ISS is no longer able to get
| serviced, or are there projects to work on in-orbit service
| vehicles?
| dragonwriter wrote:
| > So does this also mean that the ISS is no longer able to
| get serviced, or are there projects to work on in-orbit
| service vehicles?
|
| AIUI, The ISS can be serviced from the ISS if the
| appropriate supplies and personnel are sent up, but it
| doesn't have the delta-V to zip around other orbits
| servicing other satellites, so it is okay without the the
| shuttle _for itself_ , but doesn't substitute for it for
| other things needing orbital service.
| yupper32 wrote:
| ISS has airlocks that allow you to leave without removing
| all the air from the rest of the ship. Vehicles like a
| Dragon can attach their port to ISS, board, and then
| perform a space walk through the ISS's airlock.
|
| Hubble is different. It's not like it's a ship that you can
| board. So you need two things: Ability to attach yourself
| to Hubble, and ability to leave Dragon to perform a
| spacewalk. It's not clear whether you can just have
| everyone in the Dragon suit up and open the hatch. And even
| then, you still need to attach yourself to Hubble somehow.
| I think you can via the port... but then you can't leave.
| Unless you go out the other door? Can you open that from
| the inside and get out with a space suit?
|
| My rambling isn't meant to be an actual answer. It's more
| to show that it's wayyyy harder than "Let's just send up
| some people to Hubble!".
| bluGill wrote:
| These problems could be solved. However no current space
| craft is designed the right way. Maybe it is a trivial
| modification to Dragon (making it bigger...), maybe it is
| better to start from scratch. That is a question for
| domain experts who probably haven't given the idea enough
| consideration to give a good answer.
| Phillips126 wrote:
| Damn Windows updates...
|
| Getting downvoted so I guess I need to clarify I was being
| sarcastic. I work in IT and we say this a lot.
| [deleted]
| macintux wrote:
| Humor that doesn't contribute to the conversation is generally
| discouraged.
| quenix wrote:
| I'm not sure I understand this. Humor generally doesn't
| "contribute to a discussion"--it's purely that, humorous. I'm
| not sure how OP's comment contributes any less than any other
| joke one could have made.
| macintux wrote:
| Really clever humor can entertain and educate, although
| admittedly it's rare.
|
| The most common sentiment I see when people attempt cheap
| jokes is that Reddit is a more appropriate forum.
___________________________________________________________________
(page generated 2021-06-24 23:00 UTC)