[HN Gopher] The slow rise of robots in the data center
___________________________________________________________________
The slow rise of robots in the data center
Author : vanburen
Score : 73 points
Date : 2021-06-26 11:08 UTC (2 days ago)
(HTM) web link (www.datacenterdynamics.com)
(TXT) w3m dump (www.datacenterdynamics.com)
| tablespoon wrote:
| > "The server rack is more than 50 years old. There is no other
| piece of technology in data centers that has survived for so
| long," Zsolt Szabo told DCD back in 2016.
|
| Quick, someone tell him about floors, walls, ceilings, doors, and
| electric light.
| laurent92 wrote:
| Tell him about the SR-71 (entered service in 1966, and it's
| still the same instances that are flying). Most Cessnas flying
| are also 50+ years old, because if you purchase a more recent
| one, it has to comply with all the newer rules.
| tablespoon wrote:
| Yeah, but those aren't in data centers.
| donalhunt wrote:
| My sense is that robots are in use more than the article makes
| out. Anywhere there is a risk of human error or repetitive work
| that can be simplified, you'll find robots are a candidate.
|
| The main issues with robots in my experience is whether you can
| get the desired throughput while including planned and unplanned
| maintenance work, reconfigurations, etc. The upfront capital cost
| generally means you don't have the ability to have a drop-in
| replacement ready to go when a change is required. That results
| in downtime. Folk familiar with LEAN six sigma will know that
| hidden bottlenecks should be avoided or you build up large
| backlogs during downtime.
|
| Good news for datacenter engineers is that fixing robots may
| become part of your remit in the future (interesting niche given
| the mix of software and mechanical). More interesting work will
| always exist!
| dv_dt wrote:
| Are there any interesting mathematical or other tools that you
| recommend to evaluate hidden bottlenecks systematically? I
| wonder for example, about Toyota making a decision to add
| inventory buffer for chips vs other auto makers. Great decision
| in hindsight, but how did they evaluate that in context with
| other possible actions.
| pm90 wrote:
| The only way to know for sure is by experiments/simulations.
|
| The heart of any competent operations team, whether its in
| computer systems or plain old logistics, is _drills_. You can
| 't simulate every possible scenario, but you try to analyze
| and get an initial set, and the build that up as you run into
| actual production issues. What you want to avoid is running
| scripts for the first time in the middle of a disaster.
| sithadmin wrote:
| Not sure if I really buy that the Switch security bot is a
| serious project, given that over-the-top security theater seems
| to be one of Switch's main marketing points. Their employees wear
| military tactical gear; their facility walls and gates are
| designed like they're meant to house a prison; they have racks of
| what appear to be real guns and riot gear displayed prominently
| behind the security folks that check you in. Factor in the Bond-
| villain architectural aesthetic and it's really just too much to
| take seriously.
|
| Bizarro security roleplaying aside, Switch is probably the most
| professional and competent data center operator I've worked with.
| markzzerella wrote:
| I know half a dozen folks that have worked there who all
| independently described the security as a joke to impress
| clients. When I took a tour once they claimed that because of
| their government contracts they can legally commandeer fuel at
| gunpoint to run their generators if they need to, which is a
| cute fantasy.
| foobarbazetc wrote:
| Presumably the Fed/State government people who hand out the DC
| contracts like the security cosplay. :)
| dmurray wrote:
| Doesn't explain the high level of competence, though.
| fragmede wrote:
| The fact that private sector can pay much better than the
| government can seems to explain it neatly. Concepts like
| honor and service to your country doesn't pay the bills as
| nicely as selling AMZN.
| TiberiusC wrote:
| This may have been true two decades ago. Now AMZN's
| biggest customer is the government
| pram wrote:
| When I worked at Oracle our DC had similar, but less stylish
| armed guards. It was because we had equipment for Federal
| Government contracts. Another smaller (former Siebel!) DC with
| no government stuff I visited basically had zero security in
| comparison.
|
| So it might be a requirement?
| formerly_proven wrote:
| > Factor in the Bond-villain architectural aesthetic and it's
| really just too much to take seriously.
|
| -- Sith admin
| donalhunt wrote:
| [deleted]
| Firerouge wrote:
| I think you're confused, the poster you're replying to is
| talking about a company named Switch that makes robots for
| monitoring/patrolling/securing data centers.
|
| Nothing to do with packet switched networks
| donalhunt wrote:
| You are absolutely correct. I removed the post.
| systemvoltage wrote:
| Isn't security theater exactly the thing that's needed to deter
| threats? When you see guards that look like Navy Seals, it is a
| pretty strong deterrant.
| markzzerella wrote:
| Regular datacenters don't seem to be having major problems
| with technicals full of insurgents stealing hard disks, but
| maybe I'm just out of the loop.
| MR4D wrote:
| Sponsored by CBRE.
| pm90 wrote:
| It would be fascinating if we had infinitely reconfigurable data
| center network topologies. Among other things, you could have eg
| Just In Time network rebalancers that could add more capacity
| between nodes that have a lot of traffic between them. The
| reconfiguration would have to be instant, or at least take less
| time than the actual data transfer to have any benefits.
| sithadmin wrote:
| Things are already moving this direction with adoption of
| network virtualization solutions and overlays becoming the norm
| in corporate data centers.
| wmf wrote:
| There has been a bunch of research on reconfigurable optical
| datacenter networks to do this.
| klodolph wrote:
| There are various quotes about robots at Google, but the picture
| labeled "Google" is of an ordinary tape library (looks like a
| StorageTek / Oracle library). If Google has built robots, that's
| not one of them.
| trhway wrote:
| recent AWS Frankfurt event when the datacenter was flooded with
| gas to push oxygen out seems to make a good use case for the
| robots there.
| contingencies wrote:
| Why did they do that? Are they trying to kill sleeping
| sysadmins, vaguely threaten intruders, did they have a rat
| problem, or what? Seems ... inexplicable without the
| involvement of bureaucrats.
| johnohara wrote:
| Gone are the days of tape operators on roller skates, but this
| article is correct in its assessment that change in the
| datacenter moves at a very deliberate pace.
|
| Oddly enough, I found the reading pace of this article similar to
| the change it describes. Nothing negative mind you, just gently
| rolling in its delivery.
| markus_zhang wrote:
| Hi do you have any book/video talking about skaters in data
| center? I checked Youtube and only found one for switch board.
| Thanks!
| johnohara wrote:
| No. Sorry. My comment was from a conversation I had in the
| mid-80's with a couple of field service reps who worked the
| site and couldn't believe it themselves.
|
| The magtapes used to hang on row after row of metal racks.
| The operators would retrieve the tapes, hold them on their
| forearms, and bring them back to what were usually a
| centralized set of tape drives.
|
| Here's an offset into a video showing what it typically
| looked like. [0] The narrator refers to a "search tape" used
| to retrieve the person's information. The tape had to be
| retrieved from the racks.
|
| The sheer weight and overall length of the rows/racks meant
| the rooms were often lower level with vinyl flooring over
| concrete (raised tile later). Imagine walking up and down for
| an 8-hour 5pm-1am shift. Not difficult to imagine skates.
|
| Here's another video with a good representation of how it
| worked. [1]
|
| As I have come to accept, the mid-80's was pre-everything.
| Deliberate change takes time.
|
| [0]: https://youtu.be/Iddrm7mHPrY?t=111 [1]:
| https://www.youtube.com/watch?v=Cwj6pfhWBps
| dtgriscom wrote:
| A few seconds after your starting point in that first
| video, you can see a loop of tape vibrating back in forth
| in a rectangular channel. That's a vacuum column, with a
| (mild) vacuum pulling the tape loop to the right. This lets
| the servomotors that move the tape past the heads start and
| stop rapidly, while the heavy tape reels can spin up and
| down more slowly.
|
| See how the vibrating loop passes back and forth across a
| hole in the middle of the column? That's a pressure sensor.
| As the loop moves across the hole, the pressure in the hole
| changes, telling the drive whether it's time to reel in or
| dispense more tape.
|
| More info: https://en.wikipedia.org/wiki/9_track_tape
| markus_zhang wrote:
| Thanks a lot for the links!
| johnohara wrote:
| Couldn't resist. Here's another video showing a streamlined
| and "much more compact" computer room from around 1990. That
| 44GB of disk storage (DASD) probably cost $150,000 at the
| time. Somebody signed the P.O. and slept well that night. [0]
|
| Best part. You could roller skate to the music.
|
| [0]: https://www.youtube.com/watch?v=vlvUz3T4WTA
| donalhunt wrote:
| From personal experience, health and safety officers prefer
| workers to walk around facilities. Accidents while rare can
| result in hospitalisation (broken bones) when using skates,
| scooters and bicycles.
|
| I was once in a datacenter that had a golf cart to get from
| one end to another (yes - it was that long).
| markus_zhang wrote:
| Wow that's fun time~~
| tabtab wrote:
| Back in the 90's we half joked that we needed a remote controlled
| robot with 3 fingers to press Alt Ctrl Delete rather than drive
| down to the data center at 3am to reboot a jammed server. I
| sketched "R2Reboot".
| aperrien wrote:
| I'd think that a robot that could neatly wire up racks with
| cabling (ethernet, power, & fiber) would be quite doable and
| useful. Does anyone here have experience with such a project?
| sithadmin wrote:
| Having wasted way more hours of my life than I would like on
| rack-and-stack...I don't think it's possible to fully automate
| this in a scalable manner without a radical re-work of rack
| design. Which isn't going to happen any time soon, given that
| the current standards have so much momentum behind them.
|
| As things are: even in well-managed data centers, the racks
| themselves are always somewhat finnicky, with varying levels of
| precision in assembly from rack to rack that require odd
| workarounds for equipment installation more often than one
| would expect. And that's not to mention how incredibly variable
| the rack enclosures themselves can be, which has big
| implications for cable routing. And nevermind the fact that
| there's basically no standard for port placement on rackmount
| systems.
|
| Rack-and-stack labor is dirt cheap, or easily foisted off on
| your sysadmins for small deployments. I don't see a robot for
| this being competitive from a cost standpoint unless that robot
| is extremely general purpose and able to fulfill other roles.
| donalhunt wrote:
| This.
|
| Contingent workforces are cheap (minimum wage essentially).
|
| It's also cheaper to use connecters that mate on insertion
| than to use connectors that humans are accustomed to.
| Robotbeat wrote:
| Basically, a giant blade server is what a server rack
| optimized for automated server install and replacement would
| look like. Something like a cross between a vertical
| warehouse robot and a tape jukebox is what would service it.
|
| Of course, at that point it you're adding a bunch of cost and
| may be better off just designing the servers to last for like
| 10 years with enough redundancy (and hot/warm/cold spares) to
| not need any physical swapping, then replace the whole thing
| with a forklift at end of life.
| eb0la wrote:
| A friend worked in Live (now called Bing) some years ago
| between 2004-2007 I guess...
|
| He told me they had a robot like the ones from storagetek
| but for unplugging faulty blade servers and plug back new
| ones.
|
| Makes sense for large scale installations.
| wmf wrote:
| Submer seems to be working on exactly that.
| https://www.youtube.com/watch?v=zIB_BIEttFo (the robot is
| at the end)
| xyzzy_plugh wrote:
| Indeed. Even the largest DCs are putting so few racks up per
| day (still many, many racks) that throughput is rarely the
| problem.
|
| Actually getting the parts on time has been more of an issue
| in my experience, or finding out that your entire rack is
| forfeit because it fell over at the docking bay.
|
| I can count on one hand the number of times I've seen rack-
| and-stack go too slow for the consumers to actually use the
| hardware. Usually the hardware is sitting waiting to be
| provisioned for eons.
|
| The real labor is pulling bad drives/boards and rejiggering
| the network after the racks are in. So many outages are
| caused by magic hands making an error and pulling the wrong
| cord.
| bluedino wrote:
| >> I don't think it's possible to fully automate this in a
| scalable manner without a radical re-work of rack design.
|
| I think the same thing when people talking about sending a
| 'plumbing robot' to your house.
| jjeaff wrote:
| I think it would be a lot more likely that the next step would
| be a different "smart rack" where connections, or a single,
| huge bandwidth network cable would come in to the rack in a
| single place. And then have a single connection per device in
| the rack. Then software would handle the routing.
| [deleted]
| panic_on_oops wrote:
| Maybe not rack and stack, this will definitely not fully
| replace traditional datacenter operations but complement it and
| make it more streamline. Thinking about how we currently do
| disk and cable swaps,memory or even system boards misses the
| point. Of course it can't be automated because there's no
| unified standard. There needs to be one for such tasks to be
| fully automated with robots, the rack, server, network switch
| and even hot/cold aisles need to be redesigned to work with a
| standard that supports robotics arms instead of human ones. No
| more wires, maybe conductive rails.
|
| This will take infinity to accomplish (barring disruption in
| compute/storage) because -
|
| * Remote human hands are ridiculously cheap, and they can
| perform functions robots can't without extra pay or a paid for
| software upgrade (think shipment handling, etc). Only real use
| case that comes to mind is one without economic justification,
| but of necessity. Think datacenters in space or deep
| underwater, or hostile environments. So maybe innovation in
| this field will come from the government this time, and trickle
| to the private sector. Maybe.
|
| * The cost of such investment far outweighs the financial
| gains. In the very competitive cloud business, it's the
| services you offer that matters most, and the reliability of
| your systems. Sending a person with a code scanner to verify
| and do a quick disk or cable swap is not a risky endeavour.
| Unplugging the wrong cable and causing an outage never happens
| on properly designed systems, with properly set maintenances,
| so this is irrelevant (when was the last time you've heard that
| in an outage retrospect?)
___________________________________________________________________
(page generated 2021-06-28 23:00 UTC)