[HN Gopher] How Precision Time Protocol is being deployed at Meta
___________________________________________________________________
How Precision Time Protocol is being deployed at Meta
Author : mfiguiere
Score : 133 points
Date : 2022-11-21 15:41 UTC (7 hours ago)
(HTM) web link (engineering.fb.com)
(TXT) w3m dump (engineering.fb.com)
| CapmCrackaWaka wrote:
| Why do so many tech companies seem to be releasing "secret sauce"
| for free lately? I see a lot of posts lately detailing how inner
| production systems work at large companies, and while I'm
| grateful, I'm curious why the higher ups think it's worthwhile to
| release this information.
| foobiekr wrote:
| PTP isn't secret sauce. Routers I worked on were doing PTP in
| like 2009. 1588 was standardized in 2002.
| SpaceManNabs wrote:
| > 1588 was standardized in 2002.
|
| offtopic but this sentence is beautiful to me.
| m463 wrote:
| I suspect performance metrics
|
| Many higher-level engineering positions have performance
| reviews with metrics that include some sort of industry
| expertise or external collaboration.
| jtsiskin wrote:
| Facebook in particular is very open about its data center, see
| https://www.opencompute.org/
|
| See "Commoditize Your Complement" -
| https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/ ,
| https://www.gwern.net/Complement
| maximinus_thrax wrote:
| There is no 'secret sauce' here. The real 'secret sauce' stays
| secret. The goal of articles such as this one is mainly
| recruiting, of the 'look at what we're doing here, you should
| totally want to work here' variety, and it is very effective.
| throw0101a wrote:
| > _Why do so many tech companies seem to be releasing "secret
| sauce" for free lately?_
|
| Is it really their "special sauce" though? Do these types of
| releases actually give away _how these companies make money_?
|
| In this particular case, telling the world how to get to
| nanosecond levels of timekeeping doesn't really help any
| competitors take away Metabook's revenues or profits.
| benlivengood wrote:
| How FB and Google make money isn't secret either. They have a
| lot of people looking at their web pages and apps, they
| record a lot of prior interactions with the ads these people
| see, they train giant models in near-realtime to predict ad
| quality, serve the ads that maximize expected profits in an
| ad auction, and record the subsequent clicks, views, and
| attributed conversions to convince advertisers to keep
| spending.
| ElevenLathe wrote:
| OTOH if there turns out to be some other way of making money
| on nanosecond timekeeping, then keeping it secret will help
| them get into that market before competitors can.
| eastbound wrote:
| Unless they're playing 4-D chess and the goal is to make
| competitors lose time implementing CAP theorem databases,
| nanosecond precision and React, while they're pulling
| ahead.
| xboxnolifes wrote:
| This is just spending "potential money" for engineering
| talent advertisements.
| neilv wrote:
| Maybe it's the usual reasons that we see companies releasing
| stuff, not only secret sauce?
|
| I think usually it's for company PR for various purposes
| (counteract bad press, attract new hires, etc.).
|
| Sometimes to generate a bigger hiring pool that knows the stuff
| you're releasing. (And the open source story about
| crowdsourcing contributions, which sometimes might be worth the
| costs.)
|
| I've also seen it around partnerships and customer
| collaborations and competition. Including to "commoditize your
| complement", or to kill one thing with what they'd rather use.
| (And, in industry/tech standards, corporate representatives
| often have motivation to try to bias the standard to their
| employer.)
|
| In some cases, it's for individual employees' careers. Think
| how academic and some R&D jobs want research publications, or
| how some companies want people who do "talks".
|
| Sometimes also for getting code/docs public, so employees can
| still use it when they leave.
| EwanToo wrote:
| The question for 2023 is "How many companies will be investing
| in this without a clear revenue stream?"
|
| It's quite likely we're entering a period where the current
| baseline performance of core infrastructure will be considered
| "good enough" and companies won't employ people to work on
| these general improvements.
| bsder wrote:
| Resume building for people to get their next job?
|
| Lots of these companies opened up the ability for internal
| engineers to write tech blogs for the purposes of recruiting
| since the FAANGs were all in competition with one another.
| Presumably, the VPs haven't gotten around to closing the
| pipeline yet.
| bombcar wrote:
| It's an effective marketing campaign for "you should work here"
| - VERY effective.
|
| And lots of this stuff is NOT secret sauce, it's basic business
| building-blocks that they need. It's not the advertising
| formulas.
|
| I'm sure FAANG is very VERY happy that they can just run Linux
| everywhere and don't have to pay Sun or Microsoft a massive
| per-CPU fee for everything they do.
| NBJack wrote:
| I think you'd normally be right, but Meta just laid off 10k+
| people, and is currently in a company wide hiring freeze
| until at least Q1 of 2023. Much of the rest of FAANG is
| either doing one or both as well.
|
| In Meta's case, they fired a lot of "boot campers" as well,
| some only a few days into their job and before they had a
| team. Some returning interns even had their offers rescinded.
|
| Not to sound cynical, but this article feels more like "let
| us release this to show impact and keep our jobs." Or damage
| control for their engineering hiring image.
| pcthrowaway wrote:
| People are still getting hired at Facebook/Meta, they're
| probably just targeting specific people or hiring in
| cheaper countries now
| bombcar wrote:
| "Positive press" is just preemptive damage control.
|
| And anyone knows that a "company wide hiring freeze" is
| surprisingly thawable in cases where it needs to be.
| madsmith wrote:
| 1) it makes the engineers happy to be able to speak about the
| interesting projects they are working on and give back to the
| wider community.
|
| 2) it serves as a venue for attracting other talented
| candidates who are likewise minded on working on technical
| problems.
|
| 3) when I was employed of FB it was a relatively flat
| hierarchy, which is to say there weren't that many higher ups
| to convince that this should be done.
| throw0101a wrote:
| So they state:
|
| > _One could argue that we don't really need PTP for that. NTP
| will do just fine. Well, we thought that too. But experiments we
| ran comparing our state-of-the-art NTP implementation and an
| early version of PTP showed a roughly 100x performance
| difference:_
|
| While I'm not necessarily against more accuracy/precision, what
| problems specifically are experiencing? They do mention some use
| cases of course:
|
| > _There are several additional use cases, including event
| tracing, cache invalidation, privacy violation detection
| improvements, latency compensation in the metaverse, and
| simultaneous execution in AI, many of which will greatly reduce
| hardware capacity requirements. This will keep us busy for years
| ahead._
|
| But given that NTP (either ntpd or chrony) tends to give me an
| estimated error of around (tens of) 1e-6 seconds, and PTP can get
| down to 1e-9 seconds, I'm not sure how many data centre
| applications need that level of accuracy.
|
| > _We believe PTP will become the standard for keeping time in
| computer networks in the coming decades._
|
| Given the special hardware needed for the grand master clock to
| get down to nanosecond time scales, I'm doubtful this will be
| used in most data centres of most corporate networks. Adm. Grace
| Hopper elegantly illustrates 'how long' a nanosecond is:
|
| * https://www.youtube.com/watch?v=9eyFDBPk4Yw
|
| How many things need to worry the latency of signal travelling
| ~300mm?
| [deleted]
| yAak wrote:
| The way I read it: a larger "Window Of Uncertainty" impacts
| performance. Having a smaller WOU by using PTP gave them a 100x
| perf increase in their experiments with the commit-wait
| solution for "ensuring consistency guarantee."
|
| I'm completely ignorant on the topic in general though... so
| I'm probably missing something. =)
| zh3 wrote:
| It seems a little like it might mean that software errors
| caused by race conditions can be reduced by making the timing
| windows smaller. As this is a complex area (it might not appear
| so, but it is) the pragmatic solution could be to reduce the
| windows rather than fix the issue (maybe an FB engineer can
| speak OTR).
| ransom1538 wrote:
| "But given that NTP (either ntpd or chrony) tends to give me an
| estimated error of around (tens of) 1e-6 seconds, and PTP can
| get down to 1e-9 seconds, I'm not sure how many data centre
| applications need that level of accuracy."
|
| I would _NOT_ want to be working on this project at META. Seems
| like a prime place for more layoffs.
| forrestthewoods wrote:
| > How many things need to worry the latency of signal
| travelling ~300mm?
|
| Arguably every program? The slowest part of modern programs is
| memory access. L1 cache memory access is ~1 nanosecond and RAM
| is ~50 nanoseconds.
|
| Is 49 nanoseconds a lot? No, if you do it once. Yes, if every
| line of code pays the price.
| BaconPackets wrote:
| I would be really curious to drill down into one of these
| problems. Is super precise really the solution?
| klodolph wrote:
| I would also love to see an explanation of "why do we need this
| much accuracy?" that actually goes through the derivation of
| how much accuracy you need.
|
| Some of the justification for Google's TrueTime is found in the
| Spanner docs:
|
| https://cloud.google.com/spanner/docs/true-time-external-con...
|
| Basically, you want to be able to do a "snapshot read" of the
| database rather than acquiring a lock (for reasons which should
| be apparent). The snapshot read is based on a monotonic clock.
| You can get much better performance out of your monotonic clock
| if all of your machines have very accurate clocks. When you
| write to the database, you can add a timestamp to the
| operation, but you may have to introduce a delay to account for
| the worst-case error in the clock you used to generate the
| timestamp.
|
| More accurate timestamps -> less delay. From my understanding,
| less delay -> servers have more capacity -> buy fewer servers
| -> save millions of dollars -> use savings to pay for salaries
| of people who figured out how to make super precise timestamps
| and still come out ahead.
|
| This kind of engineering effort makes sense at companies like
| Google and Meta because these companies spend such a large
| amount of money on computer resources to begin with.
| jasonwatkinspdx wrote:
| Meta uses some variations on Hybrid Logical Clocks, which are
| very similar to TrueTime, so yes this does apply. Besides
| performance they very much want to avoid consistency issues
| that could result in a security breach, eg, if I block Alan
| and then post "Alan is a dookie head" you don't want some
| node seeing the second event first. Well really the bigger
| concern is someone spots this as a potential vulnerability
| point and scripts something.
| pclmulqdq wrote:
| I have never seen NTP in a datacenter of reasonable size get
| below an error of 1e-4. PTP without custom hardware can easily
| get 3 orders of magnitude better.
| jeffbee wrote:
| Great article, but I have to quibble about the casual ease with
| which they wave away the complexity of accessing these
| timestamps. The Linux APIs for accessing them is totally absurd.
| See for example the gRPC code that associates hardware (possibly)
| timestamps with messages, for tracing and other reasons. You have
| to re-arm the timestamp option before every send. And the whole
| concept of a timestamp for an ethernet frame maps poorly to
| stream sockets.
|
| https://github.com/grpc/grpc/blob/master/src/core/lib/iomgr/...
| gpderetta wrote:
| I don't see the code rearming it at every send nor it matches
| my experience. The code sets the sock option lazily, and only
| once . The option is needed to configure the driver and
| possibly the card. Then the timestamp option need to be passed
| to the send/recvmsg call to phisically retieve the send/recv
| timestamp.
|
| Far for being absurd it seems a very straightforward extension
| of BSD socket API.
| epgui wrote:
| This is incredible. If I knew I'd be working on things like this
| (as opposed to optimizing ad revenue or "user engagement" by A/B
| testing button copy), I'd possibly consider working at Meta.
| thatfunkymunki wrote:
| If this is the only thing stopping you, you should apply and
| tell the recruiter you would like to work in infra. People that
| don't want to work on product stuff don't get "roped into it",
| I know people who have worked at Meta/FB for years and have
| never touched user-facing code or ads stuff.
| capableweb wrote:
| > Meta/FB for years and have never touched user-facing code
| or ads stuff
|
| Does that really matter in the end? If you're against writing
| code for ads/optimizing for quick dopamine hits, is it really
| so different to be writing the code that shows the ads vs the
| code that sets up the infrastructure to serve the ads? Or
| even infrastructure to support the developers who write the
| code for the ads?
|
| If you don't want to support the ad-driven internet, don't
| work for companies who's entire business model is extracting
| as much data from users as possible in order to show the
| "right" ads.
| epgui wrote:
| Yeah that's exactly my concern!
|
| I want to contribute to building cool things, and building
| a really cool component becomes much less inspiring if it's
| only used to steer nuclear missiles. (dramatic but not
| unreasonable example)
| elcomet wrote:
| You could apply to work on the research infrastructure
| (for instance for AI)
| capableweb wrote:
| And instead of writing infrastructure for the developers
| who write code for ads, you now write AI software/do AI
| research for a company in order to attract enough talent
| so they can hire developers to write infrastructure for
| the developers who write code for ads?
|
| Or you know, just don't work for companies who tend to
| collect as much user data as they can?
| jtsiskin wrote:
| At Meta you choose what team you want to work on during a 3
| month "bootcamp" period, very different from Google, Amazon,
| etc. There's enough teams solving hard distributed system
| problems that you're guaranteed to never need to think about
| button copy
| loeg wrote:
| You can also be directly hired onto a team, skipping bootcamp
| (some roles).
| scheme271 wrote:
| There's other places that are working on this although they
| probably don't pay as much as Meta. CERN and a few other
| european physics projects having been using this to create the
| white rabbit project over the last decade or so (
| https://en.wikipedia.org/wiki/White_Rabbit_Project ).
| milleramp wrote:
| Companies have been using PTP to synchronize (frame sync)
| networked cameras for many years. It is much better than wiring
| up a separate ttl or differential signals. It is amazing the
| accuracy you can get with this protocol.
| RockRobotRock wrote:
| About a year ago, Linus from LTT collaborated with the Facebook
| team to make a video about the subject:
| https://youtu.be/JK3eTGkX6qY
|
| Very cool!
| qwertox wrote:
| Thanks for sharing! It's really relevant and a nice overview of
| the problem.
| vanshg wrote:
| > It's absolutely essential to ensure an open sky and install a
| solid stationary antenna
|
| What effect, if any, might a major earthquake have on the time
| calculation?
| waynesonfire wrote:
| I remember Google doing this many years ago, does anyone recall
| the papers that describes distributed storage systems that
| benefit from highly accurate clocks? Was it spanner or the next
| generation version of it? I can't remember. I recall finding it
| fascinating but out or reach since I think they were putting
| hardware based atomic clocks on their systems, sounded super
| proprietary.
| anonymousDan wrote:
| Yeah, Spanner and the Truetime API. Would be interesting to
| know whether there are many differences between the
| implementations (or indeed the abstractions offered). I always
| wonder whether there are rare occasions where such systems fail
| (e.g. the bounds reported don't actually capture the true time
| due to some bug), and what the resulting implications are for
| systems built on top of them.
| s1mon wrote:
| It's interesting that Meta is doing such serious low level R&D
| like this, but it's funny because they can't even keep their
| notifications in sync with their web site. I get a notification
| (from the mobile app, I think) that Joe has a birthday today, I
| go to the web page to wish them a happy birthday, and despite
| reloading, the web site still shows the birthdays from the
| previous day. Then I have to dig around more to find Joe's page
| to post a HBD.
|
| So as user, this level of time synchronization seems to be many
| orders of magnitude of over kill. I'm sure at some level it is
| actually important, but it seems like the database sync or
| whatever is going on with notifications is woefully lagging.
| kazinator wrote:
| It is not accurate to say that PTP is a predecessor to NTP.
|
| You need both.
|
| PTP synchronizes clocks to a ridiculous precision, like down to
| nanoseconds. To do that, it uses support in the ethernet
| hardware. Hardware adds precision stamps to the PTP-related
| ethernet frames, so that the time calculations are free of
| jitters induced by the higher network stacks.
|
| A cool thing is that PTP-aware routers/switches can actually
| rewrite the timestamps in PTP packets to subtract their own
| propagation delay accurately.
|
| Something in the network has to use NTP, if it is important for
| the devices to have the correct calendar date and time; you don't
| use PTP over the Internet to get time from a remote server.
|
| The real-time clock being synchronized isn't even the system
| clock in the first place, but a high resolution RTC that is in
| the network adapters and switches.
|
| As an additional utility on top of the PTP stack, there is the
| possibility of synchronizing the host system's clock with the
| help of PTP.
|
| PTP is used for things like generating precise signals in certain
| applications. E.g. Some devices that have PTP support can be
| programmed to generate periodic pulses which are clocked by the
| sychronized real-time clock. With PTP, two such signals from
| separate devices on the network can align very precisely. Such an
| application doesn't care about the time of day, just that the
| signals are synced.
| willis936 wrote:
| Just because high precision tasks don't care about time of day
| as much, PTP still uses TAI timestamps. The only time of day
| information missing is leap seconds, and those were a mistake
| that is being corrected. You don't need NTP for time of day
| when PTP is used.
| zie wrote:
| The point being you need a source of truth for PTP to figure
| out what the current date and time are. You won't be pushing
| PTP over the public internet, that's ridiculous. You can use
| a GPS/sat sync or atomic clock or NTP, doesn't matter, but
| something has to tell PTP what time it is. If you don't, then
| you will be subject to quartz crystal drift. It may or may
| not matter to your application(s), but for most people that
| care about time, they want the TAI timestamps to be at least
| reasonably accurate.
| bombcar wrote:
| I assume the reply to the write includes this new super accurate
| timestamp? So the read server knows it's behind a bit as the read
| request includes it also?
| zzixp wrote:
| I've been seeing a lot more of these Meta engineering articles on
| HN - I'm a fan! If you're writing them, keep up the great work!
| cheriot wrote:
| Agreed. I wonder if the motivation is competing for resources
| with Meta paying more attention to costs now.
| synaesthesisx wrote:
| Yes! I had no idea Meta actually invested in
| innovation/engineering like this. I always assumed most of
| their work was just in ad tech & "metaverse".
| coolestguy wrote:
| That's why they are promoting themselves so much on HN now -
| to encourage talent to come to Meta
| pcthrowaway wrote:
| I mean, they've created a whole bunch of the tech required to
| support their scale (React, Jest, PyTorch)
| [deleted]
___________________________________________________________________
(page generated 2022-11-21 23:00 UTC)