[HN Gopher] Apache Beam for Search: Getting Started by Hacking Time ___________________________________________________________________ Apache Beam for Search: Getting Started by Hacking Time Author : clandry94 Score : 50 points Date : 2021-01-08 17:13 UTC (5 hours ago) (HTM) web link (shopify.engineering) (TXT) w3m dump (shopify.engineering) | kordlessagain wrote: | The link for the Apache Beam project has the hostname and domain | transposed. The correct URL is: https://beam.apache.org/ | softwaredoug wrote: | Author here, thanks. I'll get it fixed. | kordlessagain wrote: | You got it fixed in no time at all! :) | waynesonfire wrote: | does it support data lineage? | runT1ME wrote: | It would be helpful if this thorough of an example could live on | the apache beam website, I think it would avoid a lot of | confusion. I certainly found the way it handles windowing with | triggers to be quite diffrent than say, Spark. | softwaredoug wrote: | Yeah I agree. I basically wrote this blog out of my challenges | learning this content. It involved a lot of code spellunking | and trial and error to figure out precisely what these concepts | meant. I do find Beam powerful, but also to be a bit esoteric | at times and difficult to follow how watermarks, windows, and | triggers all work. And we encounter sometimes unexpected | behavior that frequently causes us to revise our understanding | of these concepts. | | There's a fair amount of Stackoverflow highly voted answers out | there like "I dunno, try this trigger, see if it works" without | much understanding of how everything work underneath. Probably | cause it's tricky to grok | runT1ME wrote: | I'm not even sure the core Beam engineers understand it all! | Look at how Kafka offset acks are handled now: | | https://github.com/apache/beam/blob/master/sdks/java/io/kafk. | .. ___________________________________________________________________ (page generated 2021-01-08 23:01 UTC)