[HN Gopher] 25 Gigabit Linux internet router PC build
___________________________________________________________________
25 Gigabit Linux internet router PC build
Author : secure
Score : 421 points
Date : 2021-07-10 11:49 UTC (11 hours ago)
(HTM) web link (michael.stapelberg.ch)
(TXT) w3m dump (michael.stapelberg.ch)
| ksec wrote:
| The most surprising details was [1] from init7.
|
| >Compatibility requirements There is no obligation on you to
| procure the hardware through us, and the hardware shown here is
| not the only possible hardware for you to use. There are also
| other compatible products, as long as the requisite <<bi-di>>
| fibre optic technology conforms to the following specifications
| (recommended: Flexoptix, more router information):
|
| No Modem / ONT. Just a Router with compatible SFP optic. I wish
| more ISP do that. And not force me to use your crappy ONT or Wifi
| Router. They could of course go another router and provide actual
| decent ONT or WiFi Router. But the chance of happening, or they
| care about quality is slim.
|
| [1] https://www.init7.net/en/internet/hardware/
| tw04 wrote:
| I'd strongly recommend people take a look at the supermicro (or
| your vendor of choice) Xeon-d or AMD embedded 3000 series.
|
| Lower power draw, very quiet and more than powerful enough to
| push 25gbit with cpu left over for VMs.
| sydney6 wrote:
| To get a grip around the numbers, i quote from George Neville-
| Neil's Talk at BSDCan '15 "Measure Twice, Code Once" [1]:
|
| - 10 Gbps is 14.8 million 64 byte packets per second - 67.5 ns
| per packet or 200 cycles at 3 GHz - Cache miss is 32 ns
|
| [1] https://www.youtube.com/watch?v=LE4wMsP7zeA
| eqvinox wrote:
| You're not running a continuous stream of 64-byte packets in a
| home or SME setup. Also, assuming a 1:1 mapping to packet
| processing is a false dichotomy these days, NICs are doing an
| _unbelievable_ amount of preprocessing, particularly grouping
| related packets together.
| sydney6 wrote:
| No, of course not. A good starting point for real world
| performance benchmarking could be e.g. IMIX [1].
|
| The example above represents the solely theroretical worst
| case as a means to establish a baseline for performance
| benchmarking.
|
| Anyway, if you are referring to HW offloading capabilities of
| "modern" NIC's, using techniques like LRO would break the
| "end-to-end"-principle of a router.
|
| [1] https://en.wikipedia.org/wiki/Internet_Mix
| great-potential wrote:
| Not sure why you buy at fs.com when you have online stores in CH
| that are selling much cheaper:
|
| https://www.microspot.ch/de/computer-gaming/pc-komponenten/s...
| nimish wrote:
| Without PPS measurements its hard to say how good this is.
| 1.488MPPS is needed to saturate a gigabit connection and that
| usually needs some tricks like poll mode drives/dpdk/etc on
| commodity hardware.
| Mave83 wrote:
| why no just buy a good nvidia/mellanox nic and route directly on
| it? Modern NIC's are amazing and you can offload everything if
| you like.
| ju-st wrote:
| but it is only offloaded to some ARM cores and not done in
| hardware? can they really route and NAT 25gbit? and the manuals
| for connectx-5 or nvidia bluefield don't even mention ipv6?
| kortilla wrote:
| The article leaves out the actual network packet processing. Is
| this being done by the kernel or is dpdk being used?
| specialist wrote:
| Very cool post. Thank you.
|
| This article, and its many links, are helping noob me learn how
| to ask about I/O perf.
|
| TechEmpower benchmarks HTTP servers. While contestants continue
| to improve, I've long been curious what the _theoretical_ fastest
| HTTP server _could_ be. https://www.techempower.com/benchmarks/
|
| Basically, an update to the "C10K problem" for the year 2021.
| https://en.wikipedia.org/wiki/C10k_problem
|
| A bit like how Daniel Lemire initially thought about parsing
| JSON. Paraphrasing: Why can't a JSON parser run at wire speed?
| "Parsing JSON quickly: early comparisons in the wild"
| https://lemire.me/blog/2019/03/02/parsing-json-quickly-early...
|
| --
|
| Also, I really appreciate including the sound and cooling
| considerations. It's just great seeing the process of system
| design accommodating (balancing) multiple goals. Bravo.
| goodpoint wrote:
| > Basically, an update to the "C10K problem" for the year 2021.
| https://en.wikipedia.org/wiki/C10k_problem
|
| No, routing, natting and terminating connections are very
| different problems.
|
| Especially if we consider different implementations.
| timvdalen wrote:
| I spent a couple of hours last week trying to set up a fiber
| connection in our new office. Since I didn't want to use consumer
| hardware in an office setting (and I already had a router that
| was compatible), I opted for getting my own hardware (a media
| converter).
|
| I had _some_ prior knowledge of networking, but had never messed
| around with VLAN and MTUs before. Luckily I found this[1] gist
| for a comparable setup that saved me. I'm still only getting
| 350Mbps where I should be getting 1Gbps, but I _think_ that's
| just due to the bad networking chip in my cheap chromebook. When
| I move all my stuff to the office, we'll see what we can top out
| at...
|
| All this is to say, I definitely respect the effort the author
| put into this and it pains me to find out that my new SFP setup
| is already obsolete :).
|
| [1]:
| https://gist.github.com/Ruben-E/abb9a4a872a7c4ffff058ae291ef...
| clktmr wrote:
| The most interesting part for me came at the end: It runs
| router7+gokrazy, a self-written pure-Go userland instead of the
| traditional GNU userland.
| daneel_w wrote:
| At some point the idea of "use more case fans, then all of them
| run at lower RPM" simply turns on you and instead produce more
| noise. For the low wattages of this build a single case fan
| would've been sufficient to keep PSU and CPU fan at silent
| levels.
|
| I have a few more watts in my PC case, but not a single case fan,
| and the machine still hovers at a comfortably quiet level at 1
| meter's distance. No temperature issues.
| secure wrote:
| Yeah, I'll probably turn off some fans to see how the
| temperatures react. That's for some time in the future,
| though... :)
| cedricgle wrote:
| I wish P4 fpga boards weren't so expensive for hobbyist. You can
| do fun stuff with them and create a custom network pipeline. It's
| totally overkill for a home network (they are used to create
| datacenter fabric) but it's nice and shiny :)
| pmorici wrote:
| Kind of surprised there wasn't much attention paid to the
| software. My understanding is that with stuff like DPDK and fd.io
| you can get much better performance without going to extremes
| with your hardware. Netgate TNSR is one product that puts all the
| opensource pieces together to make it easy but it is all
| opensource software so should be usable by the average home user.
| h43k3r wrote:
| Author has written his own router software and probably wants
| to continue to develop it. They are much more pro than your
| normal average home user.
| floatinglotus wrote:
| With a Xeon, a SmartNIC, and DPDK you can hit 100 Gbps.
| virtuallynathan wrote:
| I was going to say that... with fd.io this should be pretty
| easy these days.
| formerly_proven wrote:
| > I read on many different sites that AMD's current CPUs beat
| Intel's CPUs in terms of performance per watt. We can better
| achieve goals 2 and 3 (low noise and low power usage) by using
| fewer watts, so we'll pick an AMD CPU and mainboard for this
| build.
|
| Unfortunately many reviews are very misleading here. Zen 2/3 CPUs
| have good performance per Watt, that's true. But for a machine
| like this, which will be mostly idle, this is not the interesting
| metric and Zen 2/3 systems show that you can combine good perf/W
| with poor idle power consumption (which is not true for their
| monolithic APU brethren, which are used in laptops).
|
| One of the biggest idle power hogs for these is the IO die, so
| make sure that XMP is disabled and the memory uses one of the
| slow JEDEC timings. This should be fine for a router. Check that
| the SoC/NB voltage is set to 1 V or less. Some boards set this
| higher. In the AMD CBS section of the board firmware there should
| be an item "SoC OC Mode" somewhere. Disable it. Some boards allow
| you to set a new PPT (package power target), but it's worth
| pointing out that values which are too low will make the CPU very
| slow because it essentially forces all cores to very low power
| states in order to meet the PPT since the CPU can't influence the
| baseline power (due to fabric and I/O die). The upside of using a
| reasonable PPT of e.g. 50-60 W is that you reduce power
| consumption if some errant task hogs the CPU.
|
| These settings make a big difference, but only if the CPU is
| _really_ idle. Even fairly light loads (e.g. on a desktop, moving
| the mouse on the background) has everything rev up. In deep idle
| (nothing running at all, no user interaction on a desktop) you
| might get a Zen 2 /3 CPU down to around 20 Watts, but as soon as
| anything is happening at all we're straight back to the 40-70 W
| region.
|
| Using an Intel system for this would have likely saved 10-20 W.
| iforgotpassword wrote:
| Yes I can unfortunately confirm this. Don't know how much has
| changed, but about a year ago my current employer evaluated
| about 10 desktop pcs from dell, lenovo, hp and the likes as all
| staff was supposed to get new machines. One of the important
| criteria was power consumption. The few AMD systems that were
| among the contestants had absolutely ridiculous idle
| consumption and weren't even considered any further.
| jiggawatts wrote:
| Using local retail electricity prices, the idle power draw of
| 20-40W for the AMD CPUs comes out to $20-$40 per annum.
|
| If THAT breaks the bank, run screaming from your workplace as
| fast as you possibly can. The bean counters can't count, and
| they're being penny wise and pound foolish in the worst
| possible way.
|
| AMD CPUs run circles around anything Intel makes. Total
| performance, performance per core, price/performance, and
| performance per watt.
|
| Giving all of that up for... $20? Seriously?
| ta988 wrote:
| At work people leave their computers and screens on (and
| unlocked most of the time)... (edit): I tried to educate but
| nobody really care...
| zymhan wrote:
| Hence the need for lower idle power consumption
| judge2020 wrote:
| Maybe in your office, but when I was in the office we had a
| culture of pressing Win+L or the dedicated lock key on the
| keyboard and letting Windows turn off the display after a
| minute of being on the lock screen.
| chefkoch wrote:
| That's what Group Policy is for.
| ta988 wrote:
| Yes I tried to explain both the energy and security
| reasons behind locking and hibernate/sleep.
| nfriedly wrote:
| At my job people tend to post silly things in slack, like
| an "I love ponies" giphy image, when they see _someone
| else 's_ computer unlocked and unattended.
| [deleted]
| uvesten wrote:
| Great :) I've been wondering what equipment to get when I upgrade
| to the 25 gbit connection too. Maybe a custom build is the way to
| go after all!
| phreeza wrote:
| One day init7 is going to poach you :)
| spicyramen wrote:
| Good post, bring me back old memories where I used to setup my
| own PCs back in high school. I think the graphics card and GPU
| are really not required for a router. Few years ago there was a
| Linux project called zebra router which was deprecated in favor
| of https://www.quagga.net/
| InTheArena wrote:
| I have a Unifi UDMP, which while I think is probably the best
| prosumer option right now, falls way short due to a PPPoE problem
| that limits fiber connections to under 500mb/s, and the lack of
| load balancing across multiple link networks (which I totally
| admit, is a first world problem).
|
| I am thinking about building my own, but then comes the
| maintaince of all of the hardware / software. For me that would
| be fine, but for my family, it would be a total PITA to manage,
| and any downtime would be horrible.
|
| So, stuck with UDMP for the moment, until they either fix their
| problems, or alternatively I decide to bite the bullet and build
| this on top of linux.
| lbotos wrote:
| Can you link or elaborate more on this PPPoE problem? i have a
| UDMP on the way, and I don't think I'll be affected by it with
| my simple network design, but also not sure.
| toast0 wrote:
| PPPoE de-encapuslation is likely not hardware accelerated (or
| can't be combined with other hardware accelerated packet
| processing) or if that platform is more PC like than I
| thought, it may be that the PPPoE de-encapsulation is single
| threaded either as a missed software feature or because the
| nic can't separate it into multiple queues.
|
| PPPoE is one of the worst network protocols ever, and there's
| no reason it should have been implemented on fiber. I don't
| really understand why it was implemented on DSL either; maybe
| some bizarro way to try to prevent theft of service?
| InTheArena wrote:
| It's determined by your WAN provider. Centurylink and google,
| (AFAIK) leverage PPPoE connections, and the network traffic
| there seems to be limited around 600mb/s.
|
| I replaced the connection with a cable drop, which I get
| 1.2gb/s down on.
| subhro wrote:
| Odroid N2?
| InTheArena wrote:
| Need a lot more power then that to handle 2x 1GB/s
| connections.
| coder543 wrote:
| What?
|
| The UDM Pro[0] has a quad-core Cortex A57 CPU running at
| 1.7GHz, according to the spec sheet I'm looking at right
| now. It has 4GB of RAM.
|
| The ODROID-N2+[1] has a quad-core Cortex A73 CPU running at
| 2.4GHz. It also has 4GB of RAM.
|
| The UDM Pro is like half the performance of the ODROID-N2+,
| accounting for architecture and frequency differences,
| so... I'm not sure what you're getting at?
|
| The ODROID should be _more_ than enough to handle a gigabit
| connection if the UDM Pro is even halfway capable of it,
| from a power perspective. Connecting more ethernet ports to
| it without an exposed PCIe connector is going to be clunky,
| but that 's not the issue you pointed out.
|
| [0]: https://dl.ubnt.com/ds/udm-pro
|
| [1]: https://www.hardkernel.com/shop/odroid-n2-with-4gbyte-
| ram-2/
| eqvinox wrote:
| No, you really don't. Though the problem in this case is
| the N2 only has 1 GbE port and adding more through USB,
| even 3.0, is notoriously bad (regardless of the platform.)
|
| (source: I benchmarked my N2+. It'll route and NAT 1 GBit
| quite leisurely.)
| kazen44 wrote:
| > falls way short due to a PPPoE problem that limits fiber
| connections to under 500mb/s, and the lack of load balancing
| across multiple link networks (which I totally admit, is a
| first world problem).
|
| how are you achieving load balancing? using LACP? not having
| working LACP seems kind of unacceptable in my opinion for suchs
| an expensive device.
|
| Also, older enterprise grade routing hardware can be had for
| very cheap.
| InTheArena wrote:
| It has a failover over the two WAN ports, but no load
| balancing at all across them.
| j1elo wrote:
| I'm always amazed by people who don't bat an eye on the
| perspective of having a home server, sucking up electricity 24/7.
| In this case replacing a typical router (which consumes like a
| lightbulb), with a full-fledged PC (probably consuming like x100
| the power).
|
| I guess some people around the world have quite cheap utility
| bills! For me, it's either a Raspberry Pi type of power
| consumption, or else a server that only powers on when needed.
| But I haven't learned yet how to do the latter, if possible at
| all.
|
| In fact this is a nice place to ask: how would you build a
| "something" that monitors the network for packets sent to
| powered-off machines, then somehow caches the request, powers the
| destination machine On, and finally lets the request continue to
| its target? Has this been tackled anywhere? There must be tons of
| people wanting a homeserver but living in places where
| electricity has a considerable cost...
| antonzabirko wrote:
| I mean if it's a problem just solve it. Yes, the folks in texas
| who have to pay 10x for electricity probably shouldn't get in
| if thuy can't afford it. Also, get solar/wind if you are
| concerned aboit the impact.
| zbrozek wrote:
| I have a symmetric 10 gbps connection at home. I have an EPYC
| machine that plays host to a number of virtual machine guests
| for various tasks. One of them is running opnsense to be my
| router. It's not fast enough to route at line speed, but it's
| close enough that I don't care. A consumer router or a pi or
| something would be far slower, and I would start to care.
|
| Having a tolerably powerful computer doing this means that it's
| also my web server for several sites, stores local backups and
| handles offsite backups, acts as my print server, and hosts a
| Windows virtual machine for using proprietary software (e.g.,
| for my label printers or firmware updates for random widgets
| like my Lutron light system).
|
| Quiescent appears to be something in the neighborhood of 40
| watts. It's not nothing, but it's acceptable, especially for
| the utility. I don't pay for a VPS because I have the bandwidth
| and the capacity to self-host everything I want. I spend $8/mo
| in electricity to run that machine, and that will drop to zero
| when my solar array and battery become functional.
|
| If the power consumption were 5x worse I would probably not
| have gone this route.
| tyingq wrote:
| His starting requirements include one pci card for 25gb and
| another for 4 port 10gb. I don't think there's any low power
| way to do that. Any motherboard with enough slots, CPU power,
| and PCI lanes to handle all that aggregate bandwidth isn't
| going to be low power setup. If you underpowered it, then you
| might as well back off of 25gb/10gb.
|
| So it's not really a home server. It just happens to be in a
| home.
| FpUser wrote:
| >"I'm always amazed by people who don't bat an eye on the
| perspective of having a home server, sucking up electricity
| 24/7"
|
| Some people simply run business from home and it is legitimate
| business expense. For example in Toronto server consuming 200
| Watt 24x7 comes to about $20/month. Not much to dwell about if
| you are making money as a business.
| swiley wrote:
| Many of these things are really competing with a $20/mo VPS so
| a full PC that's on 24/7 is still cheaper. Yes you could go
| further: use portable slim apps that fit on ARM SBCs but then
| you have to re-do your server config and learn a new app.
|
| Plus, at least for me, my PC is on 24/7 _anyway._
| Zenst wrote:
| For some perspective, this is a nice list of many household
| appliances and their power usage:
| https://www.daftlogic.com/information-appliance-power-consum...
| easygenes wrote:
| I have a computer in a room that's generally otherwise too
| cold... so all that power is just useful heat dumped into the
| room.
| londons_explore wrote:
| If you pay a lot for electricity, you could get heat cheaper
| with a heat pump, gas furnace, or simply adding insulation on
| the roof/walls so any heat you do add lasts longer.
| varjag wrote:
| Re-insulating a house is a substantial cost that can take a
| decade to pay back in utility bills.
| londons_explore wrote:
| Yet spending $6 on draught excluders to keep the wind out
| can pay back in just a day or two.
|
| It all depends on your baseline...
| midasuni wrote:
| Albeit not as efficient as say a heat pump
| NorwegianDude wrote:
| That depends. The heat might be used in a heat pump for all
| we know.
| deadbunny wrote:
| Significantly cheaper though.
| yjftsjthsd-h wrote:
| Cheaper to buy, but not to run
| johndhowell wrote:
| I personally run two servers with a NAS 24/7 acting as media,
| ftp, and web server. My electric bill maybe increased a couple
| of dollars, but otherwise I haven't really noticed any
| significant increase. I'm based in the southern US
| tibbon wrote:
| I've got a handful of computers running in a rack at home.
| Their total power? Around 400w.
|
| I also have a dehumidifier running all the time in the
| basement. It's power usage is 500w.
|
| No one bats and eye at a heater or dehumidifier, but a computer
| and people get worried
| sponaugle wrote:
| I agree with your sentiment. It is interesting to look at
| total power usage and power usage by device. I'm using
| IotaWatts to monitor every individual circuit in my house
| (over 200 of them), and it is easy to miss things that add
| up. With computers it is quite amazing the difference in
| power load based on cpu/disk load. When I run my PI
| calculator (which not only pegs CPU, but hits my SAN very
| hard) I see over a 1kw difference in my homelab power draw.
| 542458 wrote:
| I mean, I would bat an eye at a humidifier running 24/7
| without any sort of automatic on/off based on humidity levels
| but maybe in your climate that's more normal?
| jodrellblank wrote:
| > " _No one bats and eye at a heater_ "
|
| Then where did the sitcom style stereotype of Dad always
| turning the thermostat down come from?
| mirekrusin wrote:
| 45W is not really "sucking up electricity", it's really nothing
| compared to other things in the house. I don't think PC is
| consuming 100x more as well, less than 10x and the same or
| nothing when not used.
| ClumsyPilot wrote:
| I put together a tiny server with power consumption in mind, it
| consumes 15W and has i3-8100, that's enough to run just about
| everything I can be bothered to run: kubernetes, owncast,
| homeassistant, pihole, etc.
|
| Generally, a laptop or Intel Nuc will give you a good low power
| server, much better platforms for development than a PI.
|
| Then if you decide you need a RAID array, that's a different
| ball game.
| hatware wrote:
| I have decade old hardware that is certainly inefficient, but
| my services hosted have taught me so much it's not even worth
| the comparison. Yes, electricity is cheap in America but we all
| choose to spend money on our hobbies as we please.
|
| I'll bet there's a few nitpicks with your hobbies where you
| trade a lot of time/money/something that I don't necessarily
| understand. And that's okay.
| michael1999 wrote:
| Yes. In fact, apple's AirPort Extreme units do that for macs in
| the house though their zeroconf networking. On sleep, the
| airport borrows the address and ARPs it. On traffic to that IP,
| it holds it, sends a special wake packet to the sleeping nic,
| and then re-sends the packet to with the real MAC, and the
| waking machine picks it up.
| cptskippy wrote:
| My home has a 0.5-0.7kWh idle load, with the varying 0.2kWh
| being from the refrigerators cycling. I tried shutting down my
| NAS and Home server and it was less than 0.1kWh.
|
| I'm at a loss to explain exactly what is drawing 0.4kWh of
| power but my neighbor has a similar load on his. I'm suspecting
| all of the fancy and useless motion/astro light switches
| installed are partially to blame but there's no easy way to
| verify that.
| harikb wrote:
| There is such a thing as wake-on-lan
|
| https://en.wikipedia.org/wiki/Wake-on-LAN
| flatiron wrote:
| I run just plex and pi hole. Money I save with plex that I
| don't need a million different services and having no ads at
| home is well worth the price. One day my friends went out and I
| couldn't go. They got all drunk and shared how much they rely
| on my plex and by the time I woke up I had close to $1k in
| PayPal and a note that said "for an upgrade and your time" so
| honestly I probably making out on the deal. Went from a haswell
| pentium to a ninth gen i3. 8 gigs to 16. SSD for the root
| drive. 8 TB staging server before going to google drive. $20 a
| month for unlimited google drive is way more than the power to
| run the server. But I have about 80TB on there. Can't store
| that for less than $20 a month no matter how you slice it.
| lostlogin wrote:
| Does Google check or care what is on there? That would make
| me nervous.
| flatiron wrote:
| I use encfs. Big scary warning on Ubuntu when you install
| it that it's not 100% fool proof but for storing tv and
| movies in the cloud I think it's fine. Best part is it's
| totally seemless on my end. Uploads encrypted and my view
| is unencrypted.
| lostlogin wrote:
| Thanks.
| j1elo wrote:
| I've been looking at the offers of big providers and the
| average seems to be pretty much standard: 2 TB for $10/month.
| All across Dropbox, Google, Apple, even Microsoft (the
| individual plan) are in the same ballpark.
| flatiron wrote:
| Google drive enterprise for $20 a month for unlimited is
| the best I've found. Rclone and encfs to mount and send the
| content. Almost 0 issues.
| flatiron wrote:
| I've read that it costs them $2 to store a single TB a
| month. Which makes sense. $4 -> $10 is a reasonable markup.
| I store 80 TB for $20. They lose money on me.
| _huayra_ wrote:
| I've been using pcloud which has a lifetime plan which is a
| pretty good deal. I recommend waiting until black friday or
| August 1 (Swiss national day; they're a CH-based company)
| when they severely drop the price.
| rzzzt wrote:
| I do remember a "smart NIC" making its rounds in the news a few
| years ago -- although "few" in this case might be >10. It was
| essentially a low-power machine in itself (with no Raspberry
| Pis in sight yet) that could finish an HTTP download or receive
| e-mails without waking up the host computer. But that's more
| high-level than the buffering/relaying of packets you are
| thinking of.
|
| Edit: a bit weird, I wrote a very similar comment a year ago
| about this item. Now I feel slightly compelled to find the news
| source for it...
| rasz wrote:
| asus "killer nic", almost pure marketing wankery with little
| benefit https://www.pcgamer.com/motherboards-with-killer-
| network-ada...
| rzzzt wrote:
| "Application offloading" was one of the terms I tried when
| searching for it, and got this as a result - not the same
| thing, but similar: https://www.academia.edu/10225597/Putti
| ng_it_on_the_NIC_A_ca...
| rasz wrote:
| https://pcper.com/2007/03/bigfoot-networks-killer-
| nic-k1-and...
| teekert wrote:
| My home server is about 70W, meaning about 150EUR a year (~0.20
| eur/kWH). So about 12EUR a month. For one of my biggest
| hobbies. It's fun, but it also runs WireGuard, 3 NextCloud
| installs, Home Assistant and Mosquitto, 2 MineCraft servers, a
| FoundryVTT instance, samba, sabnzbd, Unify Controller, an Nginx
| static site, LibreSpeed, VaultWarden and soon a Django site. Oh
| and a virtual desktop (vnc) I can always leave running with
| stuff open.
|
| Do I need all of that? Meh, but it's not bad value for money-
| wise IMHO. Perhaps mostly because I just enjoy it.
|
| Edit: of course the thing itself was also quite expensive but
| not much more than a decent NAS which I think is a must for
| many people anyway.
| ta988 wrote:
| Yes we should stop chastising people like that. You have long
| hair, you use more water, shampoo and drying. You drink
| coffee how many wh for that cup and its content. We could
| always find something in others where they consume more. My
| neighbors keep all their lights on almost all the time in
| every room. The other one has AC on all day to full power
| even when it is cooler outside... So complaining that someone
| uses a 50w/70w device for their hobby... meh . However I
| still believe we and the manufacturers should work to reduce
| idle power and consumption in general.
| teekert wrote:
| Oh I never felt chastised... before...
|
| I mean I agree with not wasting energy of course, but what
| we're talking about here is really nothing compared to
| driving a car or making a couple of pots of tea a day, for
| example. I refuse to feel chastised!
| josephcsible wrote:
| > (~0.20 eur/kWH)
|
| Are such high electricity prices common in Europe? That's
| higher than it is in 49 of the 50 US states (assuming 0.20
| EUR = 0.24 USD, and with Hawaii being the exception).
| zbrozek wrote:
| That's cheaper than California!
| cinntaile wrote:
| In Western Europe at least this is quite common. Mostly
| because politicians have discovered this as a great tax
| resource.
| AnssiH wrote:
| I assume it varies a lot by region. 0.12 EUR/kWh for me
| (Finland, inc. transmission and tax).
| selectodude wrote:
| Electricity in the EU is really really expensive. That
| person is actually paying below average rates.
| teekert wrote:
| I never thought about it, as far as I know it's been stable
| like this for years now... Gas is getting more and more
| expensive though.
| mixermachine wrote:
| 0.30 Euro per kW/h in Germany
| binkHN wrote:
| > but it also runs ... MineCraft servers
|
| At least you have your priorities straight :P !
| gravypod wrote:
| You can dynamically turn on and off things with MaaS and
| monitor power though most UPSs.
| cortesoft wrote:
| My solar panels on my roof generate about 10KW/h more than I
| use. I can bank credits with the grid, but can never cash them
| out, so electricity is basically free for me.
| setBoolean wrote:
| I have a HomeKit power plug that is able to measure the power
| usage. It sits before most of my living room devices including:
|
| 2x OG HomePods (Standby) - 55" Philips OLED (Standby) - Hue
| Bridge 2. Gen (Running) - ATV 4K 1.Gen (Standby) - Netatmo
| Weather Station (Running) - Linksys MR8300 OpenWrt 21.02
| (Running) - Home Server (i5-6600, 16GB DDR3, 2x 256GB 850 EVO,
| 1Gb USB3 Ethernet Adapter as second NIC, Running)
|
| Reading with all devices combined is 35-40W.
| ab3rC1te wrote:
| You know you could just go buy an alternator and rig yourself
| up your own infinite power supply. Depends on the Amps of the
| alternator.
| rasz wrote:
| let me introduce you to "My PlayHouse"
| https://www.youtube.com/watch?v=Jhwj_aCEYc0&list=PLS2odYzlao...
| fun stuff starts at 5:00
| specialist wrote:
| Why not? Every one needs a hobby.
|
| Another benefit to Stapelberg's noodling is setting
| expectations.
|
| Imagine you're bootstrapping a municipal ISP or mid-sized org.
| Dealing with vendors and products for the first time. What's
| reasonable? Who knows? Having these projects be a reality check
| is awesome.
| philjohn wrote:
| There is a middle ground - either an Intel Atom or Intel
| Celeron based server - typically use around 10w and you still
| get niceties like SATA and even PCIe
| easygenes wrote:
| Also plenty of slightly more capable laptop class SKUs that
| are in this ballpark.
| vladvasiliu wrote:
| Can you still find "desktop" motherboards to go with them?
|
| A long time ago, I remember there being desktop MBs for the
| Pentium M.
| londons_explore wrote:
| For an always running thing, choosing a laptop
| motherboard is a nice way to guarantee all the power
| saving features will work properly.
|
| Most desktop motherboards won't allow the PCI express bus
| to sleep for example, using an extra ~4 watts, $8 per
| year in eco-friendly countries. Across the ~10 years your
| NAS will probably sit in a cupboard, that's $80, which
| isn't an awful lot, but probably would have let you buy a
| bigger SSD or hard drive which would have had more
| utility.
| vladvasiliu wrote:
| That's true, but then again I wouldn't be comfortable
| with a single-disk NAS...
|
| However, for running random VMs, etc, I think a laptop is
| quite a good proposition.
| ttsiodras wrote:
| If you are OK with external USB drives, I did just that with
| my Atomic PI [1] (Atom-based, 35$ SBC, 3.5W idle, running
| Debian on its internal EMMC, with two external USB drivers in
| ZFS-mirror configuration).
|
| [1] https://www.thanassis.space/atomicpi.html
| philjohn wrote:
| You can also get rackmount servers in 1U format with Atom
| CPU's that support 4 3.5" drives, e.g.
| https://www.broadberry.co.uk/intel-atom-rackmount-
| servers/cy...
| iforgotpassword wrote:
| > who don't bat an eye on the perspective of having a home
| server, sucking up electricity 24/7.
|
| Maybe they need it? A router isn't a home server. But even if,
| why do you keep your power efficient router running 24/7? You
| could turn it off when you're not home to save even more
| energy.
|
| > a full-fledged PC (probably consuming like x100 the power
|
| Maybe do some research before claims like these. He states this
| pc idles at 48W. Please show me a router capable of handling
| 25gbit/s that consumes 0,5W at idle.
|
| > how would you build a "something" that monitors the network
| for packets sent to powered-off machines, then somehow caches
| the request, powers the destination machine On, and finally
| lets the request continue to its target?
|
| That's what "wake on LAN" is for.
| yakubin wrote:
| _> But even if, why do you keep your power efficient router
| running 24 /7? You could turn it off when you're not home to
| save even more energy._
|
| You jest, but I turn off my router for the night and each
| time I leave home for more than a day. Not just router to be
| exact, everything that's connected to a power strip goes
| down, as I turn off all the power strips. (Not OP.)
| stefan_ wrote:
| Not a fan of the "turning the power strips off" method.
| Power supply failures are the number one reason why
| expensive electronics turn to bricks and subjecting them to
| the large inrush current that happens on the primary side
| every time you do the mechanical switch thing is a great
| way to significantly accelerate that process. Penny wise,
| pound foolish kind of thing.
| yunohn wrote:
| What do you mean? Things that don't have an in-built
| switch, are meant to be fine hot plugging and as such,
| controlling via a power strip is fine.
| yakubin wrote:
| That's the primary method of turning embedded devices on
| and off though. :) During development it's done around
| the clock and noone bats an eye. Rule of thumb: if a
| device doesn't have a power button, it's fine to turn off
| using the power strip.
| yownie wrote:
| I'd assume because development process doesn't need to
| deal with repercussions of accelerated power supply
| degradation
| elric wrote:
| I have physical switches on or near every outlet. Anything
| that's not actively being used gets turned off. With the
| exception of my oven and hob, because those are nearly
| impossible to reach. I use roughly 2kWh/day. This approach
| probably doesn't make much sense if you're using an order
| of magnitude more power. But it makes sense for me.
| a2tech wrote:
| Crazy. My wife and I used 619 kWh last month
| nixgeek wrote:
| We were at 5MWh last month, so it's all relative. Don't
| feel bad.
| liketochill wrote:
| That is impressive! Did you do that on a 200A 120/240V
| service?
|
| What is your load? Ac?
| [deleted]
| nixgeek wrote:
| It's 240V/400A split-phase service, PNW, around
| $0.11/kWh. Air conditioning is a big part of it when the
| temperatures are high. Also charging multiple bEV, a lot
| of 24x7 loads: half-dozen fridges, a couple large
| freezers, wine cellar, about 1kW of IT+AV equipment, etc.
| heartbreak wrote:
| Not the person you were replying to, but my June
| statement lists 1,800 kWh. I'm in a two-bedroom apartment
| in Texas. The primary consumer of power by far is AC,
| which is kept at 68 degrees in the summer (partly for
| temperature, partly for humidity). My effective rate
| after all the fees was $0.11/kWh.
| 83457 wrote:
| Why not just go all in and shut off at breakers?
| elric wrote:
| Breakers are not made for very frequent on/off cycles.
| They won't last as long as a light switch for instance.
| I've never seen a light switch wear out, but I have seen
| breakers wear out (at which point they trip very easily,
| and eventually seem to just permanently fail open).
| oriolid wrote:
| Fridge.
| herewego wrote:
| Then don't flip the fridge breaker.
| detaro wrote:
| Plenty places don't have a separate breaker for that.
| adrian_b wrote:
| "probably consuming like x100 the power" is greatly
| exaggerated.
|
| A typical PC might have an idle power consumption of less than
| 50 W, while a very small computer with ARM little cores might
| have an idle power of 2 ... 3 W, so at most the power
| consumption ratio would be 20.
|
| However even that is not realistic because a computer or
| appliance with less than 3 W power consumption will not be able
| to route 10 Gb/s or faster links and it will struggle even with
| multiple 1 Gb/s ports.
|
| A dedicated router appliance able to route 10 Gb/s or faster
| links will probably have an idle power of around 10 W or even
| more.
|
| For routing only 1 Gb/s links, you can use among the standard
| PC's a NUC-like model, which will have an idle power
| consumption between 5 W and 10 W, quite close to a dedicated
| appliance. This is actually what I am using for my own Internet
| router/firewall (which also runs many other services, e.g. DNS
| server and proxy, NTP, e-mail, HTTP server and proxy etc.) with
| 1 external 1 Gb/s port and 4 internal 1 Gb/s ports (4 of the 5
| ports are made with USB to Ethernet adapters).
|
| Some of the more recent NUC or similar computers have multiple
| Thunderbolt ports or 10 Gb/s USB ports, so those can be used to
| route multiple 10 Gb/s links (using Ethernet adapters), with an
| idle power of around 10 W, similar to any equivalent commercial
| routers.
|
| For 25 Gb/s Ethernet links, a commercial router is unlikely to
| be much cheaper or to consume much less than a standard PC.
| goodpoint wrote:
| > For 25 Gb/s Ethernet links, a commercial router is unlikely
| to be much cheaper or to consume much less than a standard
| PC.
|
| EDIT: I linked to a switch by mistake.
|
| This router has 12 x 10G SFP+ and 2 x 25G SFP28 ports and
| costs $595
|
| It appears to be routing between 13 to 39Gbps depending on
| the rule complexity.
|
| https://mikrotik.com/product/ccr2004_1g_12s_2xs#fndtn-
| specif...
| realityking wrote:
| That's a switch, not a router. Much simpler hardware.
|
| Here's a MikroTik Router with 28 Gbit/a throughput:
| https://mikrotik.com/product/CCR1036-8G-2SplusEM
|
| $1300 and up to 73W
| coder543 wrote:
| That's a bit of an odd example to choose.
|
| This[0] has a $600 MSRP, with twelve 10Gbps ports and two
| 25Gbps ports. It's ~32W before you add the SFP+ modules,
| and upwards of 50W if you have all modules populated.
|
| In the benchmarks they list, this can provide somewhere
| on the order of 13Gbps to 40Gbps of routing, depending on
| exactly what you're doing. (Smaller packets will lower
| these numbers, but if you care about _bandwidth_ you 're
| unlikely to be worried about smaller packets... at that
| point, you probably care more about kpps.)
|
| [0]: https://mikrotik.com/product/ccr2004_1g_12s_2xs
| ikiris wrote:
| the benchmark you even link shows absolutely abysmal
| performance curves, dropping down into even half a gig
| perf for small packets single stream.
| coder543 wrote:
| Calling any benchmark "abysmal" requires a point of
| comparison, and you have provided none... so it doesn't
| really encourage good discussion. What's the point of
| your comment? It seems to just be a way of insulting
| someone else's product.
|
| Which comparably priced router are you thinking of that
| has a "non-abysmal" performance curve for small packet
| bandwidth?
| realityking wrote:
| Fair. It was the first router I found on their site with
| listed throughout >= 25 Gbit/s.
| [deleted]
| artemist wrote:
| This is a switch and switching is hardware accelerated.
| While you can theoretically route with this this, it is
| incapable of routing at gigabit speeds, let alone 10
| gigabit speeds, once you need even a few rules.
| [deleted]
| [deleted]
| tomnipotent wrote:
| SFP requires less energy than RJ45, only 0.7W/port. The
| MikroTik 4-port CRS305-1G-4S+IN draws 10-18W.
| logifail wrote:
| > MikroTik 4-port CRS305-1G-4S+IN
|
| I have two of these at home, they're neat little bits of
| kit.
|
| They're fanless and therefore silent, which is fine until
| you realise you want to do 10GbE over existing copper
| cables with something like Mikrotik's S+RJ10 adapter. Then
| the temps start to rise...
|
| So, I've decided to go completely to fibre, even if it
| means opening up the walls of the house. Just bought a job
| lot of used ConnectX-3 cards off ebay.
| walterbell wrote:
| How do you obtain custom lengths of fibre cables that
| work with ConnectX-3?
| tomnipotent wrote:
| Use fs.com, their generic cables work with MikroTik no
| issue and they do custom orders.
| eqvinox wrote:
| The ConnectX-3 cards are not vendor locked in any way.
| (And even if they were, that'd only affect the SFP, not
| the cable.)
| lostlogin wrote:
| > ConnectX-3 cards off ebay.
|
| Those cards are so good. I got a few after seeing them
| rated in r/homelab and haven't looked back. The Synology
| took one and it was about 1/10th the price the official
| Synology one was.
| bcrl wrote:
| That's not entirely true. SFP power consumption depends on
| the type of SFP. A 10Gbps DWDM SFP+ might draw 1.8W of
| power. The why for this is actually quite interesting: the
| lasers used in DWDM SFPs have much more stringent
| requirements for temperature stability to ensure the light
| emitted doesn't drift out of spec. In order to achieve that
| temperature stability, they use a built in peltier to pump
| heat away from the laser and control the temperature.
| They're quite the marvel of modern engineering!
| oasisbob wrote:
| It would have been fun to be a fly on the wall at Cisco
| when they first realized people were jamming devices with
| heaters in SPF sockets.
| eqvinox wrote:
| Cisco didn't "realize" this was happening, they
| specified, had manufactured, and sold the very SFPs the
| GP post is describing. SFP slots in routers are designed
| with this thermal load in mind.
|
| Which is also why plugging a large amount of DWDM optics
| into a datacenter switch is a bad idea. Datacenter
| switches are _not_ designed with this in mind. You run
| into risks of both overheating the switch as well as
| overloading the PSUs. A small number of high-power optics
| ain't gonna break the switch though.
|
| And: 10Gbase-T SFPs have horrendous power consumption,
| even worse than DWDM SFPs. At these speeds, the signal
| over copper is mostly mush, and the PHY contains a none-
| too-trivial analog & digital signal processor. Which,
| again, is where limitations for 10Gbase-T SFP usage come
| from. If at all possible, avoid this shit -- there's
| absolutely no reason to have 10Gbase-T inside a rack, for
| example. Just use DAC cables or SR optics.
| tomnipotent wrote:
| Except no one is using DWDM for homelabs, as it's usually
| used for longer distances. The peltier cooler is
| definitely cool, though.
| bcrl wrote:
| True. On general principle I would recommend folks invest
| in single mode rather than multimode fiber for permanent
| installs, as the price delta on SFPs is low enough these
| days. Multimode is a complete pain as it needs to be
| upgraded for higher speeds every decade. If it's just a
| couple of patch cables in a home lab, it doesn't matter,
| but if it's run through a wall...
| naikrovek wrote:
| Have you measured how much electricity a computer doing routing
| consumes? I bet it's a lot less than you think.
|
| Also measure how much a good, high-bandwidth router uses. I bet
| that is a lot more than you think.
|
| the "why do this" question for me comes about entirely because
| of the continual manual intervention that is needed for
| solutions like this. just not worth it, to me.
| jacoblambda wrote:
| I mean I have and my router is on average drawing around 15W
| but it also isn't remotely close to being able to hit 25Gbps.
| I don't think 50W (apparently the draw of this router) is
| unrealistic for a 25Gbps router by any means.
| unixhero wrote:
| Maybe they have money and they find that spending money on
| homelabbing is worth the effort. And really it isn't THAT
| expensive either. I for one don't bat an eye on my USD200
| yearly homelab electricity cost.
|
| Of course if one wishes to perfectly optimize everything then
| having a homelab might well wasteful.
| sponaugle wrote:
| Indeed every case is unique. My home lab runs about ~4kw all
| the time, which end up being about 2.8mWh/month. Here in
| Oregon that ends up costing $340/month.
|
| This is a brand new house, and as soon as I get permits
| approved I'm adding 21kw of solar which will help offset
| that.
| baybal2 wrote:
| What a lot of other people don't realise:
|
| - There is no such thing as a 25gbps home router.
|
| What's currently on the market is a very serious overkill ever
| for 10gbps.
| great-potential wrote:
| In terms of CPU it is not totally overkill if you're using an
| IPS/IDS, bare in mind you'll also be disabling most of the
| network card offloading in a full fledged firewall and that
| will ultimately result in consuming slightly more CPU cycles.
| hocuspocus wrote:
| > I guess some people around the world have quite cheap utility
| bills!
|
| Or they work at Google and don't really have to care.
|
| Electricity is not particularly cheap in Switzerland, but not
| particularly expensive either (nothing like Germany for
| instance). If running a home lab is your hobby, why not. There
| are plenty of hobbies that are a lot more expensive.
| ianai wrote:
| The machine's clearly massive overkill for routing. Another
| commenter points out it typically runs around 50w - and the
| OP says they're also using it as a server in this thread.
| pilif wrote:
| Do you know of any smaller scale hardware capable of
| routing 25Gbit/s?
| dijit wrote:
| FWIW: raspberry pis (the 4 series) sucks in 15w which is
| why it's so bloody difficult to power them through standard
| USB power adapters (which go up to 12w).
|
| So, not hundreds of times more power hungry, but definitely
| 2-3x
| ajsnigrutin wrote:
| yep, which makes them a pain in the ass to efficiently
| cool down without active cooling (=fans)
| larschdk wrote:
| My passively cooled RPi4 with 8GiB RAM + 1GiB/USB
| ethernet dongle working as a 1Gb router never goes above
| 50 degrees C (all-metal case from aliexpress).
| geerlingguy wrote:
| Just to note that's peak power consumption. They idle
| between 1-4W, or if you're running a PoE HAT, 4-6W.
| formerly_proven wrote:
| A RPi is not equivalent to what most would use as a COTS
| router, which includes a GbE switch and some kind of
| modem (DSL or cable). The latter on its own needs a few
| Watts.
| geerlingguy wrote:
| On the contrary, OpenWRT and a good CM4-based board
| allows the Pi to run pretty well as a gigabit router :)
|
| https://www.jeffgeerling.com/blog/2021/two-tiny-dual-
| gigabit...
| aj3 wrote:
| That's not much of a router with just two ports and no
| WiFi.
| tssva wrote:
| No CM4 and accompanying carrier board needed . A regular
| RPi4 with a USB3 gigabit adapter makes for a good gigabit
| home router. I have been using one as my home gigabit
| router for the last 14 months without any issues.
| t0mas88 wrote:
| The problem is running it 24x7, then a small 50w is still
| 438 Kwh each year. That's more than a 10% increase in
| yearly use for a typical two person household...
| true_religion wrote:
| Then like another poster said, it all depends on how much
| you pay for electricity and some people have quite low
| cost electricity.
|
| For me that would be a $49 cost per year, or a cost of
| little over $4 per month.
| quaintdev wrote:
| How about people power this stuff with renewalbles like
| solar?
| manbash wrote:
| Now it's up to the person to decide whether increase in
| living cost is worth it, which... is kinda normal. I
| think I spend more money on other things that I take
| enjoyment from.
| t0mas88 wrote:
| Sure, it's just that many people don't think about this
| impact because 50W sounds low.
| [deleted]
| mirekrusin wrote:
| Maybe having single bulky, but efficient server means
| other laptops/whatnot will be used less which will cancel
| out and be electricity negative at the end of the year.
| JCharante wrote:
| Aren't 60w lightbulbs still around?
| labawi wrote:
| Do you run your lightbulbs 24/7?
|
| If so, you should get a LED alternative.
| maccard wrote:
| That's very roughly $100/year, which is < 1/6th of the
| cost of the _network cards_ in this build.
|
| My workstation bursts to 600w+ for 15-20 minutes at a
| time when compiling, for comparison.
| mirekrusin wrote:
| For developer like me, I'm actually considering this
| option, it would be great to have PC like this available
| for some docker stuff as well, ie. MSSQL instance for
| development (Azure MSSQL version that runs on M1 is shit
| slow to the point it's unusable for development). Maybe
| even some tests could be offloaded from dev machine to this
| one etc. Would be nice to have single place for backups,
| photo library etc.
| angrais wrote:
| Why not just use your development machine for
| development?
|
| Also, why not use a remote server for deployment,
| testing, and building docker images? Most roles I have
| had offer such services (remote servers) as part of their
| costing so wouldn't cost you time, money, or effort.
|
| I agree that having a single place to backup photos etc,
| is important. I use an external SSD for such a purpose as
| imo it's more useful to have it offline as I rarely add
| data and it is less likely to be comprised, e.g., it my
| machine was hacked.
| bluedino wrote:
| >> Why not just use your development machine for
| development?
|
| He said the M1 versions of MSSQL performs terribly
| great-potential wrote:
| I think if the OP uses this only as a router it is indeed a
| waste of power.
|
| What I would probably do is also use it as nas/workstation by
| using virtualization, SR-IOV is now pretty standard on these
| cards.
| secure wrote:
| I'm using the machine also as a server, so it replaces 2
| existing devices. In terms of power consumption, it's likely
| only a small increase, if at all.
| ianai wrote:
| My only rub with that is - shouldn't the router be only a
| router from a security perspective? Definitely combining
| servers for home use does make sense though.
| secure wrote:
| Ideally yes. But with the resources needed for 25 Gbps,
| not using one machine for multiple purposes seems
| wasteful. The server only stores publically available
| data, though, so not a big deal from a security
| perspective.
| ianai wrote:
| What do you mean by resources required? expansion ports
| on the mb?
| great-potential wrote:
| Yeah running your nas-bittorrent/firewall-router on the
| same OS is clearly not something I would do, especially
| that now you can use somrthing like proxmox for example
| and virtualize your pfsense/opnsense instance.
| AdrianB1 wrote:
| I am using a home server with the same CPU and I am using
| it for SQL, storage server and virtualization. In order to
| do that I changed RAM to 32GB DIMMs, 64GB in total was not
| enough and the CPU works just fine with 32GB DIMMS. The
| platform is quite limited by the number of available PCIe
| lanes, but without moving up to Threadripper (a lot more
| expensive) there is no better option, Intel is in the same
| place or worse. The good thing with Intel is that you can
| use a CPU with integrated graphics and save the PCIe lanes
| for the graphics card.
|
| My previous build (Ryzen 2700, 65W) had a great feature
| until it was gone with a BIOS update: after installing
| everything it worked with the graphics card removed. As I
| always connected only remotely, it was not a problem but a
| benefit. You can try and see what happens.
| Datagenerator wrote:
| Also, the user can use his freedom to build and do whatever
| wished for. This tone of you shouldn't do this because some
| COTS is available sounds very commercially driven. People
| have the power to create another Google, we are not
| powerless.
| smolder wrote:
| Yeah, I have an overpowered router PC (mostly in the
| capabilites sense --its a low power passively-coolable Xeon),
| but it also acts as a flash NAS, and hosts a few other
| containerized services that I prefer are always-on, it's a
| wireguard endpoint, etc. I've got a separate sometimes-on box
| for other VMs and containers. That one hosts a windows VM
| with a VFIO-attached GPU which my living room tv plugs into.
| Altogether it's lots of computing power for home use but
| draws relatively little at the wall. I'm pretty happy where I
| landed in terms of overall utility versus TCO, using this
| sort of consolidated-hardware approach.
| IgorPartola wrote:
| What kind of light bulb? :)
|
| A typical consumer router will take up something like 5-10
| watts. A PC will suck in 20-50. Remember that the PC will be a
| lot more powerful so it'll spend most of its time with low CPU
| usage.
|
| Say the worst case scenario: 45W difference. 45W * (24 * 30) /
| 1000 = 32.4 kW*h/month. At $0.10 kWh rate that's $3.24/month,
| less than a cup of Starbucks coffee.
| mixermachine wrote:
| Add HDDs and 10 GBit/s hardware and you surely reach 60 watts
| or more idling. Many home labs also don't use the newest
| hardware but old server hardware (add 10 - 40 watt idle).
|
| A NUC or other Laptop hardware on the other hand would be OK
| to run. Maybe SSD raids are (financially) feasible for
| everyday files in the near future.
|
| You are lucky to pay only $0.10. Here (Germany) I pay 0.30
| Euro per kW/h. About $0.36 dollars.
|
| I turned off my home NAS a few months back and partially
| switched to VPS services (also because of the better
| connection).
| iforgotpassword wrote:
| > Many home labs also don't use the newest hardware but old
| server hardware (add 10 - 40 watt idle).
|
| Yes, but switching to newer, more efficient hardware is not
| free, so you need to factor this in when considering a
| switch. Also, in case you don't only care about your wallet
| but also environmental impact it's getting even more
| complicated, since manufacturing your new shiny toy
| definitely is polluting the environment somewhere in Asia,
| gets shipped via ship burning horribly dirty fuel in
| international waters etc.
|
| If you don't have any use for the old hardware, chances are
| it will end up burning on some open field somewhere in
| Africa after local recyclers already extracted the good
| stuff...
| ClumsyPilot wrote:
| "your new shiny toy definitely is polluting the
| environment somewhere in Asia"
|
| Thats true whether I use dropbox or my own server
| foolmeonce wrote:
| > A typical consumer router will take up something like 5-10
| watts.
|
| You demonstrate a good worst case, but the article writer
| wants to use more than 10 gb/s so he can't actually use your
| typical router, he can have 15 gb/s with the MikroTik
| CCR2004-1G-12S which has an unknown idle W and a max around
| 50W.
|
| Looking into the problem, I can't really determine why I
| should upgrade to 10 or 25 gb/s, but if I wanted to do so now
| I would rather buy components I could reuse than buy a router
| that will be inefficient for its entire service life.
| j1elo wrote:
| Oh my I wish those $0.10 kWh... in here we have 3 prices
| through the day (depends on what time it is), and the
| cheapest one is already higher than that :)
|
| Anyway I did the math and it would be $7/month. More than
| double your estimation, but still not horrendous. Although
| for that range of prices one might be able to find a managed
| instance machine in some cloud provider...
|
| (edit: somewhere in the process I lost track of the fact this
| was a price _difference_ calculation, so it's adding $7 to
| whatever was already the cost with a more power efficient
| machine, which ofc. depends on number and type of HDs and
| other equipment)
| ta988 wrote:
| Would it be efficient to store power when price is low?
| tgragnato wrote:
| The real difference is in latency: I tend to use the same
| operator on the fixed and mobile line, with Wireguard. SSH,
| SMB and Matrix respond almost as if you were in LAN.
|
| If I add the cost for bandwidth and storage of a data
| center, then the economic choice is obvious.
| f3d46600-b66e wrote:
| My Netgear access point consumes 7-8watts. My home server, with
| 2*12TB drives, 2*3TB, and 1nvme, which also acts as NVR (for
| POE cameras), recording 24x7, and which also acts as owncloud
| server (and few other things) consumes 40-50watts (and it
| includes a 10gbps SFP+ fiber). It also includes a wireless card
| and acts as an access point. It also runs a few VMS, continuous
| integration server, pihole and other stuff.
|
| This is not x100, it's x7 times. And the utility is much much
| higher.
|
| Before, i was using amd Apu, and it consumed 20-30 watts, but
| did not support AES-NI, which made the disk access limited to
| 60MB/s :-(
|
| It's way cheaper to run everything at home than paying for the
| cloud, even if u include electricity cost.
|
| It's about $50/year for electricity (1 watt 24/7 =~ $1 per
| year)
| vbezhenar wrote:
| > In fact this is a nice place to ask: how would you build a
| "something" that monitors the network for packets sent to
| powered-off machines, then somehow caches the request, powers
| the destination machine On, and finally lets the request
| continue to its target? Has this been tackled anywhere? There
| must be tons of people wanting a homeserver but living in
| places where electricity has a considerable cost...
|
| IP protocol does not guarantee delivery. So you don't have to
| cache request. Just power on the machine, client will retry
| sending the packets until the machine is powered on and can
| respond. Just make sure that there's no gap between networking
| available and http server is still starting on.
| secure wrote:
| I have built such a gateway for SSH: it accepts the
| connection, powers on the target machine if needed, then
| forwards traffic.
|
| https://github.com/stapelberg/zkj-nas-
| tools/blob/master/wolg...
|
| The advantage over just catching any packets is that an SSH
| connection is authorized, so less noisy in terms of undesired
| wakeups.
| secabeen wrote:
| Apple has this. It's called Boujour sleep proxy, and it
| allows a always-on device like a Airport or AppleTV to
| claim the IP address of a sleeping Mac, waking it with a
| WoL packet when traffic for that device comes in. It's
| probably pretty useful, but is annoying on a managed
| network, as it spams my arpwatch system every time the IP
| is transferred between devices.
| [deleted]
| allenrb wrote:
| When I was younger, 100W _was_ a light bulb. We've come a long
| way. :-)
| [deleted]
| juancb wrote:
| I've used managed power strips for data center applications in
| the past. The strips have individually addressable power ports
| and a web interface, telnet, as well as SNMP support. They've
| been around for decades and are the solution to your problem of
| needing to remotely manage the power state of power hungry
| devices.
|
| If you set your servers to always power on after power is
| restored you can control them with that device.
|
| There's also Wake On Lan (WoL) support in a lot of systems,
| where you can use a correctly crafted "magic frame" to wake up
| any machine that received it.
| melling wrote:
| He claims one of his goals is to be power efficient
|
| "The PC consumes about 48W of power when idle (only management
| ..."
|
| https://www.eia.gov/tools/faqs/faq.php?id=97&t=3
|
| " How much electricity does an American home use? In 2019, the
| average annual electricity consumption for a U.S. residential
| utility customer was 10,649 kilowatthours (kWh), an average of
| about 877 kWh per month"
| iforgotpassword wrote:
| Ok so what am I missing here? In Germany a typical four
| person household is 2000kwh a year. Surely air conditioning
| can't make up for such a difference? Everybody already
| switched to electric cars?
| tzs wrote:
| That's about 170 kWh/month. I'm in the US and use around
| 450-500 kWh/month in the summer, which is my time of
| minimal use because it is warm enough to not need heating
| and usually not so warm as to need air conditioning.
|
| I'm curious why such a big difference. Here's what I'm
| using in a household of one.
|
| 1. Kitchen appliances: Fridge, microwave, toaster, oven,
| dishwasher. The first three are used daily. The oven and
| dishwasher once per week. I'll also occasionally use a
| bread machine and an electric kettle, maybe a couple times
| a month.
|
| 2. Washing machine and dryer. 4-6 times per month.
|
| 3. A 2017 27" iMac, a Raspberry Pi 3, cable modem, TP-Link
| A7 router, and two TP-Link SG108E switches. A USB hub and a
| case for a couple external SSDs. An external monitor.
|
| 4. A 55" LCD TV which is used a couple hours a day, and an
| A/V receiver which is always on but usually idle. A couple
| streaming boxes (Fire Stick 4K, Xfinity Flex).
|
| 5. iPhone, iPad, Apple Watch, and Surface Pro 4 chargers
| charging those devices.
|
| 6. Indoor lighting. It is almost all LED. The only
| exceptions are the lights in the fridge and dryer, one bulb
| each in the attic and crawl space that I forgot about when
| switching to LED.
|
| 7. An outdoor security light in back that is on overnight.
| I'm not sure about its power consumption, but these are
| typically under 100 W, which would be 24 kWh/month during
| the summer.
|
| (I'm a bit puzzled by that light. I'm still using the same
| bulb that was there when I bought the house, giving the
| bulb on "on" time since I've owned it of around 56000
| hours. That's quite a bit longer than expected for every
| kind of light bulb that I can think of that this might be.
| Only LED should approach that and definitely is not LED).
|
| 8. Well pump and water heater. The well pump runs maybe a
| couple times a day for maybe 5 minutes at a time, and would
| account for at most a few kWh/month. I know the water
| heater is a beast--but I think it only runs a little more
| frequently than once a day.
|
| 9. A couple box fans in windows at night to blow out hot
| air. They are 50 W each. Maybe 50 kWh/month.
|
| 10. Miscellaneous. Charger for rechargeable AA and AAA NiMH
| batteries. Electric toothbrush. A Google Home Mini as a
| kitchen timer. An Echo Dot to control lights. a Hue hub for
| the lights. Charger for the batteries for some cordless
| tools (drill, string trimmer, hedge trimmer) that are all
| used rarely.
| denimnerd42 wrote:
| Have you seen the size of an American refrigerator compared
| to European?
|
| We use 2kwh a month here in August.
| [deleted]
| zrail wrote:
| My house is very atypical for my area of Michigan. It's all
| electric with a geothermal ground source heat pump for
| heating and cooling and electric water heaters. In addition
| it has (or rather had) two of these geothermal units and
| two water heaters for two separate spaces. We also have a
| plug-in hybrid vehicle that we charge at home.
|
| Outdoor temperatures in the summer range from mid 70s F to
| low 100s F and humidity is almost always above 60%, hitting
| 90% for weeks at a time.
|
| Our electric usage in all but the coldest months of the
| year is around 2200kwh per month. I expect this to go down
| somewhat because we just upgraded one of the geothermal
| units and replaced the other with a gas furnace and
| inverter AC unit.
| lttlrck wrote:
| It looks closer to 3.5Mwh:
|
| http://shrinkthatfootprint.com/average-household-
| electricity...
|
| But still that's _far_ lower than the US at 11Mwh. Maybe A
| /C and home size accounts for most of it.
|
| We're a family of 3 and on track for around 8Mwh this year.
| We only use AC on the hottest days (Southern Californian
| climate so it's quite manageable throughout the year),
| water and dryer are gas - but I have a homelab which could
| be optimized.
| iforgotpassword wrote:
| I got that figure from my last bill, they always add
| graphs and comparisons, like a bar chart displaying usage
| of a typical 1, 2, 3, 4 person household and then your
| usage. But maybe they show lower values here to get you
| to try and save more.
|
| I'm in a two person household and usually clocking in at
| 700-900kwh, and that's with a 24/7 home server drawing
| 30W, a rpi for kodi that I keep running, dish washer,
| washing machine, induction stove. No ac, no dryer, no
| microwave, hot water and central heating with natural
| gas. Also no more desktop pc since around 2014.
| BenjiWiebe wrote:
| I don't have the numbers, but air conditioning can suck up
| a LOT of energy. I'm pretty sure air conditioning would be
| (in our household) the single biggest energy user.
| AdrianB1 wrote:
| My home server using a very similar configuration (same CPU, a
| lot less potent network) is what enables me to work from home,
| making a living, so the cost of electricity is compensated by
| the gas saved on commuting. I am also in process of adding
| solar to my house, covering a lot more than this server.
|
| If people use old servers in the kilowatt range consumption for
| fooling around having a server in the basement is questionable,
| building something with low power in mind is nothing to
| complain about.
| bastardoperator wrote:
| I have a very similar setup using a desktop as a router
| (opnsense, 2 WAN, fibre/switch) and it has a single, yet fatal
| flaw.
|
| The power button. When you lose power, and the power comes back
| on, your router does not. If you are away, you must physically
| turn it back on.
| secure wrote:
| I describe the relevant UEFI option in
| https://michael.stapelberg.ch/posts/2021-07-10-linux-25gbit-...
| :)
| fuster wrote:
| Most (all?) PCs should have a BIOS option to always restore
| power after loss.
| alexymik wrote:
| You can usually change this in the BIOS (or equivalent). Look
| for "Power on after AC loss" or similar.
| Bud wrote:
| The real story here is how great the internet service can be now
| in a civilized country if you don't let Comcast have monopolies
| and ban local internet cooperatives and coast along selling the
| same service for 15 years while raising prices.
|
| 25Gbps symmetric for about $70/month. That's significantly less
| than I pay Comcast for its crappy 900Mbps down/40Mbps up service.
| nixgeek wrote:
| At least in PNW the service isn't 900/40 any more, it's
| advertised as "1.2Gbps" and the profile actually seems to be
| 20% over that, I've seen 1.5Gbps or 170-180MB/sec downstream
| performance recently. Upstream is still <= 50Mbps and otherwise
| agree with your comment that the U.S. is falling behind in
| price-throughput on domestic internet services versus many
| other developed nations.
| walrus01 wrote:
| Upstream on cable like that will be sub 50 Mbps because
| they're very intentionally only using a small number of rf
| channels on the coax for upstream. This is why in my opinion
| docsis3 is putting lipstick on a pig and is only a short term
| stop gap solution. The better end state is proper SM fiber to
| each house.
| wormslayer666 wrote:
| There have always been regional variations in this; I've had
| the same 1.2GBps plan in a few states now and it only ever
| got close to that number in the one where Fios was an option.
|
| Comcast's only innovation in the last few years has been to
| add data caps to residential fiber service (my 300mb/s "1.2
| GBps" is capped at 1.4TB/mo).
| Bud wrote:
| Let's please not further the idea that it's acceptable to
| quote only downstream speed when citing Internet speeds.
|
| Doing that just leads to 20 more years of Comcast killing
| the concept of modern Internet service for Americans. Cite
| both numbers.
| nixgeek wrote:
| We've just been paying a $30/mo upcharge for unlimited data
| (removal of overage charges). Basically because it's more
| predictable than the $10 for every 50GB overages. A few
| months we've had 4-5TB through the service and Comcast
| doesn't seem to have slowed us down or charged more.
| underscore_ku wrote:
| 1Gb/s in Romania is 10$/month
| throwawayswede wrote:
| That's cronyism for you.
| gunapologist99 wrote:
| I feel the pain. I pay $80/mo for 200Mb/s down.
|
| But, still, installing fiber is just expensive.
|
| Much more expensive than almost anything else, and requires
| skill and experience to terminate the connections, and that's
| even overlooking any regulatory hurdles of digging along the
| right-of-way, private and public land ownership issues, the
| costs of the equipment, and the fragility of the fiber itself
| once installed.
|
| None of those costs really scale. In fact, they increase
| linearly as the pool of available installers decreases, and
| that's not even talking about the cost of connecting that fiber
| to anything on the other end.
| bluedino wrote:
| We have a couple fiber companies in our town (USA), and we have
| been shopping for internet. Currently we have fiber from AT&T.
|
| One is over a mile down the road and they want $90,000 to
| connect and then $1750 a month for 1gbps. Another company is
| two blocks away, they will only charge $8,000 to connect and
| it's $1,550 a month for the same speed.
|
| Problem is there aren't many other companies in the area that
| want service, so we would end up paying all the buildout costs.
| The other business are small retail, restaurants and shops that
| don't need more than coax (150/20 or whatever for $79/month)
| from Spectrum.
| xupybd wrote:
| I'm in New Zealand, a small island nation. I have 1gbps for
| around $70 USD per month. The install was free. I get a VoIP
| phone and Prime video subscription included.
|
| Most of my devices are wireless so I really don't take
| advantage of the speed.
| timClicks wrote:
| Which ISP are you with?
| xupybd wrote:
| 2 Degree's. They bundled it all with my cell plan.
| hackcasual wrote:
| I hit ~400mbps on my pixel 4a's wifi with 802.11ac in a
| moderately congested environment. With WiFi 6 I would
| probably saturate
| Jnr wrote:
| I pay about 17 USD for 1 Gbps in Europe.
| [deleted]
| RKearney wrote:
| I'm sure the fact that Germany has a 566% larger population
| density and is 96.19% smaller than the United States has a lot
| to do with a countries ability to bring 25Gbps internet to its
| citizens.
| hda111 wrote:
| Internet in Germany is really bad and expensive compared to
| Switzerland and even to the rest of Europe. In Germany fiber
| to the home is maybe 1-2% of all internet connections. Most
| fiber providers in Germany offer only artificial asymmetric
| connections and are very expensive.
| nix23 wrote:
| >I'm sure the fact that Germany
|
| Sorry to disappoint you, but it not Sweden or Germany but
| Switzerland...Germany has bad internet too, but hey that
| mistake happens even to the best ones (like Bush Junior)
| RKearney wrote:
| Not a disappointment at all. I see now that it is indeed
| Switzerland. The mention of the Germany based IX upgrading
| to 25Gbps towards the end of the article threw me off.
|
| But Switzerland has a 504% larger population density and
| 99.56% smaller land area as compared to the United States
| so my point still stands.
| gspr wrote:
| This is indeed great, but keep in mind that this story is about
| one of the well-known greatest: Init7.
|
| They're well-known as a for-geeks-by-geeks service. I was
| massively impressed by them when I lived in CH.
| m463 wrote:
| I think I priced it recently and it was $499/mo (+ hidden
| upcharges) for 1 gig. their lowest price was $65/month for 35/5
| XIVMagnus wrote:
| I left comcast for at&t fiber, paying $60/month for 1year. If I
| learned anything since then is to make sure you attempt to
| leave and they will start calling you with better offers..all
| of a sudden lol
| walrus01 wrote:
| Solving the last mile facilities based isp problem in detached
| houses in the USA and Canada is a hard one. In most places with
| aerial Telecom and power infrastructure, you will have three
| things coming to a house. Electricity, obviously. And then
| whatever is the local phone company, and the local incumbent
| cable TV company.
|
| If you're unlucky enough to live in a place where the phone
| company is operating old degraded DSL on copper phone lines,
| and doesn't care to overbuild it with single mode fiber for
| GPON, and the cable TV company is also something similarly
| large and slightly evil like Comcast, you're almost out of
| luck. This is a political and regulatory problem that allows
| the local franchise agreement for the phone company and cable
| tv company to be renewed in perpetuity without demanding solid
| metrics for improvement of service.
|
| If you are very fortunate, there will be an entirely single
| mode fiber-based third-party provider which competes with the
| previous two mentioned things. Doing what's called and
| "overbuild" as a new entrant for this is very capital intensive
| and requires a lot of physical outside plant cabling work and
| infrastructure at layer 1 in the OSI model. It also requires
| appropriate cooperation from whichever local entity owns and
| controls the wood utility poles (again, a political problem).
| sschueller wrote:
| The primary reason the cost for init7 is so low is that for
| example in Zurich city the fiber network is owned by the city
| (like water pipes and power lines) and paid for by the tax
| payer. Any provider can provide their service over it.
|
| Places outside the city that don't have these kinds of fiber
| connections aren't this lucky.
| walrus01 wrote:
| There are a number of county sized public utilities districts
| in Washington state which are doing essentially the same
| thing. These particular ones are also the last mile
| electrical grid operator connected to some of the big
| hydroelectric dams in the region.
| foolmeonce wrote:
| > paid for by the tax payer.
|
| Actually the Zurich build out was only financed by the tax
| payer to be done by the electric company and Swisscom.
| Internet providers must each have equal options to lease the
| last mile from whoever built it out in a given city or town;
| it isn't free.
| kbenson wrote:
| Yes and no. Local ISPs in the U.S. are also laying/stringing
| their own fiber and providing cheap (cheaper than shown here)
| service. They are starting to move into 10/25 gigabit speeds
| too, as the costly part (running the fiber) doesn't do much
| to limit the speed, you just have to update the equipment at
| each end.
| h0nd wrote:
| If you do not live in an area covered by them (which is the
| majority), it is the same as you say. Situation is not perfect
| here, either.
| sliken wrote:
| Was hoping for similar but 10 gbit. Anyone know a nice
| small/quiet linux box with 2x10 gbit for use as a router?
|
| I have a 8 port x 1gbit ubiquiti router, but ubiquiti seems to be
| going downhill, and 8 port x 10gbit routers are pretty expensive.
| My plan was a 2 port router and use vlan tagging to a 8+ port 10g
| switch. That way I can have separate networks for trusted
| ethernet, trusted wifi, untrusted ethernet, and untrusted wifi.
| tyingq wrote:
| I would go with a used SuperMicro A1SAI-2750F motherboard. It
| comes with an Atom 8-core C2750 (20w TDP) and a single 8 lane
| PCIe slot, but doesn't need active cooling. Supports ECC ram
| too. There's a passive heatsink on the CPU that works fine. I
| believe that would keep up with 2x10gb, be quiet, and power
| efficient.
|
| They are $200 or so used on eBay. So not terribly expensive for
| something that can push 10g. And it will fit in any Mini-ITX
| case.
|
| Don't be put off by the "Atom" branding. The C2750 was no
| slouch for its heyday.
| eqvinox wrote:
| Note the Atom C2xxx series is the one where Intel f*cked up
| the LPC bus I/O drivers and it dies after a few years. Don't
| buy them used unless you're comfortable soldering a resistor
| to a clock line to extend its life.
|
| This is fixed in the C0 stepping of the CPU, but finding out
| the CPU stepping is difficult even on new boards.
|
| (cf. https://www.intel.com/content/dam/www/public/us/en/docum
| ents... - this is AVR54)
| deckard1 wrote:
| I'd probably go with SuperMicro X10SDV-4C-TLN2F. It already
| has 10GbE. Xeon D-1521 2x 10GbE ports
| onboard 6x SATA3 ports Accepts ECC RDIMM DDR4
| (usually cheaper than UDIMM ECC) 1x PCIe 3.0 x16
| 1x M.2 slot
|
| There is one on ebay right now for $230 USD shipped.
|
| Also, if you're going for lower power consumption I believe
| you can disable the IPMI and the 2nd 10GbE port if you don't
| need it. There are other ways to reduce consumption too. Turn
| off anything you can on the motherboard that you don't use
| and use higher density DIMM sticks (and less of them, if you
| can, e.g. 1x16GB rather than 2x8GB). Also worth paying
| attention to the PSU's efficiency curve and making sure your
| PSU is decent quality (reliable + 80+ gold).
| eqvinox wrote:
| The cheap ones (including the one you mentioned) are
| useless because they have 10Gbase-T. You need SFP+ slots to
| hook up your provider's fiber.
|
| Which is kinda sad too because implementing 10Gbase-T is
| more expensive to implement _and_ more expensive to run
| (because the 10Gb-T PHY burns power.) Sadly, there 's not
| enough knowledge about this going around.
| deckard1 wrote:
| yes, well you can get a cheap transceiver for that.
| You're still going to have to talk to something else and
| that all goes out the window if you have existing copper
| in your house or need to have wired devices (NAS, smart
| TV, xbox, literally anything at all). You probably can't
| even find a wifi AP that has SFP+ that doesn't cost you 3
| arms and 5 legs.
| tyingq wrote:
| >There is one on ebay right now for $230 USD shipped.
|
| Ah, yeah, that's a better deal with the 2x onboard 10Gbe.
| traspler wrote:
| I'd love to get something simple that could handle a 10Gbit
| fiber7 connection and act as a 10Gbit Router & Switch in my
| home. My skills and patience in this area are a bit limited so
| going as deep as Michael is not something I could do with
| confidence. Narrowing down the hardware to something that is
| not only powerful enough, cheap enough, compatible with
| something like OPNsense and available to normal humans has kind
| of scared me off a bit.
| opencl wrote:
| AsRock makes an X570 ITX board with 2x10gig ethernet, the
| X570D4I-2T. Though at $500 for the board alone the resulting
| system is not going to be particularly cheap.
| smolder wrote:
| I would look at getting a Xeon-D board. They don't draw much,
| come in mini ITX, and mine have 2x10GbE plus 2x1GbE built in,
| for a price that's reasonable when you look at standalone NIC
| costs. The CPU is soldered on, so the cost includes that, too.
| zamadatix wrote:
| Look for an old but not outdated enterprise switch and grab a
| refurb. Most very large enterprises redo the network on roughly
| a 7 year cycle and in 2014 10g uplinks with tons of 1g devices
| was very standard. E.g. for less than 100$ you can usually pick
| up an ers 4826gts on ebay which has 24 ports of 10/100/1000
| copper with POE+ capability and 2 1/10G SFP+ ports. It has
| hardware switching and routing which will perform better than
| software, particularly on latency.
|
| As far as noise 1u devices are usually a bit whiney even when
| not pushing much air but that's usually fixed with 1 or 2 200mm
| PC fans.
|
| You'll still likely need something with NAT, preferably
| hardware NAT, for the actual internet handoff. Some of this
| class enterprise device have NAT support (not the 4826) but
| that'll usually be in CPU and perform not that great especially
| if you have like a gig connection. Or if you want to go pure
| software here any cheap mini PC with 2 gigabit Ethernet ports
| should be able to handle NAT reasonably for gig internet or
| less since it doesn't also have to do the 10g internal
| route/switch on top.
| blackcat201 wrote:
| If anyone is interested in building 10Gb router on a budget you
| can buy Mellanox OCP NIC with an adapter from OCP to PCIe for
| around 5~30 USD ( depend on your location )[0]. I recently build
| one 10G router with 4x10Gb, 2x1Gb ports for around 200 USD using
| second hand PC.
|
| [0] https://www.youtube.com/watch?v=hkf3PkuKdOA
| tohmasu wrote:
| > Our network cards need PCIe 3.0, so that disqualifies 5
| chipsets right away: only the A520, B550 and X570 chipsets
| remain.
|
| No, the linked Wikipedia page only shows the PCIe lanes connected
| through the chipset and doesn't account for PCIe lanes provided
| directly from supported CPUs. X470 supports CPUs which have PCIe
| 3.0 and a board with x8/x8 mode like the ROG Strix X470-F
| https://www.asus.com/microsite/motherboard/AMD-X470/ should work
| just fine (and has no fan).
| gigel82 wrote:
| Damn, $70 / month for 25Gbps symmetric; meanwhile, we're paying
| $130 / month for 1Gbps (400Mbps really) down / 40Mbps up in
| Comcast monopoly-land in PNW.
| rwmj wrote:
| Here I am having just had FTTH installed, but capped to a mere
| 160 Mbps. The main problem I am having is not the router (which
| handles such a "slow" speed fine) but the wiring throughout the
| house. It's all copper ethernet and because of the lengths and
| the fact that the runs must go parallel to some mains electricity
| cables, it tops out at 100 Mbps. I hope the poster has already
| put fibre around his house.
| secure wrote:
| "The poster" (hey!) has put fiber around their house :)
|
| https://michael.stapelberg.ch/posts/2020-08-09-fiber-link-ho...
|
| https://michael.stapelberg.ch/posts/2021-05-16-home-network-...
| great-potential wrote:
| Well, hate to tell you but you'll be limited 10Gb/s with these
| Intel cards because they require FEC.
| secure wrote:
| You mean 10 Gbit/s per link? Even on the 25 Gbit/s card?
|
| Where can I read more about FEC (forward error correction?) and
| how that affects link speeds?
| great-potential wrote:
| Yes the transceivers recommended by Init7 (bidi-LR) do not
| support FEC and you'll be running in degraded mode (according
| to the controller datasheet), and I don't think this is
| something that can be achieved by coding of the transceivers.
| adamcharnock wrote:
| What is the practical consumer (or even SME) use of a 10 or 25G
| connection?
|
| As a consumer I'd definitely get it for the fun of it, but what
| is the point in reality?
|
| We've just launched an ISP a couple of weeks ago here in Europe.
| Peak bandwidth use is about 3mbps per customer, and customers
| will download the same amount of data regardless of their plan.
|
| I acknowledge this could be lack of imagination on my part, but
| the progression of bandwidth availability seems to be wildly
| outstripping demand (at least when it comes to fibre
| deployments).
|
| Perhaps someone living in SF with a 25G connection can disabuse
| me of this notion.
| baybal2 wrote:
| > What is the practical consumer (or even SME) use of a 10 or
| 25G connection?
|
| Remote file storage, leasing servers for things like icecream,
| or basic HPC, self-hosting for a small business, even a small
| DC can run off it.
| tzs wrote:
| At what speed do you start to run into limits on the server
| side when it comes to remote file storage?
| adamcharnock wrote:
| Remote file storage I can just about see. But most consumers
| are streaming content (certainly TV, but possibly
| increasingly games), so where are all these multi-gigabyte
| files coming from on average 20/30-year-olds laptop/iPad? I
| don't think people are shooting _that_ much video, are they?
| baybal2 wrote:
| Renting storage, and accessing it over 10gbps at home may
| be cheaper than building your own multi-tb storage rack.
| adamcharnock wrote:
| Absolutely. But from what I'm seeing, the average
| consumer's need for storage space is decreasing not
| increasing.
| baybal2 wrote:
| Average consumer? Internet companies are responsible for
| a few percents of the labour market in some parts of the
| world.
|
| Not so few people genuinely need gigabit+ internet for
| work.
| lbotos wrote:
| is this the icecream you speak of?
| https://github.com/icecc/icecream I've never heard of it so I
| was intrigued.
| baybal2 wrote:
| Yes, so far a tool in a way better condition than distcc,
| though both are very poorly maintained.
| slumdev wrote:
| Internally:
|
| Every once in a while, I move VM images between machines. Even
| at 1Gbps, it's a drag to sit there and watch them transfer. Not
| really a big deal, but it'd be nice to see them move faster.
|
| Externally:
|
| I back up my Google Drive periodically. 500GB takes a while to
| transfer.
| scandinavian wrote:
| I would be very surprised if you could even break 2.5 gbps
| towards google drive. Saturating 25 gbps to anything the peer
| out of country would be basically impossible most places.
| adamcharnock wrote:
| I agree, and this is totally the kind of thing I would do
| too. But I don't think this represents average consumer
| behaviour.
| vitus wrote:
| > I back up my Google Drive periodically. 500GB takes a while
| to transfer.
|
| Do you download the whole thing each time, vs just changed
| files?
|
| Either way, (scheduled!) periodic backups sounds like a thing
| that could happen over trough (well, while you're sleeping),
| in which case a 200Mbps connection would be more than
| adequate for your use case (~5.6 hours for 500GB).
|
| The numbers obviously scale linearly with the size of your
| download (a common anecdote I hear re: people filling up TBs
| of hard drives is via lots of RAW photos), so in that
| scenario, you'd need to transfer 30-40TB overnight (8 hours)
| in order to saturate a 10G uplink; you'd likely saturate hard
| drive write speeds first, not to mention you'd need multiple
| HDDs/SSDs connected to even store that much data in the first
| place.
| slumdev wrote:
| It was a cinch when I had an always-on Windows box. I'd
| make a request via Takeout, save it on my Drive, and the
| Backup and Sync app would automatically download it to my
| local copy of the drive. But that box is now running
| Ubuntu, and I haven't gotten around to replacing Backup and
| Sync with one of the other (non-free, non-Google) options.
| crispyambulance wrote:
| I can't imagine ANY reason that a private household would
| actually need a 10G or 25G connection unless it's doubling as a
| workplace and they're hosting servers. Even that seems like a
| temporary stop-gap situation, get some rackspace in a
| datacenter already!
|
| Maybe they're not selling enough 25G to their business
| customers and they're trying to get private users onto 25G?
| mjevans wrote:
| For download?
|
| Working from home with large media assets of any sort.
|
| For upload? Yes, I need ANYTHING faster than Comcast's BS so
| I can actually do any work OR even backups without killing my
| ability to do other things.
| adamcharnock wrote:
| I can see that. Certainly if they already have the
| infrastructure deployed then a residential usage pattern on
| 10G is going to look identical to that on 1G (and I'd wager
| identical to that on 100M in most cases). In which case they
| can get more recurring revenue for likely only an increased
| one-off cost.
| lbotos wrote:
| When I clicked through to the provider it looked like the
| only difference between 1/10/25G was the setup cost. But I
| might have mis-read.
| rand846633 wrote:
| Init7,OP's isp is a non profit
| chinathrow wrote:
| No, they are a regular for profit company.
| detuur wrote:
| Regardless, their monthly (annual actually) fees are the
| same for 25G/25G as for 1G/1G. Only the setup fees are
| higher, which they attribute to more expensive optics.
| 10G/10G is completely identically priced to 1G/1G.
| johnwalkr wrote:
| I have 10G in Tokyo and I think it's about $60 USD per month vs
| $40 for 2G. It's definitely overkill but it sure is nice to
| download a 60GB steam game in a minute or two.
| qeternity wrote:
| The problem with 10g is that most servers aren't on 10g yet
| and if you're transiting a peering connection of a smaller
| ASN at some point, even that is likely not 10g
| martinald wrote:
| Agreed, I struggle to get anywhere close to 1gbit/sec apart
| from Steam.
| n3dm wrote:
| This hardly is relevant to your internet speeds with steam
| downloads. It is much more CPU dependent when you are
| operating at such high speeds.
| eqvinox wrote:
| A modern CPU + NIC combination will be twiddling its thumbs
| on a bulk download at 10G. There's a whole sleuth of
| optimization and offload features between the NIC and the
| CPU, and they work particularly well with a low number of
| high bandwidth data streams.
|
| The real problem starts when you're doing lots of small
| packets all over the place. Which is not something you'll
| likely run into at home, ever.
| n3dm wrote:
| No, it's still nothing to do with that for steam. Its
| their compression.
| yakubin wrote:
| I work from home and regularly need to download large files
| from work servers. Several months ago I noticed that my
| internal network was 100Mb/s, which was dreadful. Downloads of
| 500MiB files would take tens of minutes, because Tomcat would
| drop connections on such a slow download. And even without
| dropped connections it generally took a long time to download
| anything, and Zoom call quality was regularly degraded (I
| sometimes heard my colleagues as androids and vice versa). I
| fixed the issue and now I get the full 700M/s that my ISP
| offers (and 1G/s internally over a Netgear switch for NFS
| transfers). I could get a 1G/s, but it would cost me more and
| 700M/s is fine for the most time. But sometimes to get a
| shorter feedback loop on what I'm doing I could imagine getting
| a 5G/s. I think it would be optimal. But 10G is clearly
| overkill.
| e12e wrote:
| > I noticed that my internal network was 100Mb/s, which was
| dreadful. Downloads of 500MiB files would take tens of
| minutes, because Tomcat would drop connections on such a slow
| download. And even without dropped connections it generally
| took a long time to download anything, and Zoom call quality
| was regularly degraded.
|
| Was you network setup broken? Did your firewall block ICMP?
| Ancient servers like tomcat and apache should handle slow
| networks in stride, even with modern updates - and 500MB at
| 100mbps is 40 seconds?
|
| Now I can't imagine running 100mbps today for a home
| network... So upgrading makes sense. But your problems
| doesn't sound like they were caused by your lan _speed_.
| ClumsyPilot wrote:
| "Zoom call quality was regularly degraded"
|
| It sounds like something else was dodgy on your previous
| network, surely Zoom cannot use up 100mb/s connection
| yakubin wrote:
| When one side sends data at 1Gb/s and the other receives at
| 100Mb/s, it's worse than if the first side sent data at
| 100Mb/s. Here's a description from Julia Evans:
| <https://jvns.ca/blog/2018/07/12/netdev-day-2--moving-away-
| fr...>
|
| Basically my ISP router would receive data at full speed,
| while internally I would have a lot of dropped packets.
|
| Additionally, I usually use the web browser to view Jira or
| copy files over NFS during calls, and I'm sure there are
| some background apps contributing to the traffic as well.
|
| The one thing that fixed all my problems was reconfiguring
| the network driver to set 1G bandwidth on my NIC instead of
| the 100M that it set automatically. Yeah, Linux Desktop
| problems...
| iforgotpassword wrote:
| > Basically my ISP router would receive data at full
| speed, while internally I would have a lot of dropped
| packets.
|
| This is not how this works. The other end of your zoom
| call doesn't go "hey the person I'm talking to is on a
| 700mbits line, so that's how much data I'll send." By
| that logic you could extend this to the network of your
| isp, which is probably 10 or 100gbits, so you should have
| a lot of packet loss where the transition from 10gbits to
| 700mbits happens. And it's not even clear how the other
| peer could even know how fast your internet connection
| is, or the internal network of your ISP. That is what
| congestion control is for. Your peer can't know your
| modem's connection speed nor your LAN's speed. Software
| will simply observe how fast it can send data before
| packet loss starts to happen. This obviously requires
| that the there is some sort of feedback by the peer
| you're talking to. If you're using TCP, the OS will do
| that for you for free. With UDP, you have to implement
| that logic yourself in your application. And that's why
| it doesn't matter where the bottleneck is, it could very
| well be somewhere "in the middle" between two ISPs.
| tgsovlerkhgsel wrote:
| 10 Gbps can give you _some_ advantage when downloading big
| games from a network that can support it on high end hardware.
|
| Games are competing for "who can eat the most storage", it
| seems, and at 231 GB for CoD:MW (https://gamerant.com/pc-games-
| file-size-hd-space-biggest-hug...), you'll wait over half an
| hour before you can play with "only" a 1 Gbit connection.
|
| The bigger the game, the bigger the incentive to just re-
| download when needed if you can do it fast, since your storage
| is limited.
|
| The 10 -> 25 Gbps step is definitely just bragging rights,
| which I'd call well deserved given the lengths you have to go
| to to actually reach those speeds.
| louwrentius wrote:
| I really respect this build, the only thing that I don't like is
| the power consumption at idle of ~50 Watt. I understand that this
| is probably due to the used NICs but still.
|
| It might almost be interesting to setup a second router (a Pi4
| might do) for regular casual internet usage + VRRP and only turn
| on this beast when more bandwidth is required.
|
| The 25Gbit machine would be the VRRP master and the Pi4 the
| slave.
| r1ch wrote:
| One thing I've run into when building my own 10gbps router that I
| didn't see mentioned - you probably need to disable power saving
| / frequency scaling. The handling of packets is done in software,
| so when you start a download and the flood of initial data comes
| in at 10gbps, your CPU will still be running at 600 MHz and
| unable to keep up. It ramps up within 5-50 milliseconds depending
| on CPU, but for a TCP download the sender will have already
| interpreted the dropped packets as congestion and reduced their
| upload rate.
|
| You can monitor this with ethtool stats to see how many packets
| the NIC dropped due to host buffers being full.
| secure wrote:
| In none of my tests did I need to change power saving settings.
| My iperf3 transfers stayed at 10 Gbps throughout.
| r1ch wrote:
| You need to check your dropped packet statistics to know for
| sure. iperf3 TCP over LAN will not see throughput affected by
| a few dropped packets, a stream 100ms away may take longer to
| recover. High bandwidth UDP testing with iperf3 should also
| expose this, the first measurement will usually see dropped
| packets.
| bluedino wrote:
| He was only using 15% of the CPU at max speed
| r1ch wrote:
| You can't measure the CPU at 1 second granularity to see this
| problem, in fact measuring the CPU at the sampling rate
| required may be enough to raise the CPU frequency :).
___________________________________________________________________
(page generated 2021-07-10 23:00 UTC)