Conversation

Notices

1. niconiconi (niconiconi@cybre.space)'s status on Sunday, 10-Jul-2022 05:39:09 UTC niconiconi

   DO NOT run scheduled requests to a public server at the top of the hour, pick a random minute. The worst victims of this problem are community NTP servers - at every hour and especially 00:00:00 UTC, the traffic spike is just impressive.

   • Santa Claes 🇸🇪🇭🇰🎅 likes this.
   • wakame@tech.lgbt's status on Monday, 11-Jul-2022 07:33:27 UTC wakame

     in reply to

     @niconiconi Pro tip: If you run an NTP server, shut it down for five minutes around midnight for maintenance.
     That will teach them.

     Santa Claes 🇸🇪🇭🇰🎅 likes this.
   • Andreas, DJ3EI (dj3ei@mastodon.radio)'s status on Monday, 11-Jul-2022 07:34:04 UTC Andreas, DJ3EI

     in reply to

     • wakame

     @wakame
     No, it will not teach anything. This has been discussed to some extend in the public NTP server community. My take-away: There is nothing short-term that can be done to rectify the situation.

     There's a "kiss of death" packet defined in the protocol, which you can send to badly programmed rogue clients (it contains no usable time info), or you can ignore their requests. Either will cause many to simply send more requests without exponential backoff.
     @niconiconi

     Santa Claes 🇸🇪🇭🇰🎅 likes this.
   • wakame@tech.lgbt's status on Monday, 11-Jul-2022 07:34:25 UTC wakame

     in reply to

     • Andreas, DJ3EI

     @niconiconi @dj3ei It was more meant as a humorous remark, not an actual solution.

     I personally know too many people who would rather copy/paste fifteen bad stackoverflow examples than to read even the synopsis of a standards document...

     But I just head a great idea for a cyberpunk work of fiction where covert members of the FOSS army sneak into companies and other places housing old hardware and destroy (or even update) them. :blobcataww:

     Santa Claes 🇸🇪🇭🇰🎅 likes this.
   • niconiconi (niconiconi@cybre.space)'s status on Monday, 11-Jul-2022 07:34:26 UTC niconiconi

     in reply to

     • wakame
     • Andreas, DJ3EI

     @dj3ei @wakame Yep, KoD will never be respected by legacy clients already deployed on millions of zombie-like embedded & networking devices that never receive any update in their lifetimes. We have to deal with them for as long as Internet exists...

   • Ethan Blanton (elb@mastodon.sdf.org)'s status on Monday, 11-Jul-2022 07:34:51 UTC Ethan Blanton

     in reply to

     @niconiconi I follow this even for local services, as historically it didn't take very many I/O intensive operations to be trouble for a random PC. SSDs have improved that, but ...

     Santa Claes 🇸🇪🇭🇰🎅 likes this.
   • niconiconi (niconiconi@cybre.space)'s status on Monday, 11-Jul-2022 07:35:14 UTC niconiconi

     in reply to

     Also consider rate-limiting and exponential backoff in retry loops. Otherwise the results can be quite spectacular.

     Some Internet folklore from China: In 2009, a random guy paid for DDoS to attack a competing game server and disabled its DNS service. Just a regular day, right? Following that, a popular video player with 1 billion installations, appropriately named Storm, all entered an endless retry loop to phone home, DNS requests flooded China Telecom's backbone and caused a nationwide network outage. The attackers suddenly found themselves to the state's enemies.

     Santa Claes 🇸🇪🇭🇰🎅 likes this.
   • Doc Edward Morbius ⭕​ (dredmorbius@toot.cat)'s status on Monday, 11-Jul-2022 07:35:29 UTC Doc Edward Morbius ⭕​

     in reply to

     @niconiconi This is known as the #ThunderingHerdProblem

     (Especially in system-configuration contexts, but also others.)

     Santa Claes 🇸🇪🇭🇰🎅 likes this.
   • Will Orr (worr@bsd.network)'s status on Monday, 11-Jul-2022 07:35:52 UTC Will Orr

     in reply to

     @niconiconi OpenBSD has a neat crontab(5) extension for this

     https://man.openbsd.org/crontab.5

     The ~ character is used to indicate that a random value between a range (or without a range, the full range of the field) should be used as the time to execute the job.

     Santa Claes 🇸🇪🇭🇰🎅 likes this.