Post by Hanno Zulla

It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting may have on the global electricity grid.

@hzulla Similar problem for servers and clusters with on the hour spikes in compute. It was a big issue in my previous job. Especially around 14:00 when most data sources we used had been updated.

@veronica @hzulla

Yeah, I know at my last place we had to add deliberate a deliberate couple minutes of jitter to user-scheduled jobs because clients would all choose the same time!

@amethyst We had the additional fun result that since our VMs ground to a halt at 14:00, people moved to scheduled CI jobs instead, so then between 14 and 16 each day, no one could get a PR through because all the runners were queued up for the next 2h. 😅

@hzulla would it actually help? Cron would still wake up at the minute mark in order to sleep again for a few more seconds.

I suppose you’d have two smaller spikes: at the minute, and some seconds later; and the second one would be different for every machine and so average out on a global scale. The first spike would still exist, just smaller, perhaps.

@benjamineskola @hzulla I think that premise, "cron would wake up on the minute mark" is incorrect: you'd randomize the time it sleeps, in both positive and negative directions Compare how systemd-timers sleep; they already have a well-configurable randomization of wake times, exactly to combat loading a shared resource with a large number is simultaneously triggering machines.

@funkylab I mean, yeah, if you modified cron to do so. I was assuming that this was adding a random sleep to the start of the cron job itself, which was perhaps a misinterpretation. (I think OpenBSD cron already has this feature, but I’m not sure if it avoids the wake up or only delays the start of the task; and I haven’t seen it in other crons.)

Although of course this might be a premature optimisation, because it hasn’t been shown that there’s a measurable amount of energy being used by cron wakeups in the first place.

@hzulla

@benjamineskola @funkylab @hzulla maybe I'm off the mark here but cron implementations have always honored the RANDOM_DELAY env variable and it's always been recommended to set it to a non-zero value. IIRC systemd timers also honor it but I don't know if it's set by default or not.

@hzulla Cron, of course, dates back to a world in which:

- computers were a significantly smaller part of the load on the power grid
- we didn't have tightly synchronized clocks everywhere (no NTP yet)
- we didn't have architectures in which power consumption was as tightly linked to computational load

Software often outlives all sorts of assumptions!

@mattblaze @hzulla Even way back when, I generally offset my cron jobs by a few minutes, just to avoid CPU and RAM (“core storage”) conflicts with other cron jobs.

@SteveBellovin @hzulla Yeah, but back then, you mostly only had to worry about conflicts on the same machine.

@mattblaze @hzulla Which Cron? I mean there are, for good reasons, many different versions of Cron out there. There are Crons that target specific use cases the original writers didn't consider (like having computers that are turned off or in standby from time to time).

That's the wonderful thing about Free Software. It's not a "product" but instead a chorus of competing ideas and forks which explore new things.

@casandro @hzulla Um, if open source or free software or whatever buzzword the kids are calling it these days solves this problem, why wasn’t it solved already? As you pointed out, cron’s code has been re-implemented many times over the years.

Open source is nice and all, but it didn’t anticipate or solve this.

@mattblaze @casandro @hzulla This isn't really a problem that requires changes in the software but rather changes in the way it's used. There is no escaping the necessity of *some* tasks being scheduled at the exact times provided by the various crons. I used to run a fairly big website on meagre hardware and had to dither a lot of scripts to spread CPU load out, this included cron jobs. This also smoothes the power draw.

@synx508 @casandro @hzulla yes, exactly. The software changes required to have cron behave in a way that’s more friendly to the power grid are relatively trivial here. The issue is (apparently) that no one realized (or cared) that synchronized surges in computation can now have a measurable impact at scale (which is an interesting effect).

@hzulla i can add to this story ;) We once had a service that was integrated into every Windows Mobile that also stupidly checked in our service on a fixed clock. That wasn't a problem as long as Windows Mobile didn't had NTP. But then suddenly Microsoft added that feature and after a short time all phones suddenly had accurate time and DDOSed our service on the fixed interval.....