[SOLVED] Scheduled Reboots?

[Ximbiot]

I have two Hardy Heron (8.04LTS) Server installs, one of them almost clean as of last week with little other than the minimum install and Subversion on it, and both reboot almost every morning at 7:30am EST. I have been digging through cron config files and logs to figure out why but I haven't been able to track the source of the problem down.

Does anyone know why this might be happening? It is really annoying as it keeps interrupting a scheduled backup over the network of a large drive. As near as I can tell, it has been going on since I installed the machines, but I didn't notice until I tried to set up the backups.

It happens almost every day, though it appears to skip some days unpredictably. The missing reboot on the 11th below may have had to do with a power outage, but I verified in the syslogs that both machines were working this weekend before a reboot yesterday (and the long-lived backup process made it through the weekend leaving files timestamped through 7:28am Monday), and both skipped this morning:

older$ last |grep 07:30
reboot system boot 2.6.24-19-server Mon Aug 18 07:30 - 13:16 (1+05:46)
reboot system boot 2.6.24-19-server Fri Aug 15 07:30 - 13:16 (4+05:46)
reboot system boot 2.6.24-19-server Thu Aug 14 07:30 - 13:16 (5+05:46)
reboot system boot 2.6.24-19-server Wed Aug 13 07:30 - 13:16 (6+05:46)
reboot system boot 2.6.24-19-server Tue Aug 12 07:30 - 13:16 (7+05:46)
reboot system boot 2.6.24-19-server Sun Aug 10 07:30 - 13:16 (9+05:46)
reboot system boot 2.6.24-19-server Thu Aug 7 07:30 - 13:16 (12+05:46)
reboot system boot 2.6.24-19-server Wed Aug 6 07:30 - 13:16 (13+05:46)
reboot system boot 2.6.24-19-server Tue Aug 5 07:30 - 13:16 (14+05:46)
reboot system boot 2.6.24-19-server Mon Aug 4 07:30 - 13:16 (15+05:46)
reboot system boot 2.6.24-19-server Sun Aug 3 07:30 - 13:16 (16+05:46)
reboot system boot 2.6.24-19-server Sat Aug 2 07:30 - 13:16 (17+05:46)

fresh$ last |grep 07:30
reboot system boot 2.6.24-19-server Mon Aug 18 07:30 - 13:21 (1+05:51)
reboot system boot 2.6.24-19-server Fri Aug 15 07:30 - 13:21 (4+05:51)

[MJN]

Post the syslog for the few moments before a reboot and see if it gives any clues - you might just have missed them.

(I like the wtf tag by the way!)

Mathew

[Ximbiot]

Thanks! Here are the log lines you requested, starting with the cron.daily syslog restart through the latest reboots.

The older server is a mail server and thus the syslog is extremely messy with blocked connection attempts, but if I remove references to spamd and postfix, this is what is left:

lois$ egrep -v 'spamd|postfix' syslog.0 |head -10
Aug 18 07:11:49 lois syslogd 1.5.0#1ubuntu1: restart.
Aug 18 07:17:01 lois /USR/SBIN/CRON[29621]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Aug 18 07:30:18 lois syslogd 1.5.0#1ubuntu1: restart.
Aug 18 07:30:18 lois kernel: Inspecting /boot/System.map-2.6.24-19-server
Aug 18 07:30:18 lois kernel: Loaded 28743 symbols from /boot/System.map-2.6.24-19-server.
Aug 18 07:30:18 lois kernel: Symbols match kernel version 2.6.24.
Aug 18 07:30:18 lois kernel: Loaded 17960 symbols from 74 modules.
Aug 18 07:30:18 lois kernel: [ 0.000000] Initializing cgroup subsys cpuset
Aug 18 07:30:18 lois kernel: [ 0.000000] Initializing cgroup subsys cpu
Aug 18 07:30:18 lois kernel: [ 0.000000] Linux version 2.6.24-19-server (buildd@terranova) (gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7)) #1 SMP Sat Jul 12 00:40:01 UTC 2008 (Ubuntu 2.6.24-19.36-server)


The fresh install is my router, dhcp server, and name server. It is also noisy because I do a lot of packet logging. Filtering these out yields about the same thing:

shazam$ egrep -v 'IN=.*OUT=|dhcpd|named' syslog.0 |head -10
Aug 18 06:41:12 shazam syslogd 1.5.0#1ubuntu1: restart.
Aug 18 07:17:01 shazam /USR/SBIN/CRON[5908]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Aug 18 07:30:07 shazam syslogd 1.5.0#1ubuntu1: restart.
Aug 18 07:30:07 shazam kernel: Inspecting /boot/System.map-2.6.24-19-server
Aug 18 07:30:07 shazam kernel: Loaded 28743 symbols from /boot/System.map-2.6.24-19-server.
Aug 18 07:30:07 shazam kernel: Symbols match kernel version 2.6.24.
Aug 18 07:30:07 shazam kernel: Loaded 16049 symbols from 73 modules.
Aug 18 07:30:07 shazam kernel: [ 0.000000] Initializing cgroup subsys cpuset
Aug 18 07:30:07 shazam kernel: [ 0.000000] Initializing cgroup subsys cpu
Aug 18 07:30:07 shazam kernel: [ 0.000000] Linux version 2.6.24-19-server (buildd@terranova) (gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7)) #1 SMP Sat Jul 12 00:40:01 UTC 2008 (Ubuntu 2.6.24-19.36-server)

[MJN]

When are you daily cron scheduled to start?

What is curious is that both servers exhibit the same behaviour.... or do they share the same config (in terms of one being a copy of the other)?

Mathew

[Ximbiot]

When are you daily cron scheduled to start?

lois$ cat /etc/crontab
...
# m h dom mon dow user command
17 * * * * root cd / && run-parts --report /etc/cron.hourly
25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
#

What is curious is that both servers exhibit the same behaviour.... or do they share the same config (in terms of one being a copy of the other)?

They do not share a configuration, other than that they were both installed from the same Ubuntu 8.04LTS DVD and updated from the same Ubuntu Apt servers (ubuntu.com's).

Both have an empty /etc/cron.d and the common /etc/cron.{hourly,daily,monthly} files are:

shazam$ ls /etc/cron.{hourly,daily,monthly}
/etc/cron.daily:
apt aptitude bsdmainutils logrotate man-db mlocate standard sysklogd

/etc/cron.hourly:

/etc/cron.monthly:
standard

[MJN]

I'm afraid I'm stumped...

I must say I thought it was going to be something simple that you'd overlooked but it appears to be too weird for words...

Maybe someone else can come in and continue my stabbing in the dark...

Mathew

[Cheesehead]

What processes are running before the reboot? (please post, if you can)
Anything of interest in their config files?
Which were started manually?

Is this the ONLY reboot (intended or otherwise) that they do in 24 hours? - One of the logs showed another reboot ~30 min earlier.

Have someone, perhaps, saved a strange configuration change on a previous logout? Is your environment different in any other way?

[Ximbiot]

What processes are running before the reboot? (please post, if you can)
Anything of interest in their config files?
Which were started manually?

I don't really know and I'll have to get here before 7:30 and watch `top' closely to figure it out, unless you have a suggestion for how to log all running processes from 7:20 to 7:40 or something like that.

Is this the ONLY reboot (intended or otherwise) that they do in 24 hours? - One of the logs showed another reboot ~30 min earlier.

Actually, the line I think you are referring to is the cron.daily run of logrotate, which is only restarting the syslog:

Aug 18 06:41:12 shazam syslogd 1.5.0#1ubuntu1: restart.

There were some other reboots on the mail server, but mostly they happened during times I was likely in the office upgrading kernels or the like. There are three, however, that happened at an odd time when no one should have been in the office and I cannot explain them yet, but hadn't mentioned them for fear of muddying the waters:

reboot system boot 2.6.24-19-server Sun Aug 10 23:07 - 16:03 (8+16:55)
reboot system boot 2.6.24-19-server Sun Aug 10 22:47 - 16:03 (8+17:16)
reboot system boot 2.6.24-19-server Sun Aug 10 19:39 - 16:03 (8+20:23)

Have someone, perhaps, saved a strange configuration change on a previous logout? Is your environment different in any other way?

On both machines? Unlikely. I am the only one who can log into either of these machines and I certainly didn't do make such a change on purpose.

The mail server may have been in service long enough to have become slightly messy, but on the router, in particular, I started with a minimal Ubuntu 8.04LTS Server install, added Subversion, Bind, & DHCPD, then added some of my own configuration for Bind, DHCPD, the interfaces, & iptables. This machine doesn't do anything except hand out IPs and route packets and I just installed it last Thursday, so its configuration hasn't had much time to diverge from my memory of it.

[MJN]

I don't really know and I'll have to get here before 7:30 and watch `top' closely to figure it out

I don't suppose the cleaner does their rounds at 7:20am and plugs the vacuum cleaner in...?! (sorry to get so technical)

Mathew

[Ximbiot]

I don't suppose the cleaner does their rounds at 7:20am and plugs the vacuum cleaner in...?! (sorry to get so technical)

Similar thoughts had crossed my mind, but when I've encountered cleaning staff here, it has been in the evenings. I won't know that for sure until I come in and stare at the machine and `top' at 7:30am, but I doubt this would have quite such exact timing if it was being caused by regular human activity.