yum said 301, the machine said 2024

The first thing worth doing is proving the mirrors innocent, since that is where the message points. Following the chain by hand with curl, from my laptop rather than from the host that was failing, which matters more than it sounds like it does and which I will come back to:

Two hops and then the file. The first hop is the plain 301 that pushes you from HTTP to HTTPS, which every sane public mirror does now. The second is MirrorBits picking a mirror near you, and you can watch it think out loud in the headers it emits on the way past:

That is a mirror network doing precisely what it is designed to do, ranking candidates by priority and by my geography, and handing me one. curl -L walks the whole thing and comes out the other side with a 200 and a valid XML file. Nothing moved. Nothing is down.

Which means the 301 in that error message is not a fault report at all. It is a description of normal, healthy infrastructure, printed at the exact moment something completely different fell over.

There is a much cheaper version of that check, and if you have a second machine you should run it before you touch anything else. I have another host on the same LAN, same switch, same egress, same repo files. I ran the same update there.

xcp-ng-release 8.3.0-38, xen-hypervisor 4.17.6-9.3.1.xcpng8.3, the whole 26.1.11-1.3 XAPI stack, straight down from xcp-ng-updates without a murmur.

One command, and in one go I had eliminated a mirror outage, a bad repo file shipped upstream, a broken batch, DNS, my router, and my ISP. All of it, gone from the list. Whatever was wrong lived on the one host, and I had wasted my afternoon looking outward when the control test was sitting there in the next room.

I keep relearning that lesson and I keep being surprised by it, so I am writing it down this time.

Once you know the fault is local, the next question is what the failing host does differently from the working one, and the answer turned out to be about TLS.

The mirrors use Let’s Encrypt certificates on a normal ninety day cycle. mirrors.xcp-ng.org and updates.xcp-ng.org were both valid from 2026-05-31 to 2026-08-29, and the mirror I was actually landing on, xcpng-mirror.as208069.net, from 2026-07-01 to 2026-09-29. All of them comfortably current.

Here is the bit that I think trips everybody, including me. An X.509 certificate does not just have an expiry date. It has a notBefore as well as a notAfter, and a host that believes it is living before that notBefore will refuse the handshake with certificate is not yet valid. That is a sentence most of us have simply never had to read, because in fifteen years of operations the failure has always run the other way round. Certificates expire. They do not un-expire.

Then I ran the least imaginative command available on the failing host:

September 2024. Twenty two months before the certificate it was being asked to trust had even been issued. The other host, checked in the same minute, was on the correct date with chrony synced to within 46 microseconds of its upstream.

One variable, and it was the one nobody thinks to look at.

This is also why that clean curl chain earlier came off my laptop. Run the same curl -sIL on the host with the 2024 clock and you do not get 301, 302, 200 and a tidy XML file. curl follows the 301 to HTTPS exactly as before, hits the same handshake yum was hitting, and stops there with SSL certificate problem: certificate is not yet valid. Which is, in fairness, the truth, and rather better than what yum told me. If you are following along on the broken box and wondering why your output looks nothing like mine, that is why, and you can take it as a free confirmation of the diagnosis.

And a machine you have recently installed is exactly the machine most likely to be confidently wrong about the year. A dead RTC battery, a board that came back from a power cut on its factory default date, an isolated segment where NTP never resolved. Nothing stops you, and everything else works.

Because of two pieces of code in python-urlgrabber that were never really meant to meet each other.

yum on dom0 fetches through urlgrabber sitting on top of pycurl. The header callback updates the object’s idea of what it is downloading the instant it sees a Location: header, which is before anything at all has been attempted at the new address:

So after that first 301, self.scheme is https. Perfectly reasonable so far.

Then the transfer fails, and the exception handler goes to compose a message for me:

A TLS handshake that fails never gets as far as producing an HTTP response code, so RESPONSE_CODE is still sitting at 301 from the hop that succeeded. scheme is now https because the header callback moved it. The condition holds, the first branch fires, and errstr is thrown away without ever being printed.

errstr was SSL certificate problem: certificate is not yet valid.

What comes out instead is HTTPS Error 301 - Moved Permanently, which is the scheme of the hop that failed, the status code of the hop that worked, and the human readable text of neither of them.

That also explains a piece of folklore you will find on the forum, which is that changing the repo URL to https:// makes yum tell you the truth. It does, and it is worth understanding why, because it is not a fix. Point the repo straight at https:// and there is no redirect at all, so RESPONSE_CODE is 0, the else branch runs, and libcurl’s own honest message survives intact. Same failure, same host, same second. Different sentence.

The general form of this is worth carrying around in your head, because it is not specific to XCP-ng or to these mirrors at all: in yum, any transport failure that happens after a redirect is reported as the redirect. TLS, DNS, connection refused, all of it gets the same treatment. If a yum error names a 3xx, the 3xx is the last thing that went right.

I could have stopped at "set NTP", fixed the clock, taken the updates and gone to bed. I am glad I did not, because the interesting part was still underneath, and on this host NTP was already on.

Service running. Service enabled, so it survives reboots. Network fine, DNS fine, egress fine. Every check an operator would actually think to run comes back green, and on that evidence you would confidently conclude that time is not your problem and go and look somewhere else. Which is what I had spent the afternoon doing.

Then:

Zero. Here is /etc/chrony.conf in its entirety, with the comments stripped out:

There is no server line. There is no pool line. chronyd had been up for a day and a half, running flawlessly, doing exactly and only what it had been told to do, which was nothing whatsoever.

That is the failure mode worth naming out loud, because it is invisible from every angle except one. A time daemon with no sources is indistinguishable from a healthy time daemon unless you run chronyc sources, and nobody runs chronyc sources until they already suspect the clock. So the host serenely holds whatever date it was installed with, for as long as you care to leave it, and the first thing that ever tells you about it is a yum error about a redirect.

My best guess at the origin is the installer’s time step, where you can either hand it NTP servers or just set the date yourself. Choosing the second produces precisely this file. It is a reasonable thing for an installer to offer and a reasonable thing to pick when you are in a hurry, and the consequence turns up months later wearing a completely unrelated costume.

So the check is not "is NTP running". Running is not the question. The question is chronyc sources, and if that says zero, you have found it.

The whole diagnostic is one command, and I want that on the record because of how long it took me to type it:

If the date is wrong, the next command decides which of two quite different problems you have:

If that lists sources, the daemon knows where to get the time and has simply not applied it yet, and you can step the clock straight away:

makestep is the step people skip, and it matters. chrony corrects offsets by slewing the clock gently rather than yanking it, which is the right behaviour when you are drifting by milliseconds and completely useless when you are out by two years. You can sit and watch a correctly configured host stay broken indefinitely without it.

If chronyc sources came back with Number of sources = 0, which is the case this whole post is about, then makestep on its own will do exactly nothing and tell you it succeeded. There is no measured offset for it to step to, because there is nothing measuring. You have to give the daemon somewhere to look first. Put the servers into /etc/chrony.conf:

On an isolated network, point those at whatever local time source you have. Then restart the daemon and check that the sources actually came up before you trust anything downstream of them:

The stock config carries makestep 1.0 3, so once there are real sources chronyd will usually step the clock itself within its first few updates and the explicit makestep is belt to that braces. Running it costs nothing and saves you staring at date wondering how long to wait.

Editing the file rather than just stepping the clock is the part worth not skipping. Fix the time without fixing the config and you have fixed it precisely until the next reboot, at which point the host quietly goes back to being wrong and you get to have this afternoon all over again.

Or set NTP from xsconsole, which does the same thing with fewer chances to fat finger it.

Then:

On my host check-update came straight back with the full 28 package July batch, and yum update took the lot and exited 0, with the repository file exactly as it shipped. Total elapsed time once I actually knew the cause: about a minute.

Small thing worth knowing if you are scripting any of this, and pleasingly on theme for a post about misleading exit status: yum check-update returns 100 when there are updates waiting, not 0. Zero means there is nothing to do. Treat 100 as an error and you will have built yourself a second thing that reports the wrong state confidently.

Which brings me to the thing I nearly did, and you probably will too. If you pinned a mirror by hand while you were flailing around, undo it and put http://mirrors.xcp-ng.org back. Hand pinning opts you out of the geo selection you saw in those duplicate link headers earlier, and much more importantly it opts you out of failover on the day that one mirror has a bad afternoon. It is exactly the sort of "temporary" edit that is still sitting there two years later, on the afternoon it finally costs you something.

One worry you can put straight back down, because it stopped me for a few minutes before I did the fix. Dragging a host’s clock forward by two years does not break the host’s own certificate. XAPI issues it with a ten year validity, so a machine installed while it believed it was 2024 is holding one valid until 2034, and correcting the date lands comfortably inside that window. You do not need xe host-refresh-server-certificate for this. There is a real pool join certificate failure that looks alarming in a vaguely similar way, but it is a different animal and this is not it.

The best evidence for all of this is not mine, and I want to be clear about that. There is a forum thread from last November where the person who hit this worked the whole thing out themselves in under an hour, from the same host inside the same session, and left behind the three observations that make it airtight: the 301 with http:// in the repo file, the honest certificate is not yet valid with https://, and then setting NTP and reverting to the original URL and watching it just work. I spent longer than they did and arrived at the same place. There is also topic 9560, from 2024, which is the same symptom on 8.2 and simply ends, with nobody ever having named the cause.

Somebody suggested on that November thread that this deserved a page in the troubleshooting docs. They were right, and it did not happen at the time, which is more or less what the second thread looks like from the outside. There is a documentation change in review now that adds it under the common problems, next to the existing clock entry, plus a line in the update prerequisites saying the plain thing out loud: getting to the repositories needs a correct clock, and the error you get when you do not have one does not mention clocks. It is a small change and I have no idea yet whether it lands in the shape it is currently in, but it should at least mean that three minutes with a search engine gets you the answer rather than two forum threads and a shrug.

The one line takeaway, because it is the line I wish I had read first: if yum blames a redirect, run date before you touch the repo file, and if the clock is wrong run chronyc sources, because a time daemon can be running, enabled, and pointed at absolutely nothing.