„Alles kann warten, nur die Zeit nicht“
Everything can wait, only time doesn’t
This is how the Lambda Zen Temple in Glarus received me a few years ago (heck, has it been 8 years?) for a very warm (in winter!) and familiar supper, and it is one of those phrases that has stuck with me.
Indeed time flies, and by now it’s been about two decades since I started being a Sysadmin in one fashion or another.
In all that time, today happens to be the first time since then that I notice it is System Administrator Appreciation Day, on the actual day, and not several weeks after it has passed :-D.
It is also a day in which my lack of focus is “driving” (aka: helping me stomp through the day), so it’s been a busy day, in a weird way. Let’s see how.
Some people think I work for eXO.cat; but I don’t :-D, nobody does actually.
While some of these tasks are for this association, most have nothing to do with it but rather with my job, I’m just much more comfortable mentioning eXO ^^ (and sometimes: work-work happen to support eXO-work, so that’s a happy overlap).
So, here is a pretty incomplete list of sysadmin-y tasks that have been happening today.
Monitoring is up, no important alerts are triggering, some tickets can be closed as the reasonable “let’s wait if this happens again” time has passed.
Some other tickets had to be pursued, day to day things.
vaultwardenIt looks like v1.37.0 is compatible with clients with version 2026.7.0+, which means previous versions aren’t!
I noticed this because users reported it (and mentioned the web UI was a good workaround, nevertheless I ought to be aware).
Turns out, due to the FreeBSD ports tree freeze, the build servers haven’t picked up the new version, so I was faster building and deploying it myself.
dovecot-ftsFigured out why dovecot’s FTS has been funky in some mail servers:
It looks like mailboxes have grown to a point where searches would produce these messages in the logs:
Fatal: fts-flatcurve: Out of memory when indexing mail (cxxrt::bad_alloc); UID=XXXXXX (Hint: increase indexing process vsz_limit or define smaller commit_limit value in plugin config)
While indeed changing the commit limit produced a different error (progress?), it wasn’t enough:
Fatal: master: service(imap): child 95825 returned error 83 (Out of memory (service imap { vsz_limit=256 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump)
In the end, these two changes were needed:
diff --git a/type/__evilham_dovecot/manifest b/type/__evilham_dovecot/manifest
index 96b2623..6d5d614 100755
--- a/type/__evilham_dovecot/manifest
+++ b/type/__evilham_dovecot/manifest
@@ -114,6 +114,9 @@ EOF
__file "${DOVECOT_CFG_DIR}/10-master.conf" --mode 0444 --source - <<EOF
# Managed remotely, changes will be lost
+# This was causing FTS issues
+default_vsz_limit = 512M
+
## These are cdist-provided arguments
mail_plugins = \$mail_plugins $mail_plugins
@@ -248,10 +251,14 @@ plugin {
# Other plugin settings
# fts_flatcurve_<setting> = <setting_value>
+ # This was causing issues
+ fts_flatcurve_commit_limit = 100
+
# From dovecot's docs
# https://doc.dovecot.org/configuration_manual/fts/#enforce-fts
fts_enforced = yes
fts_autoindex = yes
+
}
EOF
People can now properly search their mailboxes 🥳
OTOH, this helped remind me that I want to migrate to cyrus-imap or stalwart in some near future, due to dovecot’s (somewhat) recent project changes.
While FreeBSD 15.0 is still supported, FreeBSD 15.1 is out, and I have already tested it enough to upgrade just about everywhere that can afford a couple minutes downtime.
snacA notable exception was eXO.cat’s fediverse instance, a snac server running FreeBSD; it had been left behind in 14.3 (shame on me), so now it is running 15.1 with all the latest upgrades.
Apropos eXO.cat: A very cool article was published about the association on Xarxanet, feel free to read it even if you need translation, it is a lovely grassroots project I am very fond of.
Feel free to comment on the fediverse!
Managing services that many people, communities and collectives rely on is a pretty invisible job, and it is often discouraging to only hear about issues, and not about people appreciating the work.
Can’t tell you how rare it is for people to write just in appreciation, if you think it’ll annoy them, please try to over come that sentiment and write to them anyways.
Be kind, be thankful, and do it explicitly, even if they may not have the bandwidth to reply, they will appreciate it.
Anyways, I wrote a couple emails to lovely humans doing great work because they believe in it.
jitsi instancesTalking about things used by many people, I upgraded all jitsi instances I manage, including meet.eXO.cat / meet.guifi.net, which we empirically know that is being used by a lot of people.
Noticed a server was performing to its limit due to RAM constraints, so I gave it a bit more wiggle room.
Some downtimes and maintenance tasks had to be scheduled.
And it looks like eXO.cat needs new switches and a server, so we have to schedule Datacenter work for the weeks ahead, freeing the time for that in the calendar was also necessary.
Now, after this exercise in memory, I’m off to eat something tasty and enjoy the rest of the day!