Evilham's Blog

FreeBSD adopting a port: maomao

2025-06-25T00:00:00Z

Introduction

The Fediverse is a wonderful place, and the #FreeBSD community over there is rather active :-).

Justine Smithies is one those people with whom we have had multiple constructive interactions, and she has made lovely contributions to FreeBSD’s community, starting with her enthusiasm!

Recently, she’s been toying around with maomao, a Wayland compositor.

And I promised to create a port if she committed to keeping it up to date (neat trick to get new contributors!)

Let’s document both that process, and how she can keep those patches coming in to the ports tree!

Introduction
Table of Contents
FreeBSD and the ports tree
Wayland on FreeBSD?
Maomao
Getting started with ports
Conclusions

FreeBSD and the ports tree

You may want to read my (now outdated) post on updating a FreeBD port, which goes into detail about the Source trees, but most technical bits are outdated by now, given that FreeBSD has moved away from svn and now uses git to manage code.

The short version about FreeBSD‘s Source trees (plural!). Without going into much detail: FreeBSD has a base (or src) tree and a ports tree.

The base tree contains the whole Operating System along with any necessary system utilities and they are developed as a unit.

While the ports tree contains a description of how to compile third-party software under FreeBSD and how to install it manually or how to build an installable binary package, which can later on be built on FreeBSD‘s infrastructure and served by official pkg repositories.

Wayland on FreeBSD?

Contrary to popular belief, Wayland is not something that is exclusive to Linux. While it is Linux-centric, it does support FreeBSD, even as a test target on its CI!

Some things that are worth checking on this topic (though some might be outdated, they are still worth it):

X11 and Wayland: a tale of two implementations, a talk by raichoo on writing hikari
hikari: a FreeBSD-first Wayland composer
FreeBSD Handbook: Chapter 6. Wayland
Example/Tutorial: Pure wayland desktop

There are in fact right now 140 ports with ‘Wayland’ on their description (source: quick pkg(8) magic).

Maomao

As usual with Wayland things, it is written with Linux in mind, but as it often happens, that does not mean the authors are hostile towards supporting FreeBSD, they just might not be aware that the OS can be a target.

In fact, the author recently committed some code that helps mmsg (a companion tool to maomao) bulid on FreeBSD without patches.

So if we run into issues, we can be reasonably sure that a patch will be welcome.

Getting started with ports

I have previously created several ports and update some from time to time, so I was able to take shortcuts and creating the ports was relatively quick.

Setting up `poudriere(8)`

With poudriere(8), we’ll be able to manage building the ports with jails and producing a local repository of packages, that we can easily install, as well as manage the ports tree.

Once poudriere(8) is installed with pkg install poudriere, we can check its configuration under /usr/local/etc/poudriere.conf.

Here is what I like to adapt from the sample file:

# Use the ZFS dataset zroot/poudriere
ZPOOL=zroot
ZROOTFS=/poudriere
# Which is mounted on /poudriere
BASEFS=/poudriere
# Suggested value
FREEBSD_HOST=https://download.FreeBSD.org
# DNS resolution for jails
RESOLV_CONF=/etc/resolv.conf
# Use tempfs(5) for wrdir and data
USE_TMPFS=yes
# Adapt this to your CPU count, leaving some for your regular use
PARALLEL_JOBS=6
# I have found timestamps to be useful
TIMESTAMP_LOGS=yes
# I like to keep old versions around
KEEP_OLD_PACKAGES=yes
KEEP_OLD_PACKAGES_COUNT=2
# Use the 'latest' ports branch when fetching pre-buit binaries
PACKAGE_FETCH_BRANCH="latest"

Getting the ports tree

This can be done in multiple ways, the most convenient one is getting the code from the same repository that is used by developers over anonymous git with poudriere(8):

# Fetch the ports tree
> poudriere ports -c -p default

UPDATE 2025-07-01: If we want to keep the ports tree up to date, we will want to run poudriere ports -u -p default every once in a while.

Creating the port

Luckily, this is a fairly standard port, we start by identifying a somewhat similar port, like x11-wm/swayfx, and using it as a starting point.

Here is roughly what that looks like:

PORTNAME=   maomao
PORTVERSION=    0.7.0

CATEGORIES= x11-wm wayland

MAINTAINER= ports@FreeBSD.org
COMMENT=    A streamlined but feature-rich Wayland compositor
WWW=        https://github.com/DreamMaoMao/maomaowm

# See: https://github.com/DreamMaoMao/maomaowm/issues/127
LICENSE=    GPLv3+
LICENSE_FILE=   ${WRKSRC}/LICENSE

USES=       meson pkgconfig desktop-file-utils

OPTIONS_DEFINE= X11
OPTIONS_DEFAULT=X11

X11_MESON_ENABLED=  xwayland

USE_GITHUB= yes
GH_ACCOUNT= DreamMaoMao
GH_PROJECT= maomaowm

.include <bsd.port.mk>

We then check the meson.build file, which lists all its dependencies.

And adapt entries acordingly:

BUILD_DEPENDS=  wayland-scanner:graphics/wayland \
        wayland-protocols>0:graphics/wayland-protocols \
        evdev-proto>0:devel/evdev-proto
LIB_DEPENDS=    libscenefx-0.4.so:x11-toolkits/scenefx04 \
        libwlroots-0.19.so:x11-toolkits/wlroots019

X11_LIB_DEPENDS=    libxcb-icccm.so:x11/xcb-util-wm
X11_USE=        XORG=xcb,pixman

I also identify that we need the same sample file, version and C flag patches that x11-wm/swayfx implements, so I copy and adapt those:

post-patch:
# Extract (snapshot) version from the port instead of meson.build
    @${REINPLACE_CMD} -i .nogit -e 's/git.found()/false/' \
        -e '/unknown/s/@0@/${DISTVERSIONFULL}/' \
        ${WRKSRC}/meson.build
# Fix C flags
# https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275328
    @${REINPLACE_CMD} -e 's/_POSIX_C_SOURCE=200809L/_XOPEN_SOURCE=700/' \
        ${WRKSRC}/meson.build
# Install config file as a sample
    @${REINPLACE_CMD} -e "s/'config.conf'/'config.conf', rename: 'config.conf.sample'/" \
        ${WRKSRC}/meson.build
# Respect PREFIX for system-wide config
    @${REINPLACE_CMD} -e "s,format('/etc'),format('${PREFIX}/etc')," \
        ${WRKSRC}/meson.build

We update / generate the distinfo file with make makesum.

Then the pkg-plist file:

bin/maomao
@sample %%ETCDIR%%/config.conf.sample
share/wayland-sessions/maomao.desktop

And also create the pkg-descr file:

A streamlined but feature-rich Wayland compositor

1. **Lightweight & Fast Build**

   - _Maomao_ is as lightweight as _dwl_, and its build can be completed within
     few seconds. Despite this, _maomao_ does not compromise on functionality.

2. **Feature Highlights**
   - In addition to basic WM functionality, Maomao provides:
     - Base tag not workspace (supports separate window layouts for each tag)
     - Smooth and customizable complete animations (window open/move/close, tag
       enter/leave)
     - Excellent input method support (text input v2/v3)
     - Flexible window layouts with easy switching (scroller, master, monocle,
       spiral, etc.)
     - Rich window states (swallow, minimize, maximize, unglobal, global,
       fakefullscreen, overlay, etc.)
     - Simple yet powerful external configuration
     - Sway-like scratchpad and named scratchpad
     - Minimize window to scratchpad
     - Hycov-like overview
     - Window effects from scenefx(blur, shadow, corner radius, opacity)

And the pkg-message file, which gets shown upon install:

[
{ type: install
  message: <<EOM
The default configuration works best by installing foot(1) and rofi(1)
alongside maomao.

Also, keep these default shortcuts in mind:
- alt+return: open foot terminal
- alt+q: kill client
- alt+left/right/up/down: focus direction
- super+m: quit maomao
EOM
}
]

Building the port

We do this with poudriere(8), which uses jails and ZFS snapshots.

We first have to create the jail, if we haven’t already:

# Create a builder jail for amd64 based on 14.3-RELEASE
> poudriere jail -c -j 143-amd64 -v 14.3-RELEASE
# We also can keep the jail up to date with:
# poudriere jail -u -j 143-amd64

And we can now instruct poudriere(8) to use our ports tree with this jail to build this port:

# Start jail 143-amd64, with the default ports tree, to build maomao
> poudriere bulk -j 143-amd64 -p default x11-wm/maomao

This will fetch all needed packages along with their dependencies, so we only have to build locally the port we need.

Testing the port

Once the port builds (which required a small code patch), we can install it with:

# Install the resulting maomao binary pkg
> pkg add /poudriere/data/packages/143Ramd64-default/All/maomao-.pkg

And we can then run it as usual.

Updating the port

The recurring work of updating the port lies in updating the Makefile with each release. Usually dependencies are pretty stable, other than that we just have to:

change the PORTVERSION to match the latest release
- In the case of mmsg, that’d be DISTVERSION and GH_TAGNAME, this is because it does not currently have actual releases
update the checksums with make makesum
test that the port builds and works
submit a patch to bugs.FreeBSD.org

Creating the patch

UPDATE 2025-07-01: This guide was missing how to create the patch files in order to submit them.

By default poudriere-ports(8) uses the git+https method of creating and managing the ports tree, which means that the ports tree will itself be a cloned copy of the git(1) Ports repository!

That means that we can use standard tooling to see the diffs, and to format patches!

In order to see everything that we have changed, we can use git diff from any directory in the repository.

Now, let’s use x11-wm/maomao as our current working directory.

We can see the changes to our port with git diff ..

And we can also “stage” any changes to the port with git add . (or git add x11-wm/maomao if running from the root of the Ports Tree).

“Staging” in git-speak means, that we are marking some changes as something that we want to commit to the repository.

In order to check what we are about to commit, we can run git diff --staged.

Once that looks OK, we can add a commit to our ports tree with git commit.

And in order to generate a valid .diff file, we ought to cd into the root of the Ports Tree, and run git format-patch HEAD^.., this means: “please generate the diff that only includes the very last commit”.

This will generate a file called 0001-ShortNameOfCommit.patch, which we can then submit as an attachment to [bugs.FreeBSD.org][bugs.FreeBSD.org].

Submitting the patch

Now that we created two ports: x11-wm/maomao and deskutils/mmsg, we can submit them and hope that they get merged soon.

Since Diane Bruce offered to help commit this, we add her as a reviewer. And since further maintainership is going to Justine, we add her on CC.

You can check the patch and its state here for x11-wm/maomao.

For deskutils/mmsg, we are waiting for the LICENSE to be clarified; once that’s done, we’ll be able to submit the patch to FreeBSD’s ports.

[pr_mmsg]:

Conclusions

Generally speaking, I would advise people who want to get into contributing to the ports tree, to start by updating ports that they use and care about.

Committers and maintainers are usually happy to have a patch to quickly review and test as opposed to having to do the whole work.

In this case, since the port did not exist, it might be easier to nerdsnipe someone into creating the ports, and then helping out with the maintenance ;-) which is always super needed and appreciated!

Trying out Kubernetes: etcd

2023-11-07T00:00:00Z

Introduction

As part of trying out Kubernetes, we are going to set up etcd!

etcd is a “distributed reliable key-value store for the most critical data of a distributed system “

This is where Kubernetes state lives, given how important that is and how it seems to make sense to run it separated from the Kubernetes cluster for High Availability, we will be doing just that, using WMF’s etcd guide as inspiration for decisions that are important.

Introduction
Table of contents
VM Resources
Installing
Setting up the cluster
- DNS discovery
- Provisioning parenthesis
Testing the cluster
- Basic functioning
- Killing cluster members
Conclusion

VM Resources

According to etcd’s Operations documentation at least 8G RAM is recommended for each node, and 2G should be the bare minimum, so we’ll change that for etcd hosts to something in between:

# Changing in VM_BHYVE/k8s-etcd-1/k8s-etcd-1.conf
memory=4G

The limiting factor here will likely be the mechanical hard drives :-), but this is the hardware we have.

Installing

Since we are using Debian and there is (so far?) no good reason here to use other binaries, we’ll just: apt-get install etcd-server (currently v3.4.23, that’s new enough, it defaults to v3 of the protocol).

Setting up the cluster

Since this is a small experiment, we could start the cluster with statically-defined members and that would be a breeze to set up as it is just a matter of passing the right CLI arguments / environment variables.

But we want to try out DNS discovery.

DNS discovery

I’ll use discovery-srv=lab.evilham.com and discovery-srv-name=c1. The c1 serves to tell this cluster apart from others which could exist in the future :-).

Which means we need to set up following DNS entries for this cluster with my cdist types:

# This is knot client syntax
zone-unset evilham.com lab

## Server SRV records
zone-set   evilham.com     _etcd-server-ssl-c1._tcp.lab 300 SRV 0 0 2380 k8s-etcd-1.c1.lab
zone-set   evilham.com     _etcd-server-ssl-c1._tcp.lab 300 SRV 0 0 2380 k8s-etcd-2.c1.lab
zone-set   evilham.com     _etcd-server-ssl-c1._tcp.lab 300 SRV 0 0 2380 k8s-etcd-3.c1.lab

## Client SRV records
zone-set   evilham.com     _etcd-client-c1._tcp.lab 300 SRV 0 0 2379 k8s-etcd-1.c1.lab
zone-set   evilham.com     _etcd-client-c1._tcp.lab 300 SRV 0 0 2379 k8s-etcd-2.c1.lab
zone-set   evilham.com     _etcd-client-c1._tcp.lab 300 SRV 0 0 2379 k8s-etcd-3.c1.lab

## Actual host definitions
zone-set   evilham.com     k8s-etcd-1.c1.lab       3600    AAAA    ${_k8s_v6}:1::21
zone-set   evilham.com     k8s-etcd-1.c1.lab       3600    A       10.2.1.21
zone-set   evilham.com     k8s-etcd-2.c1.lab       3600    AAAA    ${_k8s_v6}:1::22
zone-set   evilham.com     k8s-etcd-2.c1.lab       3600    A       10.2.1.22
zone-set   evilham.com     k8s-etcd-3.c1.lab       3600    AAAA    ${_k8s_v6}:1::23
zone-set   evilham.com     k8s-etcd-3.c1.lab       3600    A       10.2.1.2

I was not going to set up TLS for the cluster peers yet because traffic was internal and I run my own DNS. However! Life happened, computers happened. Apparently documentation for 3.4.X is actually wrong and this is not really supported. In theory TLS SRV entries should be tried, then non-TLS entries, in reality cluster members bail out as soon as fetching TLS SRV records fails.

Provisioning parenthesis

An advantage of systematically allocating IPs and hostnames is that our provisioning (with cdist here) gets easier:

#!/bin/sh -eu

case "${__target_host}" in
    # Expeted to be: peer-name.cluster-name.lab.evilham.com
    k8s-etcd-*.lab.evilham.com)
        __hostname --name "${__target_host}"
        ETC_DIR="/etc"  # targeting linux only atm
        # Install etcd package
        __package etcd-server
        # Prepare data from hostname
        discovery_srv_name="$(echo "${__target_host}" | cut -d '.' -f 2)"
        discovery_srv="$(echo "${__target_host}" | cut -d '.' -f 3-)"
        #   Will look like: eh-etcd-c1
        initial_cluster_token="eh-etcd-${discovery_srv_name}"
        require="__package/etcd-server" __file /etc/default/etcd \
            --onchange "service etcd restart" \
            --source - <<EOF
# Managed remotely, changes will be lost!

ETCD_NAME='${__target_host}'

# DNS discovery
ETCD_DISCOVERY_SRV='${discovery_srv}'
ETCD_DISCOVERY_SRV_NAME='${discovery_srv_name}'

# Peers use TLS
ETCD_INITIAL_ADVERTISE_PEER_URLS='https://${__target_host}:2380'
ETCD_LISTEN_PEER_URLS='https://[::]:2380'

# Clients do not
ETCD_LISTEN_CLIENT_URLS='http://[::]:2379'
ETCD_ADVERTISE_CLIENT_URLS='http://${__target_host}:2379'

# Setup TLS for peers
#   This provides encryption but not authentication of peers!
ETCD_PEER_AUTO_TLS="true"

# We deploy this first with _ETCD_STATE defined as 'new', so the
# cluster gets created. On successive runs, the state will be 'existing'.
DAEMON_ARGS="--initial-cluster-token '${initial_cluster_token}' \
             --initial-cluster-state '${_ETCD_STATE:-existing}''"
EOF
    ;;
esac

So, now we apply this manifest on all VMs on the cluster:

env _ETCD_STATE=new cdist config -i manifest/etcd k8s-etcd-{1,2,3}.c1.lab.evilham.com
[...]
VERBOSE: [5341]: config: Total processing time for 3 host(s): 33.292810678482056

Testing the cluster

We will install etcd-client on the control-plane VM, which has access to the network, and see how things behave.

Basic functioning

$ etcdctl --discovery-srv='lab.evilham.com' --discovery-srv-name='c1' \
    put "Gabon" "Bona nit"
OK
$ etcdctl --discovery-srv='lab.evilham.com' --discovery-srv-name='c1' \
    get "Gabon"
Gabon
Bona nit

$ etcdctl --discovery-srv='lab.evilham.com' --discovery-srv-name='c1' \
    --write-out=fields get "Gabon"
"ClusterID" : 17681054088137536936
"MemberID" : 9829860431962910583
"Revision" : 26834
"RaftTerm" : 61940
"Key" : "Gabon"
"CreateRevision" : 26833
"ModRevision" : 26834
"Version" : 2
"Value" : "Bona nit"
"Lease" : 0
"More" : false
"Count" : 1

Cute! It seems to be working.

Killing cluster members

Since there is no data, I want to see what happens when I kill the leader, first I have to find out who it is:

$ etcdctl --discovery-srv='lab.evilham.com' --discovery-srv-name='c1' --write-out=table \
    endpoint status --cluster

It seems like k8s-etcd-2 is the chosen one (IS LEADER) right now, let’s power it off.

Repeating the previous command is not as quick as before and I get a warning context deadline exceeded, corresponding to k8s-etcd-2 being down. Interestingly, there is a new leader in town, k8s-etcd-1!

Not for long, let’s kill it too.

Now, the cluster shows an error etcdserver: no leader.

This makes sense, it knows about two other peers, and it can’t decide to be the leader on its own; there could very well be a network split and that’d get messy when the two parts can talk to each other! This is why an etcd cluster has 2*n+1 peers!

What if I try to add something to the cluster again?

$ etcdctl --discovery-srv='lab.evilham.com' --discovery-srv-name='c1' \
    put "Egun on" "Bon dia"
... context deadline exceeded ...

So, it turned read-only? good! Can I still query it?

$ etcdctl --discovery-srv='lab.evilham.com' --discovery-srv-name='c1' \
    get "Gabon"
... context deadline exceeded ...

Nope! This is also good, in this context, it is better to have no answer than a potentially wrong one.

Let’s revive k8s-etcd-2 and check the cluster status again.

Decent, with two members an election could happen and k8s-etcd-3 is the new leader.

Now we should be able to teach our etcd to say good morning:

$ etcdctl --discovery-srv='lab.evilham.com' --discovery-srv-name='c1' \
    put "Egun on" "Bon dia"
OK

Writing works again, as does reading; good!

Now, if we bring back k8s-etcd-1, it should catch up and magically learn to say good morning.

$ etcdctl --endpoints=http://k8s-etcd-1.c1...:2379  get "Egun on"
Egun on
Bon dia

Wonderful!

Conclusion

Deploying etcd in a cluster is not terribly difficult (modulo misleading docs the implied inherent pain of running our own internal CA in production), and the effort of running the cluster outside of kubernetes is probably worth it because our kubernetes cluster state can be safer.

Further things to check here are backups and restore from the cluster. On that note, I like this quote from WMF’s wikitech:

In the sad case when RAFT consensus is lost and there is no quorum anymore, the only way to recover the cluster is to recover the data from a backup, which are regularly performed every night in /srv/backups/etcd. The procedure to bring back the cluster is roughly as follows […]

Aka: do not panic, breath in, know your emergency plans, check the emergency plan, and apply it.

Now that etcd is set up, we can actually get on with kubernetes!

These were the steps according to the plan:

[X] we start by setting up the network segments as bridges on the physical host, taking care of NAT, firewall and IPv6 routing
[X] then we set up the etcd cluster
[ ] finally, we actually setup the Kubernetes cluster

Trying out Kubernetes: Network and VMs

2023-11-06T00:00:00Z

Introduction

As part of trying out Kubernetes: planning, we are going to set up the network and VMs!

Quick reminder that this will be virtualised Linux on a FreeBSD physical host. This part is FreeBSD-specific.

Introduction
Table of contents
Network
- IP assignment
Virtual Machines
Conclusion

Network

Given that bhyve(8)‘s networking uses tap(4) devices, which we can bridge and route as needed, we will only need basic FreeBSD networking knowledge.

This actually is somewhat similar to the setup I use for IPv6-VNET jails.

That is, we will need a bridge and a tap device for each network segment:

# Adding following to /etc/rc.conf

# k8s testing
# Create tap and bridge interfaces
cloned_interface="${cloned_interfaces} tap0 bridge0 tap1 bridge1"
# Rename them
ifconfig_bridge0_name="etcdbridge"
ifconfig_tap0_name="etcdtap"
ifconfig_bridge1_name="k8sbridge"
ifconfig_tap1_name="k8stap"

# Save our network prefix data
_k8s_v6="2XXX:XXXX:XXXX"
_k8s_v4="10.2"

# Define etcd network
ifconfig_etcdtap="fib 1"
ifconfig_etcdbridge_descr="etcd cluster and k8s control-panel"
ifconfig_etcdbridge="inet ${_k8s_v4}.1.1/24 addm etcdtap up fib 1"
ifconfig_etcdbridge_ipv6="inet6 ${_k8s_v6}:1::/64 auto_linklocal"

# Define k8s network
ifconfig_k8stap="fib 1"
ifconfig_k8sbridge_descr="k8s nodes and control-panel"
ifconfig_k8sbridge="inet ${_k8s_v4}.2.1/24 addm k8stap up fib 1"
ifconfig_k8sbridge_ipv6="inet6 ${_k8s_v6}:2::/64 auto_linklocal"

Remarkable is that we are assigning /64 subnets for IPv6 and /24 for IPv4, and that we will use static IP assignment since this is just a handful of machines.

I am also using a different routing table (fib 1) in order not to mix traffic.

Note that besides valid IPv6 routing, there are NAT rules taking care of IPv4:

# k8s NAT
nat on $ext_if inet from {10.2.1.0/24, 10.2.2.0/24} to any -> ($ext_if:0)

IP assignment

We will perform static IP assignment due to the small scale:

VM	IPv4	IPv6
k8s-etcd-1	`10.2.1.21`	`${_k8s_v6}:1::21`
k8s-etcd-2	`10.2.1.22`	`${_k8s_v6}:1::22`
k8s-etcd-3	`10.2.1.23`	`${_k8s_v6}:1::23`
k8s-cp-1	`10.2.1.11`	`${_k8s_v6}:1::11`
	`10.2.2.11`	`${_k8s_v6}:2::11`
k8s-node-N	`10.2.2.20+N`	`${_k8s_v6}:2::20+N`

Now I should be able to create the Virtual Machines on each network bridge.

Virtual Machines

As mentioned before, the virtualisation will be done with bhyve(8), particularly with vm-bhyve(8), as it is very lightweight and takes care of lower level details for us.

`vm-bhyve` switches

We add following to VM_BHYVE/.config/system.conf:

switch_list="etcdbridge k8sbridge"

type_etcdbridge="manual"
bridge_etcdbridge="etcdbridge"

type_k8sbridge="manual"
bridge_k8sbridge="k8sbridge"

Which makes sure that vm-bhyve will add the network interfaces to the appropriate bridges.

Template

Then we create the vm-bhyve(8) template:

# This lives in VM_BHYVE/.templates/k8s.conf
cat k8s.conf
loader="grub"
cpu=1
memory=1G

network0_type="virtio-net"
network0_switch="k8sbridge"

# We use sparse zvols for storage, which enables us to snapshot and clone the
# whole disk while only allocating space as we start using it.
disk0_name="disk0"
disk0_dev="sparse-zvol"
disk0_type="virtio-blk"

Since we will later clone this VM, we’ll add network adapters / change bridges.

OS: Debian

We’ll use Debian’s latest release: bookworm:

curl -L \
  https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-12.2.0-amd64-netinst.iso > \
  VM_BHYVE/.iso/debian-12.2.0-amd64-netinst.iso

Creating the base VM

vm create -t k8s -s 240G k8sbase

Now we install the OS using the serial console and the downloaded .iso:

vm install -f k8sbase debian-12.2.0-amd64-netinst.iso

I use an unassigned IP for the base VM, so I can apply provisioning for the final touches.

Still over serial, we finish up basic things like enabling SSH but only with public keys, and installing my keys.

# In /etc/ssh/sshd_config
PermitRootLogin prohibit-password
PasswordAuthentication no
UsePAM no
KbdInteractiveAuthentication no

$ service sshd restart
$ mkdir .ssh
$ chmod 750 .ssh
$ wget https://evilham.com/ssh.pubkey -O .ssh/authorized_keys
$ chmod 640 .ssh/authorized_keys

Cloning the VMs

# We create a snapshot first
$ vm snapshot -r k8sbase@clean
# And then actually clone it, we start with one VM of each type
$ vm clone k8sbase@clean k8s-etcd-1
$ vm clone k8sbase@clean k8s-cp-1
$ vm clone k8sbase@clean k8s-node-1

Now, we can edit each VM’s VM_BHYVE/VM/VM.conf file in order to assign it to the corresponding network(s):

# In VM_BHYVE/k8s-etcd-1/k8s-etcd-1.conf:
network0_type="virtio-net"
network0_switch="etcdbridge"

# In VM_BHYVE/k8s-cp-1/k8s-cp-1.conf:
network0_type="virtio-net"
network0_switch="etcdbridge"
network1_type="virtio-net"
network1_switch="k8sbridge"

# In VM_BHYVE/k8s-node-1/k8s-node-1.conf:
network0_type="virtio-net"
network0_switch="k8snode1"

And using provisioning against the unassigned IP one VM at a time, we:

Reset the SSH keys and hostid so they are not shared (this is important for etcd)
Apply the assigned static IP configuration on all interfaces
Reboot and check that SSH on the new VM is possible
Snapshot the VM

Conclusion

FreeBSD’s networking and virtualisation stack are great and it often surprises me how effortless it is to get things done.

Networking is set up, one of each VM type is set up with outgoing dual stack internet and two separated network segments.

Next up, setting up etcd. These were the steps according to the plan

[X] we start by setting up the network segments as bridges on the physical host, taking care of NAT, firewall and IPv6 routing
[X] then we set up the etcd cluster
[ ] finally, we actually setup the Kubernetes cluster

Trying out Kubernetes: planning

2023-11-05T00:00:00Z

Introduction

Kubernetes turned 9 years old a couple months ago, so it’s about time I took a more serious look at it.

Kubernetes is an open-source container orchestration system for automating software deployment, scaling, and management.

I have followed its development somewhat closely and kept up with the concepts and architecture, even if I haven’t jumped to actually using it.

First it was due to waiting for things to get stable, then I decided to let the hype wear off.

Now, hype hasn’t worn off, but it is at least a topic treated with some level of nuance: Kubernetes is great, but it comes with some complexity; complexity that is not necessary in many cases. And so we have to make sure that the benefits actually outweigh the overhead for our particular use-case.

As that point nears and I have some days off, I jump in at learning the best way I know how to: by actually doing things and, this time around, by writing them down for future reference / to better ask for clarification.

Let’s create a “High Availability Kubernetes cluster”! (I guess we’ll [re]define what HA means here a couple times while doing the deed)

Note I do this from a FreeBSD laptop, against a FreeBSD physical host, but the actual Control Plane and Nodes will be running on bhyve Virtual Machines running Linux in that remote FreeBSD physical host. Should not be too relevant, it just makes networking easier =D (debatable, yes).

This will likely take several posts :-).

Introduction
Table of contents
Goals
Quick resources
The components
Necessary resources
- Network
- Virtual Machines
Conclusion

Goals

The main goal here is my own learning and exploring from an Operations perspective, with future quick-reference as a secondary goal.

Particularly: duplicating documentation or getting familiar from a user’s perspective are non-goals right now :-).

It is for these reason that some things are deliberately made more difficult, by e.g. aiming for High Availability and “production-ready” configurations from the beginning, instead of starting with something more simple and self-contained like kind and minicube.

This addresses my way of learning, which includes: reading all/most of the documentation once, gaining a broad overview of what can be done, and what should be done, then implement things as close to production as practical and possible.

Your way of learning may differ, and you may need more incremental progress and immediate feedback; use whatever works for you, if this post helps you, I’ll be happy if you let me know! I do not know many people that share my way of learning :-).

Quick resources

First, let’s compile a list of resources that will help us:

Kubernetes docs: what I’ve used periodically to stay on top of the project to keep a general overview of the system. It is fairly complete and nicely written, certainly a good reference and also a decent starting point.
WMF’s SRE guide to creating a new cluster: turns out, the Wikimedia Foundation‘s SRE team does everything in the open, and they do very interesting things. Relevant for this context: they run their own HA Kubernetes clusters and theirs is guaranteed to be a scalable setup.
How-To Geek: how to start a Kubernetes cluster from scratch: a somewhat complete, simple and to-the-point guide which helps an experienced Mensch plan ahead.

The components

Since we care about the Operations perspective, we jump straight to Kubernetes’ components:

Worker machines or Nodes: where Pods (set of running containers) actually live.
Control plane: manages worker Nodes and Pods in the cluster.

As a temporary simplification: we will run the Control plane on a single Virtual Machine, that will not be allowed to run user containers.

This makes the cluster less redundant, but we get somewhere faster. It also feels like adding this redundancy at a later stage shouldn’t be too difficult; it certainly isn’t our main goal at this stage.

Control plane

It includes:

kube-apiserver: which exposes the Kubernetes API. Can be traffic balanced.
etcd: HA key value store, this is where all cluster data lives. It is a different piece of software used outside of the Kubernetes context.
kube-scheduler: picks up new Pods and assigns them a Node to run on based on the Pod’s specifications and the cluster’s availability.
kube-controller-manager: performs multiple jobs, abstractly it seems like it is in charge of reacting to changes in the cluster of various kinds (like a Pod going down, running one-off tasks by creating their Pods, etc.), the documentation doesn’t provide an exhaustive list, so this is an extrapolation of the meaning, and could be somewhat wrong or incomplete.
cloud-controller-manager: this seems to be provider-specific, since we are running Kubernetes ourselves (the whole point), we will probably not see it.

`etcd`

The documentation for a Highly Available Kubernetes cluster gives us two options:

Stacked Control Plane nodes, which include etcd
External etcd Nodes

My systems intuition tells me that mixing Control Plane and state storage may not be the best of decisions, and I have heard horror stories.

I’d be inclined to use an external etcd for anything put in production. This is ratified by WMF’s Kubernetes deployment.

Node components

These run on each Node:

kubelet: this makes sure containers from Pods are running and healthy
kube-proxy: manages the network rules to be able to provide Services. Something tells me we’ll spend a lot of time here.
Container runtime: It seems like this is a place where we have to make a choice:
- containerd
- CRI-O

I notice on WMF techwiki: “docker is mean to be used as the CRE. Other runtime engines aren’t currently supported.” which is compatible with the statement “This guide currently covers kubernetes 1.23” and the way the deb packages are created. (the Kubernetes Infrastructure upgrade policy differs, it likely only needs an update)

This makes sense because Docker as a runtime stopped being supported by Kubernetes in v1.24! Jumping that minor version may be tricky.

Given we will start differing from WMF’s documentation, I will pick CRI-O for no particular reason and target the latest version of Kubernetes (1.28.3).

Addons

These provide cluster-level features. At first I found this confusing, but after giving it some thought: it makes sense.

This means that there can be multiple implementations (and it happens) of similar cluster services, addressing different needs and priorities.

These are the ones I think we’ll need:

DNS: else we’ll have to deal with IPs and running services gets tricky
Dashboard (UI): want to see how it works
Network plugin: this deals with Pod networking, we’ll want to check Calico as used at WMF

Necessary resources

With that, we have most information to start planning and allocating resources.

Network

Given the choice to run etcd externally, we will need two separate networks:

etcd: access will be network and firewall-controlled
Kubernetes:
- Control Plane: these must have access to etcd too and won’t run user containers
- Nodes: these will be excluded from accessing etcd

For a total of: 2 networks, where one class of machines can access both.

This is a learning environment, so we’ll NAT these IPv4 networks and assign each segment a /64 IPv6 subnet without blocking outgoing traffic.

Probably in production we wouldn’t allow that directly.

Virtual Machines

etcd: following WMF’s etcd recommendations for etcd we will use 3 VMs dedicated to etcd
Kubernetes:
- control-plane: 1 VM as a starting point. These have 2 network interfaces.
- Nodes: 1 VM as a starting point

For a total of: 5 VMs

Conclusion

Aaand, this is why Kubernetes felt like overhead for some time.

Until the cost (effort, time, energy, money) of setting up and maintaining all this is not somewhat offset by the benefits, it is the wrong tool for our scale.

Now that we are planning ahead though, these are next steps:

[X] we start by setting up the network segments as bridges on the physical host, taking care of NAT, firewall and IPv6 routing
[X] then we set up the etcd cluster
[ ] finally, we actually setup the Kubernetes cluster

YubiKey: PGP and SSH authentication

2023-10-06T00:00:00Z

Introduction

I own a few YubiKeys and use them, a lot.

If not familiar with YubiKeys, they are hardware tokens that help improve security in multiple ways:

They can do WebAuthn (modern web-based two factor authentication)
They can hold secret keys in a way that cannot be extracted, supporting these operations:
- Signature
- Encryption
- Authentication

The most common use, and what is already very useful, is using these hardware tokens for two-factor authentication. For that there are plenty of online resources (though, people certainly can use help understanding and setting that up).

What I really care about is the latter bit: securing secret keys, particularly when it comes to securing SSH access to servers, and how that fits with PGP.

Incidentally, this is where I see most online documentation falling short.

There are similar hardware tokens, YubiKeys are what I’m familiar with, and what I’ll assume here.

Introduction
Table of contents
PGP, subkeys and SSH
PGP key generation
Putting the Key into the YubiKey
Copy the public key
Using the YubiKey
- SSH authentication
Managing expiry dates and the [C]ertification key
Conclusion

PGP, subkeys and SSH

Keys (and subkeys, more below) in PGP may be marked as having one or multiple uses:

[C]ertification: a key marked like this acts as a source of truth key that can modify subkeys and the key itself
[S]ignature: a key marked like this can produce valid signatures that will be recognised as coming from the main key
[E]encryption: a key marked like this will be associated with the main key, and it will be used by other programs to encrypt things, so only the owner of the main key can read them. Note though: this is worded in a somewhat complicated fashion, because should the Encryption key be changed, messages encrypted for the older key cannot be read
[A]uthentication: a key marked like this will be associated with the main key, will be able to perform authentication e.g. against OpenSSH servers

The bit about PGP being usable for SSH [A]uthentication is not very widely known, and neither is the concept of subkeys.

Instead of thinking as a PGP key as: a single key for all things, we should think about it as: a [C]ertification key that I keep offline except when necessary, that is associated with multiple subkeys, one for each functionality.

Common state

When it comes to PGP, most advice results in keys with attributes [SCEA] or [SCA] with a [E] subkey.

What we achieve here

What we will achieve here is an offline [C] key with multiple subkeys: [S], [E], [A], with the secret parts loaded in a YubiKey (or possibly other hardware tokens).

(Roughly) why we do it

Commonly people have their PGP keys locally in a file that is secured by a password, or even separated PGP and SSH keys, protected in a similar fashion (sometimes even without a password).

When there are so many sneaky attack vectors, making a mistake once, can mean secrets being exfiltrated and possibly systems being compromised.

By not having such files anywhere but instead relying on the hardware token, we mitigate many parts of such risks.

PGP key generation

There are a bunch of security considerations as to the system used for this, many people have written better about it.

We will focus on:

Using a separated directory for the keyring and settings
Considering the directory as a temporary a in-memory thing or equivalent

Software requisites

TODO: Check debian - gnupg2 -

Setup the directory

On FreeBSD, to use an in-memory temporary directory, we can use following:

# Create directory
mkdir pgptmp
# Mount in-memory filesystem
mount -t tmpfs tmpfs pgptmp
# Secure permissions
chmod 700 pgptmp

Nothing that gets written to pgptmp, will be saved to disk, instead, it is temporary and is held in RAM.

Configuring `gpghome`

We must export the GNUPGHOME variable accordingly:

cd pgptmp
export GNUPGHOME="$(pwd)"

And it is convenient to set up certain configurations:

$ cat <<EOF > scdaemon.conf
# This is sometimes necessary
# TODO: Add mention fo scdaemon
disable-ccid
EOF

Generating the key

We use gpg --expert --full-generate-key, as it is the only way we get asked all the questions.

$ gpg --expert --full-generate-key

Please select what kind of key you want:
   (1) RSA and RSA
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
   (7) DSA (set your own capabilities)
   (8) RSA (set your own capabilities)
   (9) ECC (sign and encrypt) *default*
  (10) ECC (sign only)
  (11) ECC (set your own capabilities)
  (13) Existing key
  (14) Existing key from card
Your selection?

The first question we get asked is, which kind of key we want to create.

Unless we are cryptography with good reasons (please share!), I’d pick the ECC, which is the default, but setting our own capabilities (Option 11), that is:

Your selection? 11

Possible actions for this ECC key: Sign Certify Authenticate
Current allowed actions: Sign Certify

   (S) Toggle the sign capability
   (A) Toggle the authenticate capability
   (Q) Finished

Your selection?

We will want to generate a key with only [C]ertification capabilities, so we will ensure [S] and [A] are disabled:

Your selection? S

Possible actions for this ECC key: Sign Certify Authenticate
Current allowed actions: Certify

   (S) Toggle the sign capability
   (A) Toggle the authenticate capability
   (Q) Finished

Your selection? Q
Please select which elliptic curve you want:
   (1) Curve 25519 *default*
   (2) Curve 448
   (3) NIST P-256
   (4) NIST P-384
   (5) NIST P-521
   (6) Brainpool P-256
   (7) Brainpool P-384
   (8) Brainpool P-512
   (9) secp256k1
Your selection?

As with the key type selection, unless we have cryptography knowledge that says otherwise, we will go with Curve 25519 (Option 1):

Your selection? 1
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0)

The use-case with which I am documenting this, requires a key that does not expire, but my usual key expires usually ~2 years in the future, which I try to extend every year. This choice is yours, more often might increase maintenance burden.

Key is valid for? (0) 0
Key does not expire at all
Is this correct? (y/N) y

GnuPG needs to construct a user ID to identify your key.

Real name:

Now, PGP has a weird concept of identity, and signatures are bound to the combination of Name, Email and Comment, it is usually not convenient to change this.

After thinking about these decisions (usually Comment is left blank):

Real name: YourName
Email address: YourEmail@example.org
Comment:
You selected this USER-ID:
    "YourName <YourEmail@Example.org>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit?

Now, when we confirm the identity, the key will be generated after we enter a passphrase to protect it.

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: GNUPGHOME/trustdb.gpg: trustdb created
gpg: directory 'GNUPGHOME/openpgp-revocs.d' created
gpg: revocation certificate stored as 'GNUPGHOME/openpgp-revocs.d/KEY_ID.rev'
public and secret key created and signed.

pub   ed25519 2023-10-06 [C]
      KEY_ID
uid                      YourName <YourEmail@Example.org>

This KEY_ID will uniquely identify our PGP key.

Generating the key

Since our key only has the [C]ertification capability, we want to add some subkeys so we can take advantage of all options.

For this, we start editing the key in --expert mode, else we won’t be able to access the [A]uthentication capability:

$ gpg --expert --edit-key KEY_ID
sec  ed25519/KEY_ID
     created: 2023-10-06  expires: never       usage: C
     trust: ultimate      validity: ultimate
[ultimate] (1). YourName <YourEmail@example.org>

gpg>

And now we create one key for each of the missing capabilities ([S]ignature, [E]ncryption, [A]uthentication), trusting the current cryptography recommendations unless we have very good reasons not to, and setting the expiry according to our use-case.

Note too that the [A]uthentication capability is not available directly, but rather through the ECC (set your own capabilities) option:

gpg> addkey
Please select what kind of key you want:
   (3) DSA (sign only)
   (4) RSA (sign only)
   (5) Elgamal (encrypt only)
   (6) RSA (encrypt only)
   (7) DSA (set your own capabilities)
   (8) RSA (set your own capabilities)
  (10) ECC (sign only)
  (11) ECC (set your own capabilities)
  (12) ECC (encrypt only)
  (13) Existing key
  (14) Existing key from card
Your selection? 10
Please select which elliptic curve you want:
   (1) Curve 25519 *default*
   (2) Curve 448
   (3) NIST P-256
   (4) NIST P-384
   (5) NIST P-521
   (6) Brainpool P-256
   (7) Brainpool P-384
   (8) Brainpool P-512
   (9) secp256k1
Your selection?
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N) y
Really create? (y/N) y
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.

sec  ed25519/KEY_ID
     created: 2023-10-06  expires: never       usage: C
     trust: ultimate      validity: ultimate
ssb  ed25519/KEY_ID_S
     created: 2023-10-06  expires: never       usage: S
[ultimate] (1). YourName <YourEmail@example.org>

gpg> addkey
Please select what kind of key you want:
   (3) DSA (sign only)
   (4) RSA (sign only)
   (5) Elgamal (encrypt only)
   (6) RSA (encrypt only)
   (7) DSA (set your own capabilities)
   (8) RSA (set your own capabilities)
  (10) ECC (sign only)
  (11) ECC (set your own capabilities)
  (12) ECC (encrypt only)
  (13) Existing key
  (14) Existing key from card
Your selection? 12
Please select which elliptic curve you want:
   (1) Curve 25519 *default*
   (2) Curve 448
   (3) NIST P-256
   (4) NIST P-384
   (5) NIST P-521
   (6) Brainpool P-256
   (7) Brainpool P-384
   (8) Brainpool P-512
   (9) secp256k1
Your selection?
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N) y
Really create? (y/N) y
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.

sec  ed25519/KEY_ID
     created: 2023-10-06  expires: never       usage: C
     trust: ultimate      validity: ultimate
ssb  ed25519/KEY_ID_S
     created: 2023-10-06  expires: never       usage: S
ssb  cv25519/KEY_ID_E
     created: 2023-10-06  expires: never       usage: E
[ultimate] (1). YourName <YourEmail@example.org>

gpg> addkey
Please select what kind of key you want:
   (3) DSA (sign only)
   (4) RSA (sign only)
   (5) Elgamal (encrypt only)
   (6) RSA (encrypt only)
   (7) DSA (set your own capabilities)
   (8) RSA (set your own capabilities)
  (10) ECC (sign only)
  (11) ECC (set your own capabilities)
  (12) ECC (encrypt only)
  (13) Existing key
  (14) Existing key from card
Your selection? 11

Possible actions for this ECC key: Sign Authenticate
Current allowed actions: Sign

   (S) Toggle the sign capability
   (A) Toggle the authenticate capability
   (Q) Finished

Your selection? S

Possible actions for this ECC key: Sign Authenticate
Current allowed actions:

   (S) Toggle the sign capability
   (A) Toggle the authenticate capability
   (Q) Finished

Your selection? A

Possible actions for this ECC key: Sign Authenticate
Current allowed actions: Authenticate

   (S) Toggle the sign capability
   (A) Toggle the authenticate capability
   (Q) Finished

Your selection? Q
Please select which elliptic curve you want:
   (1) Curve 25519 *default*
   (2) Curve 448
   (3) NIST P-256
   (4) NIST P-384
   (5) NIST P-521
   (6) Brainpool P-256
   (7) Brainpool P-384
   (8) Brainpool P-512
   (9) secp256k1
Your selection?
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N) y
Really create? (y/N) y
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.

sec  ed25519/KEY_ID
     created: 2023-10-06  expires: never       usage: C
     trust: ultimate      validity: ultimate
ssb  ed25519/KEY_ID_S
     created: 2023-10-06  expires: never       usage: S
ssb  cv25519/KEY_ID_E
     created: 2023-10-06  expires: never       usage: E
ssb  ed25519/KEY_ID_A
     created: 2023-10-06  expires: never       usage: A
[ultimate] (1). YourName <YourEmail@example.org>

And finally we save the modified key

gpg> save

Putting the Key into the YubiKey

This is done by editing the key we just created:

$ gpg --expert --edit-key KEY_ID
Secret key is available.

sec  ed25519/KEY_ID
     created: 2023-10-06  expires: never       usage: C
     trust: ultimate      validity: ultimate
ssb  ed25519/KEY_ID_S
     created: 2023-10-06  expires: never       usage: S
ssb  cv25519/KEY_ID_E
     created: 2023-10-06  expires: never       usage: E
ssb  ed25519/KEY_ID_A
     created: 2023-10-06  expires: never       usage: A
[ultimate] (1). YourName <YourEmail@example.org>

We have to select and de-select each subkey and send them to the key with keytocard, and selecting the destination slot that best matches the capability:

gpg> key 1

sec  ed25519/KEY_ID
     created: 2023-10-06  expires: never       usage: C
     trust: ultimate      validity: ultimate
ssb* ed25519/KEY_ID_S
     created: 2023-10-06  expires: never       usage: S
ssb  cv25519/KEY_ID_E
     created: 2023-10-06  expires: never       usage: E
ssb  ed25519/KEY_ID_A
     created: 2023-10-06  expires: never       usage: A
[ultimate] (1). YourName <YourEmail@example.org>

gpg> keytocard
Please select where to store the key:
   (1) Signature key
   (3) Authentication key
Your selection? 1

sec  ed25519/KEY_ID
     created: 2023-10-06  expires: never       usage: C
     trust: ultimate      validity: ultimate
ssb* ed25519/KEY_ID_S
     created: 2023-10-06  expires: never       usage: S
ssb  cv25519/KEY_ID_E
     created: 2023-10-06  expires: never       usage: E
ssb  ed25519/KEY_ID_A
     created: 2023-10-06  expires: never       usage: A
[ultimate] (1). YourName <YourEmail@example.org>

Note: the local copy of the secret key will only be deleted with "save".
gpg> key 1

sec  ed25519/KEY_ID
     created: 2023-10-06  expires: never       usage: C
     trust: ultimate      validity: ultimate
ssb  ed25519/KEY_ID_S
     created: 2023-10-06  expires: never       usage: S
ssb  cv25519/KEY_ID_E
     created: 2023-10-06  expires: never       usage: E
ssb  ed25519/KEY_ID_A
     created: 2023-10-06  expires: never       usage: A
[ultimate] (1). YourName <YourEmail@example.org>

gpg> key 2

sec  ed25519/KEY_ID
     created: 2023-10-06  expires: never       usage: C
     trust: ultimate      validity: ultimate
ssb  ed25519/KEY_ID_S
     created: 2023-10-06  expires: never       usage: S
ssb* cv25519/KEY_ID_E
     created: 2023-10-06  expires: never       usage: E
ssb  ed25519/KEY_ID_A
     created: 2023-10-06  expires: never       usage: A
[ultimate] (1). YourName <YourEmail@example.org>

gpg> keytocard
Please select where to store the key:
   (2) Encryption key
Your selection? 2

sec  ed25519/KEY_ID
     created: 2023-10-06  expires: never       usage: C
     trust: ultimate      validity: ultimate
ssb  ed25519/KEY_ID_S
     created: 2023-10-06  expires: never       usage: S
ssb* cv25519/KEY_ID_E
     created: 2023-10-06  expires: never       usage: E
ssb  ed25519/KEY_ID_A
     created: 2023-10-06  expires: never       usage: A
[ultimate] (1). YourName <YourEmail@example.org>

Note: the local copy of the secret key will only be deleted with "save".

gpg> key 2

sec  ed25519/KEY_ID
     created: 2023-10-06  expires: never       usage: C
     trust: ultimate      validity: ultimate
ssb  ed25519/KEY_ID_S
     created: 2023-10-06  expires: never       usage: S
ssb  cv25519/KEY_ID_E
     created: 2023-10-06  expires: never       usage: E
ssb  ed25519/KEY_ID_A
     created: 2023-10-06  expires: never       usage: A
[ultimate] (1). YourName <YourEmail@example.org>

gpg> key 3

sec  ed25519/KEY_ID
     created: 2023-10-06  expires: never       usage: C
     trust: ultimate      validity: ultimate
ssb  ed25519/KEY_ID_S
     created: 2023-10-06  expires: never       usage: S
ssb  cv25519/KEY_ID_E
     created: 2023-10-06  expires: never       usage: E
ssb* ed25519/KEY_ID_A
     created: 2023-10-06  expires: never       usage: A
[ultimate] (1). YourName <YourEmail@example.org>

gpg> keytocard
Please select where to store the key:
   (3) Authentication key
Your selection? 3

sec  ed25519/KEY_ID
     created: 2023-10-06  expires: never       usage: C
     trust: ultimate      validity: ultimate
ssb  ed25519/KEY_ID_S
     created: 2023-10-06  expires: never       usage: S
ssb  cv25519/KEY_ID_E
     created: 2023-10-06  expires: never       usage: E
ssb* ed25519/KEY_ID_A
     created: 2023-10-06  expires: never       usage: A
[ultimate] (1). YourName <YourEmail@example.org>

Note: the local copy of the secret key will only be deleted with "save".
gpg>
Save changes? (y/N) N
Quit without saving? (y/N) y

Note that I exit without saving, because I want to back up the [C]ertification key offline, and saving the key here will remove the private part from the keyring.

It will be contained in the YubiKey and can be used with the password, but it cannot be retrieved out of it.

Copy the public key

We now want to export the public key:

$ gpg --export KEY_ID > pgp.pubkey.bin
$ gpg --export -a KEY_ID > pgp.pubkey.pem

Which we will need to publish and import on our daily driver.

Using the YubiKey

Now we can disconnect the backed up [C]ertification key, reboot into our persistent environment or change computers, depending on what our security precautions were, and move to our daily driver system.

Once there, we import the public key into our regular keyring:

# This is not the same system we were in before(!)
gpg --import -a < pgp.pubkey.bin

And regular PGP operations should request unlocking the corresponding private key with its regular password.

SSH authentication

In order to take advantage of SSH authentication, we want to use the gpg-agent as an SSH agent.

This is usually done by setting up the gpg-agent.conf

$ cat <<EOF > gpg-agent.conf
enable-ssh-support
pinentry-program /usr/local/bin/pinentry-gtk2
# Possibly on Debian-based:
#pinentry-program /usr/bin/pinentry-gtk2
# 2 min
default-cache-ttl 120
# 5 min
max-cache-ttl 300
# Grab input
grab
EOF
# Applying this might require restarting the gpg-agent, you can use:
# gpgconf --kill gpg-agent

This is usually done adding to the ${HOME}/.profile file something similar to:

# Setup the environment variable so the correct SSH agent is contacted
export SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket)
# Tell the gpg-agent to start and update its tty
echo "UPDATESTARTUPTTY" | gpg-connect-agent /bye

Note that if you use gnome, KDE or similar, their agents might have support for something like this already.

Once the SSH and GPG agents have been set up, the SSH agent will list the key contained in the YubiKey as available for authentication:

$ ssh-add -L
ssh-ed25519 AAAA...J cardno:YY_YYY_YYY

By adding that line to an authorized_keys file, we should be able to use the [A]uthentication private key to enter with SSH on that system.

Managing expiry dates and the `[C]`ertification key

This is similar to the creation and sending to card process, except we use the expiry gpg command after editing.

We can change the expiry date of multiple subkeys by selecting more than one and export the resulting public key.

Conclusion

Given the implications of such setup, some thought and personal/corporation-wide trade-offs must be considered, it is certainly not a 5 minute task.

But it also isn’t as scary as it sounds and the full described process (along with documenting it here) took about 90 minutes.

The result is actually more usable and secure than before:

In order to login to places, we need both something physical (the YubiKey) and something we know (its protection password)
If we want to be sure no keys are loaded, we just unplug the YubiKey
If we want to log in places, sign something or read an encrypted email, we plug it in
We can backup and revoke the keys as necessary, if we ever lose the YubiKey, we can revoke the subkeys and generate new ones from the offline [C]ertification key, while still having access to previously encrypted content
If something bad happens with our daily driver, odds are that not having id_rsa files or keys in the keyring will save us a lot of headaches and lateral jumps

ZFS replication tools

2023-07-04T00:00:00Z

Introduction

Back when I first started using FreeBSD and ZFS, I needed tools with an extremely low barrier of entry, that did their job well.

When it comes to creating and pruning snapshots, the job was first done by sysutils/zfs-periodic.

Over time however, even with minor improvements on my part, it has come short for my current needs.

As inspired by Dan Langille’s wonderful blog (Dan always writes great blog posts, that end up being lovely complementary documentation), I document here my somewhat bumpy road when it comes to ZFS replication tools and why you might want to use something different at each step.

Quick reminder of ZFS

ZFS is nowadays (at least on FreeBSD and Linux) actually OpenZFS. Let’s see how they define it:

OpenZFS is an advanced file system and volume manager which was originally developed for Solaris and is now maintained by the OpenZFS community. This repository contains the code for running OpenZFS on Linux and FreeBSD.

Amongst the many beautiful things of ZFS, there is the fact that it works in a Copy on Write fashion, which, amongst other things, enables it to create snapshots in an instantaneous fashion.

That is, we can assign a name to a known state with, e.g. zfs snapshot -r zroot@goodstate.

Keep on working and, if we get on to a bad state, discard any changes made since then by issuing a rollback.

This means, we can have a very handy way of having real-life save points, and work in a more relaxed fashion.

Together with zfs-send(8) and zfs-receive(8), we quickly have the basis for a remote backup system that is incremental.

Now that ZFS data set encryption (zfs-load-key(8)) is a thing, it too can be encrypted.

Snapshot creation and prunning with `zfs-periodic(8)`

Now, back in ~2019, I was mostly a Linux user that was jumping to FreeBSD and the last thing I wanted was to fiddle too much with a new file system.

Luckily, there is a very simple way to manage ZFS snapshots in an automatic fashion right in the ports system: zfs-periodic.

It profits from the periodic(8) system, to run at regular intervals, and gets configured in a minute with:

$ pkg install -y zfs-periodic
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.

[1/1] Fetching zfs-periodic-1.0.20130213.pkg: 100%    2 KiB   2.3kB/s    00:01
Checking integrity... done (0 conflicting)
[1/1] Installing zfs-periodic-1.0.20130213...
[1/1] Extracting zfs-periodic-1.0.20130213: 100%
=====
Message from zfs-periodic-1.0.20130213:

--
In order to enable periodic snapshots you need
to add these lines to your /etc/periodic.conf

hourly_output="root"
hourly_show_success="NO"
hourly_show_info="YES"
hourly_show_badconfig="NO"
hourly_zfs_snapshot_enable="YES"
hourly_zfs_snapshot_pools="tank"
hourly_zfs_snapshot_keep=6
daily_zfs_snapshot_enable="YES"
daily_zfs_snapshot_pools="tank"
daily_zfs_snapshot_keep=7
weekly_zfs_snapshot_enable="YES"
weekly_zfs_snapshot_pools="tank"
weekly_zfs_snapshot_keep=5
monthly_zfs_snapshot_enable="YES"
monthly_zfs_snapshot_pools="tank"
monthly_zfs_snapshot_keep=2

To get hourly snapshots you also need to add
something like this to /etc/crontab:

2       *       *       *       *       root    periodic hourly

We literally do that (adding the crontab(5) entry and configuring zfs-periodic(8) in /usr/local/etc/periodic.conf (*)), and we have a simple system that takes hourly, daily, weekly and monthly snapshots.

Half way there with zero effort, this is what has kept me using it for years.

(*): I prefer to use /usr/local/etc/periodic.conf for things outside the base system. This is a matter of taste and it helps easen administration.

Adding `quarter_hourly`

Given how cheap these snapshots are, I (and people at work) started wanting to have them more often, which in my case meant every 15 minutes.

Luckily, periodic(8) supports arbitrary directories:

directory  An arbitrary directory containing a set of executables to be
           run.

However, it wasn’t as simple as:

mkdir /usr/local/etc/periodic/quarter_hourly/
ln -s /usr/local/etc/periodic/daily/000.zfs-snapshot \
      /usr/local/periodic/quarter_hourly/000.zfs-snapshot

Because the actual file in there depends on the name of the directory, so it has to be customised. And also because the /usr/local/bin/zfs-snapshot script that gets installed needs to understand all the periodic.conf settings.

Luckily making these changes was fairly simple. Since I won’t be using zfs-periodic(8) any longer, and the patch has been more than tested over the years, I’m opening a Merge Request, which you can use to benefit from these changes.

Take into account that it’ll also need a corresponding crontab(5) entry:

*/15       *       *       *       *       root    periodic quarter_hourly

ZFS Replication with `zxfer(8)`

Snapshots take care of local mess ups, but they don’t serve as a backup.

For that, I took to zxfer, which is simple to use and has worked great with some caveats.

A lovely thing of zxfer(8) is that it supports an rsync(1) mode, that allows us to back up remote systems that do not support ZFS, to a ZFS-based system that takes care of snapshots and so on.

The caveats

Mostly, even with a pull-based approach, zxfer(8) doesn’t protect much against snapshots disappearing on the source, which means that there is a somewhat plausible scenario where pulling data results in breaking certain retention policies.

While there is a -g flag to protect old snapshots, it means that we have to replicate snapshot pruning on the destination system, but cannot create snapshots as they could collide with the source snapshots.

That means zfs-periodic(8) and zxfer(8) are better than nothing and work decently, but have important limitations.

Encrypted datasets

When encrypted datasets got added to OpenZFS, I realised that we need to use zfs send -w, or else the backup will happen unencrypted. So I wrote a patch for zxfer to do just that.

You can help land that patch in zxfer(8), it is basically only missing the manpage and USAGE texts, but my manpage syntax is rusty and time is not an abundantly available commodity.

I’d love to help you land this change though!

Snapshot creation and pruning with `sanoid`

After having read Dan Langille’s blog post on ZFS tools several months ago, I had been wanting to give it a try, since it looked simple to setup and more akin to my current needs.

I based my configuration mostly off Dan’s, so checked that great post :-).

Migrating `zfs-periodic(8)` snapshots to sanoid

This ends up being somewhat simple with periodic2sanoid.sh:

# Check locally and remotely that actions make sense (and results match)
#  output lists "DS" as the base dataset, it acts recursively
./periodic2sanoid.sh zroot/usr/home
[...]
DS@daily-2023-06-28 --> DS@autosnap_2023-06-28_12:24:44_daily
[...]
# Actually apply changes
env DRY_RUN=NO ./periodic2sanoid.sh zroot/usr/home

And here is the code for that script, you can (and should!) audit it:

#!/bin/sh -eu

# Copyright (c) 2023, Evilham (https://evilham.com)
# BSD 2-Clause copyright and disclaimer apply.
# See: http://www.opensource.org/licenses/bsd-license.php

DS="$1"

DRY_RUN="${DRY_RUN:-YES}"

epoch_to_time() {
    epoch="$1"
    date -j -f '%s' "${epoch}" '+%Y-%m-%d_%H:%M:%S'
}

target_name() {
    snap_epoch="$1"
    snap_name="$2"
    snap_time="$(epoch_to_time "${snap_epoch}")"
    case "${snap_name}" in
        quarter_hourly-*)
            # quarter_hourly --> frequently
            printf "autosnap_%s_%s" "${snap_time}" "frequently"
        ;;
        hourly-*)
            printf "autosnap_%s_%s" "${snap_time}" "hourly"
        ;;
        daily-*)
            printf "autosnap_%s_%s" "${snap_time}" "daily"
        ;;
        weekly-*)
            printf "autosnap_%s_%s" "${snap_time}" "weekly"
        ;;
        monthly-*)
            printf "autosnap_%s_%s" "${snap_time}" "monthly"
        ;;
        yearly-*)
            printf "autosnap_%s_%s" "${snap_time}" "yearly"
        ;;
        *)
            # Empty --> no renaming
        ;;
    esac
}


zfs list -rpHt snap -s creation -o creation,name "$DS" | while IFS='' read -r line; do
    snap_epoch="$(echo "${line}" | cut -f 1)"
    full_snap="$(echo "${line}" | cut -f 2)"
    ds_name="$(echo "${full_snap}" | cut -d '@' -f 1)"
    snap_name="$(echo "${full_snap}" | cut -d '@' -f 2-)"

    snap_target_name="$(target_name "${snap_epoch}" "${snap_name}")"

    if [ -n "${snap_target_name}" ]; then
        # We use DS to facilitate comparing local and remote actions
        printf "%s@%s\t-->\t%s@%s\n" "DS" "${snap_name}" "DS" "${snap_target_name}"
        if [ "${DRY_RUN}" != "YES" ]; then
            echo "zfs rename \"${full_snap}\" \"${ds_name}@${snap_target_name}\""
            zfs rename "${full_snap}" "${ds_name}@${snap_target_name}"
        fi
    fi
done

snapshot creation and pruning with `sanoid`

This, as Dan documented it, works wonderfully after taking into account the crontab(5) caveat and using lockf(1).

ZFS Replication with `syncoid`

When installing sanoid, we also get syncoid as a replication tool.

Issues with permissions

While trying to replicate the datasets, I realised that it expects either to run as root or to have sudo available.

Of course, that’s not ideal, as I prefer to rely on zfs-allow(8) and do not even have sudo as an available command.

There is a flag --no-privilege-elevation which helps circumvent these checks, but the fact that it fails and requires the flag to work as non-root, is kind of a red flag for me.

Issues with public key limitations

As with zxfer(8), I was setting up a pull-based approach over SSH. Since this user and that SSH key should only be allowed to use zfs send and little more, I use the authorized_keys file to force the command to a particular script.

This resulted in issues with syncoid‘s shell escaping, which lead me to this reported issue from 2022. I believe the person that reported it was in a similar position to mine.

I ended up “fixing it” (the security implications are still to be determined), by expanding the list of allowed commands and using sh -s with $SSH_ORIGINAL_COMMAND:

#!/bin/sh
case "${SSH_ORIGINAL_COMMAND}" in
    "uname")
    ;;
    "zfs get "*)
    ;;
    "/sbin/zfs get "*)
    ;;
    "zfs list "*)
    ;;
    "/sbin/zfs list "*)
    ;;
    "zfs send -n "*)
    ;;
    "zfs send -w "*)
    ;;
    " zfs send -w "*)
    ;;
    "/sbin/zfs send -w "*)
    ;;
    "echo -n")
    ;;
    "command -v lzop")
    ;;
    "command -v mbuffer")
    ;;
    "zpool get -o value -H feature@extensible_dataset "*)
    ;;
    "exit")
    ;;
    *)
        echo "Command not allowed! See ya!"
        exit 1
    ;;
esac
echo "${SSH_ORIGINAL_COMMAND}" >> /tmp/test
sh -s <<EOF
${SSH_ORIGINAL_COMMAND}
EOF

Encrypted datasets

This is also a bit of an issue, and we need to specify --sendoptions="w".

All together

syncoid -r --no-privilege-elevation --sendoptions="w" \
  sucker@HOST:zroot/usr/home backups/pools/HOST/zroot/usr/home

This kind of works, but when looking to refine it, I realised that it does not offer me any benefit regarding zxfer(8) when it comes to protecting the snapshots.

Remembering `zrepl`

After getting frustrated again, I realised this wasn’t quite looking as I was expecting (I was so sure this was doable!).

Turns out, I had read about zrepl some months ago, and checked the documentation and determined it was something I needed to use.

So, this misremembering caused me to look deeply into sanoid and syncoid.

But hey, at least now I am very sure about what I want, and I can probably quickly adapt my renaming script for a migration to zrepl.

Conclusion

I really should blog more often, if anything to save me some time.

This will have to be a two-parter, with this one being what I’ve tried and why I’m not keeping it, and the second one being how I get to setup zrepl to my liking.

UPDATE: Bonus tools from the fediverse

I posted this on the fediverse, and some people shared what they use and why!

`zfs_autobackup`

@stefano@mdon.stefanomarinelli.it mentioned zfs_autobackup, and they have a very nice blog post series documenting this, and other steps they took on migrating from Linux to FreeBSD.

`zrep`

@dk3jf@radiosocial.de mentioned zrep:

Python: yield, generators and Deferreds

2022-08-28T00:00:00Z

Introducció

El company Pedro està aprenent a fer anar Scrapy i em comenta espontàniament:

una cosa meravellosa d’emacs és que està tot molt ben documentat, i de seguida estàs mirant el codi font còmodament (tant sigui emacs lisp que c); hi ha algun equivalent a python? he intentat això:

(ara m’he posat amb el scrapy tutorial)

12345678910[13:49:28] $ ipython3
Python 3.9.2 (default, Feb 28 2021, 17:03:44)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.20.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: help(yield)
  File "<ipython-input-1-b8899ae5635b>", line 1
    help(yield)
         ^
SyntaxError: invalid syntax

El poc que sé de Scrapy és que és una eina molt potent i que paga la pena aprendre (però encara no hi he tingut ocasió!) i, encara més important: que està escrita en Python, fent servir la llibreria Twisted.

Amb aquesta informació i havent après Twisted fa més anys dels que voldria admetre, entenc immediatament que el dubte darrere la pregunta, no va de yield únicament, sinó de com funciona la programació asíncrona en Twisted.

Anem a fer-hi una ullada, amb l’explicació que a mi m’hagués agradat tenir quan ho aprenia i de pas mirem els generadors!

Taula de continguts

Introducció
Taula de continguts
help(yield)?
Ús normal de yield en Python
- Una funció generadora de nombres de Fibonacci
Per què cal el yield amb Scrapy/Twisted?
Conclusió

`help(yield)`?

Comencem per respondre la pregunta concreta: “com puc obtenir informació sobre yield des de Python?”

Doncs no anàvem malament amb help(yield)!

Aturem-nos però a pensar què passa quan fem alguna consulta que sí funciona, com ara help(""):

>>> help("")
Help on class str in module builtins:

class str(object)
[...]

Què és `help`?

Des de la terminal interactiva podem esbrinar-ho:

>>> type(help)
<class '_sitebuiltins._Helper'>

És a dir, que és una instància de la classe _sitebuiltins._Helper, un objecte!

Podem fer-hi coses curioses?

>>> a = help
>>> a("")
Help on class str in module builtins:

class str(object)
[...]
>>> dir(help)
['__call__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__',
 '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__',
 '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__',
 '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__',
 '__sizeof__', '__str__', '__subclasshook__', '__weakref__']

Interessant! Veiem que implementa __call__, és a dir, és un objecte callable:

callable(obj, /)
    Return whether the object is callable (i.e., some kind of function).

    Note that classes are callable, as are instances of classes with a
    __call__() method.

I per què no funciona `help(yield)`?

Ara que sabem que help és un objecte, i que és un callable, entenem que help(yield) no fa res més que cridar help amb yield com a argument…

Però yield és una paraula reservada! D’aquí que tinguem un error de sintaxi.

La manera correcta: `help("yield")`

Per evitar aquest error de sintaxi, podem posar la paraula yield entre cometes, d’aquesta manera és una sintaxi vàlida, i help comprovarà si correspon a un concepte conegut.

Aquesta estratègia no només ens és útil aquí, si volem saber com funciona | o + podem fer el mateix: help("|") o help("+").

>>> help("yield")
The "yield" statement
*********************

   yield_stmt ::= yield_expression

A "yield" statement is semantically equivalent to a yield expression.
The yield statement can be used to omit the parentheses that would
otherwise be required in the equivalent yield expression statement.
For example, the yield statements

   yield <expr>
   yield from <expr>

are equivalent to the yield expression statements

   (yield <expr>)
   (yield from <expr>)

Yield expressions and statements are only used when defining a
*generator* function, and are only used in the body of the generator
function.  Using yield in a function definition is sufficient to cause
that definition to create a generator function instead of a normal
function.

For full details of "yield" semantics, refer to the Yield expressions
section.

Ús normal de `yield` en Python

El text de help("yield") ens explica que es fa servir en funcions generadores, però no hi entra molt en detall. Tot i que podríem seguir amb la documentació amb help, ja és útil canviar a explicacions més casolanes!

Una funció generadora de nombres de Fibonacci

def fib():
    """Generador de nombres de Fibonacci"""
    a = 1
    b = 1
    while True:
        yield a
        (a, b) = (b, a + b)

La faríem servir així:

>>> a = fib()
>>> type(a)
<class 'generator'>
>>> next(a)
1
>>> next(a)
1
>>> next(a)
2
>>> next(a)
3
>>> next(a)
5
>>> next(a)
8
>>> next(a)
13

Si ens hi fixem, la funció fib conté un bucle infinit (while True)! Però a = fib() retorna de seguida! No només això, a acaba essent de tipus generator.

La màgia és que aquest generator és iterable, en particular podem anar-hi avançant amb next per obtenir el següent nombre de Fibonacci.

Només es generen exactament aquells nombres de Fibonacci que necessitem.

Aquest concepte és força útil i en Python modern es fa servir moltíssim.

Hi ha maneres molt, i molt interessants de fer-los servir :-) però no era el tema, el tema era per què cal yield amb Scrapy, i amb Twisted.

Per què cal el `yield` amb Scrapy/Twisted?

Doncs… Perquè a Python no hi havia async i await fa 20 anys!

La primera versió de Twisted va sortir al 2002. Aleshores el concepte de programació asíncrona no era tan comú com ho és ara, i per descomptat Python no ho suportava.

Parèntesi de programació asíncrona

Ja que hem mencionat el concepte un parell de vegades, caldrà il·lustrar-ho breument:

Tradicionalment un programa s’executa de forma seqüencial, és a dir en l’ordre en què estan escrites les instruccions.

En el món real però, especialment en el món de les xarxes, fer-ho d’aquesta manera és altament ineficient.

Ja que parlem de Scrapy, pensem en la tasca “descarregar la primera plana de 1000 pàgines web”.

Si ho fem de manera seqüencial, trigarem la suma de tots els temps de descàrrega, és a dir, si es triga 1 segon per cada pàgina, trigarem 1000 segons.

Si ho fem de manera asíncrona, es facilita paral·lelitzar aquesta tasca, el que fem és començar la descàrrega d’una pàgina web i rebre una notificació / “callback” quan aquesta descàrrega finalitza. Així, si podem fer 10 descàrregues paral·leles i triguem 1 segon per descàrrega, la mateixa tasca trigarà 100 segons (100 blocs de 1 segon amb 10 descàrregues per bloc).

Programació asíncrona en Python modern

En Python modern es pot fer programació asíncrona sense necessitar altres llibreries, aquesta implementació va estar molt influenciada per les maneres de fer de Twisted i els 10 anys d’experiència mantenint aquest tipus de codi en Python.

Molt per sobre, per definir una funció asíncrona:

async def f():
    # Pausa l'execució fins que `g` retorni
    b = await g()
    # Segueix fent altres coses
    return b**2

Podem fer servir async def per marcar la funció f com a asíncrona, i en el cos d’aquesta funció podem fer servir await per “esperar” a que altres funcions asíncrones acabin, i així poder escriure codi que sembli seqüencial.

En executar f(), no obtenim directament el resultat, sinó que obtenim una coroutine (segons com una Promise).

Aquests són els noms que més es fan servir avui dia per aquests conceptes, en part perquè es van popularitzar amb Node.js al 2009.

Programació asíncrona amb Twisted

No és molt diferent! Però per aconseguir-ho va caldre abusar conceptes que ja estaven disponibles en Python aleshores, com ara els generators!

from twisted.internet import defer

@defer.inlineCallbacks
def f():
    # Pausa l'execució fins que `g` retorni
    b = yield g()
    # Segueix fent altres coses
    defer.returnValue(b**2)

Podem fer servir el decorador defer.inlineCallbacks per “marcar la funció f com a asíncrona”, i en el cos d’aquesta funció podem fer servir yield per “esperar” a que altres funcions asíncrones acabin, i així poder escriure codi que sembli seqüencial.

En executar f(), no obtenim directament el resultat, sinó que obtenim un twisted.internet.defer.Deferred.

Conceptualment molt semblant, però força diferent a la vista. Per què?

Doncs en no tenir els conceptes dintre del llenguatge, desenvolupadors com Glyph Lefkowitz van tenir idees molt bones; si ens hi fixem, fer servir yield al cos de la funció, vol dir que f retorna un generator i no pas un resultat.

De fet, no tenim ben bé una manera de marcar “el que f retorna”, perquè cada operació que fem de manera asíncrona necessitarà un yield.

D’aquí que necessitem ambdós: defer.inlineCallbacks i defer.returnValue.

Amb el decorador defer.inlineCallbacks, el que fem és convertir aquesta funció, en quelcom que ens retorna un Deferred, i amb defer.returnValue el que fem és especificar quin serà el valor de retorn d’aquest Deferred.

Els Deferreds, com les Promises, en el fons són molt equivalents, són un objecte de Python, que ens representa un procés o càlcul que encara no ha finalitzat.

A aquests objectes els podem afegir manualment callbacks, per tal d’encadenar processos o podem fer servir yield dintre de funcions marcades com a asíncrones amb defer.inlineCallbacks per tal d’esperar el seu resultat.

Conclusió

La sintaxi “nova” de Python per programació asíncrona és molt bonica!

I Twisted, en ser una llibreria tan innovadora, encara no se’n beneficia al 100%. En part és perquè el canvi de Python 2 a Python 3 va trigar força, perquè la base de codi és molt extensa (hi ha molts protocols implementats), en part també perquè l’ecosistema async a Python va trigar en estabilitzar-se.

La realitat és que a dia d’avui es pot escriure codi que faci servir Twisted amb la sintaxi nova! Tot i això, molt sovint és útil saber d’on venim :-) perquè la compatibilitat no és perfecta, i a vegades ens trobem amb la necessitat d’escriure codi amb la sintaxi antiga; a més a més, entenent això podrem entendre els tutorials i el codi de les llibreries que necessitem, com Scrapy!

Per cert, que quedi clar que Twisted encara és un projecte rellevant! De fet, avui mateix s’ha anunciat la pre-release d’una nova versió: 22.8.0.

Fixing web pitfalls with Greasemonkey

2021-06-05T00:00:00Z

Introducció

A molts webs, particularment els de les administracions és evident que les coses es dissenyen perquè siguin pràctiques per qui les desenvolupa (teòricament més barat) i no pas per qui les utilitza (major utilitat).

Una cosa que no es coneix gaire, és que molt sovint podem afegir les nostres personalitzacions a sobre per arreglar aquestes pífies.

Aquí veurem un exemple del portal de notes de la Generalitat de Catalunya, es diu Esfer@.

Taula de continguts

Introducció
Taula de continguts
Les pífies
Les millores
Obtenint (o creant) aquestes millores
Conclusió

Les pífies

En la Educació Secundària Obligatòria és necessari posar un comentari per cada alumne, veiem com és la interfície per fer-ho:

A l’esquerra hi ha un desplegable per seleccionar la nota de cada alumne (representat en una fila) i a la dreta hi ha un botó, en color vermell ens informa que hi ha un comentari posat, en color gris que no n’hi ha.

En fer-hi click, s’obre un diàleg modal (que “ocupa” tota la pantalla) on podem posar el comentari, hem de fer click de nou en desa i anar pel següent alumne.

Sembla poc pràctic? Ho és. Addicionalment, la sessió d’usuari caduca després de 30 minuts, així que: posa comentari personalitzat a tots els alumnes del curs de la ESO en 30 minuts o tota la teva feina es perd.

És un desastre, vaja.

Les millores

Les millores fan aquesta pinta tan senzilla:

Afegim 3 botons a l’apartat d’avaluacions per grup i matèria:

Explicació millores Esfer@: un enllaç a aquest document explicatiu
Baixar CSV: genera i baixa un fitxer notes.csv on podreu emplenar les notes i els comentaris amb tranquilitat. Aquest fitxer CSV és quasi un full de càlcul, no cal explicar res més; heu de modificar/emplenar les columnes de Nota i de Comentari. Caldrà que feu servir LibreOffice (vegeu també més abaix el perquè).
Importar CSV: demana triar el fitxer notes.csv modificat i aplica els canvis de nota i comentari, posteriorment refresca la pàgina perquè ho comproveu. Recordeu que la responsabilitat és vostra!

Ha estat provat amb la ESO i el Batxillerat (per tant amb notes “qualitatives” i quantitatives), i en notes de trimestre i finals.

Obtenint (o creant) aquestes millores

Primer pas: Firefox amb Greasemonkey

Fem servir Firefox i la extensió Greasemonkey.

La extensió Greasemonkey, permet afegir “scripts” (codi) personalitzat a certs webs, d’aquesta manera podem interactuar amb aquests webs amb Interfície d’Usuari / Experiència d’Usuari poc satisfactòria i… Bé, solucionar-nos la vida una mica.

Segon pas: millores esfera

Un cop instal·lat el Greasemonkey, només cal que feu click aquí. I ara, sempre que estigueu a la part d’avaluacions de l’Esfer@ (per grup i matèria), després d’uns segons apareixeran aquests 3 botons que hem mencionat.

En visitar qualsevol enllaç que acabi amb .user.js ensenyarà un diàleg preguntant si volem instal·lar aquest codi personalitzat.

Això és una cosa que hem de fer amb cura, en aquest cas però, espero que us en fieu (i si no, comproveu o demaneu a algú que comprovi el codi!).

Tercer pas: LibreOffice

És super important fer servir LibreOffice i no pas Excel o semblants.

El motiu és que Excel no reconeix el format del fitxer fàcilment i tampoc el manté de manera senzilla; en canvi amb LibreOffice només heu de confirmar que fa bona pinta i fer click en Ok. I sobre tot en desar canvis: Desar en format CSV sempre.

Altres webs

En altres webs podeu fer coses semblants, el web de Greasemonkey te molta informació (en anglès), us podeu administrar aquest tipus de millores com vulgueu.

Conclusió

Tot te solució :-). No sempre hem d’acceptar el que ens ofereixen, que hi ha molts interessos i no sempre estan alineats amb els nostres.

En aquest cas, amb 4 clicks us podeu estalviar molta frustració, errades i podeu revisar que els comentaris (i per tant la vostra feina) estiguin ben fets.

Recordeu també que és la vostra responsabilitat comprovar que tot estigui bé. Això us pot ajudar i donar comoditat, però al final la responsabilitat és vostra!

Si això us és útil, em podeu convidar a un cafè o a dinar o també em podeu contractar en alguna de les moltes coses relacionades amb ordinadors, xarxes, sistemes, etc. que faig ^^.

FreeBSD: IPv6 in a VNET jail

2021-01-05T00:00:00Z

Introduction

Fellow human @LeJax@bsd.network has been trying to setup IPv6-enabled VNET Jails in FreeBSD.

This, turns out, can be a bit frustrating, because all the tools are there, but the tricks are not widely documented.

In an attempt to fix that, I am going to document all the tricks and sources I have found (and remember) that were necessary until I got a working and reproducible setup. Hopefully this will not only help LeJax, but also it will review my notes and allow them (or someone else) to prepare a nice general-use write up on this topic that might be useful to others.

I use cdist and iocage extensively, but the general tricks should be valid without any or either of those.

Introduction
Table of Contents
General notes
Networking options for IPv6 VNET jails
Firewall – Passing the devices
Using my cdist types
Conclusion

General notes

Reminder about FreeBSD jails

Jails on FreeBSD are a very useful OS tool that is tightly integrated with the rest of the Operating System, which means that, for the most part, issues regarding IPv6 in jails are actually issues with general networking and not specific to jails.

Note on iocage

iocage is a helper to create and manage jails, some like it, some don’t. It itself doesn’t do much magic, but has some ZFS integrations that I find useful. These notes should mostly apply to jails created with jail(8) as well.

Note on cdist

cdist is a “usable configuration management system”. It’s basically shell on steroids to manage servers. If I cite a type, the manifest and gencode-remote (optionally the explorer/*) files should be reviewed. These should contain enough comments to be understandable and to identify the relevant bits.

Networking options for IPv6 VNET jails

VNET: virtualised network stack

From vnet(9):

VNET is the name of a technique to virtualize the network stack.  The
basic idea is to change global resources most notably variables into per
network stack resources and have functions, sysctls, eventhandlers, etc.
access and handle them in the context of the correct instance.  Each
(virtual) network stack is attached to a prison, with vnet0 being the
unrestricted default network stack of the base system.

With VNET(9), our jails basically become a separated machine network-wise. We can (and maybe want to) also run a firewall on the jail itself.

It also means we have to take care of networking somehow; this is usually the tricky bit. For IPv4 that means: static addresses or DHCP, for IPv6 that might mean SLAAC.

Because of IPv6 nature, the usual tricks for IPv4 of using NAT / port forwarding are a subpar solution, so we shouldn’t necessarily just try to replicate that.

Option A: jail host acts as a router

Depending on how that is happening, our host might need to act as a router (the sysctl nodes: net.inet.ip.forwarding / net.inet6.ip6.forwarding set to 1, dhcpd(8) and rtadvd(8) running, …, see: FreeBSD router).

This might be the case if, e.g. we assign a /64 to the jails in a given host.

| jail |      | jail host  |      | vnet |
| host | <--> | bridge     | <--> | jail |
              | (optional) |

Has a           Receives            uses SLAAC
routed          rtadv               to gain IPv6
/64             for the /64         connectivity
Runs
rtadvd(8)

If we do this, some important things to take into account:

The jail host will need to forward packages (ipv6_gateway_enable=YES in /etc/rc.conf, which results in sysctl net.inet6.ip6.forwarding=1)
The jail host will have to run rtadvd(8) in a way that reaches the jails
Depending on the setup, ipv6_cpe_wanif should be set in /etc/rc.conf if the jail host needs to accept route advertisements on some interface.

In any case rc.conf(5) has more information on these settings.

Option B: bridge to public interface

This might be easier to setup if it matches our security needs and we already have a working router with SLAAC.

                                   ·--> | jail host |  firewall passes
                                  /     | interface |  traffic on bridge
| router | <--> | jail host | <--·                     (switch-like)
                | bridge    | <--·
                                  \     | vnet |       uses SLAAC to
Pre-existing                       ·--> | jail |       gain IPv6
Does SLAAC                                             Connectivity

Option C: no SLAAC, static routing

This gets a tad complicated to manage and I haven’t tested it.

The network schemes might look a lot like Option A and Option B depending on how we actually do it, but it doesn’t have to.

The main difference being that instead of SLAAC + ND + DAD, we are potentially routing things somehow else.

IPv6: `accept_rtadv` and `auto_linklocal`

This is actually the key point to get everything to work.

There is a bug in the latest iocage release (1.2 in January 2021), in which newly created jails are not properly prepared for IPv6. This is fixed in the development branch (see commit).

For regular jails, and before iocage has release beyond 1.2, we have to manually ensure that the interface is properly setup.

In any case, we do need linklocal addresses, in order for Neighbour Discovery (ND) to work.

# Inside the Jail
# Partial contents of /etc/rc.conf

# Get an IPv4 on jail start with DHCP
ifconfig_epair0b="SYNCDHCP"
# Ensure we accept route advertisements and have a linklocal address
ifconfig_emailr0b_ipv6="inet6 accept_rtadv auto_linklocal"

Firewall – Passing the devices

Nowadays it should be safe to use, e.g. pf(4) on VNET jails (TODO: citation needed, it’s somewhere on the freebsd-net/freebsd-questions ML).

We basically setup the firewall as usual, except that, by default, the devfs.rules(5) do not expose the necessary packet filter devices.

Something similar might apply for ipfw(4).

# In the Jail Host:
# Partial contents of /etc/devfs.rules
[devfsrules_iocage=5]
add include $devfsrules_jail
add path pf unhide
add path pflog unhide
add path pfsynv unhide

Changing this file requires a service devfs restart.

And we have to make sure that our jails use this set of devfs rules.

iocage set devfs_ruleset=5 ${JAIL}

In any case, care should be taken not to block too much (e.g. blocking all icmp6 will utterly break IPv6 connectivity).

Using my cdist types

I wrote a couple cdist types that take care of all this in a reliable fashion. You can find them and the source for other useful cdist types on my cdist repository.

__evilham_iocage: Setup the jail host with iocage to be ready for IPv6-enabled jails.
__evilham_iocage_jail: Create and maintain an IPv6-enabled jail.

And the manifest for the jail host looks like this:

# Setup the jail host
__evilham_iocage --zpool jails
# Create/update the necessary jails
require="__evilham_iocage" __evilham_iocage_jail \
    "jail.example.org" \
    --bridge "bridge0"

Then I can setup DNS and ssh -6 jail.example.org, or use a cdist manifest to set up the service.

Conclusion

Hopefully this contains enough pointers to help others figure out IPv6-enabled jails, and to document this in a more generic fashion for everyone else.

If this was useful for you, do let me know.

Rainer Maria Rilke's letter to Friedrich Westhoff

2020-12-12T00:00:00Z

Introduction

Putting this here because of its overall quality, significance and for easy future access.

The letter

Envelope

von: Rainer Maria Rilke

an: Friedrich Westhoff

Rom, am 29. April 1904

Body

Mein lieber Friedrich,

wir haben durch Mutter in dieser Zeit öfters von dir gehört, und, ohne Genaueres von dir zu wissen, fühlen wir doch, dass du eine schwere Zeit hast. Mutter wird dir nicht helfen können, denn im Grunde kann keiner im Leben dem anderen helfen; das erfährt man immer wieder in jedem Konflikt und jeder Verwirrung: dass man allein ist.

Das ist nicht so schlimm, wie es auf den ersten Blick scheinen mag; es ist auch wieder das Beste im Leben, dass jeder alles in sich selbst hat: sein Schicksal, seine Zukunft, seine ganze Weite und Welt. Nun gibt es freilich Momente, wo es schwer ist, in sich zu sein und innerhalb des eigenen Ichs auszuhalten; es geschieht, dass man gerade in den Augenblicken, da man fester und – fast müsste man sagen – eigensinniger denn je an sich festhalten sollte, sich an etwas Äußeres anschließt, während wichtiger Ereignisse den eigenen Mittelpunkt aus sich heraus in Fremdes, in einen anderen Menschen verlegt. Das ist gegen die allereinfachsten Gesetze des Gleichgewichts, und es kann nur Schweres dabei herauskommen.

Clara und ich, lieber Friedrich, wir haben uns gerade darin gefunden und verstanden, dass alle Gemeinsamkeit nur im Erstarken zweier benachbarter Einsamkeiten bestehen kann, dass aber alles, was man Hingabe zu nennen pflegt, seinem Wesen nach der Gemeinsamkeit schädlich ist: Denn wenn ein Mensch sich verlässt, so ist er nichts mehr, und wenn zwei Menschen beide sich selbst aufgeben, um zueinander zu treten, so ist kein Boden mehr unter ihnen und ihr Beisammensein ist ein fortwährendes Fallen. – Wir haben, mein lieber Friedrich, nicht ohne große Schmerzen, solches erfahren, haben erfahren, was jeder, der ein eigenes Leben will, so oder so zu wissen bekommt.

Ich werde einmal, wenn ich reifer und älter bin, vielleicht dazu kommen, ein Buch zu schreiben, ein Buch für junge Menschen; nicht etwa, weil ich glaube, etwas besser gekonnt zu haben als andere. Im Gegenteil, weil mir alles so viel schwerer geworden ist als anderen jungen Menschen von Kindheit an während meiner ganzen Jugend.

Da habe ich immer und immer wieder erfahren, dass es kaum etwas Schwereres gibt, als sich lieb haben. Dass das Arbeit ist, Tagelohn, Friedrich, Tagelohn; weiß Gott, es gibt kein anderes Wort dafür. Sieh, und nun kommt noch dazu, dass die jungen Menschen auf so schweres Lieben nicht vorbereitet werden; denn die Konvention hat diese komplizierteste und äußerste Beziehung zu etwas Leichtem und Leichtsinnigem zu machen versucht, ihr den Schein gegeben, als könnten sie alle. Dem ist nicht so. Liebe ist etwas Schweres, und sie ist schwerer denn anderes, weil bei anderen Konflikten die Natur selbst den Menschern anhält, sich zu sammeln, sich ganz fest mit aller Kraft zusammenzufassen, während in der Steigerung der Liebe der Anreiz liegt, sich ganz fortzugeben. Aber denke doch nur, kann das etwas Schönes sein, sich fortzugeben nicht als Ganzes und Geordnetes, sondern so dem Zufall nach, Stück für Stück, wie es sich trifft? Kann solche Fortgabe, die einem Fortwerfen und Zerreißen so ähnlich sieht, etwas Gutes, kann sie Glück, Freude, Fortschritt sein? Nein, sie kann es nicht … Wenn du jemandem Blumen schenkst, so ordnest du sie vorher, nicht wahr? Aber junge Menschen, die sich lieb haben werfen sich einander hin in der Ungeduld und Hast ihrer Leidenschaft, und sie merken gar nicht, welcher Mangel an gegenseitiger Schätzung in dieser unaufgeräumten Hingabe liegt, merken es erst mit Staunen und Unwillen an dem Zerwürfnis, das aus aller dieser Unordnung zwischen ihnen entsteht. Und ist erst Uneinheit unter ihnen, dann wächst die Wirrnis mit jedem Tage; keiner von den beiden hat mehr etwas Unzerschlagenes, Reines und Unverdorbenes um sich, und mitten in der Trostlosigkeit eines Abbruchs suchen sie den Schein ihres Glückes [denn um des Glückes willen sollte all das doch sein] festzuhalten. Ach, sie vermögen sich kaum mehr zu entsinnen, was sie mit Glück meinten. In seiner Unsicherheit wird jeder immer ungerechter gegen den anderen; die einander wohltun wollten, berühren einer den anderen nun auf herrische und unduldsame Art, und im Bestreben, aus dem unhaltbaren und unerträglichen Zustand ihrer Wirrnis irgendwie herauszukommen, begehen sie den größten Fehler, der an menschlichen Beziehungen geschehen kann: sie werden ungeduldig. Sie drängen sich zu einem Abschluss, zu einer, wie sie glauben, endgültigen Entscheidung zu kommen, sie versuchen ihr Verhältnis, dessen überraschende Veränderungen sie erschreckt haben, ein für allemal festzustellen, damit es von nun ab „ewig“ [wie sie sagen] dasselbe bleibe. Das ist nur der letzte Irrtum in dieser langen Kette von aneinander festhaltenden Irrungen. Totes nicht einmal lässt sich endgültig festhalten [denn es zerfällt und verändert sich in seiner Art] wie viel weniger lässt sich Lebendes und Lebendiges ein für alle Mal abschließend behandeln. Leben ist ja gerade Sichverwandeln, und menschliche Beziehungen, die ein Lebensextrakt sind, sind das Veränderlichste von allem, steigen und fallen von Minute zu Minute, und Liebende sind diejenigen, in deren Beziehung und Berührung kein Augenblick dem anderen gleicht. Menschen, zwischen denen nie etwas Gewohntes, etwas schon einmal Dagewesenes vor sich geht, sondern lauter Neues, Unerwartetes, Unerhörtes. Es gibt solche Verhältnisse, die ein sehr großes, fast unerträgliches Glück sein müssen, aber sie können nur zwischen sehr reichen Menschen eintreten und zwischen solchen, die jeder für sich, reich, geordnet und versammelt sind, nur zwei weite, tiefe, eigene Welten können sie verbinden. – Junge Menschen – das liegt auf der Hand – können ein solches Verhältnis nicht gewinnen, aber sie können, wenn sie ihr Leben recht begreifen, langsam zu solchem Glück anwachsen und sich vorbereiten dafür. Sie müssen, wenn sie lieben, nicht vergessen, dass sie Anfänger sind, Stümper des Lebens, Lehrlinge in der Liebe, – müssen Liebe lernen, und dazu gehört [wie zu jedem Lernen] Ruhe, Geduld und Sammlung!

Liebe ernst nehmen und leiden und wie eine Arbeit lernen, das ist es, Friedrich, was jungen Menschen nottut. – Die Leute haben, wie so vieles andere, auch die Stellung der Liebe im Leben missverstanden, sie haben sie zu Spiel und Vergnügen gemacht, weil sie meinten, dass Spiel und Vergnügen seliger denn Arbeit sei; es gibt aber nichts Glücklicheres als die Arbeit, und Liebe, gerade weil sie das äußerste Glück ist, kann nichts anderes als Arbeit sein. – Wer also liebt, der muss versuchen, sich zu benehmen, als ob er eine große Arbeit hätte: Er muss viel allein sein und in sich gehen und sich zusammenfassen und sich festhalten; er muss arbeiten; er muss etwas werden!

Denn, Friedrich, glaube mir, je mehr man ist, je reicher ist alles, was man erlebt. Und wer in seinem Leben eine tiefe Liebe haben will, der muss sparen und sammeln dafür und Honig zusammentragen.

Man muss nie verzweifeln, wenn einem etwas verloren geht, ein Mensch oder eine Freude oder ein Glück; es kommt alles noch herrlicher wieder. Was abfallen muss, fällt ab; was zu uns gehört, bleibt uns, denn es geht alles nach Gesetzen vor sich, die größer als unsere Einsicht sind und mit denen wir nur scheinbar im Widerspruch stehen. Man muss in sich selber leben und an das ganze Leben denken, an alle seine Millionen Möglichkeiten, Weiten und Zukünfte, denen gegenüber es nichts Vergangenes und Verlorenes gibt. –

Wir denken so viel an dich, lieber Friedrich; unsere Überzeugung ist die: Dass du in der Wirrnis der Ereignisse längst, aus dir heraus, den eigenen einsamen Ausweg gefunden hättest, der allein helfen kann, wenn nicht die ganze Last des Militärjahres noch auf dir läge … Ich erinnere mich, dass nach meiner eingesperrten Militärschulzeit mein Freiheitsdrang und mein entstelltes Selbstgefühl [das sich erst allmählich von Bügen und Beulen, die man ihm beigebracht hatte, erholen musste] mich zu Verirrungen und Wünschen, die gar nicht zu meinem Leben gehören, treiben wollte, und es war mein Glück, dass meine Arbeit da war: In ihr fand ich mich und finde mich täglich in ihr und suche mich nirgends anders mehr. So tun wir beide; so ist Claras und mein Leben. Und du wirst auch dazu kommen, ganz gewiss. Sei guten Mutes, alles ist vor dir, und die Zeit, die mit Schwerem hingeht, ist nie verloren. Wir grüßen dich, lieber Friedrich, von Herzen:

Rainer und Clara

Terminal-based presentations

2020-10-31T00:00:00Z

Introducción

En tiempos de Pandemia, los encuentros sociales son principalmente digitales, esto incluye también encuentros técnicos, en el contexto de “cosas de ordenadores”.

En los últimos meses hemos hecho talleres regulares de cdist, y también algún curso de IPv6 con MaadiX.

Como la mayoría de contenido es texto, las herramientas habituales para compartir pantalla no son ideales. Esto se debe a que estas herramientas no transmiten el texto, sino que transmiten vídeo; es decir, imágenes comprimidas (simplificando un poco).

Al enviar vídeo, tenemos varios problemas:

Cada persona espectadora no puede configurar la fuente a sus necesidades / preferencias
Se requiere un ancho de banda bastante superior
Si la calidad de la conexión no es perfecta, esto puede resultar en que la herramienta usada baje la calidad y el texto sea ilegible

Así que, como siempre, hacemos las cosas diferente. A petición “popular”, vamos a ver cómo.

Tabla de contenido

Introducción
Tabla de contenido
Una terminal
- Varias terminales (en una): tmux
Compartiendo la terminal: GoTTY
“Diapositivas” en la terminal: patat
Otras cosas: editando texto, …
Conclusión

Una terminal

Lo hacemos todo desde una terminal. La mayoría de personas involucradas están cómodas con la línea de comandos; y las que no, están cómodas leyendo documentación, así que es una elección bastante correcta.

Varias terminales (en una): tmux

Una terminal única puede ser algo limitante, así que usamos tmux. Un “multiplexor” de terminal.

Para la persona que lo está usando, es como si hubiera una única terminal, pero realmente, con Ctrl+b+c podemos cear una nueva terminal y con Ctrl+b+n y Ctrl+b+p respectivamente podemos movernos a la terminal siguiente (next) o la anterior (previous).

Para instalar tmux podemos usar habitualmente los paquetes del sistema. En sistemas como OpenBSD, no hace falta porque está incluido en el sistema base :-).

Compartiendo la terminal: GoTTY

Esta es la parte interesante, después de evaluar varias opciones, GoTTY es la que mejor configuración por defecto tenía y la que más control permitía a la persona usuaria.

Para instalar GoTTY en FreeBSD podemos usar pkg install gotty; en otros sistemas este paquete puede no estar disponible, pero hay versiones pre-compiladas en su repositorio. Ante la duda, compilar binarios de Go es sencillo :-). (Y hay instrucciones en la web de GoTTY)

Integración con tmux

Este es el contenido de mi configuración de tmux, en ~/.tmux.conf:

# Activar y desactivar barra con: Ctrl+F3
bind-key -n C-F3 set-option -g status
# Compartir la sessión con GoTTY con: Ctrl+F2
bind-key -n C-F2 new-window -n ttyshare "gotty tmux attach -r -t \\$(tmux display -p '#S')"

Si nos fijamos, con esta configuración añado dos atajos de teclado:

Ctrl+F3: para activar y desactivar la barra de estado de tmux
Ctrl+F2: para empezar a compartir la sesión de tmux con GoTTY.

Analicemos un poco más la línea:

bind-key -n C-F2 new-window -n ttyshare: Cuando se presione Ctrl+F2 Crear una nueva “ventana” de tmux llamada ttyshare.

Y en esa nueva “ventana” ejecutaremos el comando: gotty tmux attach -r -t $(tmux display -p '#S')

Esto significa que GoTTY empezará a ejecutarse, y para cada cliente que se conecte, se ejecutará: tmux attach -r -t $(tmux display -p '#S')

Excepto que… tmux reemplaza '#S' con el nombre de la sesión actual, es decir, donde presionamos Ctrl+F2

De esta forma, para cada persona que visite la dirección de GoTTY, empezaremos un tmux al que le decimos: “conéctate a la sesión donde presionamos Ctrl+F2 de forma sólo lectura”. Para obtener el efecto sólo lectura, usamos -r como argumento de tmux attach.

Vale la pena mencionar que: por defecto GoTTY también crea sesiones de sólo lectura, al hacerlo aquí también, conseguimos seguridad por capas: para comprometer nuestra sesión, haría falta que haya un problema en GoTTY y también que haya un problema en tmux.

Así, al presionar Ctrl+F2, se crea la ventana ttyshare y vemos lo siguiente:

2020/11/08 23:06:43 Loading config file at: ${HOME}/.gotty
2020/11/08 23:06:43 Server is starting with command: tmux attach -r -t $0
2020/11/08 23:06:43 URL: http://[::1]:8080/17lmim3q/

Es decir, que dese el navegador, puedo abrir la dirección http://[::1]:8080/17lmim3q/ y ver, en vivo, mi sesión de tmux

Configurando GoTTY

A este programa no le hace falta mucha configuración porque los valores por defecto son bastante razonables, pero hay alguna cosa que sí prefiero cambiar.

A partir de la configuración completa:

// [string] Address to listen, all addresses will be used when empty
//
// Limitar conexiones a la máquina local, via IPv6:
address = "::1"

// [bool] Enable random URL generation
//
// De esta forma, tenemos una capa de seguridad más, al requerir compartir
// un enlace con las personas que queremos compartir la terminal.
enable_random_url = true

// [string] Title format of browser window
//
// Aquí podemos personalizar el título de la pestaña en el navegador
title_format = "Hola!"

// [int] Timeout secodns for waiting a client (0 to disable)
//
// Por defecto, si no se conecta un cliente en 60 segundos, GoTTY se cierra
// esto es poco práctico si lo configuramos todo y esperamos a las otras
// personas. Al ponerlo a "0", desactivamos esta desconexión
timeout = 0

Usándolo con otras personas: Nginx como reverse proxy + TLS

Si nos fijamos, configuro GoTTY para escuchar únicamente en la máquina local via IPv6.

Justamente esto no es lo que queremos, si no que queremos que otras personas puedan acceder a nuestra terminal. Si no usamos cifrado en el transporte (TLS), nuestra comunicación es como una postal, cualquiera puede leerla.

Este trabajo lo hace nginx mucho mejor que lo podría hacer GoTTY, además ya tenemos una serie de automatizaciones para obtener certificados con Let’s Encrypt, así que es ideal usarlo para recibir las llamadas del mundo exterior y pasarlas a GoTTY.

Esto es bastante estándard, excepto que tenemos que tener en cuenta que GoTTY usa Web Sockets para actualizar las terminales de los clientes en “tiempo real”:

        location ~ /sh(/[^/]+)?/ws {
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "Upgrade";

                proxy_set_header Host              $host;
                proxy_set_header X-Real-IP         $remote_addr;
                proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;

                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header X-Forwarded-Port  443;

                proxy_pass https://MyComputerHost;
        }

        location /sh/ {
                proxy_set_header Host              $host;
                proxy_set_header X-Real-IP         $remote_addr;
                proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;

                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header X-Forwarded-Port  443;

                proxy_pass https://MyComputerHost;
        }

De esta forma, puedo enviar enlaces del tipo: https://evilham.com/sh/17lmim3q/ y las personas que lo reciban podrán seguir lo que sea que tenga en la terminal.

“Diapositivas” en la terminal: patat

También hay muchas alternativas de programas o utilidades para hacer presentaciones en la terminal, pero todas tenían algún tipo de inconveniente lo suficientemente grande como para no ser prácticas.

Todas excepto patat, que consigue hacerlo todo bien por defecto.

Sólo hace falta escribir Markdown, y ejecutar patat presentacion.md y ya estamos presentando con nuestras fantásticas diapositivas.

Para instalar esta herramienta en Linux, podemos usar algunos paquetes, o descargar el binario precompilado desde su repositorio.

Es posible que me anime a crear un port para FreeBSD pronto ^^.

Otras cosas: editando texto, …

Al interactuar con nuestros humanos seguro que salen dudas, o queremos comprobar un manual, o queremos mirar un fichero, o queremos entrar en una máquina remota, …

¿Lo mejor de estas presentaciones? Al hacerlo todo en una única sesión de tmux, podemos rápidamente presionar Ctrl+b+c, usar las herramientas habituales (man, vim, ssh, …) y volver a la presentación; y nuestros humanos sabrán de qué hablamos.

Conclusión

Estas herramientas son bastante sencillas, y una vez las hemos configurado, le dan a las otras personas plena flexibilidad a la hora de configurar el tamaño de letra y demás, y a nosotros también a la hora de salirnos un poco de lo que habíamos preparado.

No es para todas las presentaciones, pero para muchas… Seguro que es más eficiente y agradable ^^.

FreeBSD: Installing

2020-08-18T00:00:00Z

Introducció

He hagut de fer servir les meves paraules d’estiu per qüestions personals i professionals; així que la continuació d’aquell fil d’idees ha trigat una mica en arribar.

Per coses de la vida, aquests dies ha arribat un mini-server a les meves mans. Així que agafarem un petit desviament en coses “tècniques” abans de tornar al per què, que és segurament la única cosa que importa al final.

Aquí agafarem aquest mini-server i hi instal·larem FreeBSD com a sistema operatiu.

Per què és interessant? Doncs aquesta màquina tindrà certs serveis digitals que sempre està bé tenir sota el nostre control.

I el procés, des de zero, no és una cosa que es documenti habitualment en aquesta llengua ;-).

Taula de continguts

Introducció
Taula de continguts
Què és FreeBSD?
Per què FreeBSD?
- Diferències de desenvolupament amb Linux
Instal·lant FreeBSD
Conclusió

Què és FreeBSD?

FreeBSD és un “Sistema Operatiu obert i gratuït, semblant o compatible amb Unix”.

Això, pels que som resultat dels mitjans i l’educació digital que ens forma per ser consumidors passius, potser implica directament hores i hores de lectura.

Anem a fer-ho una mica més fàcil:

Sistema operatiu: sense això, un ordinador és un aparell més aviat inútil. No ens permetria engegar cap programa, llegir cap correu, no hi podríem fer res. El sistema operatiu és necessari, perquè és el que “sap parlar” amb els components físics que conformen l’ordinador. Tot i ser necessari, no sempre és evident; en general, el que volem com a usuaris és fer servir els programes que necessitem. Però en algun moment veurem que tenir la possibilitat de modificar el sistema a la nostra voluntat, també te els seus avantatges.
Obert i gratuït: aquest és un gran tema, i el món està dividit en interpretacions. Bàsicament, obert, vol dir que tenim accés al codi font del sistema, és a dir als fitxers de text que ha escrit algun humà, molt sovint de manera altruista, que després un ordinador processa per convertir-ho en instruccions que pugui executar. I gratuït, vol dir que no cal pagar res per fer-ho servir. El món es divideix en interpretacions per la manera en que el programari pot ser obert i gratuït. Pels que tinguin curiositat, internet està plena de diverses perspectives en el tema de les llicències, per exemple BSD (permissives) o GPL (copyleft).
Semblant a Unix: Unix és un sistema operatiu que es va crear vora els anys 1970, amb unes idees molt clares que han marcat el desenvolupament del món dels ordinadors. Aquell sistema, era comercial i requeria unes llicències, però es distribuïa amb el codi font, cosa que va fer que es fes molt popular a les universitats. En particular, a Berkeley (d’aquí la B de BSD – la distribució de programari de Berkeley). Per qüestions de llicència i propietat intel·lectual, el codi de BSD ja no està basat en el codi de Unix, però sí que manté les idees bàsiques i un alt nivell de compatibilitat (una cosa que es diu compatibilitat POSIX).

Per què FreeBSD?

Perquè funciona, funciona bé, i és senzill d’administrar.

Això no vol dir que altres sistemes no funcionin bé o siguin difícils d’administrar; fins a un cert punt, el tema va “a gustos”.

Bàsicament, en aquest punt de la meva vida professional, me’n surto força bé amb el sistema que em posin davant; ara, també tinc clar que FreeBSD (i OpenBSD en alguns casos) és el que em permet fer més, més ràpid, i sense preocupar-me gaire del com perquè la càrrega mental associada és mínima.

Naturalment hi ha altres alternatives que compleixin amb “obert i gratuït”, per exemple Linux; però hi ha tendències preocupants al món de Linux actualment.

Un altre factor per no fer servir Linux és que la experiència ja m’ha portat a “aprendre Linux” tres vegades a la meva vida (sense comptar haver fet servir diverses distribucions, que fins a cert punt són cadascuna un món). És a dir, canvia massa ràpid i en general sense prou bons motius; o bé perquè els canvis es fan massa ràpid i s’arriba a solucions poc ideals o bé perquè els problemes que s’hi estan solucionant, són els problemes de quatre empreses, i no els problemes dels usuaris.

Diferències de desenvolupament amb Linux

Els sistemes BSD es desenvolupen en conjunt, és a dir que quan obtenim el codi font o un binari, realment estem obtenint tot el que ens cal per compilar o instal·lar un sistema operatiu sencer amb: kernel (o nucli, més o menys la part que parla amb allò físic), i utilitats d’usuari (o world).

Sense les utilitats d’usuari, no tindríem manera d’interactuar amb el kernel, i per tant amb l’ordinador.

I, sense el kernel, els programes no poden parlar amb la part física, i per tant són capaços ni d’ensenyar-nos alguna cosa per pantalla (és una aproximació més o menys intuïtiva i no 100% acurada).

Això sembla una diferència menor, però realment és un gran avantatge; en tenir control de les utilitats d’usuari i també del kernel, hi ha també una co-responsabilitat, és a dir, si es vol introduir un canvi en el kernel, es fa necessari fer en paral·lel els canvis adients a les utilitats d’usuari.

I també si es fan canvis en una utilitat d’usuari, cal assegurar-se que les altres utilitats que fan part del sistema operatiu, mantinguin una compatibilitat i consistència general.

Això, al món de Linux no és possible, perquè les utilitats d’usuari solen ser “GNU” (o busybox) i Linux és únicament el kernel. D’aquesta manera, la responsabilitat pels canvis es mou d’una banda a l’altra, i, molt sovint, els canvis no tenen lloc, i s’acaben generant noves eines per salvaguardar les limitacions de les eines existents.

Per exemple, ifconfig, la comanda tradicional per administrar interfícies de xarxa, a Linux es considera arcaica, perquè ara existeix ip. I no és que ip sigui dolent, és que és un canvi, que només és necessari, perquè els problemes inherents al sistema de desenvolupament no són adreçables.

En canvi, als BSD, ifconfig és molt més potent del que mai ho va ser a Linux i també te molt més sentit; aquest desenvolupament conjunt del kernel i les utilitats d’usuari hi te molt a veure.

Instal·lant FreeBSD

Com sempre, el primer pas és descarregar l’instal·lador, això en en aquest cas es fa al web de FreeBSD.

Ens trobem amb la primera decisió: quina versió triem? Depèn de l’ús de cada, així que anem a entendre com funcionen.

Versions / línies de suport

Hi ha 3 línies del sistema operatiu suportades. Per mantenir-se al dia, recomanaria subscriure’s a la llista d’anuncis de FreeBSD.

Com a regla general, per un sistema que vulguem que funcioni sense problemes, recomanaria fer servir sempre l’última versió suportada de RELEASE.

CURRENT – Futur 13.0

Aquesta és la línia de desenvolupament, literalment és on vénen a parar tots els canvis, tant al kernel com a les utilitats d’usuari. Per aquest motiu, no es recomana per usuaris novells, sinó per aquells que vulguin ajudar el desenvolupament, encara que sigui només provant i trobant problemes. Havent dit això, la línia de desenvolupament sol ser molt estable, hi ha empreses com Netflix que actualitzen un parell de vegades al mes a la línia de desenvolupament i jo ho faig gairebé a diari al meu sistema principal.

Al món de BSD hi ha una dita, i és que si no vols que els bugs/problemes t’arribin, els has de trobar tu abans. Com més persones ajudin a provar la línia de desenvolupament, més segurs podem estar que les altres línies de suport no tenen grans problemes.

Quan ja ha passat un temps (mínim dos anys) i les persones que desenvolupen el sistema troben que és el moment, aquesta línia es “congela” i després d’una feina de solucionar problemes diversos, acabar d’actualitzar documentació i demés, s’acaba convertint en una versió RELEASE.

RELEASE – Actualment 12.1

Aquesta és l’última versió major (en aquest cas 12, amb 1 essent la versió menor). Des del moment en que es va publicar la versió 12.0, aquesta línia de desenvolupament te garantit el suport (actualitzacions de seguretat, disponibilitat de binaris oficials, …) un mínim de 5 anys.

És a dir, que FreeBSD 12 te suport, com a mínim, fins el 30 de Juny de 2024.

Ara, quan es publica una versió menor, com ara la 12.2 que vindrà aviat, la versió menor anterior (l’actual 12.1) deixa de ser suportada 3 mesos després.

És a dir, que tenim un període de 3 mesos per fer una actualització menor; és una cosa que cal tenir en compte. Normalment s’avisa amb mesos d’antelació.

STABLE – Actualment 11.4 [i 11.3]

Les versions STABLE són antigues RELEASE que encara estan suportades, és a dir, des que es va publicar FreeBSD 11.0, s’ha publicat la 12.0.

Però com que la versió FreeBSD 11 te suport garantit de 5 anys, seguirà existint, com a mínim, fins el 10 d’octubre de 2021.

Ara, veiem que a data d’avui hi ha dues versions de FreeBSD 11, això vol dir que fa menys de 3 mesos que es va publicar la versió 11.4 i, per tant, la versió 11.3 encara està suportada.

En efecte, FreeBSD 11.4 es va publicar el 16 de juny de 2020, i per tant FreeBSD 11.3 deixa de tenir suport el 30 de setembre de 2020.

Muntant un USB d’instal·lació

A data d’avui, amb el criteri que he explicat anteriorment, ens decidiríem per FreeBSD 12.1, així que podem descarregar la imatge d’instal·lació (també hi ha imatges per màquines virtuals pre-instal·lades).

En general, voldrem descarregar el fitxer -disc1.iso de la nostra arquitectura de hardware, si no sabem el que és, segurament sigui amd64. És a dir que volem el fitxer FreeBSD-12.1-RELEASE-amd64-disc1.iso.

Pels paranòics entre nosaltres, els anuncis d’una nova versió també es publiquen signats. I el llistat de claus PGP del projecte es troba al famòs Handbook de FreeBSD.

Comprovant la signatura, i després la integritat del fitxer, podem estar segurs que el fitxer que hem descarregat, és el fitxer que les persones que desenvolupen el sistema operatiu han generat.

Un cop l’hem descarregat, hi ha utilitats que ens ajuden a gravar aquesta imatge en un CD o, molt millor, en un USB.

Si heu descarregat la imatge sota Windows, rufus és un programa molt bo per fer-ho.

Si, en canvi, ho heu fet en Linux o en un Mac, el més senzill és fer-ho amb la comanda dd (si algú te un mètode més “gràfic”, que m’avisi ;-)). En particular:

# dd if=FreeBSD-12.1-RELEASE-amd64-disc1.iso \
     of=/dev/EL_MEU_USB bs=1m conv=sync

Cal que us assegureu de reemplaçar EL_MEU_USB amb el nom del vostre USB, recordeu que si us equivoqueu molt, això pot voler dir perdre les vostres dades!

Instal·lant!

Havent escrit la imatge al meu USB, engego el micro-server iniciant des de l’USB.

I ara ve la part “fàcil”, on un menú ens anirà guiant per instal·lar FreeBSD; la majoria de les decisions es poden canviar un cop instal·lat el sistema, així que no és molt crític, i els valors per defecte són raonables en general.

Sense tocar res veig una pantalla de benvinguda que em dóna a triar entre Install, Shell i LiveCD; volem Install.
Configuro el meu teclat
Li dono un nom a la màquina
Desactivo kernel-dbg i lib32 que no em fan falta (tampoc fa mal deixar-los)
Trio instal·lació automàtica amb ZFS (Auto (ZFS))
Vaig a Pool Type/Disks i trio stripe en el disc dur que vull
Configuro Encrypt Swap com YES
Vaig a Install: Proceed with installation
Confirmo els canvis al disc dur (si m’he equivocat, podria perdre dades!)
Espero aproximadament 2 minuts mentre el sistema s’instal·la
Configuro una contrasenya pel super usuari (no hi ha feedback visual, però està agafant els vostres caracters. Després desactivarem la contrasenya)
Trio la interfície de xarxa principal (en tinc dues)
Confirmo que vull IPv4 amb DHCP (és el més habitual)
Confirmo que vull IPv6 amb SLAAC (hauria de ser més habitual)
Confirmo la configuració que hem obtingut de la xarxa
La hora interna de la meva màquina és UTC (així no hi ha problemes de canvis horaris)
Configuro la meva zona horària
Confirmo la data i l’hora
Com a serveis activo sshd, ntpd i powerd, desactivo dumpdev (es pot canviar a posteriori)
Activo totes les opcions de seguretat a System Hardening (es pot canviar a posteriori)
No afegeixo usuaris, ho faig a posteriori
Trio Exit per indicar que ja hem instal·lat el sistema
Trio que vull una consola (shell) al nou sistema abans de reiniciar
Edito /etc/sshd/sshd.conf amb vi /etc/sshd/sshd.conf per habilitar les sessions remotes de super usuari canviant #PermitRootLogin no per PermitRootLogin yes (ho canviarem tot seguit, no és una configuració gens segura)
Tanco aquesta consola amb exit
Trio Reboot i trec l’USB d’instal·lació
El sistema inicia i ja el puc acabar de configurar remotament

És a dir, que esperem un total de 2 minuts i contestem preguntes majoritàriament senzilles; el procès triga en total uns 5 minuts si ja coneixem les preguntes i uns 20 minuts si no.

Assegurant el sistema una mica

Compte! Durant la instal·lació hem près una decició temporal que cal desfer tan aviat com sigui possible: hem de deshabilitar l’inici de sessió remot amb contrasenya.

Si no ho fem, com que hem deixat accés de super usuari remot amb contrasenya, potencialment, algú a la nostra xarxa podria intentar esbrinar la contrasenya i fer-se amb el control de l’equip.

El motiu d’aquesta decisió és, que jo faig servir autenticació amb claus públiques, però configurar aquesta autenticació només amb un teclat és una mica farragós.

Per això inicio sessió al micro-server amb usuari root i la contrasenya que he configurat abans. Executo:

# ifconfig
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
    ether XX:XX:XX:XX:XX:XX
    inet6 fe80::9640:XXXX:XXXX:XXXX%bge0 prefixlen 64 scopeid 0x1
    inet6 2a0f:de00:X:X:X:X:X:X prefixlen 64 autoconf
    inet 192.168.1.12 netmask 0xfffff000 broadcast 192.168.1.255
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

Que em diu que la IP local de la màquina és 192.168.1.12.

I llavors des del meu ordinador:

> ssh root@192.168.1.12
root@host:~ # mkdir .ssh; chmod 700 .ssh
root@host:~ # echo 'ssh-rsa AAAAB3NzaC1yc2EAAA[..] evilham.pgp_auth' \
    > .ssh/authorized_keys
root@host:~ # chmod 400 .ssh/authorized_keys

És a dir, configuro la meva clau pública per iniciar sessió (els detalls… són tema d’un altre dia).

I ara ja puc desactivar l’inici de sessió amb contrasenya amb vi /etc/ssh/sshd_config i canviant PermitRootLogin yes per PermitRootLogin without-password i reiniciant el servei amb service sshd restart.

Conclusió

Fins aquí avui, ara tenim un sistema FreeBSD instal·lat de nou, que servirà com a base per fer-hi “qualsevol cosa”.

Abans d’això caldrà configurar el sistema d’una manera una manera més segura, i d’una manera automatizada, per si mai cal reinstal·lar-ho.

Totes aquestes coses les configuro automàticament amb cdist, que és un tema prou gran per per un altre dia.

PS: 2826 paraules han estat escrites pels micos infinits.

Words of summer

2020-08-10T00:00:00Z

Introducció

Dilluns, tarda d’estiu pandèmic. Temperatura, massa alta; humitat relativa, 9001%; vent, gairebé inexistent; estat de la mar… tranquil·la.

Així és com llegeixo que una escriptora, encoratja persones des de fa un parell d’anys a escriure 1000 paraules cada dia en sessions de dues setmanes, un parell de vegades l’any. I que això, que es diu #1000wordsofsummer, comença avui.

Penso:

Hauria d’escriure més, em cal més pràctica.

M’hi apunto! Si més no, es pot intentar. El pitjor que pot passar és que ho deixis en #500wordsofsummer, que tampoc està gens malament; o que ho facis dos dies i ho deixis, que tampoc passa res. ¡Dale!

Com a motivació, fes-ho en català, que últimament no hi escrius gairebé mai.

Així doncs, sobre què escric? El que més bé se’m dóna (per a alguns valors de b) és escriure qüestions tècniques, i no és una cosa que es trobi habitualment en català.

Vet aquí que, en un acte subversiu lingüístic: escriurem qüestions tècniques en català.

Però… Qüestions tècniques sense rerefons tampoc és una cosa que em motivi del tot.

Solució: popurri d’actualitat amb motivacions socials, per tal d’aconseguir un fil conductor per quatre coses tècniques amb cohesió i coherència internes justetes però, amb una mica de sort, existents. Som-hi!

Taula de continguts

Introducció
Taula de continguts
Conjuntura socioeconòmica
La dualitat: privilegi | factor d’exclusió
Temes a treballar
Conclusió

Conjuntura socioeconòmica

Arran de les circumstàncies actuals, Catalunya te unes dades macroeconòmiques nefastes. I, no és que no hi pensi de normal, però és especialment en les crisis quan penso en coses com model econòmic, igualtat d’oportunitats, etc.

La dualitat: privilegi | factor d’exclusió

Crec que un factor clau per reconèixer els nostres privilegis és ser conscients que, conceptualment, els privilegis formen una dualitat amb els factors d’exclusió. És a dir, que l’absència d’un factor d’exclusió, és un privilegi, i viceversa.

Aquesta dualitat és útil per mi, perquè trobo més fàcil imaginar-me un factor d’exclusió, que un privilegi. Els privilegis poden ser subtils i estar interioritzats, en canvi, imaginar-nos que en la absència de quelcom, ens podríem veure exclosos, és més factible.

No és que jo sigui una persona en una posició tan privilegiada, com per influenciar aquestes macrodades o adreçar problemes sistèmics de manera eficaç.

Però sí que he tingut alguns privilegis, dels quals vull ser molt conscient, i que potser puc fer servir per eliminar el seu dual, en forma de factor d’exclusió, per algú altre.

Accés a tecnologia i coneixement

El privilegi

Vaig tenir el meu primer ordinador als 6 anys i una connexió a internet (56kbps!) als 8 ó 9 anys.

Com a experiència curiosa, durant la meva infantesa era l’únic nen amb un ordinador a casa. I, ja aleshores, vaig ensenyar a tots els humans petits del meu carrer a engegar, apagar i fer servir un ordinador. A escriure-hi, a dibuixar-hi, …

Es pot dir que ja anava descobrint, la meva línia més jáquer (la hacker de debò, no la de la RAE). És a dir, que aquest món es tracta de: compartir, obrir, descentralitzar, facilitar accés a les tecnologies de la informació i… millorar el món mateix.

Tenint en compte que aleshores vivia en un país d’Amèrica Llatina, on encara avui els habitants amb accés a internet representen només un 63.2% de la població (UE: 89.4%, CAT: 93.7%); és evident que hi tenia algun avantatge.

El factor d’exclusió

D’altra banda, aquests mesos hem vist com, fins i tot a Catalunya amb una penetració d’accés a internet tan alta (93.7%), les tasques educatives s’han vist afectades de manera negativa i han fet més evident el caràcter privilegiat de l’accés al coneixement alhora que el factor d’exclusió es feia més gran.

Arreu el món persones altruistes han intentat mitigar aquest factor mitjançant infraestructures compartides, iniciatives socials, cultura associativa, …

No tot és dolent.

Llengües

El privilegi

Parlo i escric quatre idiomes amb fluïdesa i a nivells cultes (n’entenc més, n’aprenc “seriosament” un altre).

No tot el mèrit és meu; vaig tenir la “sort” d’anar a una escola privada on, de sèrie, teníem el doble d’hores d’anglès.

També tenia accés a un ordinador i a internet a casa des de molt petit. Resulta que l’anglès és l’idioma dels ordinadors, aleshores els jocs no es traduïen gaire, a internet l’anglès era bàsicament la única llengua (la Wikipedia i Google no existien. Ja tinc una edat…). Així que tenia força motius per agafar un diccionari i anar traduint la història dels meus Jocs de Rol (RPG) o els manuals dels programes que feia servir.

** Els privilegis s’acumulen i creen un bucle de retroalimentació positiu. **

El factor d’exclusió

L’anglès ja era l’idioma dels ordinadors fa 20 anys, i encara és el cas.

Sense saber anglès, l’accés a la documentació és sovint limitat, arriba tard, o no està actualitzat. Molts programes es tradueixen quan arriben a un punt “prou estable” o directament no es tradueixen en absolut o no es tradueixen a certes llengües.

Fins i tot a països com Alemanya amb una llengua parlada per molts referents de l’àmbit, no parlar anglès és un inconvenient.

Ara, també es produeix tecnologia en alemany. També es parla de tecnologia en alemany.

Cultura de consum (no passiu)

El privilegi

De petit no tenia ningú que em solucionés els problemes amb l’ordinador o els electrònics, el que en principi sona com un factor d’exclusió, combinat amb la meva curiositat va esdevenir un privilegi.

Mai he percebut la tecnologia com un consumible, com una cosa estàtica, inalterable.

Això és, en part, perquè eren altres temps; comprar un disc dur o una ampliació de memòria requeria estalviar molt, tot el que es pogués reutilitzar, era molt benvingut.

Tampoc hi havia una indústria tecnològica orientada al consum i al recanvi, però començava a desenvolupar-se.

El factor d’exclusió

Els nostres mitjans de comunicació ens parlen de tuitàires, YouTubers, Instagrammers, Influencers, de WhatsApp, Telegram, iPhones i tantes altres empreses i productes exclusivament com si fóssim consumidors passius.

El fet que una d’aquestes plataformes (centralitzades, autoritàries i tancades) implementi una funcionalitat banal es torna digne de 5 minuts de tele-diari o de dotzenes d’articles de “tecnologia”.

És a dir, se’ns educa per consumir, per acceptar el que ens ofereixin, per no qüestionar, criticar, modificar, millorar.

El millor exemple d’això és el programa Popap de Catalunya Radio.

No és un mal programa, al contrari, és entretingut, però no eś crític, no ofereix noves perspectives, va sempre enrere; és passiu.

Si som consumidors passius, sempre estarem en una posició de desavantatge.

No endogàmia intel·lectual

El privilegi

He tingut la sort de tenir al meu costat persones fantàstiques i alienes a aquest àmbit, que han ajudat a que no caigui en les trampes habituals d’una indústria, en general, bastant malaltissa quan es tracta de posar les persones endavant.

Sense persones crítiques i amb perspectives diferents, passa que generem persones d’ufana i supèrbia infinites.

Persones que són capaces de considerar el veure persones sense sostre un problema, en comptes de considerar com un problema el fet que l’accés a l’habitatge no sigui universal.

La part positiva, a més de poder viure amb mi mateix, és que de manera totalment orgànica i espontània es generen xarxes de solidaritat amb persones que comparteixen la teva manera de veure el món.

Sense connexió humana, emocional, intel·lectual; res més te sentit.

El factor d’exclusió

Aquest… No serà ben bé un factor d’exclusió. Tristament, la societat en que vivim premia aquest tipus de comportament i aquestes persones poden sobreviure bastant bé amb altres de semblants.

El meu “consol” és que aquestes persones sovint conviuen amb una buidor interna, no accediran a una xarxa de solidaritat, no sabran el que és poder comptar amb altres persones i tenir una certa seguretat de que no estaràs sola en el món.

Sembla que en el nostre món, aquesta xarxa de solidaritat serà cada vegada més necessària, fins i tot pels benestants.

Temes a treballar

Tenint en compte aquests factors (quasi segurament no exhaustius), se’m planteja com a fil conductor:

Maneres de mitigar alguns d’aquests factors d’exclusió digital.

El com, l’anirem descobrint. Molt segurament passi per comentar l’status-quo, explicar breument per què calen alternatives, i explicar com es pot passar de ser una persona que consumeix tecnologia de manera passiva, a coproduir tecnologia o, com a mínim, a consumir-la de manera conscient i crítica.

Conclusió

Demà més, hopefully. Si hi ha algun tema que trobeu especialment interessant, alguna qüestió que no hagi previst o algun error campant, poseu-vos en contacte i m’ho penso / ho analitzo :-).

La nota que no hauria de sobrar a internet però que no sobra: els insults us els podeu quedar, la crítica constructiva, si us plau, feu-la arribar ^^.

PS: 1822 paraules han estat escrites pels micos infinits.

Phabricator: batch audit

2020-07-30T00:00:00Z

Introduction

At work I use phabricator to collaborate with my employees, usually I know what they are working on and have a rough idea that things are going fine, so we currently don’t rely on pre-commit code reviews, which are a wonderful tool, but instead rely on semi-regular audits. (Check this for an overview on phabricator reviews and audits)

This means, that every once in a while, I have to go through my audit queue and check that indeed things are going mostly alright. I raise concerns about particular changes seldom, because of a fluid communication, so I mostly just read and accept changes to keep having a reasonably up-to-date overview.

The issue is: phabricator‘s UI is very slow for this workflow.

The good news is that it has a very good API that allows automation of many things.

This is about how I implemented phabaudit.sh, a batch audit workflow for phabricator that works-for-me(tm).

Introduction
Table of Contents
Conduit: phabricator’s API
Putting it all together
Conclusion

Conduit: phabricator‘s API

Even before swagger was a thing, phabricator had a built-in page documenting the conduit API and allowing users to explore the API in a somewhat interactive fashion.

This is found at https://phabricator_instance/conduit/.

Guessing the API endpoint

Since I know audits are associated with commits, I searched for commit-related API endpoints. Indeed, diffusion.commit.search is a good place to start and it contains a simple example, which satisfies my current needs:

{
  ...
  "queryKey": "active",
  ...
}

Where "queryKey": "active" means that I want to list active audits only. There is also a mention of the order parameter which can be oldest to list entries in chronological order.

Let’s test that.

Choosing the interaction tool

The examples section in the conduit page offers curl and arc call-conduit examples, both have pros and cons.

curl is a mostly standard tool nearly guaranteed to be available, which means that it can be mostly tested directly from the shell with:

$ curl https://phabricator_instance/api/diffusion.commit.search \
    -d api.token=api-token \
    -d order=oldest \
    -d queryKey=active > audits.json

But there is another option: arc call-conduit, that is phabricator‘s very well thought out Command Line Interface.

It enables many workflows and takes care of managing the API tokens (which I don’t have to pass int he command line every time). Audits though are a workflow that is currently not supported, but call-conduit enables arbitrary API calls.

I happen to use arc, amongst others, to contribute to FreeBSD, which does use pre-commit code reviews, that enables me and the project members and community to review changes before they land, raise concerns, suggest improvements, ensure nothing fishy is trying to be committed, …

Since I already have arc, I will use that to forget about authentication.

$ echo '{"queryKey": "active", "order": "oldest"}' | \
    arc call-conduit --conduit-uri https://phabricator_instace/ \
        diffusion.commit.search > audits.json

Parsing the output

The output of both tools is JSON piped to an audits.json file which can be “easily” manipulated with jq. I write “easily” because it takes some getting used to, but using jq is certainly doable.

This gives me the information I need to an audit_commits.json file:

$ jq -c '.response.data[] | {"commit": .fields.identifier, "repository": .fields.repositoryPHID}' audits.json > audit_commits.json

Getting the `diff` (introduced changes)

In order to get the diff, so I can audit it locally, we need a couple more API calls that can also be guessed from the https://phabricator_instance/conduit/ page:

$ echo '{"commit": COMMIT, "repository": REPOSITORY}' | \
    arc call-conduit --conduit-uri https://phabricator_instance/ \
        diffusion.rawdiffquery

{
  "error": null,
  "errorMessage": null,
  "response": {
    "tooSlow": false,
    "tooHuge": false,
    "filePHID": "PHID-FILE-00000000"
  }
}

And with that filePHID, we can get the actual diff data:

$ echo '{"phid": "PHID-FILE-00000000"}' | \
    arc call-conduit --conduit-uri https://phabricator_instance/ \
        file.download

{
  "error": null,
  "errorMessage": null,
  "response": BASE_64_STUFF
}

Which is base64 encoded, so we need to decode it. By checking man 1 b64decode I realise I need the -r flag, so getting the diff file to test.diff in the hard drive looks like this:

$ echo '{"phid": "PHID-FILE-00000000"}' | \
    arc call-conduit --conduit-uri https://phabricator_instance/ \
        file.download | \
    jq -r .response | \
    b64decode -r > test.diff

Approving the changes

After opening test.diff in my favourite editor, I decide that the change is valid and I want to mark it as approved.

Again, since audits are associated with commits, this will be in diffusion.commit.edit:

$ echo '{"objectIdentifier": COMMIT_PHID,
         "transactions": [{"type": "accept", "value": true}]}' | \
    arc call-conduit --conduit-uri https://phabricator_instance/ \
        diffusion.commit.edit

Putting it all together

Now that I have all the pieces, I can finish automating it with some POSIX shell as follows (download the full script here):

#!/bin/sh
#
# Find the latest version at:
#   https://evilham.com/en/blog/2020-phabricator/batch-audit/phabaudit.sh
# With the accompanying post:
#   https://evilham.com/en/blog/2020-phabricator/batch-audit/
# 
# Copyright 2020 Evilham <code@evilham.com>
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
# 1. Redistributions of source code must retain the above copyright notice,
#    this list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright notice,
#    this list of conditions and the following disclaimer in the documentation
#    and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.

#
# Process environment variables.
#
# User may provide CONDUIT_ARGS or PHABRICATOR_INSTANCE
# If PHABRICATOR_INSTANCE is passed, CONDUIT_ARGS will be adapted so that
# arc call-conduit always uses: --conduit-uri ${PHABRICATOR_INSTANCE}
#
# If CONDUIT_ARGS is passed, it will be used verbatim (modulo appending
# --conduit-uri as explained before).
if [ -n "${PHABRICATOR_INSTANCE}" ]; then
    CONDUIT_ARGS="${CONDUIT_ARGS} --conduit-uri '${PHABRICATOR_INSTANCE}'"
fi

# Temp dir for the audit diffs
TMP_DIR=$(mktemp -d -t audits)

get_audits()
{
    echo '{"queryKey": "active", "order": "oldest"}' | \
        arc call-conduit ${CONDUIT_ARGS} diffusion.commit.search
}

filter_commit_data()
{
    jq -c '.response.data[] | 
        {
            "phid": .phid,
            "commit": .fields.identifier,
            "repository": .fields.repositoryPHID
        }'
}

get_diff_phid()
{
    jq -c '{"commit": .commit, "repository": .repository}' | \
        arc call-conduit ${CONDUIT_ARGS} diffusion.rawdiffquery | \
        jq -r .response.filePHID
}

save_diff()
{
    diff_phid="${1}"
    # Download the diff to disk
    echo "{\"phid\": \"${diff_phid}\"}" | \
        arc call-conduit ${CONDUIT_ARGS} \
            file.download | \
        jq -r .response | \
        b64decode -r > "${TMP_DIR}/${diff_phid}.diff"
    # Return path to diff in stdout
    echo "${TMP_DIR}/${diff_phid}.diff"
}

get_phid_url()
{
    jq -c '{"names": [.phid]}' | \
        arc call-conduit ${CONDUIT_ARGS} \
            phid.lookup | \
        jq -r '.response | to_entries | .[] | .value.uri'
}

accept_changes()
{
    jq -c '{
        "objectIdentifier": .phid,
        "transactions": [{"type": "accept", "value": true}]
    }' | \
        arc call-conduit ${CONDUIT_ARGS} \
            diffusion.commit.edit
}

for audit in $(get_audits | filter_commit_data); do
    audit_url="$(echo "${audit}" | get_phid_url)"
    printf "\n\nAudit URL: ${audit_url}\n"
    diff_phid="$(echo "${audit}" | get_diff_phid)"
    diff_file="$(save_diff "${diff_phid}")"
    # Open in editor for easy review
    ${EDITOR} "${diff_file}"
    # Ask for feedback as necessary
    echo -n "Do you accept this change? [Y/n] "
    read accept
    if [ -z "${accept}" ] || [ "${accept}" != "${accept#[Yy]}" ]; then
        # Close audit 
        echo "${audit}" | accept_changes
    fi
done

Conclusion

Now that I’m done designing a workflow that is reasonable for me, I better start doing some auditing /o\ and real-world testing of phabaudit.sh.

If this happens to be useful to you, let me know!

FreeBSD updating a port: Twisted python

2020-04-01T00:00:00Z

Introduction

FreeBSD is a wonderful Operating System, one of its many interesting ideas is a division between the base system and other pieces of software, which are called ports.

Since FreeBSD is an operating system, this also means that there are exactly two source trees or repositories: the main tree (base system) and the ports tree.

The base system is developed together and therefore is kept consistent, whenever an API or a behaviour changes, all pieces of the base system must be updated as well.

This makes changes somewhat seamless and easy to deal with.

On the other hand, ports provide a description of how to compile third-party software under FreeBSD and how to install it manually or how to build an installable binary package.

However, FreeBSD development, including maintaining the ports tree, is a voluntary-based effort; which means sometimes it lacks human power.

Being a good open source user sometimes goes through being aware that people’s time is limited, and helping out when things fall through the cracks.

Here I document a bit how the general process of updating a port you detect as being outdated is.

Communication and coordination

Contrary to “common wisdom” in the industry, it is not “the technical bits”, that is difficult when trying to help with a project; it is communication and coordination that are difficult things when many people are involved. And nothing big or useful enough gets built and maintained without people.

In a nutshell, when interacting with a project of this kind:

Be kind
Respect and appreciate other people’s time and energy
Don’t assume incompetence or unwillingness, when it can be explained by lack of resources (usually: time, human, money, infra)
Make sure you research before demanding time or effort from people

Interestingly, the FreeBSD project has the communication and coordination bits mostly quite well-defined.

Documentation

FreeBSD‘s documentation is really good, but there are multiple places to look for information, depending on the nature of the information.

FreeBSD Handbook: General system information and administration
Man pages: Information about specific bits of the system
FreeBSD wiki: Information that is prone to change often

Chat – IRC

Following list of channels on irc.freenode.net are a great place to start, there are usually people who will point you out to the right resource, the right person or the right place to deal with things.

Just make sure you are patient since not everyone is online always (timezones!) and express your question in as much of an accurate fashion as possible.

#freebsd: For general OS questions and support
#freebsd-ports: For questions more specific to maintenance of the ports tree
#freebsd-dev: For questions more specific to development of the OS

Bugzilla – Problem Reports (PRs)

FreeBSD’s Bugzilla is used to track problems with the base system or ports and also to document what is being worked on by whom.

It also serves a special purpose with the ports workflow: when a change is proposed to a specific port, if there is no feedback from the port maintainer within a reasonable time, that change times out and can be accepted and merged by anyone with permissions (commit bit) to the ports tree.

Phabricator – Code Reviews

Code reviews are wonderful. FreeBSD has a Phabricator instance that is used for precisely this purpose.

It requires getting used to it, but Phabricator is a very reasonable piece of software and the review workflow is incredibly useful.

Don’t be afraid, most people who might review your code will be kind and it means that the end result will be better.

Live example: Updating Twisted python

As of this writing Twisted python’s port is at version 18.9.0, because of the way Twisted manages versions, this means it dates back to September 2018.

In between, Twisted has gotten significantly better and has had some security fixes, as recently as a couple weeks ago: March 2020.

So we really should update the port to version 20.3.0.

Finding out more about the port

A good place to find out about ports is www.freshports.org.

If we go there and search for twisted, we’ll come to the port’s page.

Amongst other things, it tells us that the port is a build dependency for 4 other ports and a run dependency for 55 ports.

And it also tells us a very interesting bit, the full name of the port is: devel/py-twisted.

Each port has a main category, which is then followed by the port name, separated by a /.

Indeed, this is where we will find the code defining the FreeBSD port.

Telling people we’ll be updating the port

First things first, we search on FreeBSD’s Bugzilla for open Problem Reports (PRs) relating the port we want to update. Maybe someone is already working on it or there are subtleties that escape our uninitiated knowledge (like dependencies, external blockers, …).

In this case there are none, so we open a PR informing that we’ll be updating this port.

This is done on FreeBSD’s Bugzilla by filing a new bug against the Ports and packages “product”.

There is a very good write-up about this by koobs in the FreeBSD wiki.

Very important bits here are how to write the Summary field and how to manage the Flags of the PR.

In particular, this will be our Summary:

devel/py-twisted: WIP: Update to 20.3.0 (includes security updates)

Severity, hardware and OS are not that important here, so we leave the defaults.

Now, something will do magic and this PR will be assigned to the maintainer of the port, in this case that’s python@freebsd.org, which means there is no particular person, but a general Python team maintaining the port.

Let’s tell them about our intentions, this will be the body:

Hello, I noticed that FreeBSD's twisted version is a bit outdated, so I'll
try taking a go at it.

It is not entirely impossible that while at it I'll have to update other ports
like attrs, we'll see.

I'm also using the chance to document the whole process for myself (my future
self too) and hopefully people who would consider doing things like this.

Will be posting a patch over the next few days.

Cheers,

This way, if someone came wanting to do they same thing, they’d ask before if we are still working on it, and we won’t duplicate efforts.

Or maybe someone knows something we don’t, so they will use the chance to tell us in our PR #245252.

Getting the port tree

This can be done in multiple ways, the most convenient one is getting the code from the same repository that is used by developers over anonymous SVN:

svn checkout svn://svn.freebsd.org/ports ports

Another one is portsnap, which is in the base system, but instead of explaining that, I’ll go a bit into my favourite: poudriere.

Poudriere

Poudriere makes it simple to manage a pkg repository and to build customised packages, it also makes it simple to work on the ports tree.

There is a wonderful Poudriere guide on the FreeBSD wiki, so I’ll just do the TL;DR:

# Install poudriere from pre-built ports with pkg
> pkg install poudriere
# Create a builder jail for amd64 based on 12-1-RELEASE
> poudriere jail -c -j 121-amd64 -v 12.1-RELEASE
# Only if you have something under /usr/ports and are completely sure you never
# messed with it, clean that up with:
# rm -rf /usr/ports
# Create the ports tree
> poudriere ports -c -p default
# Or if you prefer to have it somewhere else:
# poudriere ports -c -p default -M /poudriere/ports/default

Testing that the port builds currently

Before making any modifications, let’s test that the port builds as it is, to discard that there is an issue somewhere else.

For that we can execute:

poudriere bulk -j 121-amd64 -p default devel/py-twisted

Now Poudriere will build the devel/py-twisted port and everything it depends on. The first time this will take a while, but afterwards only the ports that have changed will be rebuilt.

Actually updating the port

Since this is a Python port, the Python Ports Policy has precedence over the Porter’s handbook, both are worth a read.

We go to the place where the ports tree is located and navigate to the devel/py-twisted subdirectory.

All ports have at least following structure:

Makefile: Describes what is going to be built and how
distinfo: Contains the checksums of any needed external files (like source code!)
pkg-descr: Is what users see when they search for this port/package.

Since Twisted’s is a simple port, we just have to edit its Makefile:

Changing the version

The most obvious change is going to be… The version, that’s following diff:

 PORTNAME=  twisted
-PORTVERSION=   18.9.0
-PORTREVISION=  1
+PORTVERSION=   20.3.0
 CATEGORIES=    devel net python

We change the PORTVERSION and since there now is no PORTREVISION, because this is a new version, we remove that line.

This might be enough, but also, Twisted is not an island, it has dependencies, and they might have changed in between.

Dependencies

Port dependencies are also listed in the port’s Makefile, in particular in the BUILD_DEPENDS variable.

How we update this depends a bit on how comfortable we are with the project’s tooling.

A perfectly valid option could be just to eyeball the code from a web interface, since I contribute to Twisted when time allows, I have the code locally and can use following:

git diff twisted-18.9.0..twisted-20.3.0 -- src/twisted/python/_setup.py

Which tells me exactly how the dependencies have changed between these two versions. Resulting in following diff:

 BUILD_DEPENDS= ${PYTHON_PKGNAMEPREFIX}constantly>=15.1:devel/py-constantly@${PY_FLAVOR} \
-       ${PYTHON_PKGNAMEPREFIX}attrs>17.4.0:devel/py-attrs@${PY_FLAVOR} \
+       ${PYTHON_PKGNAMEPREFIX}attrs>19.2.0:devel/py-attrs@${PY_FLAVOR} \
        ${PYTHON_PKGNAMEPREFIX}hyperlink>=17.1.1:www/py-hyperlink@${PY_FLAVOR} \
        ${PYTHON_PKGNAMEPREFIX}incremental>=16.10.1:devel/py-incremental@${PY_FLAVOR} \
        ${PYTHON_PKGNAMEPREFIX}PyHamcrest>=1.9.0:textproc/py-pyhamcrest@${PY_FLAVOR} \
@@ -23,7 +22,7 @@
        ${PYTHON_PKGNAMEPREFIX}zope.interface>=4.4.0:devel/py-zope.interface@${PY_FLAVOR} \
        ${PYTHON_PKGNAMEPREFIX}Automat>=0.3.0:devel/py-Automat@${PY_FLAVOR}
 RUN_DEPENDS:=  ${BUILD_DEPENDS}
-TEST_DEPENDS=  ${PYTHON_PKGNAMEPREFIX}service_identity>0:security/py-service_identity@${PY_FLAVOR}
+TEST_DEPENDS=  ${PYTHON_PKGNAMEPREFIX}service_identity>=18.1.0:security/py-service_identity@${PY_FLAVOR}

 USES=      python tar:bzip2
 USE_PYTHON=    autoplist concurrent distutils

We might need to update devel/py-attrs and sercurity/py-service_identity as well. Time will tell.

Since the distfiles for the port have changed, because we now are building a different version, we also have to update the checksums file.

This is easily done by running:

make makesum

Which downloads all needed distfiles, calculates the checksums and updates the corresponding file.

Building the new version

Is the same as before our changes, Poudriere will notice that the port’s definition has changed and trigger a rebuild:

poudriere bulk -j 121-amd64 -p default devel/py-twisted

Turns out, devel/py-attrs is already at version 19.3.0, and security/py-service_identity at version 18.1.0 both of which satisfy Twisted’s requirements. So the port builds without issues \o/.

Time to party now? Well, not quite. We still have to make sure the port still works as intended.

Testing the port

We can test the port with make test, which runs upstream’s testing suite.

But also, if we are caring enough to update this port, odds are that we are using it in way or another, that’s the best kind of test.

In my case, I’ll also keep developing my custom software, which depends on Twisted, but using the new version.

Submitting the patch

After we have tested the newly built port, we should send the patch; this is done by adding an attachment to our PR with the diff (obtained by svn diff).

For a more complex patch than this one, we might want to use the Code Review flow as well as this one, and keep the patch in Bugzilla updated as it evolves.

This is because Bugzilla is where maintainer timeouts and triage happen and in general things are better tracked.

When we add the attachment, we’ll request maintainer feedback by setting the flag with the same name to ?, the patch keyword will be added automatically, and we’ll also remove WIP: from the PR’s title and add the security keyword.

To make things easier for maintainers / reviewers / ports security team, we’ll also add more information in the comments:

Changelog:
  https://github.com/twisted/twisted/blob/twisted-20.3.0/NEWS.rst

QA:

  * portlint: OK (looks fine.)
  * testport: OK (poudriere: 3.3.3, amd64)


Related Security issues:

  CVE-2020-10108
  CVE-2019-9512
  CVE-2019-9514
  CVE-2019-9515
  CVE-2019-12387
  CVE-2019-12855
  https://twistedmatrix.com/trac/ticket/9420

And now someone will hopefully check our PR soon and commit it.

Conclusions

Sometimes the quickest way to have something be done is doing it yourself :-D. This doesn’t even take that long, so get into the habit of considering updating ports and proposing patches yourself.

Also, these things have effects, e.g. I know for a fact that Matrix Synapse depends on Twisted! This was probably being a blocker :-p.

Switching Android phones -- Getting rid of Google (mostly)

2020-01-05T00:00:00Z

Introduction

After being fed up with mostly non-serviceable phones and my previous one suffering a bit of an accident; it was the perfect time to align consumer values with portable device because the Fairphone 3 started selling in late 2019.

This is about setting up an Android phone with a stock OS, but getting rid of as much of Google as possible, while still having the option to use the “necessary” regular things.

As a friend put it:

“I don’t believe in free. Not anymore.”

Introduction
Table of Contents
Hardware
Setting up the phone
Being a responsible consumer

Hardware

I went with the Fairphone 3, one of their catchy slogans kind of say it all:

“If your phone doesn’t care about the planet, it’s not that smart”

They care about the source of the materials and the working conditions of the supply chain; besides that, they actively encourage electronics recycling by taking your old phone against a voucher.

Besides the ethical bits, a few great points that matter-to-me (tm) as just a consumer:

It supports dual-SIM and micro-SD card simultaneously
It is highly serviceable, having a perfect iFixit score
It feels robust
It does its job

Setting up the phone

F-Droid – The Libre App store

Setting up F-Droid is about the very first thing I do on any android device.

F-Droid is an alternative “app store” for Android, its main features:

All apps in the main repository are Free and Open Source
You get warned if they contain “undesirable features” (ads, relying on closed-source servers, …)
You can host and setup your own repository.

Disabling (most) Google apps

Disable auto-update

The very first thing to do here, is to setup Google’s Play Store to not use automatic updates.

I discovered that even if you disable apps, Android will “helpfully” reinstall and enable them if there is an update available.

This is done by:

Opening the Play Store app
Clicking the “hamburger” menu (the three horizontal lines)
Picking “Settings”
Under “General” find “Automatic app updates”
Pick “Do not update apps automatically”

This is a necessary trade-off between: keeping apps from the Play Store up-to-date and having Gooogle sneak their apps back in. Which still does happen from time to time with, e.g. Chrome.

Disable and remove unnecessary Google apps

This is done in the phone’s settings:

Settings
Apps and notifications
Show all apps

Once we have the list of all apps, we have to iterate through those that are unwanted and perform following:

Touch the app on the list
Touch on “Deactivate”
Confirm that you want to Deactivate
When asked if you want to uninstall any updates to the app, say “Yes”. This has the decent side-effect that you get some space back.
Touch on “Force exit” and confirm
Go to “Storage”
Delete data and delete cache
Go back to the apps list

And this is the list of forced-upon-me apps for which I do this; since the Fairphone 3 does not contain crapware of their own, it’s just Google’s:

Digital Wellbeing
Drive
Duo
Fotos
Gboard (care! Install an alternative before!)
Gmail
Google
Google Play Films and Series
Calendar
Contacts
Maps
YouTube
YouTube Music

Installing alternative software

Some of the alternative software on the F-Droid repositories is actually significantly better than that of Google.

The best example of this would be NewPipe, a replacement for the YouTube app. Amongst the most appreciated features from people I’ve shown this to is the ability to play videos as a pop-up window or in the background (with the screen off).

List of great apps on F-Droid

A quick and dirty list of apps on F-Droid that are a must for me:

Alarmio: alarms, timers, …
andOTP (replaces Google Authenticator): second factor authentication
AntennaPod: podcasts
Simple Gallery (replaces Google Fotos): offline photo browser
Ghost Commander: file browser
K-9 Mail (replaces Gmail): E-mail client supporting encryption and most email providers. It looks like Gmail is going to stop being email, and they will stop supporting the standard protocols. So, you are forced to use Gmail if that’s where your mail is. Notice how “embrace extend extinguish” is not from your usual source.
Simple Calendar (replaces Google Calendar): offline or CalDAV-based calendars.
KISS Launcher (any app launcher): search-based app launcher. It’s simple, and zeroes the mental burden of finding the app you need.
Simple Contacts (replaces Google Contacts): offline contacts.
Nextcloud (Google Drive): cloud-based storage.
OpenKeychain: for encryption / signature checking while on the go.
PassAndroid: Manage tickets
Simple Keyboard (replaces Gboard): it lets you type without having to correct every single word.

Being a responsible consumer

The F-Droid project itself and several of these apps are long-standing Open Source projects, with ways to donate some money to the developers. Please do so at least for a few!

FreeBSD home router -- Legacy IPv4: NAT+DHCP

2020-01-02T00:00:00Z

Introduction

I’ve been writing about setting up a FreeBSD-based eXO router, first the upstream and PPPoE details were covered, then how to setup the gateway, access point and an IPv6-only network. This builds upon a system that can properly route IPv6 and IPv4 as well as has a means to connect other clients and possibly providing them with an IPv6 address and route.

The goal here is to setup the bits of the router needed for everyday usage.

This is part of a series on home routing with FreeBSD.

FreeBSD eXO router – Get to ping6

FreeBSD home router – Access Point and IPv6-only

FreeBSD home router – Legacy IPv4: NAT+DHCP (this one)

Introduction
Table of Contents
Home router
Local legacy IPv4 networking: NAT (half-rant)
Local legacy IPv4 networking: NAT (implementing)
- DHCP: providing the local IPv4 addresses
- NAT with pf: forwarding packets properly
Remaining work
- Internal network bits

Home router

As mentioned on the first post, this setup is based on an APU2d4, with three ethernet interfaces: igb0 for egress/wan, igb1 for local devices and, igb2 as a management interface.

The current state is that the machine can setup the connection to eXO and can correctly accept the IPv6 and IPv4 routes and delegations, furthermore it creates an access point wlan0 that is bridged to igb1 (bridge is called lair) and any devices that connect to these interfaces are able to use only the IPv6 internet.

Local legacy IPv4 networking: NAT (half-rant)

Network Address Translation, one of those things that are secretly quite complex and a bit of a PITA to debug.

The Internet was made to be End-To-End, to have all members of the network be equal in the sense that they can provide and consume services the exact same way [old knowledge; citation needed].

NAT gets in the way of that and it was a necessary evil. Virtually all ISPs that provide you with an internet connection, will also assign you a single IPv4 address (either dynamically or statically), but you surely have more than one device at any given home/company location.

Since, except in very specific cases, it’s not valid for multiple devices to work with the same IP, NAT shows up everywhere.

In a nutshell, NAT makes sure that your 192.168.X.Y IPv4 doesn’t leave the local network (because it’s “non-routable”, aka shouldn’t leave the premises) and then connects on your behalf to whatever other peers you requested.

Then, when the answer comes back, it’s “smart enough” to know that that reply has to go to your 192.168.X.Y instead of to some 192.168.W.Z in the network.

That’s trickier than it sounds and indeed means that, without some kind of permanent port-forwarding, it is not possible to reach devices behind the NAT for better and for worse.

Since RIPE ran out of IPv4 in 2019, this is specially important: now ISPs are starting to use Carrier-Grade NAT, some even “sell it” as a feature because it means they can give you a crappier internet at lower prices.

Crappier internet? Well, if you or someone you know/trust manage your NAT, it is possible to do things like port-forwarding, if in reality, your NAT is behind another NAT that hosts thousands of customers; there is no real option for that.

Random trivia: The Nintendo Switch will happily tell you how crappy your NAT is :-) on a scale from “A: mostly works” to “F: how did this happen?”. CGNAT will automatically place you somewhere between “C” and “D”.

Local legacy IPv4 networking: NAT (implementing)

So, eXO is in this regard only different, in that they assign a static IPv4 address to the connection that is directly addressable by any device connected to the IPv4 internet.

Which means, I do need NAT if I want IPv4 support.

Side note for some other time: NAT64 and DNS64 make it mostly unnecessary to have IPv4 (and NAT!) in the first place. This has been tested out successfully at length by many, but specially by the awesome people at ungleich.ch at last Hack4Glarus and with their IPv6 only hosting Virtual Private Server (VPS) offering.

DHCP: providing the local IPv4 addresses

Before actually translating addresses, we have to provide those addresses to devices in the network. That’s what DHCP is good for.

The along with the DHCP6 client (dhcp6c), DHCP server is the second bit that we need that is not part of the base system.

In this case, I’ll use OpenBSD‘s dhcpd, which is packaged as a port for FreeBSD.

As with dhcp6c, installing dhcpd is a matter of:

pkg install dhcpd

And the magical man pages are man 5 dhcpd.conf and man 8 dhcpd and, even if it is focused on isc-dhcpd-server, the Chapter on DHCP from the FreeBSD Handbook.

Familiar pattern by now, setup /usr/local/etc/dhcpd.conf (since dhcpd is not part of the base system, its config lives in /usr/local/etc) as follows:

# Add a suffix for local name queries
option domain-name "evilham.local";
# TODO: change to own DNS. Using quad9 for now
option domain-name-servers 9.9.9.9, 114.112.112.112;

# Default to a 1 day lease
default-lease-time 86400;
# Limit the lease to, say a month if someone
# requests a longer lease.
max-lease-time 2592000;

# 2^16 should be enough addresses for everyone (' ^.^)
option subnet-mask 255.255.0.0;

# This will have to be the internal IPv4 of the router
option routers 192.168.0.1;

subnet 192.168.0.0 netmask 255.255.0.0 {
  # I want to "reserve" some addresses for ${reasons}
  range 192.168.77.0 192.168.99.255;
}

And setup /etc/rc.conf:

# Modify this existing line to add the router's IPv4 and subnet
ifconfig_lair="inet 192.168.0.1/16 addm igb1 addm wlan0"

# Start giving out IPv4 addresses
dhcpd_enable="YES"
#   And explicitly do that only locally
dhcpd_flags="lair"

Then start the dhcpd service:

service dhcpd start

Now any device connected to the bridge interface will get an IPv4 address \o/.

BUT! We have no NAT yet, so, things will get weird soon! (*)

(*): they got weirder than expected, ppp was doing NAT?

That doesn’t sound right :-D. Let’s trust pf more than ppp for that job.

Modified /etc/ppp/ppp.conf so that the exo profile also contains a line: nat enable no This should have been the default if I understood documentation properly.

TODO: Figure out why this was the default behaviour and fix documentation/open a bug

Besides talking to our local network, these addresses are not good for anything. Since they are not routable, any routers must refuse to forward the packets any further.

NAT with pf: forwarding packets properly

FreeBSD has two main firewalls, I like pf. It originated in OpenBSD like so many great things, and was ported to FreeBSD.

Kristof Provost’s talk on automated firewall testing, explains a bit how it came to be that OpenBSD‘s pf and FreeBSD‘s pf are actually different. The TL;DR is: nobody has done the work to upgrade it, it works mostly fine, and the diff is not as small as one would expect.

OpenBSD‘s pf has a bunch of goodies, like support for NAT64 :-). Hopefully things catch up soon.

Back to topic: man 4 pf, man 5 pf.conf and man 8 pfctl are the magical man pages.

And a super-tiny but permissive example on how to do NAT was already done by Kamila here, so go steal that into /etc/pf.conf.

With an important addition for dhcp6c:

# TODO: update with provisioning example
# In practise, this should be limited to link local addresses
pass in quick proto udp to port dhcpv6-client

Now we have to apply those rules, since I was already loading pf, albeit with an empty set of rules, this means running:

service pf reload

And boom, now everything in the network can talk to the outside over IPv4.

Remaining work

Finish securing the firewall with pf
Create and publish cdist types for all the things

Internal network bits

Monitoring (Prometheus)
DNS (unbound)
Traffic dashboards (taking into account the percentiles)

FreeBSD home router -- Access Point and IPv6-only

2020-01-01T00:00:00Z

Introduction

Last time I wrote about setting up a FreeBSD-based eXO router, this builds upon a system that can correctly ping and ping6 to the internet.

The goal here is to setup the bits of the router needed for everyday usage.

This is part of a series on home routing with FreeBSD.

FreeBSD eXO router – Get to ping6

FreeBSD home router – Access Point and IPv6-only (this one)

FreeBSD home router – Legacy IPv4: NAT+DHCP

Introduction
Table of Contents
Home router
Remaining work
- Legacy IPv4
- Internal network bits

Home router

As mentioned on the previous post, this setup is based on an APU2d4, with three ethernet interfaces: igb0 for egress/wan, igb1 for local devices and, igb2 as a management interface.

The current state is that the machine can setup the connection to eXO and can correctly accept the IPv6 and IPv4 routes and delegations, and is able to ping and ping6 devices on the internet.

This is similar to what Kamila did with her router, but has a few twists and only focuses on the home router bits, leaving the installation and 3G and PPPoE bits somewhere else.

Create the bridge

Creating network bridges on FreeBSD is pretty straightforward with man 8 ifconfig and man 5 rc.conf being the magical man pages.

To do it once:

# Create the bridge with a deterministic name
ifconfig bridge create name lair
# Add the necessary interfaces
ifconfig lair addm igb1 addm wlan0 inet6 auto_linklocal

To do it permanently, add in /etc/rc.conf:

cloned_interfaces="bridge0"
# Have a deterministic name
ifconfig_bridge0_name="lair"
# Add the necessary interfaces
ifconfig_lair="addm igb1 addm wlan0"
# While at it, ensure we have a link local address
ifconfig_lair_ipv6="auto_linklocal"

Setup the wireless network

This is done with hostapd, which is also in the base system. The magical man pages for this are man 8 hostapd and man 5 hostapd.conf as well as man 5 rc.conf.

First of all, we setup things in /etc/rc.conf:

# Setup the wireless interface
wlans_ath0="wlan0"
ifconfig_wlan0="HOSTAP"
# Respect local regulations and all that -- ES is the country code
create_args_wlan0="wlanmode hostap country ES"
# Enable the access point daemon
hostapd_enable="YES"

Now, we create /etc/hostapd-wlan0.conf as follows:

interface=wlan0
debug=0
ctrl_interface=/var/run/hostapd
ctrl_interface_group=wheel
ssid=eXO-wlan
wpa=2
wpa_passphrase=WirelessPassword
wpa_key_mgmt=WPA-PSK
wpa_pairwise=CCMP

Become a gateway

Now most of the things are in place in order to forward traffic for other clients in the local network.

A very important step that is missing is becoming a gateway! Here also man 5 rc.conf is a magical man page.

Add following to /etc/rc.conf:

# Act as gateway
gateway_enable="YES"
ipv6_gateway_enable="YES"
# Setup dependent: Accept Route Advertisements even with
#   IPv6 forwarding enabled.
#   In my case, this is not needed as DHCP6 is used for prefix delegation.
#   The value should be the interface where your ISP will be doing RA.
#ipv6_cpe_wanif="exoppp"

This is effective after a reboot, using sysctl and so on is possible, but…

Provide IPv6 connectivity with SLAAC

This is also done with the base system, in this case man 5 rtadvd.conf, man 5 rc.conf and man 8 rtadvd are the magic man pages.

It further follows the pattern, we setup /etc/rc.conf:

rtadvd_enable="YES"
rtadvd_interfaces="lair"

And the software in /etc/rtadvd.conf:

# rtadvd will try to get the data from the kernel's routing table
# TODO: use own DNS servers, this uses quad9.
lair:\
    :mininterval#5:\
    :maxinterval#10:\
    :rdnss="2620:fe::fe,2620:fe::9"\
    :mtu#auto:\
    :rltime#60:

And then start it with

service rtadvd start

At this point we also modify /usr/local/etc/dhcp6c.conf to assign a delegated /64 to the lair interface as opposed to the igb1 one. See previous post for the base config.

Now, when another system connects to the lair bridge (that includes the access point on wlan0), it will automagically receive the /64 prefix information and derive its addresses as necessary (temporary, non-temporary, random, it’s all black magic).

BUT! Keep on reading, this is important!

Secure the firewall

Up until now only our router was accessible from the internet without a firewall, but it also had no services running (besides SSH with Public Key based authentication); with the previous step though, besides providing internet to the network, we provide our internal network to the internet :-).

Remember that up until now /etc/pf.conf was empty.

My firewall rules are non-trivial and keep evolving, so they won’t be replicated here, instead there will be some cdist type with them soon.

Needless to say, man pf.conf, man pf and man pfctl are the magical man pages and block all the things.

TODO: publish the cdist types and link them here

Remaining work

Legacy IPv4

Setup dhcpd to hand out local IPv4s
Setup NAT
Create cdist types for all the things

Internal network bits

Monitoring (Prometheus)
DNS (unbound)
Traffic dashboards (taking into account the percentiles)

FreeBSD eXO router -- Get to ping6

2019-12-31T00:00:00Z

Introduction

eXO – expansió Xarxa Oberta (open network expansion) is an non-profit association based in Barcelona and since 2019, it’s also a RIPE member.

eXO […] promotes open telecommunication networks, technological sovereignty, access to wholesale Internet services and reduces the digital gap.

They are also pretty much the only decent way to access native IPv6 in Barcelona and since RIPE ran out of IPv4 in 2019, it’s also pretty much the only decent Internet Service Provider around.

Also, the fact that you are not a customer but a member who can make things work out if necessary, kicks ass over <rant> Customer Service that keeps you waiting for hours and then insults your time and intelligence, or a service that forces you to use crappy routers, uses CGNAT or one that basically mobs you into joining them by aggressively calling you multiple times a week. (Who does these terrible things? Search for “biggest ISPs in Spain”) </rant>

Also: you get to pick your router and, besides something running OpenWRT (default), do it your self and e.g. run OpenBSD or FreeBSD.

This is roughly about how internet with eXO works and mainly about how to setup a FreeBSD system as a router for it.

PS: eXO does more than just fiber, there are also community-owned wireless mesh networks, but that’s for another day.

This is part of a series on home routing with FreeBSD.

FreeBSD eXO router – Get to ping6 (this one)

FreeBSD home router – Access Point and IPv6-only

FreeBSD home router – Legacy IPv4: NAT+DHCP

Introduction
Table of Contents
Hardware
Software and installation
- Hardware-specific bits
- External network bits
Remaining work
- Optional
- Internal network bits

Hardware

Fiber network

Since Barcelona is one of those regions, particularly cities, that missed the “let’s own our digital infrastructure” wagon, and it’s prohibitively expensive to install your own, the physical layer gets leased from a wholesale provider which, in turn, pays to the owner of the network (see “biggest ISPs in Spain”) for its use.

Depending on where in the metropolitan area, that provider changes, in the case of the city of Barcelona, it’s Sarenet.

Once Sarenet has done its job (which was extremely quick!), there is an optical fiber cable arriving at its future usage location, eXO gets notified and receives some details like the connection’s required VLAN and they also setup users and assign a fixed IP block from their pool to the connection.

Going Ethernet

Before we can use our regular router, an Optical Network Terminal is needed. In this case eXO offered me a Huawei EchoLife EG8010H which is kind of cute and does the job. Plus side: now I own the li’l box (no BS equipment rental or crappy hardware).

Router

I’m re-purposing electronics, whose intended initial use has developed beyond its capabilities.

In this case, that’s an APU2d4 from PCEngines, with a 1T mSATA drive for home storage and booting off a 32G SD card. Some more drives will be attached but are not relevant.

I decided to boot off a 32G SD card, because if anything happens, I can have it back up in no time. And it slightly simplifies the management of the data drive.

Software and installation

Since, because of the somewhat large mSATA, I’ll be using the APU as home storage as well, I will prefer FreeBSD over OpenBSD for this use-case, as the former will give me the wonders of ZFS, minus some of the easy routing awesomeness of the latter, while still being good at it.

Hardware-specific bits

Installing FreeBSD on a PCEngines APU2 is somewhat easy with the right equipment and a couple important tips.

Kamila covered that when she did her router, so do read that.

Once the OS is installed and before rebooting into the newly installed system, I prepare a few things for provisioning over local network:

Setup my root SSH key in /root/.ssh/authorized_keys
Ensure password logins are disabled in SSH
Enable root login over SSH
Add pf_enable="YES" to /etc/rc.conf and kldload pf
Create an empty /etc/pf.conf
Make sure things like sendmail and the like are fully disabled
I make sure I can use the router as a client over a local network with

# In /etc/rc.conf
ifconfig_igb2="DHCP"
ifconfig_igb2_ipv6="inet6 accept_rtadv"

External network bits

I’ve been switching to cdist for provisioning, so this might be easier for you once I finish publishing the types.

Since I’ll be doing the provisioning over local network on igb2 (mangement), igb0 is assumed to be the egress, wan interface and igb1 will be treated as a client-facing interface.

Wholesale provider requirements (PCP + VLAN)

The wholesale provider requires a specific IEEE 802.1p Priority Code Point and VLAN for the connection.

The magical man pages are man 8 ifconfig and man 4 vlan as well as the Chapter on VLANs from the FreeBSD Handbook.

This is achieved one time with:

# VLAN 24 -- Called exovlan because of ppp
# We also setup IEEE 802.1p priority code point
ifconfig igb0.24 create name exovlan vlan 24 vlanpcp 3

And permanently by adding following to /etc/rc.conf:

# Wholesale provider requirements
#   Create VLAN inteface
#   It doesn't look like ppp likes dots in interfaces, let's give this
#   interface a friendly name.
vlans_igb0="exovlan"
#   Setup the VLAN number
create_args_exovlan="vlan 24"
#   IEEE 802.1p priority code point
ifconfig_exovlan="vlanpcp 3"

PPPoE

This is what is actually used by eXO to provide the service.

The magical man pages are man 8 ppp and the Chapter on PPP from the FreeBSD Handbook.

And this is what the resulting /etc/ppp/ppp.conf looks like:

default:
  # An address is required
  set ifaddr 0 0 255.255.255.255

exo:
  # I'll be using my own DNS, otherwise use "enable dns"
  disable dns
  # Reminder that ppp does not like dots in interface names
  # so we created the VLAN 24 as exovlan and not igb0.24
  set device PPPoE:exovlan
  # Set a deterministic name
  iface name exoppp
  # Replace with your own data
  set authname ${EXO_USER}
  set authkey  ${EXO_PASS}
  set dial
  set login
  # Use LQR and echo
  set echoperiod 15
  set lqrperiod 15
  # mssfixup should be enabled by default
  enable echo lqr mssfixup
  set timeout 0
  # Pause of 5, increment +5, -5 times
  set redial 5+5-5 0
  # Wait random (1-30) to reconnect
  set reconnect random 0
  add! default HISADDR
  add! default HISADDR6

Now PPPoE can be used once with:

ppp -ddial exo

And permanently by adding following to /etc/rc.conf:

# PPPoE bits
ppp_enable="YES"
ppp_mode="ddial"
ppp_profile="exo"

Now pinging some IPv4 address should work :-).

IPv6 connectivity

IPv6 connectivity is slightly more difficult as eXO must delegate the /56 prefix to us.

In this case, I’ll want to quickly assign a /64 to the remaining ethernet interface, igb1.

Sadly dhclient in the base system does not include this feature, and that’s the kind of situation when ports are needed.

Good thing we now have some legacy (v4) network connectivity!

Let’s get the dhcp6 port:

# The first time pkg runs, it will bootstrap itself, since it's
# actually maintained as a port.
pkg install dhcp6

Since this does not belong to the base system, its settings are under /usr/local/etc and there is indeed a dhcp6c.conf.sample with nearly all we need.

By reviewing that, man 8 dhcp6c and specially man 5 dhcp6c.conf, following results:

# This is /usr/local/etc/dhcp6c.conf

interface exoppp {
    # This will solicit a non-temporary address on the wan interface
    send ia-na 0;
    # This will solicit a prefix on the exoppp interface
    send ia-pd 0;
};

# An id-assoc is always needed, even for NAs
id-assoc na 0 {
};

# This will use that delegated prefix to assign it the igb1 ethernet.
id-assoc pd 0 {
    prefix-interface igb1 {
        # This is how many bits we have "free"
        # Since IPv6 addresses are 128 bits and eXO delegates a /56,
        # we have 8 bits of freedom until a /64.
        # The remaining 64 bits, will be used by SLAAC locally
        sla-len 8;
        # This *decimal* will be turned binary to a sla-len bits
        # representation and be used along with the delegated prefix.
        sla-id 0;
    };
};

In order to use this, following must be added to /etc/rc.conf:

# Just enabling dhcp6c has issues when reconnecting
#dhcp6c_enable="YES"
dhcp6c_interfaces="exoppp"

And it can be tested out with

service dhcp6c start

Now ifconfig igb1 should have a globally routable unicast IPv6 in a /64 under your control and ping6 should work (real-time update: happy 2020!).

Update (2020-01-28): Just enabling dhcp6c has issues with IPv6 on reconnect after connection loss. Bright side of this is that we detected an anomaly on eXO‘s ppp server that will be solved in the upcoming planned maintenance.

In order to reconnect properly, let’s use ppp’s hook functionality.

On /etc/ppp/ppp.linkup:

MYADDR:
  # This should block as little as possible, we use !bg to keep going
  # "in the background" so ppp is not blocked.
  !bg /usr/sbin/service dhcp6c onerestart

On /etc/ppp/ppp.linkdown:

MYADDR:
  # dhcp6c can take a bit to close, if the link came back up
  # meanwhile things can be messy. We won't use !bg here.
  ! /usr/sbin/service dhcp6c onestop

Remaining work

Create a bridge interface for ethernet and wlan.
Setup the wireless network
Setup SLAAC
Secure the firewall
Create cdist types for all the things

Optional

Setup dhcpd to hand out local IPv4s
Setup NAT

Internal network bits

Monitoring (Prometheus)
DNS (unbound)
Traffic dashboards (taking into account the percentiles)

ThinkPad A485 and FreeBSD

2019-08-03T00:00:00Z

Introduction

I’ve always thought diversity is an important thing, which is why Linux’ hegemony on servers, as great as it is, can also be a threat for future network stability.

Most people having something to do with computers have probably come across the BSD term at one point or another, yet not everyone is aware of OpenBSD and FreeBSD being reasonable alternative Operating Systems with their own strengths.

Since my previous laptop was way too fragile to be a mobile computer any longer, I had to relegate it to a virtualisation server and start the hunt for a powerful, mobile and flexible laptop.

Easier said than done, after much researching I settled in May 2019 for a ThinkPad A485 as being the closest thing to a good trade-off for me. Amongst other things it features an AMD Ryzen™ processor and 32G RAM.

There is a great article covering support for this laptop on OpenBSD.

Since it’s not the case for FreeBSD and, at that point, support for this laptop was not ideal, this is intended as some kind of documentation that will hopefully help other people in the near future, that is in the transition to full support.

The wish to document this started when someone asked on the freebsd-current Mailing List about getting the A485 to work with FreeBSD.

Note: This can be copied / adapted to be on the FreeBSD wiki. Let me know if you adapt it. Sadly the username policies go against my strong preference for pseudonymous internet. Not that figuring out “real names” would be difficult for really interested parties, it’s just a matter of preference and online identity.

Introduction
Table of Contents
Installation
- 12-RELEASE
- 13-CURRENT
Operation
Graphics
- 12-RELEASE
- 13-CURRENT
Wireless
SD Card reader
Overview of relevant bugs and issues

Installation

12-RELEASE

Installing FreeBSD on an AMD Ryzen™ laptop is not as easy as it could be these days, support in 12-RELEASE is not quite there due to some peculiarities in ACPI, FreeBSD bug #231760 tracks that.

That bug report also lists the necessary workaround to manage to boot and install FreeBSD on the A485:

The hw.pci.mcfg kernel parameter must be set to 0, that’s done from the boot loader by typing set hw.pci.mcfg=0.

After installation, it is quite important to edit /boot/loader.conf and add a line with hw.pci.mcfg=0, that way the system will be able to boot without manual intervention.

13-CURRENT

For those willing and knowledgeable enough to compile and debug things, 13-CURRENT is where development happens, and diff D20327 just landed! This means I can stop maintaining my own branch :-).

As of this writing, the latest installation media dates August 1st 2019, which is about 48 hours too old, so the above kernel parameter is still needed to install FreeBSD on a new A485 for those tracking 13-CURRENT, but it can be removed from /boot/loader.conf after compiling and installing HEAD.

Whenever an image is built after August 3rd 2019, it should not need the kernel parameter, check here to see if it has happened.

Note: 13-CURRENT is not intended to be used if you can’t fix things yourself or are not willing to debug them. If, however, like me you are curious about the development process and want to be detecting issues before they hit 12-RELEASE, it can useful to use 13-CURRENT.

Operation

After installing FreeBSD on the A485, the system behaved well over all but I was experiencing some random kernel panics, which I could reliably trigger by stressing a WireGuard VPN connection; that sent me down the wrong path until Konstantin mentioned in bug #239351 that AMD is very secretive about processor features and erratas, which are sometimes fixed in BIOS, so I should make absolutely sure I had the latest BIOS.

In early June 2019, my A485 came with BIOS version 1.16, but in July 2019 versions 1.22 and 1.24 came out, apparently one of these versions fixed something in the way the processor is used, which totally fixed these random panics.

Graphics

X is the best terminal multiplexer around, luckily this is supported out of the box these days both in 12-RELEASE and 13-CURRENT.

Support for the graphics card is done through the drm-kmod port, which automagically installs and loads the appropriate drivers.

Without this, external screens or graphics acceleration cannot be used.

12-RELEASE

Under 12-RELEASE this worked perfectly fine with drm-kmod version 4.16.g20190722 until I upgraded the BIOS to 1.24. After this upgrade, the system started hanging after boot when trying to load the graphics driver.

On a message on the mailing list someone had suggested previously to test version 5.0 of drm-kmod, as the first wrong hint at tackling the random panics involved this port.

Getting 5.0 on 12-RELEASE involves building the port from source, which I didn’t try, but should be doable.

13-CURRENT

After the BIOS upgrade to 1.24, 13-CURRENT with drm-kmod also stopped booting, since this is a development release, it was slightly easier to fix than compiling a newer version of the port, indeed there is a drm-devel-kmod port, which has version 5.0.g20190722.

With this newer version of drm-kmod, FreeBSD boots into X without issues.

Wireless

The A485 comes with a Realtek RTL8822BE, which is not supported, luckily enough it can be swapped by an Intel Dual Band Wireless-AC 8265, which is.

Adding support for this wireless card is as simple as adding following to /boot/loader.conf:

# Wireless Intel AC8265
# Not strictly necessary, but the Realtek that is shipped is not
  supported
# You can easily (and carefully) change the wireless cards.
if_iwm_load="YES"
iwm8265fw_load="YES"

SD Card reader

The SD card reader is currently not supported, bug #204521 is tracking this and has a bounty of 175 USD (mostly from arrowd, but I added to the bounty).

Basically, driver rtsx should be ported from OpenBSD.

Overview of relevant bugs and issues

#231760 - FreeBSD halts at ACPI on AMD Ryzen laptops

Solved by D20327, and therefore in 13-CURRENT images built after August 3rd 2019.

Workaround for 12-RELEASE is booting with set hw.pci.mcfg=0 and setting hw.pci.mcfg=0 in /boot/loader.conf.

#239351 - PANIC spin lock held too long - A485 BIOS bug

Solved by upgrading BIOS to 1.24 or newer.

? - RTL8822BE is not supported

Can be swapped by an Intel AC8265, ideally there would be drivers for Realtek cards, but that’s harder to manage.

#204521 - Support for Realtek SD card reader

Bounty of 175 USD, rtsx needs porting from OpenBSD to FreeBSD.

AcCoreConsole: Simple plugin for workload simulation

2016-03-15T00:00:00Z

Introduction

Following up on my previous post, I’ll introduce a small AutoCAD .Net plugin that can run on the AcCoreConsole.

This could work as a hands-on, quick and dirty tutorial to writing .Net plugins for AutoCAD, I recommend reading the official documentation though. If you are already comfortable writing .Net plugins for AutoCAD, you can skip to the Customising execution (Command Line Arguments) section.

The idea of this plugin is to perform a computation that takes a while and exit; nothing fancy.

The reason for this, is to test my implementation of something similar to AutocadIO that runs locally and in a much smaller scale; the main motivation for that is that many companies are not (yet) ready to handle over their files to Autodesk and having the files processed in their amazing cloud.

At some point, my implementation should provide for an easy way to choose between using all local resources and sending jobs to AutocadIO; that would give maximum flexibility to both users and developers as well, all while giving end-users (companies) a peek into the potential of massively parallel running processes.

Introduction
Table of Contents
Previous setup
Project setup
The actual workload simulation plugin
Closing

Previous setup

The officially supported way to write .Net plugins for Autocad is using Visual Studio, something like Mono would probably work as well.

Basically, you need a copy of the ObjectARX SDK which is a set of dlls that define AutoCAD’s API.

There are two ways to get (and license!) the ObjectARX SDK:

Downloading it directly from the official website.
Using nuget, which sets up everything for you. This currently only supports AutoCAD 2016; the binaries will usually be compatible with AutoCAD 2014 and 2015 though.

I recommend using nuget since it’s the most straightforward way to set up the development environment.

If you do choose the other way, you will get a zip file with some samples, documentation and lots of other files. Most of these files are for ObjectARX developers, which is the underlying C++ API for AutoCAD.

The most important files in the zip are in the inc directory: AcCoreMgd.dll, AcDbMgd.dll, AcTcMgd.dll. You should make sure to add references to these dlls in your Visual Studio project.

Project setup

The Visual Studio project needs to have Class Library as its output type, this means that after compiling we will get a dll file containing the classes that make up the plugin.

This dll gets imported in runtime by AutoCAD or the AcCoreConsole, which will also call the relevant bits of your code.

Code

There are basically two classes that make a .Net plugin for AutoCAD: the Plugin class and the Commands class.

The `Plugin` class

This class is optional, the basic idea is that this class is instantiated once and should contain data that is static for the plugin.

The Plugin class makes more sense when running a command inside AutoCAD as opposed to the AcCoreConsole; the reason for that is that this class gets instantiated just once for the whole AutoCAD session, allowing you to share data across command executions in different documents and so on.

Since in AcCoreConsole you only open one document, I think its usefulness is rather limited.

A typical Plugin class would look like this:

using Autodesk.AutoCAD.Runtime;

[assembly: ExtensionApplication(typeof(MyPlugin.Plugin))]
namespace MyPlugin 
{
  public class Plugin : IExtensionApplication
  {
    public void Initialize()
    {
      // Initialize some resources to be shared across documents
      // and executions of any commands define by this plugin.
    }

    public void Terminate()
    {
      // Dispose of any resources that need to be manually released.
      // Remember that AutoCAD is already closing at this point!
    }
  }
}

The class name Plugin is totally arbitrary and could easily be something else.

The important part here is that the class implements the IExtensionApplication interface.

Signalising the ExtensionApplication attribute on a class is recommended, but not mandatory.

The `Commands` class

This doesn’t have to be a class per se but I do like to create a unique class that contains any custom defined commands.

My typical Commands class looks like this:

using Autodesk.AutoCAD.Runtime;
using Autodesk.AutoCAD.ApplicationServices.Core;

[assembly: CommandClass(typeof(MyPlugin.Commands))]
namespace MyPlugin
{
    public class Commands
    {
        [CommandMethod("MyCommand", "MyCommand", "MyCommand",
            CommandFlags.Modal & CommandFlags.Session)]
        public static void MyCommandMethod()
        {
          // Do something with the current drawing
        }
    }
}

In addition to the class name Commands, the method name MyCommandMethod is arbitrary. The defined commands don’t even have to be all in a single class.

To define a command, we just use the CommandMethod attribute on a static method in a class as shown.

As before, the CommandClass attribute is recommended but not mandatory

To my understanding, the Commands class never gets instantiated since AutoCAD and AcCoreConsole invoke the static method directly.

The class attributes

The main benefit of using the ExtensionApplication and CommandClass attributes is load time. That way AutoCAD (and AcCoreConsole) know exactly where the Plugin’s entry point is and where any commands are defined.

If you use class attributes once, you must use them all the time in the plugin, otherwise any commands defined in a class not signalised with the CommandClass attribute won’t be defined.

More information on the class and method attributes can be found in the official documentation, as well as in Kean‘s incredibly informative blog.

The actual workload simulation plugin

The workload

Because of my Math background, I love the Fibonacci sequence. Turns out, it’s also a very popular programming exercise that shows how badly recursive algorithms can make something easy complicated.

Which is why I picked this as a way to occupy the CPU in the plugin.

/// <summary>
/// Workload simulation. Naïve calculation of the n-th Fibonacci number.
/// A significant workload can be achieved with n >= 42.
/// </summary>
/// <param name="n"></param>
/// <returns></returns>
private static ulong Fib(uint n)
{
    if (n < 2)
        return n;
    return Fib(n-1) + Fib(n-2);
}

First Plugin version

Now, a very quick way to have a plugin that simulates workload would be to calculate Fib(60); that would look something like this:

using System;
using Autodesk.AutoCAD.Runtime;
using Autodesk.AutoCAD.ApplicationServices.Core;

[assembly: CLSCompliant(true)]
[assembly: CommandClass(typeof(TestAcCorePlugin.Commands))]
namespace TestAcCorePlugin
{
    public class Commands
    {
        [CommandMethod("TestAcCore", "TestAcCore", "TestAcCore",
            CommandFlags.Modal & CommandFlags.Session)]
        public static void TestAcCoreCommand()
        {
            Application.DocumentManager.MdiActiveDocument.Editor.WriteMessage(
        string.Format("\nFib {0}:\t{1}", 60, Fib(60)));
        }
        /// <summary>
        /// Workload simulation. Naïve calculation of the n-th Fibonacci number.
        /// A significant workload can be achieved with n >= 42.
        /// </summary>
        /// <param name="n"></param>
        /// <returns></returns>
        private static ulong Fib(uint n)
        {
            if (n < 2)
                return n;
            return Fib(n-1) + Fib(n-2);
        }
    }
}

But I want this plugin to be a bit more flexible than that and since it will be running in the AcCoreConsole…

Customising execution (Command Line Arguments)

To my knowledge, there is no defined way to pass information to AcCoreConsole plugins; a suggestion Kean made when we met last year in Prague was to pass arguments in the Command Line to the plugin.

This is possible because the AcCoreConsole ignores any arguments it doesn’t recognise. That leaves us with the issue of telling AcCoreConsole’s arguments and those for our plugin apart.

One way to do that is the good ol’ UNIX way of using two consecutive dashes: -- to signalise the beginning of our arguments.

That leads us to this method:

/// <summary>
/// Ignore any command line arguments before the last "--" word.
/// </summary>
/// <returns>A list of the gotten arguments.</returns>
private static List<string> ParseArgs()
{
    var args = new List<string>(Environment.GetCommandLineArgs());
    var parsedArgs = new List<string>();
    var lastIndex = args.FindLastIndex((x) => x == "--");
    if (lastIndex > 0 && lastIndex + 1 < args.Count)
    {
        parsedArgs.AddRange(
          args.GetRange(lastIndex + 1, args.Count - (lastIndex + 1)));
    }
    return parsedArgs;
}

This method reads the arguments from the Environment variable, finds the last appearance of -- and ignores everything behind that. That way we don’t have to worry about parsing and understanding AcCoreConsole’s arguments.

Furthermore, it provides us with a (probably) forward compatible way of passing arguments. If Autodesk decides to add more Command Line arguments to the AcCoreConsole in the future, that’s OK since we just ignore everything before the last --.

Now, to use the gotten arguments we have to modify the TestAcCoreCommand method:

public static void TestAcCoreCommand()
{
    var args = ParseArgs();
    var fibs = new List<uint>();
    if (args.Count > 0)
    {
        foreach (var arg in args)
            fibs.Add(uint.Parse(arg));
    }
    else {
        for (uint i = 0; i < 45; ++i)
            fibs.Add(i);
    }
    foreach (var i in fibs)
    {
        Application.DocumentManager.MdiActiveDocument.Editor.WriteMessage(
            string.Format("\nFib {0}:\t{1}", i, Fib(i)));
    }
}

In this case, we treat the arguments as a list of integers that tells us which members of the Fibonacci sequence are to be calculated.

When no argument is passed (or if the command runs inside AutoCAD), we calculate the first 45 Fibonacci numbers.

Using the plugin

After compiling, a TestAcCorePlugin.dll file will be generated. This file can be loaded with netload from both inside AutoCAD and the AcCoreConsole; that will define the testaccore command.

To automate loading and command calling, I created a Script file called ac.scr, with following content:

secureload 0
netload "${PathToDll}\TestAcCorePlugin.dll"
testaccore
secureload 1

The secureload part is because we are not (yet) signing this dll; which we should.

Now with an open Command Prompt, I can invoke my plugin with:

"${PathToAutoCAD}\accoreconsole.exe" /s "%{PathToScript}\ac.scr" -- 30 40 45

Which will produce following output:

AutoCAD Core Engine Console - Copyright Autodesk, Inc 2009-2013.
Regenerating model.

Command: netload Assembly file name: "${PathToDll}\TestAcCorePlugin.dll"
Command: testaccore

Fib 30: 832040
Fib 40: 102334155
Fib 45: 1134903170

Command: _quit

Great! We just used AcCoreConsole to calculate Fibonacci numbers.

Closing

The main point of this post is the usage of Command Line Arguments to customise the execution of a given script running in AcCoreConsole.

This allows for many interesting use-cases: we can, for example, run three different AcCoreConsoles, each with a different number as an argument; that would make the calculations in parallel as opposed to serially.

As a matter of fact, I am doing just that; next post will start introducing what I call OwnACCCL (OwnAcCoreConsoleLauncher). Which takes care of starting the AcCoreConsoles with the proper arguments and not exhausting the system’s resources while at it.

AutoCAD, .Net and Cloud technologies

2016-03-14T00:00:00Z

Introduction

It has been quite a while since I last updated this blog and in between a lot has changed personally, professionally and in technology. For instance: German is now the language I speak the most and it has taken its toll on my English, I apologise for that in advance.

Another thing that has changed is my preference of technology when it comes to AutoCAD automation: back when I wrote this post it was Lisp, nowadays I use mostly .Net plugins.

The downside [of Lisp, back in 2013 my go-to AutoCAD automation tech] is that it’s quite a mess to have a GUI for the scripts or routines you develop. I mean, sure, it can be done (there’s DCL after all) but it’s not as flexible or easy as one would hope. Luckily, for most use cases a simple custom toolbar and text input will make users happy.

Turns out, all those issues with GUI development along with some caveats with batch scripting (on hundreds and thousands of DWG files) made Lisp more of a burden than a blessing.

Also, AcCoreConsole happened and Autodesk has been investing in cloud technologies heavily, so AutocadIO and View and Data APIs will be coming strong.

Introduction
Table of Contents
Why .Net?
AcCoreConsole and the Cloud
Closing

Why .Net?

Performance and flexibility

Autodesk invested very heavily in creating a powerful, flexible, yet simple .Net API for AutoCAD. It allows developers to do nearly everything that is possible with ObjectARX (the low level C++ API) with a higher level language like C#, F# or VB.Net.

Better tools and support

If you want to write Lisp for Autocad, you will most likely use Notepad++ or the Visual Lisp IDE, which has probably not received significant update since before year 2000; it looks like it’s basically supported for backwards compatibility and to avoid annoying the remaining Lispers amongst us.

With .Net, you can use Visual Studio and its debugging features which are, in my opinion, far better.

It is also easier to structure, maintain and reuse code thanks to shared class libraries.

Purpose oriented routines

Most of my Lisp routines were very generic and defined a couple AutoCAD commands with the most common options. These routines could be quickly extended to do exactly what the user wanted with Lisp one-liners; my experience so far has been that end-users never do that and they end up forgetting commands they don’t use that often.

The reason for that is that most end-users have no experience with simple programming, left alone with Lisp; you can’t really blame them.

Which is why I now focus on providing users with a GUI application that allows them to perform a specific task and adjust the parameters for that in the process. This is much easier in .Net, specially when using WPF for the GUIs.

That being said, I still provide commands that allow programmatic usage of the routines; that means I sometimes use Lisp as some kind of meta-scripting to further automate .Net routines.

Packaging, distribution and deployment

Common issues with Lisp scripts were:

Users found routine installation too complicated.
It was difficult to provide users with updates.
As long as users were in our office, everything worked fine because of custom AutoCAD workspaces; remote or in-client-premises workers were sometimes not as lucky.

Some of these issues were addressed by Autodesk with the introduction of the Autoloader (see the white paper), but others (e.g. updates) persist and are quite difficult do solve with Lisp but not so in .Net.

AcCoreConsole and the Cloud

Back in 2013, Autodesk had already been separating AutoCAD’s GUI from the logic; one of the reasons for that, was to provide a Mac version of AutoCAD. As a result of that separation, the AcCoreConsole was born.

The AcCoreConsole is a headless, minimalistic version of AutoCAD, that can fully interact with a DWG database. For regular users, this means nothing; for developers however, this opened up a world of possibilities.

Suddenly, running a Lisp script in a few hundred files was as easy as preparing a small .bat file that called the AcCoreConsole with the right arguments and invoked the script. All with the performance gains of not having any GUI and not having to render each opened drawing — these performance gains are huge.

This, along with the rise of Amazon Web Services, gave some very clever people in Autodesk the base idea for AutocadIO and View and Data.

AutocadIO is basically a way of running scripts massively in the cloud and View and Data is a way of accessing or displaying drawings in a browser.

The potential of these APIs both in and outside of the engineering world is enormous and they will surely gain importance in the next couple years.

A very cool example prepared by Kean Walmsley is Jigsawify; information on some design decisions and technical details can be found on his blog.

Closing

This post serves as a bit of an update and introduction for the work I’ve been doing in the past few months, stay tuned.

AutoCAD: Blocks. Part I: Basics

2013-01-31T00:00:00Z

Introduction

If you’re a somewhat advanced AutoCAD user, you probably fully understand blocks and its usage. If that’s the case, you probably should skip to the next part of the series on blocks.

Basically, a block is a way of grouping objects (e.g. lines, texts, circles, other blocks, etc.), giving such group a name and it allows for easy manipulation of a big amount of objects.

On this part, we’ll see some advantages of using blocks and lastly but more importantly: block attributes.

Introduction
Table of Contents
Advantages
Block Attributes
Closing

Advantages

Use of blocks has several advantages, here are some I came up with:

Smaller file size

The block definition is saved just once and every graphic instance of a block (called block reference) consists only of basic properties like: block name, insertion point, scale (x,y,z) and rotation (plus, the general graphic object properties). That means that when a block is used several times, less information has to be stored in the file.

Instant update of symbols

Again, because the block definition is saved just once, if you need to modify a symbol you have used a hundred times, you need only to modify the block definition in order to reflect those changes consistently and instantly across the drawing.

Conceptual and graphic separation

If a drawing is complex and layers are not enough for conceptual separation, it may be useful to create blocks for things that represent different parts of the same object. Done correctly, it can avoid confusion and create a better workflow.

Block Attributes

What are attributes?

Attributes are a way of having some text associated with a block, the beautiful thing of attributes is that, unlike the other parts of a block (excluding dynamic blocks), they can change from one block reference to another.

These attributes may or may not be visible, constant, you can change their position, even change their aspect for each block reference, they are quite flexible.

Example

Consider a valve and its TAG number, it may be desirable to use a block for the symbol but texts are something static that looks exactly the same in all of the instances. One solution could be to use a different block for each tag, but that’d defeat the point of using blocks. Here is where attributes come handy, we just add one to the block definition, place it where we want it, insert it and change the text at will.

The catch

The flexibility of attributes comes with a cost, since you are able to relocate them, change their colour, text style, angle, alignment, etc. for each block reference, there’s a bit of a disconnection between the block definition and the block reference when it comes to attributes.

For instance, say you define a block valve without the tag attribute, make a couple insertions of the block and then realise you need that tag attribute. So you go to the block definition, add it and save the definition.

What happens to the existing block references? Absolutely nothing, they won’t have your tag attribute. Similarly, relocating an attribute or changing its aspect won’t affect the existing block references. Instead, all of the new ones will be exactly as the block definition.

There’s a way to force a global update of a particular attribute natively with AutoCAD, it’s under: Modify / Object / Attribute / Block Attribute Manager. The downside is that it will reset all of the properties for a given attribute, that is, if you relocated the attribute in a cluttered zone so that it’s visible, it’ll go back to the default position and overlap with something else, so be careful!

You could, of course, write your own Lisp routines to update just a given property like colour or layer.

Closing

In a few follow up posts, I’ll cover how to create and modify blocks, how to insert them from an existing file, how to access and modify its attributes, and a few other things, all with lisp scripting. Stay tuned.

AutoCAD automation

2012-06-27T00:00:00Z

Introduction

As you can check on my CV, I’ve been working as a draughtsman/IT guy/CAD manager for around five years now.

We spend most of our time “drawing” things with Autodesk’s AutoCAD. Be it a P&ID, an electric diagram, civil, structural or piping drafts, a 3D model, you name it!

Being a somewhat small company, in five years we’ve faced different challenges, like getting projects that would require more human hours than we have the means to cover — either by lack of personnel or deadlines.

Sometimes we’ve solved those challenges by getting more people, sometimes by working more hours than usual. Sometimes though those things are just not possible — something else is needed.

Introduction
Table of Contents
Automation in AutoCAD
- Small sample
Conclusion
Further reading

Automation in AutoCAD

That something else, would be automation in AutoCAD. That is, using some amount of programming to make tasks easier, faster and less monotonous.

There are plenty of APIs for AutoCAD, the eldest (and my favourite!) being Lisp (more accurately AutoLisp/Visual Lisp).

Other options include VBA, .NET and ObjectARX. VBA for AutoCAD is deprecated though, so that leaves us with either .NET and ObjectARX as alternatives.

I mentioned that my favourite option is Lisp and that it’s the oldest; the thing is that ever since the Visual Lisp extensions, it’s incredibly easy to program in a somewhat object-oriented way and to manipulate graphic objects without having to worry too much about AutoCAD’s inner functioning.

The downside is that it’s quite a mess to have a GUI for the scripts or routines you develop. I mean, sure, it can be done (there’s DCL after all) but it’s not as flexible or easy as one would hope. Luckily, for most use cases a simple custom toolbar and text input will make users happy.

If you really need GUIs or deep control of what’s going on under the hood, I’d strongly recommend learning about the ObjectARX API. It’s well-documented and it’s incredibly powerful — you have to program in C++ though.

Small sample

Here are some bits from what I usually have on my acaddoc.lsp file. That is, the customisations that are loaded with each drawing.

Most are really silly things that happen to save little time per single use but one gets to use quite a lot depending on the kind of work currently at hand.

You may download this file here.

; small-sample.lsp: Is a small sample of things that can be done with
; Visual Lisp. It includes part of Evilham's acaddoc.lsp.
;
; LICENSE:
; Copyright (c) 2012, Evilham (https://evilham.com) 
; BSD 2-Clause copyright and disclaimer apply.
; See: http://www.opensource.org/licenses/bsd-license.php


; This loads the Visual Lisp extensions for AutoLisp
(vl-load-com)

(defun c:``()
; Toggle isometric / orthogonal axis (called snap in AutoCAD)
  (setvar 'SNAPSTYL (* (- (getvar 'SNAPSTYL) 1) -1)))

(defun c:<<(/ z)
; Print the Z coordinate of a given point:
  (setq z (nth 2 (getpoint "\nPoint to get z from: ")))
  (princ (strcat "\nz: " (rtos z 2 2)))
  (princ))

(defun ss->vla-objectlist(ss / lst i lim)
; Convert a selection set to a list of vla-objects
  (setq i -1 lim (sslength ss))
  (while (< (setq i (1+ i)) lim)
    (setq lst (cons (vlax-ename->vla-object (ssname ss i)) lst)))
  (reverse lst))

(defun txtnum(ss prefix suffix nstart nstep ndecimals / i n obj)
; Takes a selection set / vla-object list, two strings (prefix/suffix)
;   and three numbers nstart, nstep, ndecimals.
; It puts in the text/mtext objects' TextString property the following:
;   (strcat prefix NUM suffix)
;   where NUM is nstart + POS_IN_QUEUE * nstep. With ndecimals decimals.

; If ss is a selection set, convert it to a list.
  (if (= (type ss) 'PICKSET)
    (setq ss (ss->vla-objectlist ss)))
; If ss is not a list at this point. Or nstart, nstep, ndecimals are not all
; numbers, set ss to nil. That skips the code and avoids errors.
  (if (not (and (listp ss) (numberp nstart) (numberp nstep)
                (numberp ndecimals) (setq ndecimals (fix ndecimals))))
    (setq ss nil))
  (setq i 0)
  (foreach obj ss
    (if
      (not (vl-catch-all-error-p 
        (vl-catch-all-apply
          'vlax-put-property
          (list obj
                'TextString
                (strcat prefix
                        (rtos (+ nstart (* i nstep)) 2 ndecimals)
                        suffix)))))
      (setq i (1+ i))))
  i)
(defun c:txtnum(/ ss prefix suffix)
; Asks for a prefix and a suffix.
; Calls txtnum with nstart=1 nstep=1 ndecimals=0
  (setq ss (ssget)
    prefix (getstring "Prefix: ")
    suffix (getstring "Suffix: ")
  )
  (txtnum ss prefix suffix 1 1 0)
  (princ))

(defun r:gen(ang / ss po i lim)
; Rotates all selected items by ang 
  (if (and (setq ss (ss->vla-objectlist (ssget)))
           (setq po
                 (vlax-make-variant
                   (vlax-safearray-fill
                     (vlax-make-safearray
                       vlax-vbDouble
                       '(0 . 2))
                     (getpoint "Base point: "))))
           (numberp ang))
      (foreach obj ss
        (vl-catch-all-apply
          'vlax-invoke-method
          (list
            obj
            'Rotate
            po
            ang)))))
(defun deg->rad(ang)
; Conversion from degrees to radians.
  (* (/ ang 180.0) pi))
; The following commands rotate by 90, 180 and 270 degrees respectively:
(defun c:r1() (r:gen (deg->rad 90)) (princ))
(defun c:r2() (r:gen (deg->rad 180)) (princ))
(defun c:r3() (r:gen (deg->rad 270)) (princ))
(princ)

Some of these commands look really silly, like r1, r2 and r3, however once you get used to them it’s much faster to just type r1, select the objects, and click on the base point than having to also type in the degrees or activating ORTHO and clicking on the good direction (also minding the fact that OSNAP may get you a different point and hence a wrong angle).

In others, like txtnum, you’ll notice how a much more generic function is also defined. That’s meant to allow for quick in-place modifications to satisfy slightly different needs.

For example, if you wanted the text “X meters” with X starting at 0 and increasing by 0.5 each time, you could just type:

(txtnum (ssget) "" " meters" 0 0.5 1)

And if it happens to be something you need a lot in a session, just assign that to a temporary alias like this:

(defun c:<1() (txtnum (ssget) "" " meters" 0 0.5 1) (princ))

And if you really use it a lot, just add that line to your acaddoc.lsp and now you have yet another silly command to make your life easier.

Conclusion

There’s much power in AutoCAD’s APIs, basically the limit between what you can and cannot do is in your imagination (albeit it’s cliché, it’s also mostly true).

Any seemingly simple task that is bothering you because it’s too boring (read, easy and mechanical) can most likely be automated in plenty of different ways! Heck, even pretty complex things can be automated (up to a certain point) given enough analysis and understanding of the needs.

If you find yourself in such a case, do drop me a line! If you provide me with a good description of what you have to do (hopefully with a file example; example, I won’t do your job for you!), how you think you can automate it (I reserve the right to do it in a totally different way ;)), and I happen to: find it interesting enough, have the time and disposition to do what you’re asking for; I may as well just do it!

How I got into programming

2012-05-28T00:00:00Z

Introduction

At first, I wanted to start blogging about things related to my job but then I realised how introducing the topic out of the blue would have required explaining How I got into programming and how I use it at work.

So, here is my attempt at doing just that.

I just want to add, that I program for fun and because it makes my life easier. Quite often someone else’s as well. That does mean that I program at work, but as a mean to help getting my job (or someone else’s) done faster and better, i.e. it’s not my main function — at least it isn’t as of this writing.

Introduction
Table of Contents
Early years
- Logo parenthesis
Maturing and serious programming
At work
Summing up

Early years

I belong to the new generations —which means that my early years are actually kind of the new days— hence I have no cool stories about computers that needed a whole room in an University, had no experience with keypunch cards; heck, I hadn’t even been conceived by the time the internet started to get mainstream!

That means that my very first experience with a computer… Was actually an IBM PC! Yep, a somewhat small thing with 64MB RAM and a 15” screen that was able to handle a 800x600 pixel resolution (and 16bit colour depth!).

As the curious kid I was, the first thing I did when we got that thing (running Windows 95!) was to… try and find out what to do with it. Yeah sure, it turned on, you could move the mouse around, make a few drawings on Paint or whatever but… these things had to be able to do something else.

It wasn’t long after its arrival that I learned by heart every single configuration that was available and got bored of the computer. Luckily, a time after that I received a copy of some version of Logo.

Now the fun started! Why draw geometric figures that won’t be precise with Paint, when I had the opportunity of drawing anything I could imagine just by telling that agreeable turtle how to move?

Maybe it was a hidden desire to rule the world and have everyone do as I said but it was a damn good feeling to see that turtle do as commanded. Even when it made something unexpected because, hey, I had told it to do that even if it wasn’t what I wanted.

In other words, it wasn’t the turtle that was an idiot, the problem was the 7 or 8 years old creature between the chair and the keyboard — and at the time I knew it as well.

Later on, I discovered that it wasn’t only that turtle’s nature, it was computer’s nature to do exactly as they’re told (even if you tell them wrong!); the little dictator inside of me was very pleased.

Logo parenthesis

While writing this, nostalgia kicked in and I found a version of logo writer. Here’s a little something I did in the time needed to remember syntax and basic commands:

And here’s the code for that:

cg pu setpos [-200 -95] pd setc 8
repeat 125 [repeat 6 [left 120 repeat 7 [fd 20 right 60]] \
pu fd 60 left 60 fd 40 right 120 fd 20 left 60 pd] pu home setc 1 pd

As a side note, this turtle’s space is topologically equivalent to a donut!

Maturing and serious programming

While starting high school, I felt I had tons of time in my hands and learned how to program in Pascal and Basic (because it’s what I had at hand!), a couple years later discovered Visual Basic and learned to love it, hate it and eventually forgot it — I thought, erroneously, that I’d never need it again.

After that, I went to the University (not this time, the first time) and learned about the existence of Linux. Free Software philosophy, bash, Python amongst other things followed that discovery. I almost immediately fell in love with the Operative System —specially when one of the best professors I’ve ever had introduced me to Debian—. Have used nearly nothing else for my daily life since then.

When I got back to study at another University, notions of scientific programming turned out to be part of the Mathematics curriculum which got me to learn how to program in C.

Currently, I’m at yet another University (this time just an abroad semester) where I get to learn how to program re-usable, generic and efficient scientific software in C++ and some Java which I had already learned for personal purposes.

At work

I’m lazy… yep, not a good thing to say (specially if the section is called “At work“) but there’s more to it than just the definition of lazy. The thing is: I strongly dislike doing things inefficiently.

The way I see it: the faster I finish doing the things I don’t enjoy as much, the faster I get to do something more interesting — or even nothing at all!

That would be optimal in an objective-driven environment, which sadly is not the case at my current job, therefore I’m just… well, more productive.

So, how do I get to be more productive? I just assume that for every single repetitive thing that has to be done than once manually, someone will probably make a mistake at some point — most likely more than once and each time in different sneaky ways.

If the cost in time of automating the task (and then finishing the task in this more automatic way) is roughly equal to the time it takes someone to do the task by hand (including an error check). I just go for it and automate it… Not only I get to have some fun, but I get to be sure there are no errors (or know the kind of errors there may be) and the best thing: it may even be useful later and save us a few hours!

At first, of course, I had no idea how to do such things, nor was I able to estimate the time cost of a task accurately, the latter ability came with time and experience, the former was product of my already mentioned insatiable curiosity and a few hours of research plus tons of hours of practise — most of those hours were my own personal time or dead time at a company.

Summing up

I had the privilege of being in touch with computers from an early age and always had an interest for programming, being something that somehow suits my personality and way of doing things.

My studies have brought me somewhat closer to the programming world, and by chance my job allows me to do some programming somewhat regularly… Because, who on their right mind would argue against doing something in a day rather than three or four?

Migrating Windows (or any partition) to a Virtual Machine

2012-05-09T00:00:00Z

Introduction

At the company I work for, being sick of making clean Windows installs, we decided to willingly violate Windows XP’s EULA for the greater good and put together a few open-source tools (basically ntfsclone, ntfsreloc, ntfsresize, gparted and of course, Linux), wrote a couple of witty scripts and came out with a “free” and nearly legal way of (re)installing Windows on our machines.

Such a method consists basically of having in each machine besides the live Windows installation, a striped out Linux system with a backup image of its Windows (legally registered!). Of course, to save time, we sometimes use that same image to install Windows in more than one than one machine and once it’s been installed, we change the license data and create a new internal backup image with its own license info.

So, I have that set up on my machine as well, except that I have a full, lovely, amazingly useful Debian installation. The problem is, that I can’t be bothered to close, abandon whatever I’m doing and leave my happy place just to boot 5 minutes into Windows, figure out how to do something or test a new script and boot back into Linux to resume my other activities.

Here comes Virtualisation to the rescue, being something I had played with in the past, it wasn’t totally new and I already knew about the different options out there and their pros and cons.

So I decided to give it a shot, but once again, making a fresh Windows install with all of the software needed to make it useful… is just too much of a burden.

Introduction
Table of Contents
Issue
Solution
Conclusion

Issue

Summing up, I’ve got a Linux installation (which will be the Host OS) and a gzipped ntfsclone Windows XP image (which will, of course, be the Guest OS).

That being, nearly the equivalent of having a live Windows installation, and wanting to migrate it to a Virtual Machine.

There are already several articles about how to do that, but none of the ones I found solved a very simple issue:

You can’t just get the image of a disk partition and boot it!

As a matter of fact, the lovely guys at VirtualBox already tell you that exactly:

Either pull the drive from the windows machine or copy the data with a low level image tool (like dd) to a USB drive or other removable media. If making an image, DO NOT image just the partition, this will not work!

What they seem to be suggesting here, is that you make the image of the full hard drive (as opposed of just one partition), just to get the piece of it you want. That may mean having a 240 Gb of raw data, instead of just the 25 Gb you’re interested in… That’s nearly 90% inefficiency.

Solution

Analysis

To get around this, we need to understand why there is this problem to begin with.

You’ll see, it’s not as simple as that virtualisation software not wanting to boot your image because it is a faulty, buggy or incomplete program (as many would dare to suggest without digging in any further!); the problem here, is how hard drives work, how they were designed and how a real computer understands them.

There are many good articles around, and it’s not my intention at all to duplicate that information, so I’m just going to make a quick introduction to the topics, in order for the solution to make a bit of sense.

Hard drives

So, nowadays, everyone knows that a hard drive can have several partitions that look like “different hard drives” inside of the Operative System, but the information about the different partitions, their format, size, position in the physical hard drive, etc. has to be stored somewhere.

Where? Well, most likely you’ve heard about the infamous Partition Table or Master Boot Record (MBR), and I say infamous because probably the one time you heard about those things, you had to curse a lot due to data loss or all those wasted hours.

There we have it, there’s a mystical thing at the very beginning of our hard drive describing where our partitions are and how they are!

Then, when we try to get our Virtual Machine to boot that partition image we made, it’ll complain about it not being a properly formatted disk or something amongst those lines. Of course, there’s no MBR!

Master Boot Record

It’s pretty well explained on Wikipedia, but it’s full of historical data and things that, whilst being interesting, are not related to our goal. I’d recommend then, taking a look at these articles.

In short, the MBR is a 512 byte long section with a standard structure:

Code area (0x000—0x1BD)
Partition Table (0x1BE—0x1FD)
Boot Record Signature (0x1FE-0x1FF).

Those hex numbers in parentheses, correspond to the offset within the MBR in which the sections are located, which is also the absolute offset.

We’ll see later that we only have to worry about the Partition Table, so let’s take a look into it.

Partition Table

The Partition Table is really where the information about our disk partitions is written, it has enough room to define four partitions called primary partitions, one of those can be an extended partition, which will contain another partition table with information about all the logical partitions, but we don’t really care much about it right now, if interested read the articles above or make a quick internet search.

This sector then, has also a standard structure:

Entry for Primary Partition #1 (0x1BE—0x1CD).
Entry for Primary Partition #2 (0x1CE—0x1DD).
Entry for Primary Partition #3 (0x1DE—0x1ED).
Entry for Primary Partition #4 (0x1EE—0x1FD).

That means, that whatever defines the first primary partition, is between sectors 0x1BE and 0x1CD of the MBR.

Those entries have, of course, a structure that is better explained here, but here it goes for completeness’ sake:

Partition State: 0x80 if it’s the boot partition, 0x00 otherwise (1 byte).
Starting sector CHS coordinates (3 bytes).
Partition Type (1 byte).
Ending sector CHS coordinates (3 bytes).
Starting sector LBA coordinates (4 bytes).
Partition length in sectors (4 bytes).

What are those CHS, LBA things you ask. Well, in þe old times, it was actually needed to refer to a disk sector by its CHS coordinates (Cylinder, Head, Sector) which is hardware-dependant. However, nowadays software cares more about LBA (Logical Block Addressing) because it’s easier and the abstraction layers do the hard part.

Also, as Dan Strick said on the FreeBSD mail list (and I believe it just because my BIOS agrees):

Modern BIOS geometry most frequently uses 255 heads and 63 sectors/track because that maximizes the addressable part of the disk drive using the basic int13 function.

Real case MBR

Cool, we now know that there are three things we need, and roughly how they are, but it was all too abstract, so, as an instructive exercise, why don’t you go to your terminal and execute

$ dd if=/dev/sda count=1 | hd | less

Note that you may get into permission errors, just turn root, use sudo or get privileges to the disk group or whatever helps you get raw access to the disk. Also note that if you mess up the if= and write of= instead, you may be killing your MBR :), read man dd for more info.

So, mine looks a bit like this (I skipped a part as it’s mostly incomprehensible):

00000000  eb 63 90 d0 bc 00 7c 8e  c0 8e d8 be 00 7c bf 00  |.c....|......|..|
00000010  06 b9 00 02 fc f3 a4 50  68 1c 06 cb fb b9 04 00  |.......Ph.......|
00000020  bd be 07 80 7e 00 00 7c  0b 0f 85 0e 01 83 c5 10  |....~..|........|
00000170  be 95 7d e8 34 00 be 9a  7d e8 2e 00 cd 18 eb fe  |..}.4...}.......|
00000180  47 52 55 42 20 00 47 65  6f 6d 00 48 61 72 64 20  |GRUB .Geom.Hard |
00000190  44 69 73 6b 00 52 65 61  64 00 20 45 72 72 6f 72  |Disk.Read. Error|
000001a0  0d 0a 00 bb 01 00 b4 0e  cd 10 ac 3c 00 75 f4 c3  |...........<.u..|
000001b0  00 00 00 00 00 00 00 00  f7 a4 85 a3 2f d2 80 20  |.........J..... |
000001c0  21 00 17 fe ff ff 00 08  00 00 00 00 80 02 00 fe  |!...............|
000001d0  ff ff 83 fe ff ff 73 0a  80 02 92 69 04 00 00 fe  |......s....i....|
000001e0  ff ff 17 fe ff ff 05 74  84 02 c1 3e 00 00 00 fe  |.......t...>....|
000001f0  ff ff 05 fe ff ff fe bf  84 02 02 c8 1c 10 55 aa  |..............U.|

The interesting part is at offset 0x1b0, which is the row in which the partition table starts, notice the section at 0x1B8, where we see f7 a4 85 a3 2f d2, that’d be this disk’s identifier (I must confess I don’t know if, or how this is important) and right after that, starting at 0x1BE, we find the start of the partition table.

If we try the same thing (hexdump the first 512 bytes) on our image (again, some bits have been skipped):

00000000  eb 52 90 4e 54 46 53 20  20 20 20 00 02 08 00 00  |.R.NTFS    .....|
00000010  00 00 00 00 00 f8 00 00  3f 00 ff 00 00 08 00 00  |........?.......|
00000020  00 00 00 00 80 00 80 00  f8 ff 7f 02 00 00 00 00  |................|
00000030  00 00 0c 00 00 00 00 00  8e f0 1b 00 00 00 00 00  |................|
00000180  eb f2 c3 0d 0a 45 72 72  6f 72 20 64 65 20 6c 65  |.....Error de le|
00000190  63 74 75 72 61 20 64 65  20 64 69 73 63 6f 00 0d  |ctura de disco..|
000001a0  0a 46 61 6c 74 61 20 4e  54 4c 44 52 00 0d 0a 4e  |.Falta NTLDR...N|
000001b0  54 4c 44 52 20 63 6f 6d  70 72 69 6d 69 64 6f 00  |TLDR comprimido.|
000001c0  0d 0a 50 72 65 73 69 6f  6e 65 20 43 74 72 6c 2b  |..Presione Ctrl+|
000001d0  41 6c 74 2b 53 75 70 72  20 70 61 72 61 20 72 65  |Alt+Supr para re|
000001e0  69 6e 69 63 69 61 72 0d  0a 00 00 00 00 00 00 00  |iniciar.........|
000001f0  00 00 00 00 00 00 00 00  83 9f ad c0 00 00 55 aa  |..............U.|

Which doesn’t look like a partition table… Now, that’d explain why our virtualisation software refuses to boot it!

Generating the MBR

As mentioned before, we only have to worry about the partition table, but we do need a valid code area; luckily, there is already some software available to do it for us. That’d be ms-sys, which, by the way is not packaged by Debian due to license issues (citation needed; read it long ago, can’t be arsed to look for it now) but it’s just a matter of downloading the source code and compiling.

ms-sys has several options, the one I’m interested in is -m. Now, turns out, ms-sys needs a file to write the data, so let’s create a zeroed one.

$ dd if=/dev/zero of=mymbr count=2048
$ ./ms-sys -f -m mymbr

Notice the -f argument, if it weren’t there, ms-sys would complain about the file not being a disk device, but it’s ok, we know (or hope we know) what we’re doing. Also, we created a 1MB zeroed file (count=2048 in dd), that’s because it’ll be the start of our image, and leaving 1MB at the beginning seems to be a sane thing to do (e.g. gparted does it that way).

Having done that, we get:

00000180  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001b0  00 00 00 00 00 2c 44 63  00 00 00 00 00 00 00 00  |.....,Dc........|
000001c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

That is, no disk identifier, no partition table, but filled in code area and Boot Record Signature; not bad.

We just have to fill in the missing data with a hex editor (e.g. ghex).

As for the disk identifier, I guess we can fill in those bytes (0x1B8—0x1BD) pseudo-randomly, but have really no clue if that’s what dedicated software do :).

So, as for the Partition Table, we just need to fill in the first entry (0x1BE—0x1CD), as a single partition is all we need.

Partition State: 0x80 it’ll be the boot partition.
Starting sector: 0x002021 the partition will start at 1MB, see details for CHS encoding here.
Partition Type: 0x07 for NTFS, use fdisk and then l for more options.
Ending sector: odds are, your partition is big enough as to not fit in these three bytes, in those cases 0xFEFFFF is what should be there.
Starting sector: 0x00080000 that’d be 2048 as stored on a little-endian computer, it’s 2048 because our partition will start after 1MB (2048 sectors of 512 bytes).
Partition length: this will vary depending on your partition size, first find out the size of your partition in bytes and then divide it by 512, or better yet, find the size of your partition in sectors! Hint: man fdisk, man ls. This field must be encoded as stored on a little-endian computer as well.

After all that mess, my partition entry looks like this:

000001b0  00 00 00 00 00 2c 44 63  f2 aa cd f3 12 83 80 00  |.....,Dc........|
000001c0  20 21 07 fe ff ff 00 08  00 00 d8 0b 54 02 00 00  | !..........T...|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

Joining the new MBR and the image

So, we make a copy of the mymbr file as myimg.hdd for example. Remember that we created this file as a zeroed 1MB file and then edited the first 512 bytes corresponding to the MBR.

And then, we create, extract, convert or whatever the image, and add it to that file using dd, in my case:

$ gunzip -c Image.gz | ntfsclone -r -O - - | dd of=myimg.hdd bs=1048576 seek=1

The gunzip and ntfsclone parts are just because that’s how I have the image already, in practise you just need to pass the raw data to dd via standard input or create the image directly by indicating the input file (see man dd). Notice the arguments bs=1048576 seek=1 for dd, here, we’re telling dd to start writing on myimg.hdd after 1MB, leaving our 512 byte long MBR plus some zero data intact; by the way, by increasing the block size, the whole process is considerably faster.

Also, do note that we can overwrite the MBR of this newly created virtual hard drive at any time:

$ dd if=mymbr of=myimg.hdd count=1 conv=notrunc

Note that if you leave out the conv=notrunc parameter, you’ll lose all your precious data.

Use the image

Now just use whatever virtualisation solution you want with this raw image or maybe even convert it to another format (like vdi), it’s a perfectly valid virtual hard drive with a single partition.

Conclusion

It does look like quite a mess, but creating a virtual hard drive with a bootable partition from an image or a real hard drive just takes around two minutes, that is, not taking into account the time-consuming dd step, but then again, there we just have to sit back and relax.

Just as a side note, this does not guarantee by any means a migration from a live installation to a Virtual Machine, it does guarantee however that the virtualisation software will try to boot that partition.

As a matter of fact, my migration was “unsuccessful” at first as I got in VirtualBox the dreadful BSOD, but I could boot using kvm — albeit, the guest was pretty slow.

It was caused by the IDE/ATA drivers. So I booted with kvm and used the MergeIDE solution mentioned here. After doing that, I’m able to boot into that image from VirtualBox and it actually runs fast in kvm.

Evilham's Blog

FreeBSD adopting a port: maomao

Introduction

Table of Contents

FreeBSD and the ports tree

Wayland on FreeBSD?

Maomao

Getting started with ports

Setting up poudriere(8)

Getting the ports tree

Creating the port

Building the port

Testing the port

Updating the port

Creating the patch

Submitting the patch

Conclusions

Trying out Kubernetes: etcd

Introduction

Table of contents

VM Resources

Installing

Setting up the cluster

DNS discovery

Provisioning parenthesis

Testing the cluster

Basic functioning

Killing cluster members

Conclusion

Trying out Kubernetes: Network and VMs

Introduction

Table of contents

Network

IP assignment

Virtual Machines

vm-bhyve switches

Template

OS: Debian

Creating the base VM

Cloning the VMs

Conclusion

Trying out Kubernetes: planning

Introduction

Table of contents

Goals

Quick resources

The components

Control plane

etcd

Node components

Addons

Necessary resources

Network

Virtual Machines

Conclusion

YubiKey: PGP and SSH authentication

Introduction

Table of contents

PGP, subkeys and SSH

Common state

What we achieve here

(Roughly) why we do it

PGP key generation

Software requisites

Setup the directory

Configuring gpghome

Generating the key

Generating the key

Putting the Key into the YubiKey

Copy the public key

Using the YubiKey

SSH authentication

Managing expiry dates and the [C]ertification key

Conclusion

ZFS replication tools

Introduction

Table of contents

Quick reminder of ZFS

Snapshot creation and prunning with zfs-periodic(8)

Adding quarter_hourly

Setting up `poudriere(8)`

`vm-bhyve` switches

`etcd`

Configuring `gpghome`

Managing expiry dates and the `[C]`ertification key

Snapshot creation and prunning with `zfs-periodic(8)`

Adding `quarter_hourly`

ZFS Replication with `zxfer(8)`

Snapshot creation and pruning with `sanoid`

Migrating `zfs-periodic(8)` snapshots to sanoid

snapshot creation and pruning with `sanoid`

ZFS Replication with `syncoid`

Remembering `zrepl`

`zfs_autobackup`

`zrep`

`help(yield)`?

Què és `help`?

I per què no funciona `help(yield)`?

La manera correcta: `help("yield")`

Ús normal de `yield` en Python

Per què cal el `yield` amb Scrapy/Twisted?

IPv6: `accept_rtadv` and `auto_linklocal`