Ceph - Upgrade feature levels

Upgrading a distributed system like a ceph cluster implies upgrading all daemons and clients. If all systems are on a common level, you may safely upgrade the release level.

Determining current ceph feature level is straight forward.

ceph osd get-require-min-compat-client

This will tell you your current minimum client level. In our case it's jewel (still).

# ceph osd get-require-min-compat-client
jewel

Let's get this updated.

Upgrade the OSD min-compat value

Whenever the feature level is upgraded, this is done by setting the minimum required version that may connect to ceph. This ensures all components are compatible when working together.

In our actual use case we wanted to enable the ceph balancer in upmap mode. This is supported starting with luminous.

# ceph balancer mode upmap 
Error EPERM: min_compat_client "jewel" < "luminous", which is required for pg-upmap. Try "ceph osd set-require-min-compat-client luminous" before enabling this mode

Unfortunately, trying to upgrade the compat level failed.

# ceph osd set-require-min-compat-client luminous
Error EPERM: cannot set require_min_compat_client to luminous: 3 connected client(s) look like jewel (missing 0x800000000000000); add --yes-i-really-mean-it to do it anyway

As we don't want to enforce the setting (which would block the affected client when connecting next time and introduce a delayed impact of this setting), let's get into diagnosis and find out what clients ceph is talking about.

Detecting client versions

The first step is to check out the versions of all ceph components using ceph versions.

# ceph versions
{
    "mon": {
        "ceph version 15.2.6 (cb8c61a60551b72614257d632a574d420064c17a) octopus (stable)": 3
    },
    "mgr": {
        "ceph version 15.2.6 (cb8c61a60551b72614257d632a574d420064c17a) octopus (stable)": 2
    },
    "osd": {
        "ceph version 15.2.6 (cb8c61a60551b72614257d632a574d420064c17a) octopus (stable)": 3
    },
    "mds": {
        "ceph version 15.2.6 (cb8c61a60551b72614257d632a574d420064c17a) octopus (stable)": 2
    },
    "overall": {
        "ceph version 15.2.6 (cb8c61a60551b72614257d632a574d420064c17a) octopus (stable)": 10
    }
}

In our case we're running octopus release and all components are up to date.

Let's check the ceph features right now.

# ceph features 
{
    "mon": [
        {
            "features": "0x3f01cfb8ffadffff",
            "release": "luminous",
            "num": 3
        }
    ],
    "mds": [
        {
            "features": "0x3f01cfb8ffadffff",
            "release": "luminous",
            "num": 3
        }
    ],
    "osd": [
        {
            "features": "0x3f01cfb8ffadffff",
            "release": "luminous",
            "num": 3
        }
    ],
    "client": [
        {
            "features": "0x27018fb86aa42ada",
            "release": "jewel",
            "num": 3
        },
        {
            "features": "0x3f01cfb8ffadffff",
            "release": "luminous",
            "num": 4
        }
    ],
    "mgr": [
        {
            "features": "0x3f01cfb8ffadffff",
            "release": "luminous",
            "num": 2
        }
    ]
}

You may have spotted that we've clients connected with feature level from the jewel release.

Lookup clients using sessions command

As we're talking about connected clients, our next step is to check out the sessions of our monitors. We're using the sessions command on the monitor instances for this purpose: ceph tell mon.0 sessions

# ceph tell mon.1 sessions 
[
    {
        "name": "mon.2",
        [...]
        "remote_host": ""
    },
    {
        "name": "client.54440",
        "entity_name": "client.user",
        "addrs": {
            "addrvec": [
                {
                    "type": "v1",
                    "addr": "10.56.34.18:0",
                    "nonce": 494494558
                }
            ]
        },
        "socket_addr": {
            "type": "v1",
            "addr": "10.56.34.18:0",
            "nonce": 494494558
        },
        "con_type": "client",
        "con_features": 2810685664681798362,
        "con_features_hex": "27018fb86aa42ada",
        "con_features_release": "jewel",
        "open": true,
        "caps": {
            "text": "allow r"
        },
        "authenticated": true,
        "osd_epoch": 0,
        "remote_host": ""
    },
    {
        "name": "client.144495",
        [...]
        "remote_host": ""
    }
]

Quite luckily - we've found clients that are connected to mon.1 that are running the jewel release.

In our case the clients are running on debian 10 which comes with an older version of ceph. Solution here is to switch the driver and use ceph-fuse (which might be slower - in our use case we prefer the current build).

After upgrading our clients, we can just continue with feature upgrade.

Performing feature upgrade

Running the actual upgrade is really straight forward - we just tell ceph to forbid clients below a certain feature set (version).

# ceph osd set-require-min-compat-client luminous
set require_min_compat_client to luminous

Checking out the value can be done using the get-require-min-compat-client value.

# ceph osd get-require-min-compat-client
luminous

I'm quite happy with this.