RHEL 7 Interesting CGroup behaviour.

The other day I was looking at some Systemd service properties with the systemctl show command. One of them, CPUShares, got my interest.

Caution: The following tutorial shouldn’t be run on a production server! The CPU will be used at 100%!

To better understand how all this works, I created a basic Systemd unit file called /etc/systemd/system/testSpeed.service:

[Unit]
Description=Test Speed
After=syslog.target

[Service]
ExecStart=/usr/bin/openssl speed 
ExecStop=/bin/kill -WINCH ${MAINPID}

[Install]
WantedBy=multi-user.target

Then, I created another copy of this file and updated the Systemd configuration:

# cd /etc/systemd/system; cp testSpeed.service testSpeed2.service
# systemctl daemon-reload

I started both new services on a fresh standard install of Centos 7 on a VM with 1 vCPU:

# systemctl start testSpeed && systemctl start testSpeed2

Each of the two new services were using almost 50% of the CPU:

  PID USER      PR  NI S %CPU %MEM     TIME+ COMMAND      
24598 root      20   0 R 49.8  0.3   0:08.42 openssl      
24601 root      20   0 R 49.8  0.3   0:08.40 openssl      

I checked the default CPUShares of the new services:

# systemctl show testSpeed | grep CPUShares
CPUShares=1024

At this point, I asked myself: Is RHEL 7 really different from RHEL 6 and all the Unix systems in general? Is RHEL 7 limiting any new service by default? After some time, I came to the conclusion that everything was exactly as before: there is no CPU limit (or any other limit) on your service by default.

Because I wanted to learn how CGroups were working, I decided to apply a CPU constraint:

# systemctl set-property testSpeed CPUShares=512
# systemctl daemon-reload

Note1: The daemon-reload option is only there to avoid a warning message but doesn’t change anything.
Note2: You don’t need to restart any service.

Now, the testSpeed service gets 33.3% of the CPU and the testSpeed2 gets 66.7%:

  PID USER      PR  NI S %CPU %MEM     TIME+ COMMAND      
24601 root      20   0 R 66.7  0.3   1:23.58 openssl      
24598 root      20   0 R 33.3  0.3   1:22.20 openssl      

What exactly happened?

By default, Systemd services are part of the system.slice hierarchy. When you set a CPU constraint on one service, you not only assign a CPU resource controller to this service but also to all the services in the system.slice hierarchy. It is not possible to apply a CPU constraint on one service without affecting all the others.

The CPUShares concept represents more a CPU priority than a limit.

Concerning the CPU usage, this is the result of a computation:

• testSpeed gets 33.3% because 512/(512+1024)=1/3
• testSpeed2 gets 66.7% because 1024/(512+1024)=2/3

It is better to know this behaviour because otherwise you would perhaps be very surprised at the worst moment!

I would like to thank Radoslaw Kujawa for helping me understand what I was suspecting.

Look at my CGroups page to get some other tips on this topic.