Since Systemd v213 and consequently with RHEL 7.2, a new resource control option called CPUQuota is now available.

Last year I wrote a post on the CPUShares option.

I decided to revisit it with this new CPUQuota option.

Caution: The following tutorial shouldn’t be run on a production server! The CPU will be used at 100%!

As in the previous post, I created a basic Systemd unit file called /etc/systemd/system/testSpeed.service:

[Unit]
Description=Test Speed
After=syslog.target

[Service]
ExecStart=/usr/bin/openssl speed 
ExecStop=/bin/kill -WINCH ${MAINPID}

[Install]
WantedBy=multi-user.target

Then, I created another copy of this file and updated the Systemd configuration:

# cd /etc/systemd/system; cp testSpeed.service testSpeed2.service
# systemctl daemon-reload

I started both new services on a fresh standard install of Centos 7.2 on a VM with 1 vCPU:

# systemctl start testSpeed testSpeed2

Each of the two new services were using almost 50% of the CPU time (excerpt of the top command execution):

  PID USER      PR  NI S %CPU %MEM     TIME+ COMMAND      
24598 root      20   0 R 49.8  0.3   0:08.42 openssl      
24601 root      20   0 R 49.8  0.3   0:08.40 openssl      

I checked the default CPUQuota property of the testSpeed service:

# systemctl show testSpeed | grep CPUQuota
CPUQuotaPerSecUSec=infinity

Note: Don’t ask me why the option is called CPUQuota and the property CPUQuotaPerSecUSec, I don’t know!

Because I wanted to learn how CGroups were working, I decided to apply a CPU constraint:

# systemctl set-property testSpeed CPUQuota=10%

Note: You don’t need to restart any service. The % character is not optional.

Now, the testSpeed service gets 10% of the CPU time and the testSpeed2 gets 90%:

  PID USER      PR  NI S %CPU %MEM     TIME+ COMMAND      
24601 root      20   0 R 90.0  0.3   1:23.58 openssl      
24598 root      20   0 R 10.0  0.3   1:22.20 openssl      

The CPUQuota property of the testSpeed & testSpeed2 services is now as follows:

# systemctl show testSpeed | grep CPUQuota
CPUQuotaPerSecUSec=100ms
DropInPaths=/etc/systemd/system/testSpeed.service.d/50-CPUQuota.conf
# systemctl show testSpeed2 | grep CPUQuota
CPUQuotaPerSecUSec=infinity

What exactly happened?

With the CPUShares option, you assigned a percentage of CPU time to a service. With the CPUQuota option, you now set a duration in millisecond. This duration is the maximum of CPU time allowed to a service per second. This service can get an amount of CPU time below but not above this limit.

The behaviour of the CPUQuota option is much easier to understand than the CPUShares‘.

Look at my CGroups page to get some other tips on this topic.

Some additional information is available in the systemd.resource-control man page.

One of the main changes that occurred in the RHCE 7 exam was the new objective regarding MariaDB. You can be a very experimented system administrator and never deals with databases. Not only you are asked to install the database system but you are also supposed to know how to create a database, create tables and manipulate SQL instructions.

SQL syntax is not so easy to learn. Even though nobody expects you to be a SQL master, correctly applying a create table statement requires some time.

Hopefully, MariaDB provides a nice solution to this problem through the help command. When you don’t remember a syntax detail, the help command should be a reflex.

MariaDB [db]> help update
Name: 'UPDATE'
Description:
Syntax:
Single-table syntax:

UPDATE [LOW_PRIORITY] [IGNORE] table_reference
SET col_name1={expr1|DEFAULT} [, col_name2={expr2|DEFAULT}] ...
[WHERE where_condition]
[ORDER BY ...]
[LIMIT row_count]

Multiple-table syntax:

UPDATE [LOW_PRIORITY] [IGNORE] table_references
SET col_name1={expr1|DEFAULT} [, col_name2={expr2|DEFAULT}] ...
[WHERE where_condition]

Sometimes like with the grant command, you even get some examples!

As usual, don’t rely too much on this feature, otherwise you will waste too much time and won’t be able to complete your exam in time.

Thanks to Sam and krz for sharing their knowledge.

Most system administrators know the tail command. By default, it displays the 10 last lines of a file.

The -f option is less known. Besides displaying the last 10 lines of a file, it mainly provides a way to watch what is written into a file in real-time:

# tail -f /var/log/messages
Apr  4 20:30:01 vm systemd: Stopping user-0.slice.
Apr  4 20:31:35 vm systemd: Created slice user-0.slice.
Apr  4 20:31:35 vm systemd: Starting user-0.slice.
Apr  4 20:31:35 vm systemd: Started Session 961 of user root.
Apr  4 20:31:35 vm systemd-logind: New session 961 of user root.
Apr  4 20:31:35 vm systemd: Starting Session 961 of user root.
Apr  4 20:31:35 vm dbus[590]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)
Apr  4 20:31:35 vm dbus-daemon: dbus[590]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)
Apr  4 20:31:35 vm dbus[590]: [system] Successfully activated service 'org.freedesktop.problems'
Apr  4 20:31:35 vm dbus-daemon: dbus[590]: [system] Successfully activated service 'org.freedesktop.problems'
Apr  4 20:33:53 vm systemd: Started Session 962 of user root.
Apr  4 20:33:53 vm systemd-logind: New session 962 of user root.
Apr  4 20:33:53 vm systemd: Starting Session 962 of user root.
...

When debugging complicated situations involving several files, the tail command proves to be a invaluable tool. In fact, you can specify several files after the -f option and a header will be displayed each time something is written into a different file:

# tail -f /var/log/messages /var/log/secure
==> /var/log/messages <== 
Apr 4 20:51:07 vm systemd-logind: Removed session 963. 

==> /var/log/secure <== 
Apr 4 20:51:11 vm sshd[21933]: Accepted password for root from 94.23.44.29 port 47411 ssh2 

==> /var/log/messages <== 
Apr 4 20:51:11 vm systemd: Started Session 966 of user root. 
Apr 4 20:51:11 vm systemd-logind: New session 966 of user root. 

==> /var/log/secure <== 
Apr 4 20:51:11 vm sshd[21933]: pam_unix(sshd:session): session opened for user root by (uid=0)

==> /var/log/messages <==
Apr 4 20:51:11 vm systemd: Starting Session 966 of user root.

...

For example, OpenStack engineers use this kind of command to debug their configuration on compute nodes:

# tail -f /var/log/{ceilometer,neutron,nova,openvswitch}/*.log /var/log/syslog

When troubleshooting, using the tail command should become a reflex!

When preparing an exam like the RHCSA or the RHCE exams, it is relatively difficult to remember the commands with all their options, the configuration procedures with all their details, and finally the mistakes to avoid. This is even worse if you have little or no real experience. In this case, you have to make up for your lack of experience by being clever.

One of the main tips is to know how to find the relevant information in the current system itself through man pages, default configuration files with theirs comments (Apache, Samba, etc), and configuration examples coming with some packages (Postfix, MariaDB, etc).

Recently one reader, Carlos G Mendioroz, left a very good comment about teaming configuration. If you don’t remember some details, several configurations are available in the /usr/share/doc/teamd-*/example_configs directory.

For example, to get an example of a round robin configuration, type:

# more /usr/share/doc/teamd-1.9/example_configs/roundrobin.conf
{
"device": "team0",
"runner": {"name": "roundrobin"},
"ports": {"eth1": {}, "eth2": {}}
}

It is clear that this doesn’t explain everything nor replace practice. If you spend your time searching for man pages information or searching for example configuration, time will run out and you won’t be able to complete the different tasks in time.

When using the new RHEL 7.2 release and running the systemctl status command, some slight changes were recently visible:

# systemctl status sshd
● sshd.service - OpenSSH server daemon
Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2016-02-13 10:56:40 CET; 6s ago
Docs: man:sshd(8)
man:sshd_config(5)
Main PID: 1286 (sshd)
CGroup: /system.slice/sshd.service
└─1286 /usr/sbin/sshd -D

Feb 13 10:56:40 server.example.com systemd[1]: Started OpenSSH server daemon.
Feb 13 10:56:40 server.example.com systemd[1]: Starting OpenSSH server daem....
Feb 13 10:56:40 server.example.com sshd[1286]: Server listening on 0.0.0.0 ....
Feb 13 10:56:40 server.example.com sshd[1286]: Server listening on :: port 22.
Feb 13 10:56:44 server.example.com sshd[2457]: Accepted password for root f...2
Hint: Some lines were ellipsized, use -l to show in full.

As you can see above, a new string vendor preset: enabled is now displayed. Here it means that the preset mechanism has been applied on the sshd package.

The preset mechanism came with Systemd and the RHEL 7.0 release, the RHEL 7.2 release just makes it visible. It is a way for a Linux distribution (Debian, CentOS, etc) to centrally define whether a package should be automatically enabled or not at installation time.

So, through the well known directory hierarchy (/usr/lib/systemd/system-preset, /etc/systemd/system-preset, /run/systemd/system-preset), a Linux distribution can specify how packages initially behave.

A typical preset file would appear like this:

# more /usr/lib/systemd/system-preset/90-default.preset
# System stuff
enable sshd.service
enable atd.*
enable crond.*
enable chronyd.service
enable rpcbind.*
enable NetworkManager.service
enable NetworkManager-dispatcher.service
enable ModemManager.service
enable auditd.service
enable restorecond.service
enable bluetooth.*
enable avahi-daemon.*
enable cups.*

# The various syslog implementations
enable rsyslog.*
enable syslog-ng.*
enable sysklogd.*

# Network facing
enable firewalld.service
enable libvirtd.service
enable xinetd.service
enable ladvd.service
...

A directive enable or disable precedes each package in this file as appropriate.

You shouldn’t need to modify this file in most cases but it is still interesting to know that it exists.

Sources: systemd.preset man page and Freedesktop.org website.

Among the changes coming with the RHEL 7.2 release, there is the new systemctl edit command.

Like the systemctl cat command, you specify a service unit file as argument. But, unlike the systemctl cat command, you will need to think twice before using it in a production environment. All successful editing converts into a service restart!

So, without any other argument than the unit file, you create or edit a drop-in snippet called override.conf that will complement the current service configuration. If necessary the associated service directory is created (for example /etc/systemd/system/httpd.service.d in the httpd case) which is pretty handy.

You need to set up one of the following environment variables before: SYSTEMD_EDITOR, EDITOR or VISUAL. If you don’t do it, the nano, vim, and vi editors will be tried one after the other.

If you add the –full option, it is now the service unit file that you are editing and not a drop-in snippet anymore.

Finally, you can use the –runtime option, if you want to make some changes to a service unit file in the /usr/lib/systemd/system directory and don’t want the change to survive a reboot (this doesn’t work if the unit file is located in the /etc/systemd/system directory because of precedence between /etc and /run).

This command is definitively a useful addition for speeding up the tuning of service unit files in a development environment.

In a previous post, I presented the various mechanisms used for customizing Systemd unit files.
These mechanisms are so rich that it is not easy to get the real picture of the final unit file.
You have to search in different locations (/usr/lib/systemd/system, /etc/systemd/system, /etc/systemd/system/myservice.service.d/).

Coming with the RHEL 7.2 release, the new systemctl cat command serves two main purposes:

• to allow an administrator to quickly display the content of a given unit file,
• to take into account drop-in files when debugging unit files.

For example, let’s install the mariadb-server package:

# yum install -y mariadb-server

Now, let’s create the /etc/systemd/system/mariadb.service.d directory:

# mkdir /etc/systemd/system/mariadb.service.d
# cd /etc/systemd/system/mariadb.service.d

Then, let’s create the foo.conf file and paste the following lines into it:

[Service]
LimitNOFILE=10000

Finally, let’s display the mariadb unit file:

# systemctl cat mariadb.service
# /usr/lib/systemd/system/mariadb.service <== unit file location
[Unit]
Description=MariaDB database server
After=syslog.target
After=network.target

[Service]
Type=simple
User=mysql
Group=mysql

ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n
ExecStart=/usr/bin/mysqld_safe --basedir=/usr
ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID

# Give a reasonable amount of time for the server to start up/shut down
TimeoutSec=300

# Place temp files in a secure directory, not /tmp
PrivateTmp=true

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/mariadb.service.d/foo.conf <== drop-in file
[Service]
LimitNOFILE=10000

Now, you can see the unit file location displayed at the top and the content of the drop-in file at the end.

The systemctl cat command should become a reflex for all the system administrators when they want to know the precise configuration of a given service.

Note: Most of the original comments have been removed from the mariadb.service file here for brevity’s sake. I invit you to look at it in details because it provides a lot of useful comments, reminding you of the available Systemd customization mechanisms.

From my understanding, Systemd was created to make the maintenance of Linux distributions easier.

With previous init systems, you had to know all packages that could interact with a given subsystem, and you had to know them for each distribution for the same subsystem.

For subsystems as complicated as NFS, it was certainly a real pain.

At the same time, Systemd came with added features solving several problems of previous init systems:

• dynamic dependencies management,
• parallel execution of initial processes making boot faster,
• xinetd replacement through socket-activation,
• respawn of services as needed,
• declarative configuration files instead of complicated scripts,
• powerful customization of service configuration files,
• etc.

Concerning Unix philosophy, gurus could discuss “Do only one thing but do it well” principle during hours against Systemd integration model, this would not change anything.

So, all was pretty good except one point: to know how to administer a complicated subsystem, you now had to look at the /usr/lib/systemd/system directory, grep on keywords and guess which unit files needed to be started and enabled at boot! This could become very tricky.

Some problems started with the iSCSI subsystem where a service called target.service came out(what a strange and confusing name!). This unit file didn’t even come from one of the two main iSCSI packages (targetcli and iscsi-initiator-utils) but from a dependency (python-rtslib)! To understand that you had to enable it to get your iSCSI target configuration saved took some time.

Then came some NFS interface changes with RHEL 7.1. Some service configuration files were renamed, some disappeared, others were created(*). What a mess! And no explanation at all! You had to be a NFS guru and keep reading the NFS mailing list to understand what was going on.

Now, you must ask yourself several questions:

• Why the public Red Hat documentation doesn’t yet advertise the main services and targets required when administering a subsystem like NFS? Search for nfs-client.target on the Red Hat website to get your own idea.
• Is it really the Open Source spirit? 
• Besides the .service, .target, .socket, why there isn’t a .info file describing how to administer a given subsystem?

Systemd has made maintenance of Linux distributions easier, why wouldn’t it make life of administrators easier now?
 
(*)The first command to type when taking the RHCE 7 exam is now:

# cat /etc/redhat-release