Traditionally, to test the configuration of Postfix you had to install a full featured DNS server. This was due to the MX records mechanism. When you send a mail outside, the MTA (Mail Transfer Agent) sends a request to the DNS server responsible (the term normally used is authoritative) for the domain name specified in the recipient’s email address to get the MX records. These MX records give you the name of the servers handling the mails for the recipient’s domain. So, if you’ve got a domain called example.com, you need a way to define which servers in this domain are in charge of the mail delivery. The mail is sent to the mail server with the lowest value. If that failed, the mail is then sent to the one with the slightly higher value, etc.

For example, if you’ve got two servers managing mails for your domain, the DNS configuration is as follows:

example.com 86400 MX 10 mail.example.com
                     20 mail2.example.com

This DNS requirement makes Postfix configuration slightly more complicated because you need to be sure of your DNS configuration and it’s sometimes delicate in a lab environment.

However, there is a directive in Postfix called disable_dns_lookups that stops this behavior altogether. When set to yes, Postfix only relies on the local /etc/hosts file, which makes everything easier.

The RHEL 6.8 has just been released and brings the following main benefits:

• it enhances security by replacing openswan with libreswan as VPN endpoint solution,
• it makes integration with Active Directory easier through SSSD improvements (cached authentication lookup, authentication via smart cards) and support for adcli,
• it adds the new system archiving tool, Relax-and-Recover, enabling systems administrators to create local backups in an ISO format, simplifying disaster recovery operations,
• it now supports xfs filesystem sizes up to 300TB through the Scalable File System Add-on.

With the RHEL 6.8 release, this also marks the transition of Red Hat Enterprise Linux 6 into Production Phase 2. According to Red Hat Enterprise Linux lifecycle, this mainly means:

• no additions of new functionality beyond correcting defects,
• no implementation of previously existing features on a new hardware generation.

Rumors about Btrfs being deprecated are false and are only due to a poor wording in the technical notes.

Finally, you will find all the details in the RHEL 6.8 Releases Notes & RHEL 6.8 Technical Notes.

Since Systemd v213 and consequently with RHEL 7.2, a new resource control option called CPUQuota is now available.

Last year I wrote a post on the CPUShares option.

I decided to revisit it with this new CPUQuota option.

Caution: The following tutorial shouldn’t be run on a production server! The CPU will be used at 100%!

As in the previous post, I created a basic Systemd unit file called /etc/systemd/system/testSpeed.service:

[Unit]
Description=Test Speed
After=syslog.target

[Service]
ExecStart=/usr/bin/openssl speed 
ExecStop=/bin/kill -WINCH ${MAINPID}

[Install]
WantedBy=multi-user.target

Then, I created another copy of this file and updated the Systemd configuration:

# cd /etc/systemd/system; cp testSpeed.service testSpeed2.service
# systemctl daemon-reload

I started both new services on a fresh standard install of Centos 7.2 on a VM with 1 vCPU:

# systemctl start testSpeed testSpeed2

Each of the two new services were using almost 50% of the CPU time (excerpt of the top command execution):

  PID USER      PR  NI S %CPU %MEM     TIME+ COMMAND      
24598 root      20   0 R 49.8  0.3   0:08.42 openssl      
24601 root      20   0 R 49.8  0.3   0:08.40 openssl      

I checked the default CPUQuota property of the testSpeed service:

# systemctl show testSpeed | grep CPUQuota
CPUQuotaPerSecUSec=infinity

Note: Don’t ask me why the option is called CPUQuota and the property CPUQuotaPerSecUSec, I don’t know!

Because I wanted to learn how CGroups were working, I decided to apply a CPU constraint:

# systemctl set-property testSpeed CPUQuota=10%

Note: You don’t need to restart any service. The % character is not optional.

Now, the testSpeed service gets 10% of the CPU time and the testSpeed2 gets 90%:

  PID USER      PR  NI S %CPU %MEM     TIME+ COMMAND      
24601 root      20   0 R 90.0  0.3   1:23.58 openssl      
24598 root      20   0 R 10.0  0.3   1:22.20 openssl      

The CPUQuota property of the testSpeed & testSpeed2 services is now as follows:

# systemctl show testSpeed | grep CPUQuota
CPUQuotaPerSecUSec=100ms
DropInPaths=/etc/systemd/system/testSpeed.service.d/50-CPUQuota.conf
# systemctl show testSpeed2 | grep CPUQuota
CPUQuotaPerSecUSec=infinity

What exactly happened?

With the CPUShares option, you assigned a percentage of CPU time to a service. With the CPUQuota option, you now set a duration in millisecond. This duration is the maximum of CPU time allowed to a service per second. This service can get an amount of CPU time below but not above this limit.

The behaviour of the CPUQuota option is much easier to understand than the CPUShares‘.

Look at my CGroups page to get some other tips on this topic.

Some additional information is available in the systemd.resource-control man page.

One of the main changes that occurred in the RHCE 7 exam was the new objective regarding MariaDB. You can be a very experimented system administrator and never deals with databases. Not only you are asked to install the database system but you are also supposed to know how to create a database, create tables and manipulate SQL instructions.

SQL syntax is not so easy to learn. Even though nobody expects you to be a SQL master, correctly applying a create table statement requires some time.

Hopefully, MariaDB provides a nice solution to this problem through the help command. When you don’t remember a syntax detail, the help command should be a reflex.

MariaDB [db]> help update
Name: 'UPDATE'
Description:
Syntax:
Single-table syntax:

UPDATE [LOW_PRIORITY] [IGNORE] table_reference
SET col_name1={expr1|DEFAULT} [, col_name2={expr2|DEFAULT}] ...
[WHERE where_condition]
[ORDER BY ...]
[LIMIT row_count]

Multiple-table syntax:

UPDATE [LOW_PRIORITY] [IGNORE] table_references
SET col_name1={expr1|DEFAULT} [, col_name2={expr2|DEFAULT}] ...
[WHERE where_condition]

Sometimes like with the grant command, you even get some examples!

As usual, don’t rely too much on this feature, otherwise you will waste too much time and won’t be able to complete your exam in time.

Thanks to Sam and krz for sharing their knowledge.

Most system administrators know the tail command. By default, it displays the 10 last lines of a file.

The -f option is less known. Besides displaying the last 10 lines of a file, it mainly provides a way to watch what is written into a file in real-time:

# tail -f /var/log/messages
Apr  4 20:30:01 vm systemd: Stopping user-0.slice.
Apr  4 20:31:35 vm systemd: Created slice user-0.slice.
Apr  4 20:31:35 vm systemd: Starting user-0.slice.
Apr  4 20:31:35 vm systemd: Started Session 961 of user root.
Apr  4 20:31:35 vm systemd-logind: New session 961 of user root.
Apr  4 20:31:35 vm systemd: Starting Session 961 of user root.
Apr  4 20:31:35 vm dbus[590]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)
Apr  4 20:31:35 vm dbus-daemon: dbus[590]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)
Apr  4 20:31:35 vm dbus[590]: [system] Successfully activated service 'org.freedesktop.problems'
Apr  4 20:31:35 vm dbus-daemon: dbus[590]: [system] Successfully activated service 'org.freedesktop.problems'
Apr  4 20:33:53 vm systemd: Started Session 962 of user root.
Apr  4 20:33:53 vm systemd-logind: New session 962 of user root.
Apr  4 20:33:53 vm systemd: Starting Session 962 of user root.
...

When debugging complicated situations involving several files, the tail command proves to be a invaluable tool. In fact, you can specify several files after the -f option and a header will be displayed each time something is written into a different file:

# tail -f /var/log/messages /var/log/secure
==> /var/log/messages <== 
Apr 4 20:51:07 vm systemd-logind: Removed session 963. 

==> /var/log/secure <== 
Apr 4 20:51:11 vm sshd[21933]: Accepted password for root from 94.23.44.29 port 47411 ssh2 

==> /var/log/messages <== 
Apr 4 20:51:11 vm systemd: Started Session 966 of user root. 
Apr 4 20:51:11 vm systemd-logind: New session 966 of user root. 

==> /var/log/secure <== 
Apr 4 20:51:11 vm sshd[21933]: pam_unix(sshd:session): session opened for user root by (uid=0)

==> /var/log/messages <==
Apr 4 20:51:11 vm systemd: Starting Session 966 of user root.

...

For example, OpenStack engineers use this kind of command to debug their configuration on compute nodes:

# tail -f /var/log/{ceilometer,neutron,nova,openvswitch}/*.log /var/log/syslog

When troubleshooting, using the tail command should become a reflex!

When preparing an exam like the RHCSA or the RHCE exams, it is relatively difficult to remember the commands with all their options, the configuration procedures with all their details, and finally the mistakes to avoid. This is even worse if you have little or no real experience. In this case, you have to make up for your lack of experience by being clever.

One of the main tips is to know how to find the relevant information in the current system itself through man pages, default configuration files with theirs comments (Apache, Samba, etc), and configuration examples coming with some packages (Postfix, MariaDB, etc).

Recently one reader, Carlos G Mendioroz, left a very good comment about teaming configuration. If you don’t remember some details, several configurations are available in the /usr/share/doc/teamd-*/example_configs directory.

For example, to get an example of a round robin configuration, type:

# more /usr/share/doc/teamd-1.9/example_configs/roundrobin.conf
{
"device": "team0",
"runner": {"name": "roundrobin"},
"ports": {"eth1": {}, "eth2": {}}
}

It is clear that this doesn’t explain everything nor replace practice. If you spend your time searching for man pages information or searching for example configuration, time will run out and you won’t be able to complete the different tasks in time.

When using the new RHEL 7.2 release and running the systemctl status command, some slight changes were recently visible:

# systemctl status sshd
● sshd.service - OpenSSH server daemon
Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2016-02-13 10:56:40 CET; 6s ago
Docs: man:sshd(8)
man:sshd_config(5)
Main PID: 1286 (sshd)
CGroup: /system.slice/sshd.service
└─1286 /usr/sbin/sshd -D

Feb 13 10:56:40 server.example.com systemd[1]: Started OpenSSH server daemon.
Feb 13 10:56:40 server.example.com systemd[1]: Starting OpenSSH server daem....
Feb 13 10:56:40 server.example.com sshd[1286]: Server listening on 0.0.0.0 ....
Feb 13 10:56:40 server.example.com sshd[1286]: Server listening on :: port 22.
Feb 13 10:56:44 server.example.com sshd[2457]: Accepted password for root f...2
Hint: Some lines were ellipsized, use -l to show in full.

As you can see above, a new string vendor preset: enabled is now displayed. Here it means that the preset mechanism has been applied on the sshd package.

The preset mechanism came with Systemd and the RHEL 7.0 release, the RHEL 7.2 release just makes it visible. It is a way for a Linux distribution (Debian, CentOS, etc) to centrally define whether a package should be automatically enabled or not at installation time.

So, through the well known directory hierarchy (/usr/lib/systemd/system-preset, /etc/systemd/system-preset, /run/systemd/system-preset), a Linux distribution can specify how packages initially behave.

A typical preset file would appear like this:

# more /usr/lib/systemd/system-preset/90-default.preset
# System stuff
enable sshd.service
enable atd.*
enable crond.*
enable chronyd.service
enable rpcbind.*
enable NetworkManager.service
enable NetworkManager-dispatcher.service
enable ModemManager.service
enable auditd.service
enable restorecond.service
enable bluetooth.*
enable avahi-daemon.*
enable cups.*

# The various syslog implementations
enable rsyslog.*
enable syslog-ng.*
enable sysklogd.*

# Network facing
enable firewalld.service
enable libvirtd.service
enable xinetd.service
enable ladvd.service
...

A directive enable or disable precedes each package in this file as appropriate.

You shouldn’t need to modify this file in most cases but it is still interesting to know that it exists.

Sources: systemd.preset man page and Freedesktop.org website.

Among the changes coming with the RHEL 7.2 release, there is the new systemctl edit command.

Like the systemctl cat command, you specify a service unit file as argument. But, unlike the systemctl cat command, you will need to think twice before using it in a production environment. All successful editing converts into a service restart!

So, without any other argument than the unit file, you create or edit a drop-in snippet called override.conf that will complement the current service configuration. If necessary the associated service directory is created (for example /etc/systemd/system/httpd.service.d in the httpd case) which is pretty handy.

You need to set up one of the following environment variables before: SYSTEMD_EDITOR, EDITOR or VISUAL. If you don’t do it, the nano, vim, and vi editors will be tried one after the other.

If you add the –full option, it is now the service unit file that you are editing and not a drop-in snippet anymore.

Finally, you can use the –runtime option, if you want to make some changes to a service unit file in the /usr/lib/systemd/system directory and don’t want the change to survive a reboot (this doesn’t work if the unit file is located in the /etc/systemd/system directory because of precedence between /etc and /run).

This command is definitively a useful addition for speeding up the tuning of service unit files in a development environment.