New book about KVM.

Until now there was almost no book about KVM in English. This lack is fulfilled.

Three people, working for Red Hat, have published a book called Mastering KVM Virtualization in August and this book provides an extensive coverage of this technology.

So, you will learn the basic stuff like the various ways to set up the network and the storage parts. But you will also get best practices, some performance tuning tips or some advanced configurations with OpenVswitch and oVirt.

This book is the new reference on this topic.

Posted in Others

RHEL 7.3 Firewalld new features.

From RHEL 7.0 to RHEL 7.2, Firewalld didn’t really evolve (v0.3.9.7 -> v0.3.9.14). It was mainly a matter of bug fixes.

As usual with RedHat, Systemd already showed it, new Firewalld features are triggered by backport difficulties: as new bugs are found, fixes are applied but, at some point, this becomes too difficult to maintain, upgrade to a complete new software version is necessary, bringing a new set of features as an additional bonus.

The new version of Firewalld (v0.4.3.2) included in RHEL 7.3 comes with the following features:

  • performance improvements: Firewalld starts and restarts significantly faster thanks to the new transaction model which groups together rules that are applied simultaneously.
  • ebtables support: tables of rules similar to iptables but for Ethernet frames, ebtables, are now supported and can be used in direct chains and rules.
  • better zone management: zone settings (connections, interfaces and sources) can be specified in NetworkManager, in Firewalld or in the ifcfg files.
  • ipset support: ability to create a set of IP addresses or networks used as zone sources, within rich and direct rules.
  • MAC address management: ability to specify a MAC address to define a source.
  • new firewall-cmd options: –info-zone displays details about a given zone, –info-service about a given service and –info-ipset about a given ipset.
  • easier troubleshooting: with the new LogDenied directive in the /etc/firewalld/firewalld.conf file, the user can easily debug and log denied packets.

As usual, the Firewalld dedicated page has been updated with the new available features and lots of details.

Posted in RHEL7

RHEL 7.3 officially released.

Yesterday, Red Hat announces the official release of RHEL 7.3.

To know more about this new version you can read a summary of the RHEL 7.3 changes or the RHEL 7.3 Release Notes.

Be careful when upgrading to RHEL 7.3, a problem can happen concerning the network interface names preventing the interfaces from working after reboot. Hopefully, the fix is simple and already available (see details).

Posted in RHEL7

New RedHat Summit 2016 videos.

It’s more than three months after the event that RedHat publishes some new videos.

And we are talking about 60 videos!

People like me, who were disappointed by the lack of good technical videos right after the event, can now uncover the real reason: due to the number of recorded sessions during the summit, some were not released and were almost forgotten by the subcontracting production company.

Among these videos, we will find:

You can watch all these videos and fitfty others on the RedHat Summit channel on Youtube and get the associated pdfs by searching for the presentation name here.

Enjoy!

Posted in RHEL7

Bad experience with RedHat hands-on labs & exam.

Note: This is a translation of an article about the experience of a French guy following RedHat hands-on labs and taking the RHCE exam (source in French here).
This document was published on October the 12th 2016 and is under CC-BY-SA license.

I came across a LOT of technical and logistical/administrative problems when following the online RHCE training (RH299) and when taking the RHCE Exam (EX300F) in “individual exam” mode (in fact, I did not even take it).

Bugs in the RH299 training labs

(https://www.redhat.com/fr/services/training/rh299-rhce-certification-lab)

I chose to take this course “online”: I had access to the training materials for 3 months and to dedicated virtual machines for hands-on labs (with automated verification). The concept was really good and gave a lot of organizational flexibility. However, it was almost the same price as a “traditional” training in a classroom with an instructor.

The content of the training material was of good quality: clear explanations, appropriate practical work (apart from the videos that are not translated into French).
However, I came across a lot of bugs in the training labs.

I contacted the RedHat support about these problems (mid-March 2016). Someone quite competent replied point by point. I below summarizes these issues (the most important ones are at the beginning):

  • Unable to use the AltGr key to enter special characters which are essential to hands-on labs: ‘{‘, ‘@’, ‘#’, etc. This problem was fortunately fixed in late March or early April.
  • Error checking: the script that controls the outcome of the exercise is mistaken with respect to statement (for example: it searches for a connection named “review” when the statement says to create one named “exam”). So with a correct result in line with the statement, the tool still answers that this is wrong. I had the case on at least 3 assignments. RedHat confirms that it is definitely due to some translation “bugs”.
  • Bad user interface: for example, if you quickly click twice on the button to stop a VM, it will unfortunately reset it without asking for any confirmation. You will lose all the work in progress.
  • A lot of time wasted due to the use of VMs:
    For most hands-on labs, they ask to reset the VM before anything else. Except that it takes a few minutes to reset, and then 4 to 5 minutes to start, and then we must also configure the keyboard in French (it takes a long time, especially on first start). These are points that RedHat knows and attributes to their hosting infrastructure. They consider the possibility of reviewing their hands-on labs to avoid asking the reset.
    Each VM automatically stops after a timeout (the user can adjust it, but with a maximum). Except that the user is not informed when this timeout will be achieved. And when it is reached, the machine is abruptly stopped. RedHat told me that they are considering a pop-up to warn the user.
    And other technical problems were experienced such as the loss of network connectivity between the two VMs that serve the hands-on labs: they blamed the hosting infrastructure.
  • Unable to copy/paste text into or from a VM. Again, they know it, but explain that the software they use does not allow this feature.
  • After 4 hours of connection to the RedHat training site, the authentication cookie expires. Instead of asking the user to authenticate again, no feedback is given and all actions on the VMs are without effect anymore (and without error).
  • Crashes in the nmtui tool (for configuring network connections), which prevent some hands-on labs to be carry out with this tool (it is still possible to do them with nmcli, but it is much less intuitive). This is probably a bug in nmtui, but I had never came across them outside of these VMs.

Suffice to say that this training was not for me as effective as it should have been.
A student shouldn’t have to debug the platform on which he is supposed to learn.

Many of these problems should have already been detected upstream by RedHat: it should have been enough to do some tests on the hands-on labs in a French configuration (keyboard, language, etc), before putting it online. They would have necessarily found that they could not type the necessary special characters and there were various bugs.

Note that the RedHat website at the end of May 2016 indicated that an update of this platform was underway: it is possible that some of these problems are now resolved (but I have no way to check it myself).

Finally, after ranting about RedHat about all these problems, I got they let me take the exam for free, as a gesture of goodwill (they also extended my access to the online training).

The EX300F exam

(https://www.redhat.com/en/services/training/ex300-red-hat-certified-engineer-rhce-exam)

Expected skills “Slightly out of date”

I noted several inconsistencies between the list of skills tested in the examination and the content of the RH299 training. In particular, the following topics were not covered in the training, and were yet displayed as required for the examination:

  • Use /proc/sys and sysctl to modify and set kernel runtime parameters
  • Produce and deliver reports on system utilization (processor, memory, disk, and network)
  • (Network Services) Configure host-based and user-based security for the Service
  • (HTTP) Configure private directories
  • (HTTP) Configure group-managed pleased

After asking several people and called RedHat again on this subject, one of them finally told me specifically (mid-May 2016):

It appears that the information on the web page is slightly out of date.

The web page in question being the link above the “Objectives” tab.

In more detail, the RedHat‘s response was:

Unfortunately it appears that the information on the web page is slightly out of date. I will bring this to the attention of our web team. As a general answer to your question there are many factors that influence how well a given candidate will perform on the exam. Training alone is not usually sufficient to guarantee success but if you have relevant experience then training can make a difference. The other thing to consider is that RH299 is a rapid track course that is intended for experienced candidates.
The course covers material that we also cover in longer format in three separate four day courses. Going from twelve days of content to four days of content means that there are some topics that may not be covered or may not be covered in great depth.

Please see in-line for answers to some of the specific points you mention.
– Use /proc/sys and sysctl to modify and set kernel  runtime parameters
This is not an exam competency.

– Produce and deliver reports on system utilization (processor,memory, disk, and network)

This is not an exam competency per-se but this is the sort of information you might require in the course of performing other exam items and we expect candidates to be self-sufficient in this regard.

– (Network services) Configure host-based and user-based security for the service
This is a general competency. Some network services may support host-base or user-based security and we may ask you to implement access using these mechanisms.

– (HTTP) Configure private directories
– (HTTP) Configure group-managed content
Private directories are not an exam competency. We may request that you create directories that only allow certain users or user groups to manage content. I do not know if this is covered in RH299 but it is covered in RH254 (System Administration III).

Apparently the web page in question was amended in early June: no more objective on the sysctl topic and “Configure private directories” was renamed to “Configure access restrictions on directories” but the other objectives did not change.

Problems due to the “individual exam” mode

Small registration problems

I chose the “individual exam” mode (ex-kiosk) as exam dates in Lyon were all canceled (too few registrations). This method is much more flexible since you can select the date and time freely, which is very convenient. It is remotely monitored by a webcam.

RedHat goes through the same company as the Linux Foundation: ExamsLocal (Innovative Exams). However, unlike the Linux Foundation that allows to take the exam from home, it must be conducted in one of the RedHat examination centers (only Paris in France). I guess they consider it more secure, but it’s much less convenient (and more expensive) for those who don’t live in the Paris area.

During registration, available dates were changing all the time: at the beginning of my browsing on the registration site (managed by ExamsLocal), the date that interested me was available and then after a few minutes all schedules with that date became unavailable, then available again after a few minutes. Bug? I don’t know: I booked my date as soon as possible, as a precaution.

I received an immediate confirmation by email, but it gave the time in another timezone (UTC-1): not very convenient. Fortunately, the following emails were well in the French timezone.

Keyboard problem

On the day, after blocking 3 days of study, I went to Paris (no chance, railway strike, but ultimately not much trouble).

The exam went on a dedicated PC, where everything is locked so you can’t cheat. You have to log on to ExamsLocal (as expected) to take the exam.

Problem: we were faced with a AZERTY keyboard, but the OS was configured for QWERTY and no access to system settings to change it. In short, you had to type your password, translating positions from AZERTY to QWERTY. As a security aware sysadmin, I had special characters in my password, I had a hard time and typed my password in the login field before copying/pasting in the password field (I couldn’t find a better way).

Note: RedHat assured me later that this problem has now been fixed, which seems confirmed by the “Azerty keyboard” mention when seeking a session of this type.

SSO login problem that prevents the exam from being taken

And here it is: unable to log if you have a login from the ExamsLocal Linux Foundation.

Indeed, during the RHCE registration, I mentioned the ExamsLocal account, previously used when passing the Linux Foundation exams: as I was already identified with it, I didn’t need to create a new account. There were no warnings on this in RedHat or ExamsLocal websites. The ExamsLocal website correctly added the exam to my account and sent me reminders by email before the exam, etc: everything seemed in order.

Except that the PC used to take the exam in “individual exam” mode was locked with forbidden access to the Internet. ExamsLocal website uses a Single-Sign-On procedure provided by the Linux Foundation website: in this case, connection was redirected to a page on the Linux Foundation website… that was blocked by the system.

At StartX (the company that manages the RedHat exams in Paris), they told me that it had happened once, and they had reported it to RedHat. But RedHat did not correct the problem they already knew neither changed its website (to inform people about this problem), modified the filter rules in their PC or warned people about this issue (in my case, nobody let me know).

StartX team told me that I had to create another account on the ExamsLocal website (without SSO) and my exam would be transferred. Yes, except that they can’t do it themselves: this must go through ExamsLocal, which are in the United States, and there should be a period of “a few days” (wait later you will laugh). Joy, happiness. I could not take the exam because of this and went home very angry.

Obviously, I could log on the examslocal website with my personal PC (I carried it with me at StartX), and I could have taken the exam with it. But no, RedHat’s refusal: only the RedHat dedicated PC is allowed for the exam. On the other hand, StartX people have no access to the PC used for the examination.

Icing on the cake that evening, I received an email from RedHat who told me I failed my exam, because I only had 46 points, while it was 210 in order to pass it: how could I have 46 points without taking the exam? I asked several people from RedHat, but none replied.

Over 4 months to take into account some of my complaints

I immediately demanded compensation and the opportunity to retake the exam. Through the only means of communication that I had: a claim form from the RedHat website. It was the beginning of an obstacle race.

No news after 2 weeks (except for a “Please allow me to Investigate the box and will get back to you soon”), I asked again. Still no response after 3 weeks, so I sent a letter with recorded delivery. Then I got a call from a person from RedHat London who apologized for all these failures, assured me that the issue was taken very seriously by the “top management” and I would be quickly informed. He gave me his direct phone number. I was somewhat reassured (I was wrong).

A few days later, I received an email from another person from RedHat, who apologized for forgetting to reply to my request (!). This email said that, as compensation, RedHat would offer me a free retake of the EX300 exam and another of my choice. 2 exams to be taken before the beginning of December, 2016 (we were early July) …

After discussions with my interlocutor from London and another one (I’ll spare you the wasted time and ping pong games between these interlocutors), the final position of RedHat was:

  • Reimbursement of transportation related to the exam that I could not take (only on proof)
  • Ability to retake the exam for free in the next 12 months
  • And, as compensation, the possibility of taking another exam of my choice in the next 12 months

RedHat seems to think that a free exam is a nice gift as it is worth several hundred euros. This may be true for a person who needs additional certifications. In my case, I would probably do nothing of this. Especially since, after all these problems, it is not the first thing I want … If I do not actually make use of it, I will try to offer it to someone else (but I have nobody around me currently interested in).

I had rather sought financial compensation for days spent preparing for the exam (I’ll have to prepare again, given the time), and to take it. The answer was clearly no.

A month later, I still had nothing. So I sent a 2nd letter with recorded delivery, call my interlocutor from London and got an email confirming that I will receive very soon what had been promised.

It took a total of over 3 months, 2 letters with recorded delivery, much time and energy lost, to get the codes to take the exam again.

To date, more than 4 months after the initial exam, I still have not been reimbursed for transportation costs (I’ll spare you the many reminders and “internal procedures” to be followed for a simple transfer) .

Initially, I wanted to wait to be reimbursed before publishing this article (not to influence the procedure in progress, whether in one direction or another). But never mind, it seems to me that the exam candidates need to be aware of the SSO login problem, not to suffer the same setbacks as me.

In the end, I intend to take the RHCE exam, I will not give up. But, given the time elapsed, I must restart my study from scratch. And I think I need a break after all that: it will be for later 😉

RedHat portal and its invisible conditions

(https://www.redhat.com/wapps/ugc/protected/account.html)

Besides the above problems, this one is much less serious, but equally symptomatic.

From this portal, once authenticated, you can access the list of the various certifications.

Problem: when you click on “Certifications”, RedHat asks you to read and validate “Logos and Guidelines – Terms and Conditions” with a link to read these conditions … this doesn’t redirect to the right place. In short, RedHat asks you to validate conditions that you can’t read.

I first reported this problem in mid-December 2015. At the time, the link showed a downright “Proxy Error”.

The “Proxy Error” has been fixed mid-January 2016, but the link now returns to https://www.redhat.com/rhtapps/certification/… I still don’t see the “terms and conditions” to read.

Late February 2016, RedHat sent me an email with a draft of these conditions:

We’re in the process of rewriting these, but here is what we currently have. Please see attachment.

To date (mid-October 2016), despite my numerous reminders, the bug is still present, and I don’t know how to read the final conditions RedHat asks me to confirm.

What to think about this?

I haven’t any doubt about the RedHat employees’ skills nor the architectural complexity of RedHat websites, or the difficulty of implementing secure online exams, but it’s quite ironic that all these technical problems occur with an editor certifying system administrators …

I mainly see the result of administrative burdens, lack of communication between services, complexities of outsourcing and a dilution of responsibility. This kind of symptom unfortunately occurs in many large companies. But, at such a level, I admit that I would not have imagined that.

I also think that the methods that I have chosen (online training and individual examination) are not frequently used and may not have received the same attention as the more “classical” modes. That doesn’t excuse anything, of course …

In the end, as you could imagine, I’m not fully pleased with this experience …

Posted in RHEL7

Systemd.conf 2016 videos.

Last week the Systemd.conf conference took place in Berlin with a lot of interesting presentations.

Among these videos, Davide Cavalca from Facebook discussed Systemd deployment at scale (41min). Besides the motivation of the migration from CentOS 6 to 7, he gave some feedbacks about difficulties he faced (logind timeout problem after 2000 connections, /sbin & /bin not in the PATH variable by default in a unit file, etc).

Marcel Holtmann from Intel presented a New Wireless Daemon for Linux (40min) replacing wpa_supplicant.

Vincent Batts from RedHat talked about What’s next for containers? (32min). He gave some examples of commands used in the container ecosystem.

Lennart Poettering and Tom Gundersen showed What you didn’t know about networkd (47min) and some new network features included in the latest version of Systemd.

If you are interested in the embedded world, there are definitely other presentations from this conference and you can watch most of them here.

Posted in RHEL7

Firewalld documentation website.

Since RHEL 7.0, Firewalld has been subject to controversies.

Newcomers find it easy to work with because it masks complexity: ports used by protocols are stored in configuration files, network masquerading is started through the simple –add-masquerade option, permanent and temporary configurations are clearly differentiated with the –permanent argument, etc. No need to remember the various iptables network chains or to be an expert in network packets to enable or disable a given protocol anymore.

However, some seasoned administrators don’t like it because it breaks iptables habits, add new concepts like zones, direct rules, rich rules and make some configurations almost impossible like ipset (to match entire sets of addresses at once) or MAC filtering, at least in the current RHEL 7.x versions.

In one word, Firewalld is generally easier to use than iptables but not always!

As Firewalld is part of the RHCSA & RHCE curriculums, even though iptables can still be used, it’s worth spending some of your time to learn it.

Thomas Woerner, Firewalld‘s author, has created a website to provide some documentation, explain the main concepts and offer some perspective about the future versions of his software: www.firewalld.org.

This is definitely a place to visit.

Posted in RHEL7

Systemd service debugging tips.

Troubleshooting a systemd service can be tricky.

Because the Restart=always directive is sometimes set in the unit file, you don’t know if a service is running fine or if it is stopping and restarting all the time.

The systemctl status command doesn’t help you much:

# systemctl status myservice
● myservice.service
Loaded: loaded (/etc/systemd/system/myservice.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2016-09-09 20:20:30 CEST; 44s ago
Main PID: 21366 (node)
CGroup: /system.slice/myservice.service
├─21366 /opt/nodejs/current/bin/node /opt/myapp/index.js -config=myconf.js
└─21376 /opt/nodejs/node-v4.5.0-linux-x64/bin/node index.js -config=myconf.js

Sep 09 20:20:30 myvm systemd[1]: Started myservice.service.
Sep 09 20:20:30 myvm systemd[1]: Starting myservice.service...
Sep 09 20:20:31 myvm nodejs[21366]: index.js: configurationFilename=myconf.js

First, you can add the -l/–full and -n 20/–lines 20 options to the systemctl status command. They respectively stop truncating the journal output and display 20 lines instead of 10 by default.

But if the service is regularly stopping and starting, these options won’t help you.

Hopefully, you can use the -u option and specify the service name in the journalctl command:

# journalctl -u myservice
Sep 09 16:52:47 myvm systemd[1]: Started myservice.service.
Sep 09 16:52:47 myvm systemd[1]: Starting myservice.service...
Sep 09 16:52:47 myvm nodejs[4076]: index.js: configurationFilename=myconf.js
Sep 09 16:53:47 myvm nodejs[4076]: can't access database
Sep 09 16:53:47 myvm nodejs[4076]: exited
Sep 09 16:53:48 myvm systemd[1]: myservice.service holdoff time over, scheduling restart.
Sep 09 16:53:48 myvm systemd[1]: Started myservice.service.
Sep 09 16:53:48 myvm systemd[1]: Starting myservice.service...
Sep 09 16:53:48 myvm nodejs[4261]: index.js: configurationFilename=myconf.js
Sep 09 16:54:48 myvm nodejs[4261]: can't access database
Sep 09 16:54:48 myvm nodejs[4261]: exited
Sep 09 16:54:48 myvm systemd[1]: myservice.service holdoff time over, scheduling restart.

Due to an error (“can’t access database”), it is now clear that the service is stopping and starting.
If it were not the case, adding Environment=SYSTEMD_LOG_LEVEL=debug in the service stanza of the unit file could provide some useful messages.

Posted in RHEL7

Third website anniversary.

As Redhat just announced the RHEL 7.3 Beta release (release notes are available here), this week marks the third anniversary of the website creation.

Tutorials are regularly updated even though writing of new ones has seriously slowed down due to lack of time.

I hope you still enjoy the website.

Posted in RHEL7

Rsyslog tip.

When you are about to deploy an application, you’ve got a lot of problems to solve.
How are you going to deal with backups, monitoring, filtering admin connections?

One of these problems concerns the management of system and application messages.
There are many available options. One of them is to use rsyslog.

With rsyslog, you can store system and application messages into local files or/and send them to a remote server according to the configuration located in the /etc/rsyslog.conf file or the /etc/rsyslog.d directory.

However, what happens if your central rsyslog server is not available because of maintenance or failure? You loose all your platform messages during this time! This is not good.

But, there is a solution: you can perfectly configure two or several remote rsyslog servers in your client configuration (still in /etc/rsyslog.conf) as follows:

# ### begin forwarding rule ###
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
*.* @@remote-host1:514
$ActionExecOnlyWhenPreviousIsSuspended on
& @@remote-host2:514
& @@remote-host3:514
$ActionExecOnlyWhenPreviousIsSuspended off
# ### end of the forwarding rule ###

Then, check the syntax:

# rsyslogd -N 1
rsyslogd: version 7.4.7, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: warning: ~ action is deprecated, consider using the 'stop' statement instead [try http://www.rsyslog.com/e/2307 ]
rsyslogd: End of config validation run. Bye.

This way, all the messages go to the remote-host1 server by default. If the remote-host1 server doesn’t answer, messages are sent to the remote-host2 server, then to the remote-host3 server if the previous server doesn’t reply.

You can find all the details in the tutorial about Configuring a system to log to a remote system.

There are certainly other options but this one is pretty simple and works fine.

Note: Rsyslog was an RHCE 6 objective but doesn’t appear in the RHCE 7 objectives anymore.

Posted in RHEL7

RHCSA7: Task of the day

Allowed time: 5 minutes.
Create two users "tom" and "engine". "tom" has the UID/GID 3000 and "engine" the UID/GID 4000. "engine" doesn't have an interactive shell.

RHCE7: Task of the day

Allowed time: 10 minutes.
Set up a NFS server that exports the /opt directory in read-only mode.

Poll for favorite RHEL 7 book

What is your favorite RHEL 7 book to prepare RHCSA & RHCE exams?

View Results

Loading ... Loading ...

Poll for most difficult RHCSA 7 topic

What do you think is the most difficult RHCSA 7 topic?

View Results

Loading ... Loading ...

Poll for most difficult RHCE 7 topic

What do you think is the most difficult RHCE 7 topic?

View Results

Loading ... Loading ...

Recent Comments