Troubleshoot Process in AWS Linux EC2

A system administrator working with Linux servers must have an extensive knowledge on Linux processes and how to work with them. Even if you have just started working on AWS or Microsoft Azure, you must have heard about starting a process, checking the process status, stopping a process or even killing one, in Linux instances. In this blog, we are going to discuss different operations that you can do on Linux processes.

What is a process?

A process is referred to a task which runs on a Linux operating system. Example: in a web server, httpd is a process whereas in a MySQL server, mysqld is a process. A process in a Linux instance is similar to the processes that you see in Task Manager in a Windows system.

There are two types of Linux processes:

1. Foreground processes: They are not started automatically. Such processes are initialized using a terminal session and require a user connected to the system to start such processes.
2. Background processes: Such processes are started automatically, and they don’t expect any user to be connected to the system via terminal.

Let’s have a look at how to run a command or process in the background, check the background processes and how to bring them to foreground.

When you know a command will take time to execute, for example, extracting a jar file, it’s better to run it in the background so the terminal will be free for you to work. Use your command followed by an ‘ampersand’ (&) for achieving this.

#<your_command> &                    Example: #sleep 60 &

To check the commands executed in the background, use the below command:

#jobs

To bring a background running command to foreground, use the below command:

#fg

The numbers [1],[2] and [3] in the above screenshot are the job IDs and ‘+’ symbol denotes which process was backgrounded or foregrounded last. The ‘fg’ command will select the command/process which was last sent to background and bring it to foreground. If we specify the job ID after ‘fg’, then that particular background job will be foregrounded.

In case you have executed a command and later realise that you should have sent it in the background, use ‘Ctrl-Z’ in the running process and then type ‘bg’. That will send that command execution to background.

There is one more important command which you can use to run processes / commands which take time. ‘nohup’ lets you run the command even after you have logged out of the system. For example, you have SSH-ed to a system and you think the connection might drop before the command execution is completed, you can mention ‘nohup’ before your actual command and then it will run the whole thing even after you have disconnected.

Syntax for ‘nohup’ command is:

#nohup <your_command_here> &

What is a systemd process?

Systemd is the first process which starts when a Linux system boots up, and directly or indirectly, it is the parent of all processes in that Linux machine. The process ID(PID) of systemd is 1. Instead of systemd, earlier it used to be the ‘init’ process which had to be replaced because of it’s shortcomings. Systemd comes with lot of advantages like parallel processing at boot, a better API and simpler boot process.

How to start, stop and restart processes in Linux?

Starting, stopping or restarting processes in Linux has become fairly easy since most of the distributions have migrated to systemd. For most of the Linux distributions today, systemctl is the service manager and command to start and stop a process would be:

#sudo systemctl start <your_application_service>

#sudo systemctl stop <your_application_service>

Similarly, command to restart/reload a service is:

#sudo systemctl restart <your_application_service>

#sudo systemctl reload <your_application_service>

The above reload command is used when the application is capable of reloading it’s configuration files without it restarting.

If you are unsure whether the application will be able to reload its configuration without restarting, then you can use the below command:

#sudo systemctl reload-or-restart <your_application_service>

Some service needs to be always started when the system boots up, so that you don’t have to manually start them every time the system restarts. For that purpose, you can use the below command:

#sudo systemctl enable <your_application_service>

The opposite can also be done. If you don’t want a service to be started automatically on every system boot, then use the below command:

#sudo systemctl disable <your_application_service>

To check the status of a service, use the below command. It will give you an overview of the service, it’s the current status, few logs and any issues which you can take actions upon.

#systemctl status <your_application_service>

There is an important command to note here, which should be used after you have modified a service file or directory. The command is :

#sudo systemctl daemon-reload

This command is used for reloading the systemd process. That will refresh it and stop it from referring to the deleted files.

Now that we have seen the basics of the processes in Linux, let’s see how to use the terminal to troubleshoot process-related issues in Linux. I am going to talk about some important commands which you can use in Linux to troubleshoot errors related to processes.

• Top command

This comes pre-installed on every Linux distribution and it can be used to monitor processes and system resources. The command syntax is simply ‘top’ and when you press enter, you can see a screen just like below:

You can see various information like system uptime, active user sessions count, system memory usage and processes running along with their individual details like PID, user, CPU and memory utilization.

Similar to ‘top’ command, another command ‘htop’ gives you an interactive output with all the process related details in meaningful text graph which is easier to understand. You can also use mouse clicks to select different tabs to see more details.  For ‘htop’ command to work, you should install it first by using the command:

#sudo yum install htop                   [For CentOS or RHEL systems]

OR

#sudo apt-get install htop             [For Ubuntu systems]

There are various tasks that you can perform after running ‘top’ command. Some of them are:

1. Shift + M – To sort the processes by memory usage.
2. Shift + P – To sort the processes by CPU usage.
3. Shift+ N – To sort the processes by process ID(PID).
4. Shift+ T – To sort the processes by running time.

So, you are going through the details given in the ‘top’ command output, and while coming to conclusions, the output gets refreshed before you note down the PID. This is because the default value of refresh time is 3 seconds. You can change it by typing ‘d’ on a running ‘top’ command and giving the desired refresh time, allowing you more time to analyze the output.

You can also kill a process by pressing ‘k’ on a running ‘top’ command and then providing the PID of the process, without exiting the top output display. Refer to the below screenshot for the same:

• ps command

ps stands for process status. This command is used to check the status of running processes in a system. The output is given in 4 columns, namely,

1. PID -process ID
2. TTY – terminal for user
3. TIME – time since the process is running
4. CMD – a command that launched the process.

Ps command will show the processes started by the user who is logged in to the system.

To see all the processes, use the below command:

#ps -ef

To get information about a particular process, use ‘grep’ command along with ps:

#ps -ef | grep httpd         [to know about httpd service in the system]

Additionally, to display all the processes in a tree format, use the below command:

#pstree

Pstree command can also be combined with various arguments, like -V for displaying the version, -s for displaying the parent processes of the specified process etc.

• nice command

By default, all processes in Linux are considered equal and are given same amount of CPU time. We can change this behaviour by forcing the kernel to prioritize some processes compared to others. The command used for this is ‘nice’ followed by the parameters. The nice value for the processes varies from minus 20 to plus 19. By default, all processes have a default nice value of zero (0). The more the nice value, the more CPU time the process will get. This is slightly confusing, but in Linux terms, minus 20 is considered a “nicer” value because it allows other processes to receive more CPU time. ‘nice’ and ‘renice’ command syntaxes are:

#nice – 12 abcd                [ this will start a process abcd with a nice value 12]

#renice 10 -p 1456           [ this will change the nice value of process with 1456 to 10]

• free command

Before installing an application on the system, we should check the free memory available. There comes the use of ‘free’ command in Linux. It provides the total memory, used memory and available memory in the system. The syntax of the command is:

#free

Sample output:

• kill command

Now that we know how to check the running processes, check their PIDs, check the available memory etc, let’s see how to terminate a process, or in Linux terms, how to ‘kill’ a process. The command for that is simply ‘kill’ followed by the process ID. In the below screenshot, I have mentioned the PID of firefox application which I had started earlier.

#kill <process_id>

The default command ‘kill’ sends signal to the process to be terminated. The process may choose to ignore it so the command does not always guarantee that the process will be terminated. There are two kinds of signals which I would like to talk about here: SIGTERM and SIGKILL.

SIGTERM is the default signal when we fire the ‘kill’ command. As you would have already guessed, the name is made up of the words ‘signal’ and ‘terminate’. It’s a soft kill method in process termination. There is no need to explicitly mention ‘-15’ in the kill command because that would be redundant.

SIGKILL is used when you must immediately stop a process from running. There is no graceful termination when you use this signal. For ‘kill’ command to use SIGKILL signal, use ‘-9’ as I have showed below:

#kill -9 <process_id>

It’s recommended not to use SIGKILL always to terminate a process, because it kills the child processes as well whereas, in the case of SIGTERM, the process gets time to send information to it’s parent and child processes, which is a better way of terminating processes.

Hope the steps mentioned above are clear. Watch this space for more such blogs.

If you want to learn more about Linux Servers kindly check our website.

If you have any comments or questions, then do write it in the comment.