About Nginx number of workers

Reading Time: 4 minutes

Introduction

This article provides information about the  number of workers a typical nginx system administrator should setup.
As per any web server, the tuning of the worker processes is a sort of voodoo art.Ask hundreds system administrators and you will get probably hundreds of different opinions. Before we go deep in the topic, let’s understand what is an nginx worker. An nginx worker is a process in memory which “takes care”  of the clients’ requests. There is a minimum of one worker process in a Nginx environment. It is part of the roles of a good system administrators to decide how many workers you actually need.
NGINX offers the ability to configure how many worker processes must be spawned through the core configuration parameter “worker_processes” (http://nginx.org/en/docs/ngx_core_module.html#worker_processes).
Now, the temptation for many newcomers would be to set a very high number of worker processes or ignore the parameter at all.
The result would be to have an either over-complicated process management (or not enough CPU power) or to not take the best from the HW available.
This is pretty much similar to the endless story of how many APACHE workers you need to configure in a typical MPM_Prefork environment. Reality is, there is no magic recipe for it.
The number of workers you should configure on your NGINX server depends on different factors:
– The role of the server
– The number of CPU available
– The storage system landscape
– The disk I/O requirements (pay attention to caching which is disk I/O intensive)
– SSL encryption support
– Data compression support (gzip)
Why all these factors should be considered?
Let me try to explain this point by point.

The role of the server

The role of the server and the architecture of your web server solution is very important in the counting of the number of workers you should configure. For instance, a stack where NGINX is running on the same machine serving your Django stack (through WSGI for instance) is very much different from a stack where NGINX is running on a different machine providing you the django application. In the first case there will be competition in the usage of the cores, in the second case, the cores are available for a typical reverse proxy scenario and can be all allocated pretty much for NGINX usage.
Would be very inefficient to have a full LEMP (Linux, nginx, MySQL, Perl/Python/PHP) stack on your server having 4 cores and allocate 4 nginx workers. What about MySQL needs? What about your PHP/Python needs?

The number of CPU availablenginx

This parameter is very important since it does not make much sense to overload your system with more “loaded” processes than the number of CPUs. Nginx is very efficient in using CPUs, if your workers don’t take a full CPU most probably you don’t need it. Remember that Nginx workers are single thread processes.

The storage system landscape

Parallelism is not only related to CPUs. If your web server uses a lot of cached items will do surely high I/O on your storage system. What about having the full content of your web server, OS included, on the same physical volume? Would you benefit in having many processes all demanding pretty much 100% of your single physical volume I/O bandwidth? Try to think about it. Try to split the I/O load on multiple disk systems before you actually think about increasing the number of workers. If I/O is not a problem (i.e. some proxy services), then ignore this point.

The disk I/O requirements

This is linked very much to the previous points. Disk I/O is one of the few situations where Nginx can find itself in a lock/wait situation. There are possible ways to workaround this situation but none of them is a magic recipe (see AIO and sendfile)

SSL encryption support

Is your web server heavily using SSL? If so, remember that SSL requires additional CPU power compared to plain HTTP. Why this? Try checking the SSL encryption algorithms supported nowadays, you will see a lot of math over there, then you will realized why we talk about CPU consumption. To make a long story short, if you use SSL, consider that the CPU usage will be higher than not using it. How much higher? this depends very much on your application usage pattern, the number of GET/POST operations performed, the average size of your responses etc.

Data compression support

The support for data compression serving responses is a very good approach to limit the used bandwidth of your environment. It is nice and good but it costs in terms of CPU cycles.
Every response needs to be compressed and depending on the level of compression you set, the algorithm behind it will cost you in terms of CPU consumption.
The complexity and the computational cost of the GZIP support is ruled by the configuration parameter “gzip_comp_level” of the ngx_http_gzip_module module (http://nginx.org/en/docs/http/ngx_http_gzip_module.html#gzip_comp_level)
 
It accepts a value from 1 to 9. The higher you go, the best results (probably) you will get, the higher will be the CPU load on your average transactions.
 

Conclusion 

Having mentioned the above points, it is way too evident that there is no magic recipe for it.
 
 
 
Let me say that a very common and easy approach is to allocate a worker per CPU. This is the basic setup and I am pretty sure that it works flawlessly on most of the environments.
 
But, as already said, this is not always the best approach. It works but it is not the best.
 
For instance on a typical reverse proxy, people tend to allocate one and a half worker for each core or, as well two workers per core. Let me say, if you don’t do much SSL, 2 workers per core works Why this? Because most of the times NGINX will be waiting for the back-end systems to generate the response of the request and because the machine running Nginx is not busy running your application’s logic and CPU payload is at minimum.
 
If you do extensive caching, considering the way NGINX does it (very efficient but I/O intensive), I would go for a more conservative approach, I would go for 1 worker per core and I would monitor the situation, if the machine behaves well, I would try to go for a 1.5 workers/cores ratio.
 
If you would like to have more information about this topic or your would like to have an advice on the number of workers you would need to load, try to comment on this page, I will try my best to support you.


There is an excellent diagram reporting how the master and workers processes work. Please refer to the following:


Create a small RAM disk for ultra fast I/O

Reading Time: 1 minute

In this simple guide I will explain how to create a virtual ram disk based on ext3 file system to be used for very fast I/O.

This example is very useful though it has important limitations, for example:

  • It does not allow big ram drives, as far as I know, the limit is 64 MB
  • Whatever is stored in the ram drive, is destroyed as soon as your system is restarted

In this example, we create a ram drive of 64 MB, we mount and set permissions so that your current user is the owner of the newly created file system.

Step 1: create the new file system, based on memory:

As you can see, the disk size is 64 MB, we use as device the /dev/ram1 (you can use by default up to ram15) and the file system used is ext3.

Step 2: Create the directory to be used to mount the new file system

Step3: Mount the file system

Step4: take the ownership of the new file system

Replace the [USERNAME] part with your actual user name…..if you don’t know who you are, try to check it with the whoami command.

This article, very simple, has a lot of limitations due to the small size of the disks you can create.
Planning to write an article on how to create bigger volumes with  tmpfs.

Side Note: Nowadays, using a 64bit OS, having a lot of RAM on your PC allows you to create quite big ram drives having incredible performances during read/write operations, sometimes it is useful to leverage on this to have ultra fast user experience.

Lenovo Thinkpad T410 and Ubuntu 12.10

Reading Time: 2 minutes

(This article is rather old and was posted on another personal blog, I think it could help people anyway). Just as a side note, I stopped using Ubuntu since a while due to Unity. Now I work on Fedora 20 and Gnome 3, I love it.

I have upgraded my laptop, a Lenovo T410 mounting integrated intel graphics, to Ubuntu 12.10.

As soon as the OS started I started getting errors on the xorg server about the drivers installed. The only option was to start the PC in low resolution. In addition to that many other problems were there like support for external monitors and so on.

Checking the ubuntu unity support utility the “Unity 3D supported”  was reported as No and the overall system performances were very bad.

I have luckily solved the problem in very long manner and therefore I want to write here what I have done to solve my problems hoping to help anyone having the same isse.

To start with, the drivers installed by UBUNTU (the gallium one) are not the right one, they doe their job but don’t cover all the aspects of the system configuration.

To solve the issue I have done the following.

1. Install the required packages for the next steps:

Run in a terminal the following command:

This step is required in order to prepare the environment for the actual compile of the other components.

2. Go the intel OpenSource Technology website

https://01.org/linuxgraphics/downloads

3. In the download page, download the packages MESA, libdrm and xf86-video-intel – 2.20.12.

4. extract the content of the 3 files into separate directories.

5. Build and install of the libdrm package

6. Build and install the video intel (the 3rd package you downloaded)

7. Do the same for MESA (last one), it will take a very long time

8. Restart your ubuntu machine. At this point, going into the system settings/details applet of your ubuntu client you will notice that ubuntu will use a new driver (and you should notice the new look and feel too!)

If everything is OK, you should have something like this:

 

 

The love and hate for shared libraries

Reading Time: 2 minutes

Introduction

Since yesterday I have been playing around on the WII controller integration on my PATRIA 3D engine.

This is required in order to have a fully working accelerometer support for our games testing since it is quite an effort to perform real testings on the smartphones especially when you are still in the middle of the caos of code development.

I have been trying the libwiimote on my ubuntu 64 bit but unfortunately, for some reasons, it does not look like the best solution for me (thought the library is well done). In fact it has a lot of sleep/wait time in the implementation which cause the frame reate of the game Fantasy Racing to drop to 15/18 FPS on my laptop. It looks like that, if no input is made on the device, the wiimote_update function freezes my process.

I have then decided to go for libwiiuse and I find it fantastic though I have not yet completed the integration.

 

How to compile

To cut a long story short, I have compiled the library, using the well done documentation, I have done all the required configuration steps in my eclipse environment but every time I have run a compiled program using the shared object library libwiiuse.so  I used to get this error message:

error while loading shared libraries: libwiiuse.so: cannot open shared object file: No such file or directory

This problem has kiled me since I spent a lot of time looking for an explanation to the root causes of it.

After I while I thought…..holy crap Maurizio, can you be such a stupid? The new library is a shared library which links other shared libraries, let’s try to refresh the dynamic linker cache by running the command ldconfig

.

And….it actually solved it.

Lesson learnt, whenever you have problems linking shared libraries, always remember to update the linker cache!!!

Android and HW differences

Reading Time: 2 minutes

“This article was created on my old blog in 2012”

Since the release of the first game on the Android Market I have faced the usual problems on the HW differences of the different Android devices on the market.

I think this is the actual limitation of the Android platform, a lack in the HW standards.

This limitation forces every developer to perform endless sessions of quality testing on a multitude of devices that ofter represent a serious limitation for and indie team.

Not all the people of starting teams can afford to have tens of devices for testing and, even if they can, no one can guarantee that the application won’t crash on a specific sub version of the HW or a specific version of Android.

As if it was not enough the presence of a multitude of custom Roms, custom access levels (root access), anti virus and anti spy-wares make the situation out of control and almost impossible to be handled by simple human beings.

Besides, if you decide to go for a custom code written in C, eventually using the Android NDK, things gets harder and harder.

Just to report an example of this, when we released the version 1.7 of Fantasy Racing, we started received bad reviews (four one star in one day, terrible!) and we didn’t know how to react.
In the comments our users reported things like “always crashes” or “does not start” while on our devices the game was rock solid and very stable.

We realized that all the users reporting the errors were running the game either on Galaxy SII or Galaxy Note which are 2 very common devices in the Android market.
We started digging around and we found many forums talking about bugs specific for these devices. Bugs were pointing to ADMOB compatibility or the soundpool bug. We released fixes for these problems but apparently nothing changed, we continued to receive bad reviews from our users.

We decided then to procure some devices having the same specs and we found out that on a Galaxy SII of a friend of ours, the game went very smoothly while on a Galaxy SII of another friend it constantly crashed.

To cut a long story short, we found out that the problem was depending on our PATRIA3D Engine which used to ask the GPU to draw an “not existing” Vertex Buffer Object in some game conditions and that the GPU on these devices (Mali 400) raised an error blocking the game.

The same problem didn’t happen on our testing devices using different GPU families such as PowerVR, Adreno or NVidia.

In simple words, the problem was caused by a bug in our code but we had different behaviors depending on the GPU installed on the devices (though I think it is mostly related to the OpenGL driver implementation of the device).

We have now fixed the problem and the game is rock solid on those devices but unfortunately the bad reviews remains like a stain we cannot remove.

We still cannot explain why on same Galaxy S2 devices the game worked fine but start thinking about them as mysteries of the Android World.

Compile Linux static libraries

Reading Time: 1 minute

I am preparing the code I have to use for the Global Game Jam 2013 which will occur next week end, 25-27 of January 2013.

The idea is to use my 3D Engine, PATRIA for the development of the game but I would like to avoid to share the last 2 years efforts with everyone in the world. This considering as well that PATRIA3D is not only the results of my efforts but the result of the efforts of all the team members of the Team Novasoft.

Considering that PATRIA is a modular system where each component of the engine has its own .c and .h, the best solution is to compile the .c files of the core engine which most probably won’t be altered during the JAM is static libraries.

In this way I will use just the .a (static library) and the header file .h

In order to do this operation, the following steps are required:

1. Compile the .c file you want to create the library

2. Once the file is compiled, you have the .o, the object file compiled by the GNU C Compiler.

3. Now, to create the static library, you need to use the ar command. Remember that a static library must start with the prefix lib and must end as “.a”

 

4. At this point you have your library. The best choice in my humble opinion is to copy it into the OS /usr/lib so that it will be usable at system level. Copy it over there.
5. Include the library into the Eclipse project (if using eclipse. Otherwise just include the -llibname in the linker instructions
N.B. remember to remove in the eclipse library reference the lib and the extension!
Now you can build your project with your libraries and the code is safe!

Publish files on port 80 using netcat

Reading Time: 1 minute

Let’s use an example, you need to pass a file to one of your friends….

but, you don’t have a web space

and,you don’t have samba installed nor you are in the same network

or, you don’t want to create an account for just on file transfer…

but…. you have linux! WOW great.

All you need is to have netcat installed (available easily on most of Linux distros…) and use this command:

Open a shell, move to root (su command) and use the following command:

while true; do nc -l -p 80 < thefileyouwanttoshare; done

Now, just provide your IP address to your friend and ask your friend to open a browser to your just provided IP address…and woila’!

I was so surprised by this simple, obvious command that I love it.

Cheers guys.

Maurizio

Root access ubuntu

Reading Time: 1 minute

By default on Ubuntu systems the root account is disabled.

To run commands on console as root you have to use the sudo command in order to have root privileges on the required operation.

For instance, you can edit the X11 configuration file using root credentials with the following command:

sudo gedit X11/xorg.conf

Sometimes, you need to run a batch of commands on your console as root, to do this, usually on all the unix like systems you use the command <span style="font-weight:bold;">su</span> [Super User] and then you type the root’s password.

In Ubuntu, by default, this option is disabled since the root account itself is disabled.

In order to enable the root account with your password and then use the su command, you have to run the following command:

sudo passwd root

In other words, using the sudo command, you run, with root priviliges the command passwd (set a password for a user) on the root account.

Type twice your password and that’s all, you will be able to use the su command and swap to root account at your convenience.

Monitor Bandwidth utilization using shell

Reading Time: 2 minutes

This simple article illustrates how to monitor the network throughput using some simple shell applications.

 

Many times you might need to see what is the actual speed of your network connection or, more often, you just need network transfer rate info to know when a damn file transfer will complete.

As usual, this article is based on UBUNTU and therefore I use in my examples the DEB package manager (for RPM system, I  am sorry  )

 

Ok, let’s start, what we are going to use, two simple tools, nload and iftop.

 

Both nload and iftop are not shipped to you by the most common linux/unix distributions therefore, you need to install them.

To install the 2 applications, you need any shell program, in my examples I use bash, feel free obviously to use the same commands on whatever is your preferred shell program.

 

Install nload:

sudo apt-get install nload

nload is quite a small application, it takes less than 150 KB of disk space (in my case, with the current version in the Ubuntu’s applications repository).

 

Install iftop

sudo apt-get install iftop

iftop is even smaller than nload, less than 120 KB

 

Once you installed the 2 applications, you can easily use them.

Just remember that in order to use iftop, you need super user access rights.

 

>>> nload <<<

 

To load nload, just run the command:

nload

After that, you will get a screen like the following:

 

 

As you can see, nload just displays, using text mode graphs, the current throughput of the default NIC (in my case eth0) reporting inbound and outbound traffic).

I find this program awesome and I love it. It just does its job, in a perfect way, full stop.

 

 

 

>>> iftop <<<

 

Iftop application aims, from my perspective, to do something similar to the "top" command,

To load iftop, use the command:

sudo iftop

This should load the iftop application with a look similar to the following:

 

 

I love this application too, it makes easy the monitor of the bandwidth utilization and, in addition to that, it can also report the destinations of the network traffic.

 

 

Both applications are great, which one is the one to use, just depends on you and in the spefici, on your needs.