Using APC to speed up Magento

Alternative PHP Cache (APC) is a rather new and yet already popular technology in web development. APC is widely used as an opcode (operation code) cache engine or as they are commonly called – as PHP accelerator which means it stores compiled PHP byte code in server’s memory and thus speeds up work of any PHP script. But in fact APC could also be used as a usual cache engine, not just as accelerator.

Some people make grave mistake by trying to improve the speed of their Magento by installing a pack of different cache engines and accelerators and enabling them all together. At best, Magento will work a little faster, but it will consume a lot of memory as every caching engine will store its data there. At worst, Magento will become unstable and begin to fall in crashes in unpredictable ways.

Let's have a look at how Magento performance could be optimized. Magento can use two types of cache. The first type is an opcode cache (PHP accelerator). For this purposes APC, XCache or eAccelerator should be used. Not all of them – just one! This improves overall performance of any PHP script work on the server including Magento. As a result, the speed of Magento could increase many fold.

The second type of cache is Magento’s cache. Please note that in early versions of Magento there was no division between fast and slow caches – there was only one type of cache. Nowadays it is possible to set up two levels of cache via /app/etc/local.xml file. The fast level cache is a cache which is primarily used by Magento until it’s not depleted. There are many different cache engines which could be used for this purpose – APC, Memcached, etc.  As soon as there’s no available memory in system for the fast cache Magento begins to use the slow level cache. For the slow level cache a database or a file system are commonly used. By default Magento uses only file system for caching and that’s a rather slow solution.

Additional way to speed up Magento is to enable Magento’s Compiler. Basically, compiler just copies all the classes to one directory and merges some of them to a big `scope` files. Then Magento autoloads all the classes from this directory. Someone could say that while using PHP accelerators there is no reason to use compiler. It’s true except for one thing – accelerator stores all files of classes by the absolute path to a certain file. That means Magento will still have to calculate the path to a file on every class autoload before using the accelerator. Magento contains about 10 000 classes. If we presume that 10% of these classes are loaded on every launch of Magento, then about 1 000 of files should be loaded. And what if there are 10, 50, 100 users browsing the site at once? That’s quite a lot and the compiler helps to reduce time by removing this path calculating process.

As the most common and native PHP technology for the fast cache nowadays is the APC, let’s have a look at how to enable APC support for Magento. First of all it’s necessary to install the APC extension on the server. Some info on requirements and installation could be found on the official APC page on the php.net website. Now, when we have APC working on our server it will automatically cache all *.php files after they were interpreted to bytecode.

It is possible to check if APC stores all the PHP files correctly by using the apc.php utility script which could be found in any APC package from here. Edit this script and change administrator’s username and password at the beginning of the script. The first page of this script shows some cache statistics and main settings’ values and percentage of APC. If there is a message like “No cache info available. APC does not appear to be running.” on this page instead of cache info then it’s necessary to check whether APC extension for PHP was installed properly. Also ensure that apc.enabled runtime parameter in extension’s configuration is set to 1, as disabling this option will also show this “No cache info…” error at apc.php. When cache info shows up, there will be a couple of diagrams showing percentage of hits/misses while using cache and amount of memory dedicated to and used by APC. Every reload of this page should increase the number of hits and thus bring up the percentage of hits. If the cache is empty while running apc.php for the first time there will be used ~350Kb – size of the apc.php script itself. Now we should log in to the script using the button in the upper-right corner of the script and credentials we had set in the script’s code.

Host Status Diagram

APC after the first run

There are two main diagrams in the apc.php script – Host Status Diagrams. The left diagram shows APC memory usage, amount of free and used memory. It also shows fragmentation of memory. The right diagram shows the number of successful calls to cache. If some .php script or some data entry is being called from cache and not found then this attempt marked as “miss”. Then most likely this file will be included from the file system and will be cached or some data entry will be stored in cache for future use. Successful attempts to get some data from cache are called “Hits”. The ideal condition for this diagram is ~99% hits and ~1% misses – that means all data was cached once, then always called from cache and there are no cache flushes.

There are two important sections in the top menu of this script - `System Cache Entries` and `User Cache Entries`. In the first one we can see all .php files cached by APC on this moment, numbers of hits/misses and some timing information for these files. Clicking some file name will point us to the page with some detailed info on this file. If this section is empty at this moment, then it seems we have a problem – APC either does not cache files or does not shares it between PHP requests. Some info on this problem could be found at the end of this article and it’s necessary to solve it to move forward. But for now let’s presume we have this section working and there are some files. The second, `User Cache Entries` section should be empty for now – it should show entries of data which are stored at APC using usual memory cache not an opcode cache.

Host Status Diagram with Cache Enabled.

normal APC stats

To begin using this type of APC cache we should make some modifications to /app/etc/local.xmlfile. It’s necessary to add the following rows under the `global` node:

<global> 
    ...
    <cache> 
        <backend>apc</backend> 
        <prefix>MY_STORE_NAME_</prefix> 
    </cache>
    ...
</global>

That’s it! Now we should clear Magento’s cache at the `System->Cache Management` section of the Magento backend. If we take a look at the `User Cache Entries` section of the apc.php script now, we should see some entries of data there. Clicking the entry will point us to the details of this entry, including all the data stored inside the entry.

There is also another modification which could be made to the /app/etc/local.xml file – we could setup slow cache type. If it is not set directly, Magento will use file system to store cache when the limit of memory dedicated to APC is exceeded. The second option is to use database as a slow cache storage – it’s necessary to add <slow_backend>database</slow_backend>line under the `cache` node like

<global>
    ...
    <cache>
        <backend>apc</backend>
        <slow_backend>database</slow_backend>
        <prefix>MY_STORE_NAME_</prefix>
    </cache>
    ...
</global>

 

Without this parameter Magento will use the file system to store slow cache. While this is not the best option to store cache in the database, but it is faster than storing it in file system. You may ask why to use slow cache at all if there is APC on the server. The sad truth is that Magento always writes data to BOTH fast and slow caches. So if a slow cache type is not defined at the local.xml file then Magento will use file system to store data and this will decrease Magento’s load speed. And there’s no obvious way to stop this process, but there’s a little trick of how to prevent Magento from writing slow cache:

<global>
    ...
    <cache>
        <backend>apc</backend>
        <slow_backend>database</slow_backend>
        <slow_backend_store_data>0</slow_backend_store_data>
        <auto_refresh_fast_cache>0</auto_refresh_fast_cache>
        <prefix>MY_STORE_NAME_</prefix>
    </cache>
    ...
</global>

 

This trick works only for database slow type cache. In this case Magento will NOT store anything at the database neither will it use the file system. Note that if APC  exceeds its memory limit,  it will flush the entire cache. While this is not the worst thing that might happen with Magento, there is one problem: if this happens once, these flushes will repeat in a short time thus decreasing the advantage of APC usage. In such case Magento may become even slower than without APC at all. So it’s vital to set such memory limit for APC to always have some free memory for cache. For example, allocate an additional 20% of memory for APC if you see that the cache is almost full.

Now let’s have a look at some problems which could appear while setting up or use of APC. We presume that APC is up and running on the server and the apc.php script is accessible.

1. The storage is always empty no matter how we are using it – for opcode, or just for cache.

There is one common issue which can cause this situation. Most likely PHP on the server works in the Fast-CGI mode. In this mode parallel PHP processes don’t have access to each other’s data stored at APC. This is an old APC/PHP bug and there is no good solution for this problem: either change PHP mode or use other cache engine.

2. `Unable to allocate memory for pool` error in logs and crashes of Magento with this error.

There are few things which could cause this error, and most of them are related to incorrect APC configuration. First of all – there might be just no free memory to store some file/data and meanwhile apc.ttl system option isn’t set to zero. This setting regulates the behavior of an APC when it runs out of memory. Basically and by default it is set to zero which means the entire cache should be flushed in this situation. If this setting is changed to some other value e.g. 3600 (seconds), then only entries which are idle (neither new nor accessed for the specified period of time) in the last hour will be flushed. While setting this option to non-zero value may increase the speed of an APC work by eliminating unnecessary cache flushes, and it is recommended to set it this way, it is also recommended to increase the memory limit for APC to prevent unnecessary flushes at all.

Another source of the problem is incorrect memory allocation for APC. If apc.shm_size is set for example to 512Mb and there’re only 256Mb of shared memory available for APC then this error may also occur.

3. `Cannot redeclare class` errors.

Most likely something in the code calls for a certain PHP files by including these files using include_once function with relative paths. APC stores files by their paths. If one file called twice with two different relative paths, then even while using include_once APC will load the same file/class twice. The second situation which can cause this issue is when some extension loads some files/classes separately from autoloader. If it loads a class before an autoloader – everything is ok, autoloader will not load the file again. But if it loads a class AFTER an autoloader – there’s a high risk of redeclaration error to appear. While Magento itself cannot create such problem, some  third-party extensions can.

4. apc.php script shows a high level of cache fragmentation.

This problem could also be caused by apc.ttl option if the memory limit is low. Let’s presume we have set this option to 3600 seconds. When APC will run out of memory it will flush all files unused in the last hour, i.e. 20-30% of files. These files were located not in a certain part of the memory, but were distributed by the order in which they were cached. Now we have 20-30% of free memory, but this space is represented by small areas of free memory equal to the size of files which were stored at these areas before flush. Now we are trying to cache some new file – APC will split this file to few parts if it cannot store this file in the first free fragment of the memory. That is the cache fragmentation. The solution for this problem is to allocate enough memory for APC to prevent data flushes at all. Generally this problem is always caused by replacing data in cache no matter what exactly have caused this replacement. It could be data flushes or usage of some function like apc_compile_file which forces files to re-compile. The basic recommendation is to check and increase the memory limit for APC and if the fragmentation is not finished, then it’s a good reason to have a look at extensions code to find the reason of fragmentation.

Detailed Memory Usage

0% fragmentation - ideal condition

 

5. Why the speed of Magento hasn’t increased dramatically after enabling APC?

 There are many reasons for this and there are some recommendations on how to increase the speed. First of all check the apc.php script – does APC have already cached any data at this moment at all? We have described this situation earlier and this is a rather common problem. Then please check the number of hits/misses and the capacity of the cache. If the cache has many misses and their number grows quickly while the cache is at full capacity, then it’s obvious that overall speed of the cache is decreased by permanent replacement of some files because cache overflow. The similar problem is when cache often flushes itself because of overflowing. The only difference between these two cases is a value of the apc.ttl option. The solution to both problems is to increase the memory limit for APC.

 And some words about APC settings optimization. As we have mentioned many times in this article – the most vital optimization for APC is allocating enough memory to store all the data for an opcode and usual cache types and even to have some free memory (approximately 20-30%). To know how much memory your Magento really uses it's a good idea to allocate a large amount of memory to APC (e.g. 512Mb) and then check a couple of times the real usage of cache in a day's period. And then decrease the memory limit to a reasonable value. So set apc.shm_segments = 1 and apc.shm_size to have some free buffer. Set apc.ttl to some value above zero – this will prevent APC from flushing all the data if there’s no free memory.

It is also a good practice to use APC compiled with a MMAP (Memory Mapping) support, not with SHM (SysV Shared Memory). Furthermore this practice is recommended by APC developers. In this mode APC will use another mechanism of memory allocating which reclaims memory faster and there’s almost no limit on resources with this option. But if you are using shared memory anyway then it’s useful to know how to set up some memory to be shared. To set e.g. 128Mb of shared memory once you can use the following command:

    sysctl -w kernel.shmmax= 134217728

To save this setting between server restarts you can use the following commands:

    echo 'kernel.shmmax= 134217728' >> /etc/sysctl.conf
    sysctl -p

 On production environment set apc.stats to zero – this will prevent APC from automatic checks of each file’s status. While this option is enabled APC will check on each Magento load each cached file for possible changes and re-cache it if a file was changed. As Magento contains about 10000 classes disabling these checks may lead to a significant speed boost. Please note that to refresh the cache while apc.stats is disabled it’s necessary to flush cache manually via apc.php or restart the server.

Set apc.max_file_size = 5M, apc.num_files_hint = 10000 this will allow APC to store large files (e.g. big files created by Magento compiler) and will increase speed of APC a little by “hinting” to it how many PHP files approximately there are in the system.

In conclusion, there are many different recommendations for APC setting on the internet and we don't want to argue which one is better or worse. And this is our version of such a recommendation for a production server. If you have some advice or proposals on how to make this configuration better – we will gladly discuss them in comments.

/app/etc/local.xml:

<global>
    ...
    <cache>
        <backend>apc</backend>
        <slow_backend>database</slow_backend>
        <slow_backend_store_data>0</slow_backend_store_data>
        <auto_refresh_fast_cache>0</auto_refresh_fast_cache>
        <prefix>MY_STORE_NAME_</prefix>
    </cache>
    ...
</global>

APC settings:

apc.enabled = 1
apc.cache_by_default = 1
apc.shm_segments = 1
apc.shm_size = [real consumption of certain Magento installation +20-30%]
apc.mmap_file_mask = /tmp/apc.XXXXXX
apc.ttl = 7200
apc.user_ttl  = 7200
apc.num_files_hint = 10000
apc.max_file_size = 5M
apc.stat = 0   #set to 1 during development
Migrating the Mail System to a New Server: Unexpected Problems. Previous Post
Speed Up Magento 2 Times in 3 Simple Steps Next Post