Guide to PHP Security

Quicksearch

Calendar

Comments

Mike about PHP's Output Buffering

Mon, 12.12.2011 05:12

How about a PHP_OUTPUT_HANDLER _ABSORBENT flag for trunk? ob_start(null, 1024, PHP_OUTPU T_HANDLER_STDFLAGS | PHP [...]

gphilip about PHP's Output Buffering

Mon, 12.12.2011 03:56

Hi, "What was unusual is th at 16 calls to ob_start() up c hewing through almost 700kb of memory" Can you give [...]

Ilia Alshanetsky about PHP's Output Buffering

Thu, 08.12.2011 11:49

In PHP 5.4 the default buffer size was reduced to 0x4000 (16 kb), which in part is responsi ble for the performance [...]

Chris Jones about PHP's Output Buffering

Thu, 08.12.2011 10:00

An error is thrown in PHP 5.4 but not PHP 5.3: $ cat t.ph p $ php54 t.php PHP Warn ing: ob_start(): no arr [...]

Chris Jones about PHP's Output Buffering

Thu, 08.12.2011 09:55

For people not following PHP d evelopment, output buffering c hanged a lot in PHP 5.4. Runni ng this: $a = microti [...]

Sebastian about PHP's Output Buffering

Thu, 08.12.2011 06:04

I dont understand your post be cause the first parameter for ob_start is not the chunk size , it is the callback fun [...]

Ilia Alshanetsky about PHP's Output Buffering

Thu, 08.12.2011 05:21

You can try the test at: http ://pastebin.com/Kjpmea3h in this test the new code is abo ut 25% faster, not to me [...]

Ilia Alshanetsky about PHP's Output Buffering

Thu, 08.12.2011 04:19

Think of a templating system t hat needs to generate various sub-templates and place result ing data in various diff [...]

Ilia Alshanetsky about PHP's Output Buffering

Thu, 08.12.2011 04:10

There is a really good reason why it is slower. Your text is quite big, but you set your i nitial buffer size to me [...]

Ilia Alshanetsky about PHP's Output Buffering

Thu, 08.12.2011 04:05

You are correct, it should say 10kb not 20... I'll adjust.

Wednesday, December 7. 2011

Posted by Ilia Alshanetsky in PHP

Comments (13)
Trackbacks (0)

PHP's Output Buffering

While profiling our application I came across a a rather strange memory usage by the ob_start() function. We do use ob_start() quite a bit to defer output of data, which is a common thing in many applications. What was unusual is that 16 calls to ob_start() up chewing through almost 700kb of memory, given that the data being buffered rarely exceeds 1-2kb, this was quite unusual.

I started looking at the C code of the ob_start() function and found this interesting bit of code inside php_start_ob_buffer()
initial_size = 40*1024;
block_size = 10*1024;

Which directs PHP to pre-allocate 40kb of data for each ob_start() call and when this proves to be insufficient, increase by 10kb each time. Ouch!

PHP does allow you to say how much memory ob_start() can use, via 2nd parameter to the function. However, if you exceed that size, PHP will promptly flush the captured data to screen, which means that unless you are really good at predicting your buffer sizes or vastly overestimate, there is a risk that the data will be dumped to screen by PHP if you use this option.

Since I am not really good at guessing, I've decided to make a small, backwards compatible tweak to PHP's code that allow specification of custom buffer sizes, but allow the buffer size to be increased if the initial buffer size proves to be insufficient, ensuring that the data can be safely buffered. This functionality is implemented through a change (see patch below) to the 1st parameter of the ob_start() function, which normally is used to provide the callback function. With the patch in place the parameter, can be a number, which defines the desired buffer size. With the patch, ob_start(1024) means that the 1kb buffer should be used and when it is exceed keep allocating 1kb at a time to allow for additional data to be stored. This solution does mean you cannot use custom, resizable buffer sizes with a callback function, however it does provider a backwards (PHP API wise) compatible way of implementing the functionality in PHP 5.2 and 5.3.

Here is a simple before & after example:

PHP:



<?php



     ob_start(null, 1024);

     echo str_repeat("a", 1500);

     var_dump(strlen(ob_get_clean()));



?>

will print 0, since buffer is exceeded and is flushed to screen

PHP:



<?php



     ob_start(1024);

     echo str_repeat("a", 1500);

     var_dump(strlen(ob_get_clean()));



?>

will print 1500, buffer was exceeded and subsequently increased, allowing data to remain in the buffer

CODE:

Index: main/output.c
===================================================================
--- main/output.c (revision 320624)
+++ main/output.c (working copy)
@@ -155,10 +155,14 @@
initial_size = (chunk_size*3/2);
block_size = chunk_size/2;
} else {
- initial_size = 40*1024;
- block_size = 10*1024;
+ if (output_handler && Z_TYPE_P(output_handler) == IS_LONG && !chunk_size) {
+ initial_size = block_size = Z_LVAL_P(output_handler);
+ } else {
+ initial_size = 40*1024;
+ block_size = 10*1024;
+ }
}
- return php_ob_init(initial_size, block_size, output_handler, chunk_size, erase TSRMLS_CC);
+ return php_ob_init(initial_size, block_size, (output_handler && Z_TYPE_P(output_handler) != IS_LONG ? output_handler : NULL), chunk_size, erase TSRMLS_CC);
}
/* }}} */

Trackbacks

Trackback specific URI for this entry

No Trackbacks

Comments

Display comments as (Linear | Threaded)

Thanks for the post! Maybe I'm reading it wrong, but the snippet for the block size says 10*1024, but in the body of the post you say the block size is 20 kb.

#1 Sam Hennessy on 2011-12-07 19:08 (Reply)

You are correct, it should say 10kb not 20... I'll adjust.

#1.1 Ilia Alshanetsky (Homepage) on 2011-12-08 04:05 (Reply)

Co solution but what is he use-case for using ob so extensively?

#2 Evan (Homepage) on 2011-12-07 19:50 (Reply)

Think of a templating system that needs to generate various sub-templates and place resulting data in various different sections of the page.

#2.1 Ilia Alshanetsky (Homepage) on 2011-12-08 04:19 (Reply)

Hi! i've tested this with a little script: http://pastebin.com/w27a5MXP

Seems that the execution time increase if you set the chunk size.

Tests
- without chunk size ~0.1 ms
- with chunk size ~2.0 ms

But test it your self

#3 Sebastian on 2011-12-08 02:20 (Reply)

There is a really good reason why it is slower. Your text is quite big, but you set your initial buffer size to mere 1kb, so PHP needs to keep resizing the buffer resulting in a slow down due to memory re-allocation. If you set the initial to a size more reflective of your output size you will see a performance benefit.

As it stands you have a text size that is 3.2kb in size that you display 50 times for a total of 150+kb. With default allocation schema there are 12 re-allocations, with 1kb buffer there are 150 (10x more), hence the speed drop.

#3.1 Ilia Alshanetsky (Homepage) on 2011-12-08 04:10 (Reply)

You can try the test at:
http://pastebin.com/Kjpmea3h

in this test the new code is about 25% faster, not to mention more memory efficient.

#3.2 Ilia Alshanetsky (Homepage) on 2011-12-08 05:21 (Reply)

I dont understand your post because the first parameter for ob_start is not the chunk size, it is the callback function. It seems to be that ob_start doest not throw a error if callback is not valid.

Example: ob_start('asdf'); (function not av.)

#3.2.1 Sebastian on 2011-12-08 06:04 (Reply)

An error is thrown in PHP 5.4 but not PHP 5.3:

$ cat t.php

$ php54 t.php
PHP Warning: ob_start(): no array or string given in /tmp/t.php on line 3

Ilia's patch to PHP 5.3 allowed the first parameter to be overloaded.

#3.2.1.1 Chris Jones (Homepage) on 2011-12-08 10:00 (Reply)

For people not following PHP development, output buffering changed a lot in PHP 5.4. Running this:

$a = microtime(1);
for ($i = 0; $i < 100000; $i++) {
ob_start();
echo str_repeat("a", 1024);
ob_end_clean();
}
var_dump(microtime(1) - $a);
echo memory_get_peak_usage(true), "\n";

against an unpatched PHP 5.3 and PHP 5.4 gives:

$ php53 test.php
float(0.66586494445801)
524288
$ php54 test.php
float(0.29983496665955)
262144

#4 Chris Jones (Homepage) on 2011-12-08 09:55 (Reply)

In PHP 5.4 the default buffer size was reduced to 0x4000 (16kb), which in part is responsible for the performance improvement.

5.4 still suffers from the limitation that you cannot specify a custom buffer size, and have it be automatically increased when the data in the buffer exceeds the buffer size.

#4.1 Ilia Alshanetsky (Homepage) on 2011-12-08 11:49 (Reply)

Hi,

"What was unusual is that 16 calls to ob_start() up chewing through almost 700kb of memory"

Can you give more information please under what circumstances it happened?

http://pastebin.com/XPhtzpAh

I think ob_get_clean takes care of freeing the memory right away, so an ob_call in one request can result in max 40kB "extra peak", which is nothing in case of a regular webapplication.

#5 gphilip on 2011-12-12 03:56 (Reply)

How about a PHP_OUTPUT_HANDLER_ABSORBENT flag for trunk?

ob_start(null, 1024, PHP_OUTPUT_HANDLER_STDFLAGS | PHP_OUTPUT_HANDLER_ABSORBENT);

Though, due to current implementation, PHP_OUTPUT_HANDLER_ALIGN_TO_SIZE, i.e. 4k, is the allocation barrier.

#6 Mike on 2011-12-12 05:12 (Reply)

Add Comment

Name
Email
Homepage
In reply to
Comment	Standard emoticons like :-) and ;-) are converted to images. E-Mail addresses will not be displayed and will only be used for E-Mail notifications.
	Remember Information? Subscribe to this entry