★ wanayoo — archive 1999 https://github.com/phpredis/phpredis/issues/951Nouvelle recherche | Portail wanayoo
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segfault in cluster mode (3.0.0) #951

Closed
vkirovskiy opened this issue Sep 19, 2016 · 6 comments
Closed

Segfault in cluster mode (3.0.0) #951

vkirovskiy opened this issue Sep 19, 2016 · 6 comments

Comments

@vkirovskiy
Copy link

@vkirovskiy vkirovskiy commented Sep 19, 2016

Hello everyone

We often receive segfaults like this

Sep 19 16:21:45 v-6-07-12-d5110-93 kernel: [10117415.046740] php-fpm7.0[33132]: segfault at 7fa76d53f498 ip 00007fa75abe1623 sp 00007ffd395eabd0 error 6 in redis.so[7fa75ab4e000+a3000]

Our enviroment:
ubuntu 14.04 LTS
php-fpm 7.0
php-redis 3.0.0 (compiled from pecl stable release)
Tried it on redis server 3.0.3, 3.2.0, 3.2.3

Redis is build in cluster mode, we have 2 setups, prod and stage.
Prod has 6 servers (3 masters and 3 slaves).
Stage has 6 redis instances on single server.
Segfaults appear on both of setups.

Backtrace

Program terminated with signal SIGSEGV, Segmentation fault.

0 0x00007fd26cbe1623 in cluster_shuffle_seeds (seeds=0x7fd266d85f88, len=0x7fffa6ae777c) at /vagrant/php-redis/cluster_library.c:862

862 /vagrant/php-redis/cluster_library.c: No such file or directory.
(gdb) bt full

0 0x00007fd26cbe1623 in cluster_shuffle_seeds (seeds=0x7fd266d85f88, len=0x7fffa6ae777c) at /vagrant/php-redis/cluster_library.c:862

   z_seeds = 0x7fd27bb094d0
   map = 0x7fd27bad3060
   i = 7
   count = 7
   index = 8

1 0x00007fd26cbe16bd in cluster_init_seeds (cluster=0x7fd27bb0f000, ht_seeds=0x7fd266d85f88) at /vagrant/php-redis/cluster_library.c:880

   redis_sock = 0x7fd27ba780e0
   str = 0x56201ee608b5 <destroy_op_array+757> "I�\204$�"
   psep = 0x7fd27ba15e40 "\030?�f�\177"
   key = "��\236}�\177\000\000`^�{�\177\000\000\230f�{�\177\000\000B��\036V\000\000@�$\037 V\000\000�D�{�\177\000\000\060^�{�\177\000\000\225��\036V\000\000�3�{�\177\000\000\060^�{�\177\000\000\002\000\000\000\000\000\000\000���\036 V\000\000�F#\037 V\000\000�F#\037 V\000\000 ]�{�\177", '\000' <repeats 18 times>, "Kt�\036 V\000\000\001\000\000\000\000\000\000\000�F#\037 V\000\000\000\000\000\000\000\000\000\000���\036 V\000\000\200\201�{�\177\000\000\220[�{�\177\000\000H", '\000' <repeats 15 times>...
   key_len = 0
   count = 0
   i = 0
   z_seeds = 0x7fd27ba15e40
   z_seed = 0x7fd27ba780e0

2 0x00007fd26cbc6a50 in redis_cluster_init (c=0x7fd27bb0f000, ht_seeds=0x7fd266d85f88, timeout=0, read_timeout=0, persistent=0) at /vagrant/php-redis/redis_cluster.c:353

No locals.

3 0x00007fd26cbc70a1 in zim_RedisCluster___construct (execute_data=0x7fd27ba15df0, return_value=0x7fd27ba15de0) at /vagrant/php-redis/redis_cluster.c:455

   object = 0x7fd27ba15e10
   z_seeds = 0x7fd27ba15e60
   name = 0x7fd2665d27f8 ""
   name_len = 0
   timeout = 0
   read_timeout = 0
   persistent = 0 '\000'
   context = 0x7fd27bb0f000

4 0x000056201ee5bafb in dtrace_execute_internal ()

No symbol table info available.

5 0x000056201eee4f64 in ?? ()

I guess it's a block of code where error is appeared (line 862)

/* Iterate over our source array and use our map to create a random list */
    for (zend_hash_internal_pointer_reset(seeds);
         zend_hash_has_more_elements(seeds) == SUCCESS;
         zend_hash_move_forward(seeds))
    {
        >> z_seeds[map[index++]] = zend_hash_get_current_data(seeds); <<
    }

Help please to determine the reason and resolve this issue.
Thanks!

Vladimir.

@vkirovskiy
Copy link
Author

@vkirovskiy vkirovskiy commented Nov 8, 2016

Additional debug info
I hope it will help you
If any additional info is needed ask please
It's still urgent for us
Thanks

static zval **cluster_shuffle_seeds(HashTable *seeds, int *len) {
    zval **z_seeds;
    int *map, i, count, index=0;

    /* How many */
    count = zend_hash_num_elements(seeds);

    /* Allocate our return value and map */
    z_seeds = ecalloc(count, sizeof(zval*));
    map = emalloc(sizeof(int)*count);

    /* Fill in and shuffle our map */
    for (i = 0; i < count; i++) map[i] = i;
    fyshuffle(map, count);

    /* Iterate over our source array and use our map to create a random list */
    for (zend_hash_internal_pointer_reset(seeds);
         zend_hash_has_more_elements(seeds) == SUCCESS;
         zend_hash_move_forward(seeds))
    {
        z_seeds[map[index++]] = zend_hash_get_current_data(seeds);
    }

    efree(map);

    *len = count;
    return z_seeds;
}

(gdb) p z_seeds[map[index++]]
Cannot access memory at address 0x7ffc418c28c8

(gdb) p count
$13 = 7

(gdb) p index
$4 = 8

(gdb) p *map@9
$6 = {0, 3, 6, 5, 4, 1, 2, -16747776, 0}

(gdb) p *seeds
$34 = {gc = {refcount = 2, u = {v = {type = 7 '\a', flags = 2 '\002', gc_info = 0}, type_info = 519}}, u = {v = {flags = 28 '\034', nApplyCount = 0 '\000', nIteratorsCount = 0 '\000', reserve = 0 '\000'}, flags = 28},
nTableMask = 4294967294, arData = 0x7f6322877388, nNumUsed = 7, nNumOfElements = 7, nTableSize = 8, nInternalPointer = 4294967295, nNextFreeElement = 7, pDestructor = 0x55f8c218dca0 <_zval_ptr_dtor>}

(gdb) p *z_seeds@9
$10 = {0x7f6322877388, 0x7f6322877408, 0x7f6322877428, 0x7f63228773a8, 0x7f63228773e8, >0x7f63228773e8, 0x7f63228773c8, 0x700000001, 0xfffffffe00000012}

@vkirovskiy
Copy link
Author

@vkirovskiy vkirovskiy commented Nov 14, 2016

Some investigation and workaround

Sometimes function zend_hash_has_more_elements(seeds) returns SUCCESS when no more elements are defined (or available). Unfortunately we have no resources and time to find the main reason.

Small patch.
We tested it and it's seems we didn't lose any functionality, all redis nodes are used, segfaults disappeared.

*** ./1/redis-3.0.0/cluster_library.c   2016-06-10 20:08:21.000000000 +0000
--- ./redis-3.0.0/cluster_library.c 2016-11-11 12:36:01.376468711 +0000
***************
*** 859,866 ****
--- 859,868 ----
           zend_hash_has_more_elements(seeds) == SUCCESS;
           zend_hash_move_forward(seeds))
      {
          z_seeds[map[index++]] = zend_hash_get_current_data(seeds);
+ 
+   if ( index == count ) break;
      }

      efree(map);
@yatsukhnenko
Copy link
Member

@yatsukhnenko yatsukhnenko commented Nov 14, 2016

@vkirovskiy, could you try this code?

-    for (zend_hash_internal_pointer_reset(seeds);
-         zend_hash_has_more_elements(seeds) == SUCCESS;
-         zend_hash_move_forward(seeds))
-    {
-        z_seeds[map[index++]] = zend_hash_get_current_data(seeds);
-    }
+    ZEND_HASH_FOREACH_VAL(seeds, z_ele) {
+        z_seeds[map[index++]] = z_ele;
+    } ZEND_HASH_FOREACH_END();

@vkirovskiy
Copy link
Author

@vkirovskiy vkirovskiy commented Nov 14, 2016

@yatsukhnenko sure! thanks for the answer.
Will compile and test it today

@yatsukhnenko
Copy link
Member

@yatsukhnenko yatsukhnenko commented Nov 14, 2016

@vkirovskiy BTW develop brunch is also support php7 and some of bugs already fixed

@vkirovskiy
Copy link
Author

@vkirovskiy vkirovskiy commented Nov 15, 2016

@yatsukhnenko Hi. Good patch, segfaults didn't appear after stress testing.
All nodes are used.
I found a bit failure connections, about 10 of 65000 connections was lost, but i guess it's cause of high connection rate.
Thank you for resolving this issue!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.