Opened 4 years ago

Closed 3 years ago

#4760 closed defect (expired)

Crash when EMM is enabled

Reported by: petrkr Owned by:
Priority: blocker Component: Reader
Severity: high Keywords:
Cc: Sensitive: no

Description

Revision

more versions

When the issue occurs

everytime when EMM from new version are enabled - does not have exact version, but version from like r11213 works but r11537 not. After like 5 - 10 minutes crash EMM of upstream OSCam server

How the issue is reproducable

have PCSC card oscam in version r11552, r11494 or r10943 and connect to them from ARM version r11537 (Octagon SF4008 and SF8008) not tested (yet) VU+

stack trace:
https://pastebin.com/vdkr8SEY

Attachments (4)

oscam-r11555-server.log (999.3 KB ) - added by petrkr 4 years ago.
oscam-r11556-client1.log (2.2 MB ) - added by petrkr 4 years ago.
oscam-r11558-server.log (44.4 KB ) - added by petrkr 4 years ago.
oscam-r11558-client1.log (261.7 KB ) - added by petrkr 4 years ago.

Change History (28)

comment:1 by theparasol, 4 years ago

If you ask me the issue is not the emm but this malformed invalid ecm:

02:20:19 41204145 c      (ecm) get cw for ecm:
02:20:19 41204145 c      (ecm)   80 00 00

comment:2 by petrkr, 4 years ago

Could be, but why it occurs only when AU is enabled and only from new version. If i'll use old oscam (from vu+) with AU enabled. Then it works for long time without any issue. But if I'll use AU from new version it will make all other oscams crash on buffer overflow (see stacktrace)

Setup:

Usb pcsc card -- oscam (lets say server1 and server 2) one server r10943 second latest trunk

Have 3 oscam clients

Vu+ old oscam r11213
Sf4008 new oscam r11537
Sf8008 new oscam

When vu+ has enabled AU everything works fine

When sf 4008 or 8008 enables AU.. within 5-10minutes both servers crash with buffer overflow.

strange is, if i'll try do 'man-in-middle' oscam like stb -- mim -- usb reader. Then that oscam in middle NOT crash, only that where cards are.

Also one is i686 a second is x86_64

Can i help somehow with debugging?

comment:3 by theparasol, 4 years ago

Try latest trunk if the issue is still there.
If still present go way back and work upwards to newer revision in attempt to pinpoint the exact revision that triggers the issue.

comment:4 by petrkr, 4 years ago

Hello, since I think it was something on sat side I tried recompile oscam since working version from VU+ (r11213) - which works on SF4008 fine too..

By compiling random versions between r11213 to latest turnk I figured out that problem was probably first introduced in commit r11287 (Add overflow protection to ECM and EMM pids). Since this commit it starts to do that problems. Latest working version is r11286.

comment:5 by theparasol, 4 years ago

11287 is for sure not the reason. Its even a buggy commit, please try to advance from 11288

comment:6 by theparasol, 4 years ago

update your servers to r11555, it got protection against empty ecm packets.
I dont care where they got from but it makes no sense to process them at all.

comment:7 by petrkr, 4 years ago

Will try.
I trying reproduce it whole night and it seems now it's like more random.

Also seems when I start client with -d255 it does not occurs (Murphy law in pactice...)... Now I've got another crash even with latest trunk (how to send securely files?)
I really love random problems :(

comment:8 by theparasol, 4 years ago

Luckily in coding there is no such thing as random or magic.

comment:9 by petrkr, 4 years ago

Some more logs:
https://pastebin.com/j00er2f0 - r11555 (server)
https://pastebin.com/6JJxs0dH - r11537 (client from stb store)

comment:10 by petrkr, 4 years ago

Well I need Murphy again... After update to latest version on stb too it works for 30 minutes without freeze...

I look like some commits between r11537 and r11556 fixes that issue (maybe?).

If that will work we can lower blocker to some high/medium. Because I think server should NOT crash when older client is connected to it. There are lot of old stb where you can not recompile latest version of OSCam. But for servers on Rpi/PC that possibility exists.

Still can do some more debugs and tests - if you suggest some what could cause that.

Last edited 4 years ago by petrkr (previous) (diff)

comment:11 by theparasol, 4 years ago

Added some more protection against zero length requests in dvbapi and in newcamd too.
Try r11557

Last edited 4 years ago by theparasol (previous) (diff)

by petrkr, 4 years ago

Attachment: oscam-r11555-server.log added

by petrkr, 4 years ago

Attachment: oscam-r11556-client1.log added

comment:12 by petrkr, 4 years ago

Well Murphy came... So it crashed again, but for now I had -d255 on both server and client. Server r11555 Client r11556:
Logs are attached

Later I will try 11557 on both, server + client

comment:13 by petrkr, 4 years ago

well

combinations:

cli : server : status : comment

r11557 : r11557 : not work : timeout/not found
r11556 : r11557 : not work : timeout/not found
r11557 : r11555 : works (but dnk about crash yet)

comment:14 by theparasol, 4 years ago

Well I successfully committed code that invalidated all ecm request presented to newcamd :(

Try r11558

comment:15 by petrkr, 4 years ago

meanwhile combination 15557 vs 15555 crashed again. (server side). On Client side there is NO any log message introduced for dvbapi ("Received filter data with section length 0 -> invalid length").

Server already have other "empty ecm"

15:49:32 2F5ED9C1 c  (newcamd) ncd_process_ecm: er->msgid=170 len=5 ecmlen=3
15:49:32 2F5ED9C1 c      (ecm) get cw for ecm:
15:49:32 2F5ED9C1 c      (ecm)   81 B0 00 
15:49:32 2F5ED9C1 c      (chk) 1234@000000 allowed by user 'testclient1' filter caid 1234 prid 000000
15:49:32 2F5ED9C1 c      (chk) trying server filter 1234@000000
15:49:32 2F5ED9C1 c      (chk) 1234@000000 allowed by server filter 1234@000000
15:49:32 2F5ED9C1 c      (chk) trying reader 'externi3irdeto' filter 1234@000000
15:49:32 2F5ED9C1 c      (chk) 1234@000000 allowed by reader 'externi3irdeto' filter 1234@000000
15:49:32 2F5ED9C1 c      (ecm) request_cw stage=2 to reader externi3irdeto ecm hash=D41D8CD98F00B204E9800998ECF8427E
15:49:32 2F5ED9C1 c     (work) start reader thread action 5
15:49:32 2F5ED9C1 c     (main) starting thread client work
15:49:32 2F5ED9C1 c     (main) client work thread started
15:49:32 7E671C6A r     (work) data from add_job action=5 client r externi3irdeto
15:49:32 7E671C6A r   (reader) externi3irdeto [irdeto] cardreader_do_checkhealth: reader->card_status = 2, ret = 1
15:49:32 7E671C6A r   (reader) externi3irdeto [irdeto] cardreader_do_ecm: cardreader_do_checkhealth returned rc=1
*** buffer overflow detected ***: /root/svn/oscam-svn-new/Distribution/oscam-1.20_svn11555-x86_64-pc-linux-gnu-pcsc terminated

by petrkr, 4 years ago

Attachment: oscam-r11558-server.log added

by petrkr, 4 years ago

Attachment: oscam-r11558-client1.log added

comment:16 by petrkr, 4 years ago

nope... still same.. logs attached

comment:17 by theparasol, 4 years ago

yes, you proved me wrong once again.
My assumption that sctlen reflected the size of the ecm was totally wrong.
Its always minimal 3 :(

Hope you are still in the mood to try r11559

Last edited 4 years ago by theparasol (previous) (diff)

comment:18 by petrkr, 4 years ago

Now trying r11558 attached to GDB but again... more than half-hour no crash. Hopefully after I will send this comment it will crash soon :)

Last edited 4 years ago by petrkr (previous) (diff)

comment:19 by petrkr, 4 years ago

Finally after 2 hours got backtrace (version r11558) Now I will try latest 60th version

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7f152be38700 (LWP 24980)]
0x00007f152c6a0457 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007f152c6a0457 in raise () from /lib64/libc.so.6
#1  0x00007f152c6a1659 in abort () from /lib64/libc.so.6
#2  0x00007f152c6dd239 in ?? () from /lib64/libc.so.6
#3  0x00007f152c75eaa6 in __fortify_fail () from /lib64/libc.so.6
#4  0x00007f152c75cdfc in __chk_fail () from /lib64/libc.so.6
#5  0x0000000000482d37 in memcpy (__len=18446744073709551615, __src=<optimized out>, __dest=0x7f152be377f5) at /usr/include/bits/string3.h:53
#6  irdeto_do_ecm (reader=0x1b5dca0, er=<optimized out>, ea=0x7f152be37c90) at reader-irdeto.c:692
#7  0x0000000000475524 in cardreader_do_ecm (reader=reader@entry=0x1b5dca0, er=er@entry=0x7f1518000e70, ea=ea@entry=0x7f152be37c90) at reader-common.c:468
#8  0x0000000000475631 in cardreader_process_ecm (reader=reader@entry=0x1b5dca0, cl=0x1b643a0, er=0x7f1518000e70) at reader-common.c:520
#9  0x00000000004bf366 in reader_get_ecm (reader=reader@entry=0x1b5dca0, er=<optimized out>) at oscam-reader.c:1171
#10 0x00000000004c35d4 in work_thread (ptr=<optimized out>) at oscam-work.c:292
#11 0x00007f152cc0932c in start_thread () from /lib64/libpthread.so.0
#12 0x00007f152c74f62d in clone () from /lib64/libc.so.6

comment:20 by theparasol, 4 years ago

This terrific log confirms my thoughts about what goes wrong.
There is no protection on malformed ecm on the reader code.
What happens is somehow a client sends a ecm that just contains the sctlen bytes and fully omits the needed ecmpayload.
For a result the memcopy will eventually copy large block of random data triggering the crash.
If I got it right this time from now on oscam is protected against such crafted ecms.

Last edited 4 years ago by theparasol (previous) (diff)

comment:21 by petrkr, 4 years ago

Now question is. Why 3 years old version does not send these malformed ECM data when AU is enabled. It must be introduced somewhere. But for confirm this I would need compile same "buggy" version for VU+ too to confirm/deny it's problem on OSCam side or STB's DVBAPI side.

comment:22 by theparasol, 4 years ago

I think its wise to add an ecmwhitelist for your reader. That way no ecm can be processed with wrong size.

Why nowadays wrong ecm data is fetched? If I remember correctly data received from the client is handled different. It attempts to read additional bytes until the transmitted data is successful completed. Perhaps the drivers of the box react funny to this and send zero's.

comment:23 by petrkr, 4 years ago

After 6 hours of version r11560 it does not crash. (both cli and server).

Also added to server debug message

Index: module-newcamd.c
===================================================================
--- module-newcamd.c    (revision 11560)
+++ module-newcamd.c    (working copy)
@@ -1333,6 +1333,11 @@
        }
 
        ecmlen = SCT_LEN((&buf[2]));
+
+       if (ecmlen < 4 ) {
+               cs_log_dbg(D_CLIENT, "ncd_process_ecm: ECMLEN Less than 4");
+       }

and tried older version on client... and I can confirm this message occurs sometimes but server not crashed... So it seems it will work.

server oscam-svn-new # tail -f /media/disk/oscam-r115560.log |grep -i "less than"
01:52:54 193644DD c  (newcamd) ncd_process_ecm: ECMLEN Less than 4
01:52:54 193644DD c  (newcamd) ncd_process_ecm: ECMLEN Less than 4

Regards

comment:24 by Opti, 3 years ago

Resolution: expired
Status: newclosed
Note: See TracTickets for help on using tickets.