Opened 4 years ago
Closed 3 years ago
#4760 closed defect (expired)
Crash when EMM is enabled
Reported by: | petrkr | Owned by: | |
---|---|---|---|
Priority: | blocker | Component: | Reader |
Severity: | high | Keywords: | |
Cc: | Sensitive: | no |
Description
Revision
more versions
When the issue occurs
everytime when EMM from new version are enabled - does not have exact version, but version from like r11213 works but r11537 not. After like 5 - 10 minutes crash EMM of upstream OSCam server
How the issue is reproducable
have PCSC card oscam in version r11552, r11494 or r10943 and connect to them from ARM version r11537 (Octagon SF4008 and SF8008) not tested (yet) VU+
stack trace:
https://pastebin.com/vdkr8SEY
Attachments (4)
Change History (28)
comment:1 by , 4 years ago
comment:2 by , 4 years ago
Could be, but why it occurs only when AU is enabled and only from new version. If i'll use old oscam (from vu+) with AU enabled. Then it works for long time without any issue. But if I'll use AU from new version it will make all other oscams crash on buffer overflow (see stacktrace)
Setup:
Usb pcsc card -- oscam (lets say server1 and server 2) one server r10943 second latest trunk
Have 3 oscam clients
Vu+ old oscam r11213
Sf4008 new oscam r11537
Sf8008 new oscam
When vu+ has enabled AU everything works fine
When sf 4008 or 8008 enables AU.. within 5-10minutes both servers crash with buffer overflow.
strange is, if i'll try do 'man-in-middle' oscam like stb -- mim -- usb reader. Then that oscam in middle NOT crash, only that where cards are.
Also one is i686 a second is x86_64
Can i help somehow with debugging?
comment:3 by , 4 years ago
Try latest trunk if the issue is still there.
If still present go way back and work upwards to newer revision in attempt to pinpoint the exact revision that triggers the issue.
comment:4 by , 4 years ago
Hello, since I think it was something on sat side I tried recompile oscam since working version from VU+ (r11213) - which works on SF4008 fine too..
By compiling random versions between r11213 to latest turnk I figured out that problem was probably first introduced in commit r11287 (Add overflow protection to ECM and EMM pids). Since this commit it starts to do that problems. Latest working version is r11286.
comment:5 by , 4 years ago
11287 is for sure not the reason. Its even a buggy commit, please try to advance from 11288
comment:6 by , 4 years ago
update your servers to r11555, it got protection against empty ecm packets.
I dont care where they got from but it makes no sense to process them at all.
comment:7 by , 4 years ago
Will try.
I trying reproduce it whole night and it seems now it's like more random.
Also seems when I start client with -d255 it does not occurs (Murphy law in pactice...)... Now I've got another crash even with latest trunk (how to send securely files?)
I really love random problems :(
comment:9 by , 4 years ago
Some more logs:
https://pastebin.com/j00er2f0 - r11555 (server)
https://pastebin.com/6JJxs0dH - r11537 (client from stb store)
comment:10 by , 4 years ago
Well I need Murphy again... After update to latest version on stb too it works for 30 minutes without freeze...
I look like some commits between r11537 and r11556 fixes that issue (maybe?).
If that will work we can lower blocker to some high/medium. Because I think server should NOT crash when older client is connected to it. There are lot of old stb where you can not recompile latest version of OSCam. But for servers on Rpi/PC that possibility exists.
Still can do some more debugs and tests - if you suggest some what could cause that.
comment:11 by , 4 years ago
Added some more protection against zero length requests in dvbapi and in newcamd too.
Try r11557
by , 4 years ago
Attachment: | oscam-r11555-server.log added |
---|
by , 4 years ago
Attachment: | oscam-r11556-client1.log added |
---|
comment:12 by , 4 years ago
comment:13 by , 4 years ago
comment:14 by , 4 years ago
Well I successfully committed code that invalidated all ecm request presented to newcamd :(
Try r11558
comment:15 by , 4 years ago
meanwhile combination 15557 vs 15555 crashed again. (server side). On Client side there is NO any log message introduced for dvbapi ("Received filter data with section length 0 -> invalid length").
Server already have other "empty ecm"
15:49:32 2F5ED9C1 c (newcamd) ncd_process_ecm: er->msgid=170 len=5 ecmlen=3 15:49:32 2F5ED9C1 c (ecm) get cw for ecm: 15:49:32 2F5ED9C1 c (ecm) 81 B0 00 15:49:32 2F5ED9C1 c (chk) 1234@000000 allowed by user 'testclient1' filter caid 1234 prid 000000 15:49:32 2F5ED9C1 c (chk) trying server filter 1234@000000 15:49:32 2F5ED9C1 c (chk) 1234@000000 allowed by server filter 1234@000000 15:49:32 2F5ED9C1 c (chk) trying reader 'externi3irdeto' filter 1234@000000 15:49:32 2F5ED9C1 c (chk) 1234@000000 allowed by reader 'externi3irdeto' filter 1234@000000 15:49:32 2F5ED9C1 c (ecm) request_cw stage=2 to reader externi3irdeto ecm hash=D41D8CD98F00B204E9800998ECF8427E 15:49:32 2F5ED9C1 c (work) start reader thread action 5 15:49:32 2F5ED9C1 c (main) starting thread client work 15:49:32 2F5ED9C1 c (main) client work thread started 15:49:32 7E671C6A r (work) data from add_job action=5 client r externi3irdeto 15:49:32 7E671C6A r (reader) externi3irdeto [irdeto] cardreader_do_checkhealth: reader->card_status = 2, ret = 1 15:49:32 7E671C6A r (reader) externi3irdeto [irdeto] cardreader_do_ecm: cardreader_do_checkhealth returned rc=1 *** buffer overflow detected ***: /root/svn/oscam-svn-new/Distribution/oscam-1.20_svn11555-x86_64-pc-linux-gnu-pcsc terminated
by , 4 years ago
Attachment: | oscam-r11558-server.log added |
---|
by , 4 years ago
Attachment: | oscam-r11558-client1.log added |
---|
comment:17 by , 4 years ago
yes, you proved me wrong once again.
My assumption that sctlen reflected the size of the ecm was totally wrong.
Its always minimal 3 :(
Hope you are still in the mood to try r11559
comment:18 by , 4 years ago
Now trying r11558 attached to GDB but again... more than half-hour no crash. Hopefully after I will send this comment it will crash soon :)
comment:19 by , 4 years ago
Finally after 2 hours got backtrace (version r11558) Now I will try latest 60th version
Program received signal SIGABRT, Aborted. [Switching to Thread 0x7f152be38700 (LWP 24980)] 0x00007f152c6a0457 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x00007f152c6a0457 in raise () from /lib64/libc.so.6 #1 0x00007f152c6a1659 in abort () from /lib64/libc.so.6 #2 0x00007f152c6dd239 in ?? () from /lib64/libc.so.6 #3 0x00007f152c75eaa6 in __fortify_fail () from /lib64/libc.so.6 #4 0x00007f152c75cdfc in __chk_fail () from /lib64/libc.so.6 #5 0x0000000000482d37 in memcpy (__len=18446744073709551615, __src=<optimized out>, __dest=0x7f152be377f5) at /usr/include/bits/string3.h:53 #6 irdeto_do_ecm (reader=0x1b5dca0, er=<optimized out>, ea=0x7f152be37c90) at reader-irdeto.c:692 #7 0x0000000000475524 in cardreader_do_ecm (reader=reader@entry=0x1b5dca0, er=er@entry=0x7f1518000e70, ea=ea@entry=0x7f152be37c90) at reader-common.c:468 #8 0x0000000000475631 in cardreader_process_ecm (reader=reader@entry=0x1b5dca0, cl=0x1b643a0, er=0x7f1518000e70) at reader-common.c:520 #9 0x00000000004bf366 in reader_get_ecm (reader=reader@entry=0x1b5dca0, er=<optimized out>) at oscam-reader.c:1171 #10 0x00000000004c35d4 in work_thread (ptr=<optimized out>) at oscam-work.c:292 #11 0x00007f152cc0932c in start_thread () from /lib64/libpthread.so.0 #12 0x00007f152c74f62d in clone () from /lib64/libc.so.6
comment:20 by , 4 years ago
This terrific log confirms my thoughts about what goes wrong.
There is no protection on malformed ecm on the reader code.
What happens is somehow a client sends a ecm that just contains the sctlen bytes and fully omits the needed ecmpayload.
For a result the memcopy will eventually copy large block of random data triggering the crash.
If I got it right this time from now on oscam is protected against such crafted ecms.
comment:21 by , 4 years ago
Now question is. Why 3 years old version does not send these malformed ECM data when AU is enabled. It must be introduced somewhere. But for confirm this I would need compile same "buggy" version for VU+ too to confirm/deny it's problem on OSCam side or STB's DVBAPI side.
comment:22 by , 4 years ago
I think its wise to add an ecmwhitelist for your reader. That way no ecm can be processed with wrong size.
Why nowadays wrong ecm data is fetched? If I remember correctly data received from the client is handled different. It attempts to read additional bytes until the transmitted data is successful completed. Perhaps the drivers of the box react funny to this and send zero's.
comment:23 by , 4 years ago
After 6 hours of version r11560 it does not crash. (both cli and server).
Also added to server debug message
Index: module-newcamd.c =================================================================== --- module-newcamd.c (revision 11560) +++ module-newcamd.c (working copy) @@ -1333,6 +1333,11 @@ } ecmlen = SCT_LEN((&buf[2])); + + if (ecmlen < 4 ) { + cs_log_dbg(D_CLIENT, "ncd_process_ecm: ECMLEN Less than 4"); + }
and tried older version on client... and I can confirm this message occurs sometimes but server not crashed... So it seems it will work.
server oscam-svn-new # tail -f /media/disk/oscam-r115560.log |grep -i "less than" 01:52:54 193644DD c (newcamd) ncd_process_ecm: ECMLEN Less than 4 01:52:54 193644DD c (newcamd) ncd_process_ecm: ECMLEN Less than 4
Regards
comment:24 by , 3 years ago
Resolution: | → expired |
---|---|
Status: | new → closed |
If you ask me the issue is not the emm but this malformed invalid ecm: