Skip to content

Navigation Menu

Sign in
Sign up

How to se threefive to validate the CRC_32 of a SCTE35 message #102

Answered by futzu
fxengelbrecht asked this question in Q&A
Discussion options

Hi.

How would you go about validating the CRC_32 of a SCTE35 message?
As an example see /DA3AAAAAAAAAP/wDwUAAAAAf//+AClWCAIrAAAAFwIVQ1VFSQAAAAB/yAAAKTLgAQEwMAAAlDpDfA==
This is a splice_insert message with a segmentation_descriptor.
threefive shows the CRC_32 as: "crc": "0x943a437c" as found in the SCTE35 instance. However the correct CRC_32 is 0x9c216676 if you calculate the CRC_32 based on the content of the message. (This according to the middleman.tv online SCTE35 parser)
I would like to parse a list of SCTE35 and determine which are correct, vs. those with bad CRC_32.
Any guidance would be appreciated.
You must be logged in to vote

I was worried about false negatives. As I mentioned, threefive re-calculates everything it can when encoding, and also fixes several common errors I've seen over and over. I decided to check the crc32 from the raw bites, much like you thinking with the hex.

a@fu:~$ python3
>>> from threefive import Cue
>>> cue=Cue('/DA3AAAAAAAAAP/wDwUAAAAAf//+AClWCAIrAAAAFwIVQ1VFSQAAAAB/yAAAKTLgAQEwMAAAlDpDfA==')
>>> cue.decode()
True
>>> cue.info_section.crc
'0x943a437c'
>>> from threefive.crc import crc32
>>> hex(crc32(cue.bites[:-4]))
'0x943a437c'
>>> cue.info_section.descriptor_loop_length
23
>>> cue.encode()
'/DA5AAAAAAAAAP/wDwUAAAAAf//+AClWCAIrAAAAGQIXQ1VFSQAAAAB/yAAAKTLgAQEwMAAAAACRUPll'
>>> cue.i...

Replies: 4 comments 13 replies

Comment options

Let me ask you this, what do you plan on doing when you find a bad crc? How would you handle it and why would you handle it?
What do you perceive as the danger? I'm just curious.
You must be logged in to vote
1 reply
Comment options

I need to inform the upstream service provider that the SCTE35 is malformed. No idea yet on the impact to downstream entities ingesting this content, I just noticed this issue...

As for the validation of the CRC, I thought the following may work.

  1. Convert the base64 encode string to hex string
  2. Use threefive to decode to json
  3. Use threefive to encode the json to hex string.
  4. If the last 8 digits in the hex string from step 3 differs from the last 8 digits of the hex string in step 1 then the CRCs are different and the CRC of the source SCTE35 is incorrect.
    Will test this procedure this weekend.
Comment options

Sorry, I was such a dick yesterday, I assumed you giving me shit and I just wasn't in the mood. I apologize.

Just decode and then encode man, see if the strings match. When threefive encodes, it calculates everything that it can, lengths, types, and crc32. Here's all you have to do.

>>>> from threefive import Cue
>>>> c = Cue('/DAgAAAAAAAAAP/wDwUAAAEDf0//MqD0yAEDAAAAAHXkvig=')
>>>> c.decode()
True
>>>> c.encode()
'/DAgAAAAAAAAAP/wDwUAAAEDf0//MqD0yAEDAAAAAHXkvig='

here's the Cue you had

>>>> cue = Cue('/DA3AAAAAAAAAP/wDwUAAAAAf//+AClWCAIrAAAAFwIVQ1VFSQAAAAB/yAAAKTLgAQEwMAAAlDpDfA==')
>>>> cue.decode()
True
>>>> cue.encode()
'/DA5AAAAAAAAAP/wDwUAAAAAf//+AClWCAIrAAAAGQIXQ1VFSQAAAAB/yAAAKTLgAQEwMAAAAACRUPll'
You must be logged in to vote
0 replies
Comment options

I was worried about false negatives. As I mentioned, threefive re-calculates everything it can when encoding, and also fixes several common errors I've seen over and over. I decided to check the crc32 from the raw bites, much like you thinking with the hex.

a@fu:~$ python3
>>> from threefive import Cue
>>> cue=Cue('/DA3AAAAAAAAAP/wDwUAAAAAf//+AClWCAIrAAAAFwIVQ1VFSQAAAAB/yAAAKTLgAQEwMAAAlDpDfA==')
>>> cue.decode()
True
>>> cue.info_section.crc
'0x943a437c'
>>> from threefive.crc import crc32
>>> hex(crc32(cue.bites[:-4]))
'0x943a437c'
>>> cue.info_section.descriptor_loop_length
23
>>> cue.encode()
'/DA5AAAAAAAAAP/wDwUAAAAAf//+AClWCAIrAAAAGQIXQ1VFSQAAAAB/yAAAKTLgAQEwMAAAAACRUPll'
>>> cue.info_section.crc
'0x9150f965'
>>> cue.info_section.descriptor_loop_length
25

One of the more common mistakes that threefive corrects is when people leave off sub_segment_num and'sub_segments_expected from a segmentation descriptor, they are used depending on the value of segmentation_type_id.
sub_segment_num and sub_segments_expected are each 1 byte.
That's why the descriptor_loop _length was 23 before encoding and 25 after, threefive adds them before encoding.

  • So to recap,
    • middleman.tv is wrong in multiple ways,and I can't explain their results.
    • The crc for the cue from your provider is correct for what they encoded, but they did not include sub_segment_num, and sub_segments_expected in the splice descriptor when they encoded it.

I just saved you a month worth of aggravation trying to figure that one out. You owe me a favor.

try this instead:

Updated!

#!/usr/bin/env python3
from threefive import Cue
from threefive.crc import crc32
cue_list = [
 "/aBadCue",
 "/DAgAAAAAAAAAP/wDwUAAAEDf0//MqD0yAEDAAAAAHXkvig=",
 "/DA3AAAAAAAAAP/wDwUAAAAAf//+AClWCAIrAAAAFwIVQ1VFSQAAAAB/yAAAKTLgAQEwMAAAlDpDfA==",
 "/DA3AAAAAAAAAP/wDwUAAAAAf//+AClWCAIrAAAAFwIVQ1VFSQAAAAB/yAAAKTLgAQEwMAAAlDpDfA==",
]
def vrfy(data):
 stats = {
 "decoded": False,
 "valid_crc": False,
 "complete": False,
 }
 cue = Cue(data)
 try:
 cue.decode()
 except:
 return data, stats
 stats["decoded"] = True
 in_crc = cue.info_section.crc
 stuff = cue.bites[: len(cue.bites) - 4]
 stats["valid_crc"] = (False, True)[hex(crc32(stuff)) == in_crc]
 cue.encode()
 stats["complete"] = (False, True)[cue.info_section.crc == in_crc]
 return data, stats
def do(cue_list):
 """
 do return a dict of cue, valid
 """
 cue_list = set(cue_list) # remove duplicates
 results = {out[0]: out[1] for out in [vrfy(data) for data in cue_list]}
 return results
if __name__ == "__main__":
 print(do(cue_list))
  • output
a@fu:~$ pypy3 vrfy.py 
{'/DAgAAAAAAAAAP/wDwUAAAEDf0//MqD0yAEDAAAAAHXkvig=': {'decoded': True, 'valid_crc': True, 'complete': True}, '/DA3AAAAAAAAAP/wDwUAAAAAf//+AClWCAIrAAAAFwIVQ1VFSQAAAAB/yAAAKTLgAQEwMAAAlDpDfA==': {'decoded': True, 'valid_crc': True, 'complete': False}}

 
 

 
 

 

 

 
 

 

 

 

 
 

 

  You must be logged in to vote
 

 

 

 
 

 

 

    

 
 

 

 

 

 
 8 replies
 
 

 

 

 

 

 

 

 
 

 
 

 
 
 
 




 

 
 

 

 

 

 

 


 

 

 

 
 
 Comment options
 

 
 


 
 
 

 

 

 

 
 

 

 

 



 

 

 
 
 
 
 
 
Where did you get that from, middleman?


If it didn't come from me or directly from the specification, it's not relevant and I don't care to hear it.

I had so much trouble trying to verify threefive's results using other SCTE-35 parsers, that I wrote a completely

different parser in Go called cuei.

Trust threefive until you have a reason not to, and then come tell me that reason and I'll fix it.

 
 

 
 

 

 

 

 
 

 

 
 
 

 

 

 

 

 

 

 
 

 


 



 

 

 
 

 

 

 

 

 


 

 

 

 
 
 Comment options
 

 
 


 
 
 

 

 

 

 
 

 

 

 



 

 

 
 
 
 
 
 
You're creating your own problems. Had you not been listening to middleman, none of this would have been an issue. This is why I get angry about online parsers, it's always a big waste of time. We wasted hours on this.

 
 

 
 

 

 

 

 
 

 

 
 
 

 

 

 

 

 

 

 
 

 


 



 

 

 
 

 

 

 

 

 


 

 

 

 
 
 Comment options
 

 
 


 
 
 

 

 

 

 
 

 

 

 



 

 

 
 
 
 
 
 
I prefer using threefive, however, when a vendor tells me "but middleman says..." then I need to verify.


As for the statement "sub_segment_num and sub_segments_expected are only included in the SCTE35 message if segmentation_type_id in [0x34, 0x36, 0x38, 0x3A]. " I got that from the SCTE35 spec. As an example, in the 2022 edition, table 20 on page 59 the pseudo code for the segmentation_descriptor shows:

...

segment_num 8 uimsbf

segments_expected 8 uimsbf

if(segmentation_type_id == ‘0x34’ || segmentation_type_id == ‘0x36’ || segmentation_type_id == ‘0x38’ || segmentation_type_id == ‘0x3A’) {

sub_segment_num 8 uimsbf

sub_segments_expected 8 uimsbf

}

 
 

 
 

 

 

 

 
 

 

 
 
 

 

 

 

 

 

 

 
 

 


 



 

 

 
 

 

 

 

 

 


 

 

 

 
 
 Comment options
 

 
 


 
 
 

 

 

 

 
 

 

 

 



 

 

 
 
 
 
 
 
image 

 
 

 
 

 

 

 

 
 

 

 
 
 

 

 

 

 

 

 

 
 

 


 



 

 

 
 

 

 

 

 

 


 

 

 

 
 
 Comment options
 

 
 


 
 
 

 

 

 

 
 

 

 

 



 

 

 
 
 
 
 
 
    

  • 

    Here's the Spec
    
image 
    

    • 

  • 

    Here's middleman's UPID for that Cue
    

    • 



 {
 segmentation_upid_type: "UserDefined", <!---- Type 1 has been deprecated, this is old.
 segmentation_upid_length: "0x1", <!------- if the length is one, 
 Data: "MA==", <!------why is the data 4 bytes long?
 MPU: null, <! this makes no sense, MPU is a type of UPID
 SegmentationUPIDFormat: "NotSpecified", <!--- I have no idea what this is supposed to be
 Value: "" <!--- I have no idea what this is supposed to be
 }


Here's threefive


 "segmentation_upid_type": 1, <!---- Type 
 "segmentation_upid_type_name": "Deprecated", <---- I add the type_name to make it easier
 "segmentation_upid_length": 1, <----- Length -- the length is 1 byte
 "segmentation_upid": "0", <---- SegmentationUpid() -- This is the data, 1 byte

 
 

 
 

 

 

 

 
 

 

 

        

 
 

 

 

 

 

 

 

 
 

 


 



 



 
 

 

 Answer selected by
 futzu




 



 

 

 

 

 

 

 

 

 

 


 

 

 

 
 
 Comment options
 

 
 


 
 
 

 

 

 

 
 

 

 

 



 

 

 
 
 
 
 
 
2022 is out of date man. There have been big changes since then.

Tell your client just because it's on the Internet, doesn't mean it's true.

There are multiple errors in middleman's output, that are not a good comparison.


I still for the life of me can't figure out what kind of math middleman is doing,

they are using like the 2020 spec. Comcast's online parser is out of date too.


https://iodisco.com/cgi-bin/scte35decoder uses threefive.

 
 

 
 

 

 

 

 

 
 

 

  You must be logged in to vote
 

 

 

 
 

 

 

      

 
 

 

 

 

 
 4 replies
 
 

 

 

 

 

 

 

 
 

 

 

 

 

 


 

 

 

 
 
 Comment options
 

 
 


 
 
 

 

 

 

 
 

 

 

 



 

 

 
 
 
 
 
 
Thank you very much. I've downloaded the new SCTE35 and 67 specs and will "read the fancy manual" on both.

And thank you for writing threefive.

 
 

 
 

 

 

 

 
 

 

 
 
 

 

 

 

 

 

 

 
 

 


 



 

 

 
 

 

 

 

 

 


 

 

 

 
 
 Comment options
 

 
 


 
 
 

 

 

 

 
 

 

 

 



 

 

 
 
 
 
 
 
I appreciate you saying that.


SCTE-35 doesn't make much sense , it's smart to read the specs.

 
 

 
 

 

 

 

 
 

 

 
 
 

 

 

 

 

 

 

 
 

 


 



 

 

 
 

 

 

 

 

 


 

 

 

 
 
 Comment options
 

 
 


 
 
 

 

 

 

 
 

 

 

 



 

 

 
 
 
 
 
 
I don't know if you seen my hls stuff, but there's a bunch of it M3ufu, x9k3, umzz, showcues, and sideways.

 
 

 
 

 

 

 

 
 

 

 
 
 

 

 

 

 

 

 

 
 

 


 



 

 

 
 

 

 

 

 

 


 

 

 

 
 
 Comment options
 

 
 


 
 
 

 

 

 

 
 

 

 

 



 

 

 
 
 
 
 
 
showcues is great for debugging hls and scte-35

 
 

 
 

 

 

 

 
 

 

 
 
 

 

 

 

 

 

 

 
 

 


 



 



 
 

 


 



 

 

 
 

 

 

 


 

 
 

 

 
 Sign up for free
 to join this conversation on GitHub.
 Already have an account?
 Sign in to comment
 


 
 

 



 

 

 

 

 

 Category
 

 
 
 Q&A
 


 

 

 

 

 Labels
 

 

 None yet
 



 
 
 
 



 
 
 
 
 
 
 
 
 

 

 


 

 

 

 
 
 
 
 

 


 

 

 

 

 




AltStyle によって変換されたページ (->オリジナル)
/