XML and Dash and SCTE-214 #108

futzu Sep 30, 2024
Maintainer Author

@davemevans I know exactly what you need to do, make a __ threefive.xsd__ , that handles the ##any properly and get rid of the component and other old stuff and we can post on one of my servers and include in the releases. I'll try to push it on SCTE and everyone else. That's would be __HUGE. We need a threefive schema, that can be validated against.
Make sure your name is in it as the author too. Maybe give us a namespace or something. Whatever you think best.
Hell Yes. Can you do that?

futzu Sep 30, 2024
Maintainer Author

Even if we only use it to validate threefive output, that would save us so much time. Just like you said , people are going to want to validate, let's give them a way to do it.

futzu Sep 30, 2024
Maintainer Author

@davemevans
On the MID upid thing, we could call all the UPIDs in xml with the segmentation descriptor node as an arg and attach child nodes directly to it. All the upiids have to attach to it anyway.

 sd=Node('SegmentationDescriptor',attrs=sd_attrs)
 the_upid = self.mk_the_upid()
 upid_node = the_upid.xml()

that's how I do it right now, but we could do it like

 sd=Node('SegmentationDescriptor',attrs=sd_attrs)
 the_upid = self.mk_the_upid()
 sd = the_upid.xml(sd) <---- just pass the sd Node in and attach child nodes directly to it and return the node.

davemevans Sep 30, 2024

Cool cool - will have some faith, let you work, and come back in a bit when things are more stable.

My use case is to take an arbitrary binary cue, convert it to XML representation, then run an arbitrary xpath evaluation against that eg /SpliceInfoSection/SegmentationDescriptor[@segmentationTypeId=48], //SegmentationUpid[@segmentationUpidType=8 and .='0000000012345678'] and such.

davemevans Sep 30, 2024

@davemevans On the MID upid thing, we could call all the UPIDs in xml with the segmentation descriptor node as an arg and attach child nodes directly to it. All the upiids have to attach to it anyway.
 sd=Node('SegmentationDescriptor',attrs=sd_attrs)
 the_upid = self.mk_the_upid()
 upid_node = the_upid.xml()
that's how I do it right now, but we could do it like
 sd=Node('SegmentationDescriptor',attrs=sd_attrs)
 the_upid = self.mk_the_upid()
 sd = the_upid.xml(sd) <---- just pass the sd Node in and attach child nodes directly to it and return the node.

Yes, that approach makes sense - tell the child its parent and have it attach itself.

futzu
Sep 30, 2024
Maintainer Author

class Upid:
 """
 Upid base class handles URI UPIDS
 """
 def __init__(self, bitbin=None, upid_type=0, upid_length=0):
 self.bitbin = bitbin
 self.upid_type = upid_type
 self.upid_name = upid_map[upid_type][0]
 self.upid_length = upid_length
 self.bit_length = upid_length << 3
 self.upid_value = None
 def decode(self):
 """
 decode Upid
 """
 self.upid_value = self.bitbin.as_charset(self.bit_length, charset)
 return self.upid_name, self.upid_value
 def encode(self, nbin ):
 """
 encode Upid
 """
 if self.upid.value:
 self.upid.value = self.upid.value.encode("utf8")
 nbin.add_bites(self.upid.value)
 def xml(self):
 """
 xml return a upid xml node
 """
 ud_attrs = { 'name': self.upid_name, # this is for clarity
 'segmentation_upid_type': hex(self.upid_type),
 'segmentation_upid_format':'hexbinary',
 'segmentation_upid_length':self.upid_length,}
 return Node('SegmentationUpid',attrs= ud_attrs, value=self.upid_value)

6 replies

davemevans Sep 30, 2024

What I didn't realise at the time I said that is that these things are constructed every time they are needed rather than hanging around. So it probably didn't actually really matter.

This could help but as it is now, it is possibly worse - it's not really clear to me how this can work when it is called in SegmentationDescriptor.xml() since there is no bitbin from which to decode any value. So you have to explicitly set the_upid.upid_value before calling the_upid.xml() which feels a bit janky.

Also, self.upid.value appears to be a typo.

For MPU support, there will need to be a way to pass in at least one more attribute (format_identifier) which I don't see how to do now, but perhaps I miss something. Would also be nice to be able to set the upid format to text for some types.

futzu Sep 30, 2024
Maintainer Author

You're over thinking it .

It's constructed every time by design, I didn't want to worry about changed values.
we don't pass in format_identifier, it already in the upid after decoding
the is the Upid instance, and a Upid xml Node. All the data is in the instance, it is copied to thee xml node.
You don't need bitbin, it's used by the instance, not the xml node.
this is just for the xml() method call, when you writing xml, the cue has already been decoded.
Here is an MPU upid decoded, format_identifier is already there.

 "segmentation_upid": {
 "format_identifier": "RTLN",
 "private_data": "0x3148010000000031333736393230323534343935483100"
 },

Here's a mid decoded

 "segmentation_upid_type_name": "MID",
 "segmentation_upid_length": 17,
 "segmentation_upid": [
 {
 "upid_type": 14,
 "upid_type_name": "ADS Info",
 "upid_length": 5,
 "segmentation_upid": "LA309"
 },
 {
 "upid_type": 8,
 "upid_type_name": "AiringID",
 "upid_length": 8,
 "segmentation_upid": "0x2e538481"
 }
 ],
 "segmentation_type_id": 52,

davemevans Sep 30, 2024

In the segmentationDescriptor.xml example above, the value isn't in the Upid instance because it isn't set when you mk_the_upid, and no decode is called (as you say it's already decoded). In this case, self.upid_value in Upid instance will be None unless it's set explicitly.

futzu Sep 30, 2024
Maintainer Author

Scratch that last part. It's better to have one special instance that to change them all.
Have the MID return a list of Nodes, one for each UPID,
This is a better way

# Segmentation Descriptor xml has this in it 
 def xml(self):
 """
 Create a Node describing a SegmentationDescriptor
 """
 .
 .
 .
 
 sd=Node('SegmentationDescriptor',attrs=sd_attrs)
 the_upid = self.mk_the_upid()
 upid_node = the_upid.xml()
 if isinstance(upid_node,list): #<-- check if it's a list 
 _=(sd.add_child(node) for node in upid_node) #<----- add each mid upid as a child of sd 
This is the MID class xml()
 def xml(self):
 """
 xml return a upid xml node

 """
 mid_nodes =[]
 for u in self.upid_value:
 u_attrs = ["upid_type": u['upid_type'],
 "name": u['upid_type_name'],}
 value= u["segmentation_upid"]
 node= Node('SegmentationUpid', attrs=u_attrs, value=value)
 mid_nodes.append(node)
 return mid_nodes

futzu Sep 30, 2024
Maintainer Author

Sorry, I'm doing three things at once.

don't call mk_the_upid,, you're right. I'll sort this out, but I gotta go for a little bit,
Make us a schema and I'll sort this out tonight after work

futzu
Sep 30, 2024
Maintainer Author

This is easy, they all have the same vars

in segmentationDescriptor.xml()

 def xml(self):
 """
 Create a Node describing a SegmentationDescriptor
 """ 
 ....
 
 self.segmentation_upid_type = bitbin.as_int(8)
 self.segmentation_upid_length = bitbin.as_int(8)
 the_upid = self.mk_the_upid(bitbin=None)
 # populate the upid's values
 the_upid.upid_name= self.segmentation_upid_type_name # mk_the_upid passes segmentation_upid_type and length in already
 the_upid.upid_value= self.segmentation_upid
 # call the_upid.xml()
 upid_node = the_upid.xml()
 if isinstance(upid_node,list):
 _=(sd.add_child(node) for node in upid_node)
 else:
 sd.add_child(upid_node)
 ......

This is the MID class xml()

 def xml(self):
 """
 xml return a upid xml node

 """
 mid_nodes =[]
 for u in self.upid_value:
 u_attrs = ["upid_type": u['upid_type'],
 "name": u['upid_type_name'],}
 value= u["segmentation_upid"]
 node= Node('SegmentationUpid', attrs=u_attrs, value=value)
 mid_nodea.append(node)
 return mid_nodes

1 reply

futzu Sep 30, 2024
Maintainer Author

Does that make sense?

futzu
Oct 1, 2024
Maintainer Author

Dave,
does this look right?

<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="8" xmlns="http://www.scte.org/schemas/35">
 <TimeSignal>
 <SpliceTime ptsTime="3522714355"/>
 </TimeSignal>
 <SegmentationDescriptor segmentationEventId="1207959743" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="52" segmentNum="0" segmentsExpected="0" subSegmentNum="0" subSegmentsExpected="0" segmentationDuration="16317027">
 <DeliveryRestrictions webDeliveryAllowedFlag="false" noRegionalBlackoutFlag="true" archiveAllowedFlag="true" deviceRestrictions="3"/>
 <SegmentationUpid upidType="14" name="ADS Info">LA309</SegmentationUpid>
 <SegmentationUpid upidType="8" name="AiringID">0x2e538481</SegmentationUpid>
 </SegmentationDescriptor>
</SpliceInfoSection>

1 reply

Dave Evans needs validation

futzu Oct 1, 2024
Maintainer Author

There is super easy way SCTE could fix this, is to make SegmentationDescriptor.segmentation_upid, segmentation_upids, a list.

futzu
Oct 1, 2024
Maintainer Author

6 replies

futzu Oct 2, 2024
Maintainer Author

okay. We'll change it back

futzu Oct 2, 2024
Maintainer Author

done.

futzu Oct 2, 2024
Maintainer Author

@davemevans
We need the segmentation message on the descriptor. People need to be able to visually inspect a manifest, to know when the break starts.I know this from HLS. Without the segmentation, you need to memorize like thirty different codes.

We could just make comment nodes, couldn't we?
For the segmentation descriptor segmentation message and upid name ?

Also , you need to tell me what you're trying to accomplish so I can keep it mind.

futzu Oct 2, 2024
Maintainer Author

I can do it like this

<!-- Created with Unified Streaming Platform (version=1.14.2-30503) -->

I'll just put the stuff that isn't in the xsd in comment nodes. Easy fix.

, that works just fine.

Any problem with comment nodes?

davemevans Oct 2, 2024

Yeah, this is a great shout 👌

futzu
Oct 1, 2024
Maintainer Author

Most of the UPiDa are now working.

Upid Xml output

0x3 : ABCD0123456H
<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095" xmlns="http://www.scte.org/schemas/35">
 <TimeSignal>
 <SpliceTime ptsTime="0"/>
 </TimeSignal>
 <SegmentationDescriptor segmentationMessage="Program Start" segmentationEventId="3" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="16" segmentNum="0" segmentsExpected="0" segmentationDuration="2702700">
 <SegmentationUpid name="AdID" segmentationUpidType="3" segmentationUpidFormat="hexbinary" segmentationUpidLength="12">ABCD0123456H</SegmentationUpid>
 </SegmentationDescriptor>
</SpliceInfoSection>
0x6 : 0x3a8d000000000000
<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095" xmlns="http://www.scte.org/schemas/35">
 <TimeSignal>
 <SpliceTime ptsTime="0"/>
 </TimeSignal>
 <SegmentationDescriptor segmentationMessage="Program Start" segmentationEventId="6" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="16" segmentNum="0" segmentsExpected="0" segmentationDuration="2702700">
 <SegmentationUpid name="ISAN" segmentationUpidType="6" segmentationUpidFormat="hexbinary" segmentationUpidLength="12">0x3a8d000000000000</SegmentationUpid>
 </SegmentationDescriptor>
</SpliceInfoSection>
0x7 : MV0004146400
<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095" xmlns="http://www.scte.org/schemas/35">
 <TimeSignal>
 <SpliceTime ptsTime="0"/>
 </TimeSignal>
 <SegmentationDescriptor segmentationMessage="Program End" segmentationEventId="3" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="17" segmentNum="0" segmentsExpected="0">
 <SegmentationUpid name="TID" segmentationUpidType="7" segmentationUpidFormat="hexbinary" segmentationUpidLength="12">MV0004146400</SegmentationUpid>
 </SegmentationDescriptor>
</SpliceInfoSection>
0xe : ADS-UPID:aa85bbb6-5c43-4b6a-bebb-ee3b13eb7999
<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095" xmlns="http://www.scte.org/schemas/35">
 <TimeSignal>
 <SpliceTime ptsTime="0"/>
 </TimeSignal>
 <SegmentationDescriptor segmentationMessage="Program End" segmentationEventId="11" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="17" segmentNum="0" segmentsExpected="0">
 <SegmentationUpid name="ADS Info" segmentationUpidType="14" segmentationUpidFormat="hexbinary" segmentationUpidLength="45">ADS-UPID:aa85bbb6-5c43-4b6a-bebb-ee3b13eb7999</SegmentationUpid>
 </SegmentationDescriptor>
</SpliceInfoSection>
0x4 : 060a2b34.01010105.01010d20.13000000.d2c9036c.8f195343.ab7014d2
<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095" xmlns="http://www.scte.org/schemas/35">
 <TimeSignal>
 <SpliceTime ptsTime="0"/>
 </TimeSignal>
 <SegmentationDescriptor segmentationMessage="None" segmentationEventId="3" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="215" segmentNum="None" segmentsExpected="None">
 <SegmentationUpid name="UMID" segmentationUpidType="4" segmentationUpidFormat="hexbinary" segmentationUpidLength="32">060a2b34.01010105.01010d20.13000000.d2c9036c.8f195343.ab7014d2</SegmentationUpid>
 </SegmentationDescriptor>
</SpliceInfoSection>
0xd : [{'upid_type': 14, 'upid_type_name': 'ADS Info', 'upid_length': 5, 'segmentation_upid': 'LA309'}, {'upid_type': 8, 'upid_type_name': 'AiringID', 'upid_length': 8, 'segmentation_upid': '0x2e538481'}]
<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="8" xmlns="http://www.scte.org/schemas/35">
 <TimeSignal>
 <SpliceTime ptsTime="3522714355"/>
 </TimeSignal>
 <SegmentationDescriptor segmentationMessage="Provider Placement Opportunity Start" segmentationEventId="1207959743" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="52" segmentNum="0" segmentsExpected="0" subSegmentNum="0" subSegmentsExpected="0" segmentationDuration="16317027">
 <DeliveryRestrictions webDeliveryAllowedFlag="false" noRegionalBlackoutFlag="true" archiveAllowedFlag="true" deviceRestrictions="3"/>
 <SegmentationUpid upidType="14" upidTypeName="ADS Info">LA309</SegmentationUpid>
 <SegmentationUpid upidType="8" upidTypeName="AiringID">0x2e538481</SegmentationUpid>
 </SegmentationDescriptor>
</SpliceInfoSection>
0x8 : 0x072d08c7
0xc : {'format_identifier': 'RTLN', 'private_data': '0x3148010000000031333736393230323534343935483100'}
<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095" xmlns="http://www.scte.org/schemas/35">
 <TimeSignal>
 <SpliceTime ptsTime="5469869394"/>
 </TimeSignal>
 <SegmentationDescriptor segmentationMessage="Provider Advertisement End" segmentationEventId="1859042901" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="49" segmentNum="11" segmentsExpected="25">
 <SegmentationUpid name="AiringID" segmentationUpidType="8" segmentationUpidFormat="hexbinary" segmentationUpidLength="8">0x072d08c7</SegmentationUpid>
 </SegmentationDescriptor>
 <SegmentationDescriptor segmentationMessage="Distributor Placement Opportunity End" segmentationEventId="1858902152" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="55" segmentNum="1" segmentsExpected="1">
 <SegmentationUpid name="AiringID" segmentationUpidType="8" segmentationUpidFormat="hexbinary" segmentationUpidLength="8">0x011836b21611</SegmentationUpid>
 </SegmentationDescriptor>
 <SegmentationDescriptor segmentationMessage="Provider Advertisement Start" segmentationEventId="1859063048" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="48" segmentNum="12" segmentsExpected="25" subSegmentNum="2" subSegmentsExpected="42" segmentationDuration="1821600">
 <SegmentationUpid name="AiringID" segmentationUpidType="8" segmentationUpidFormat="hexbinary" segmentationUpidLength="8">0x072d5612</SegmentationUpid>
 </SegmentationDescriptor>
 <SegmentationDescriptor segmentationMessage="Content Identification" segmentationEventId="1859063094" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="1" segmentNum="0" segmentsExpected="0">
 <SegmentationUpid name="MPU" segmentationUpidType="12" segmentationUpidFormat="hexbinary" segmentationUpidLength="27">{'format_identifier': 'RTLN', 'private_data': '0x3148010000000031333736393230323534343935483100'}</SegmentationUpid>
 </SegmentationDescriptor>
</SpliceInfoSection>
0xb : {'TSID': 241, 'reserved': 3, 'end_of_day': 23, 'unique_for': 511, 'content_id': 'human012'}
<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095" xmlns="http://www.scte.org/schemas/35">
 <TimeSignal>
 <SpliceTime ptsTime="0"/>
 </TimeSignal>
 <SegmentationDescriptor segmentationMessage="Program Start" segmentationEventId="3" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="16" segmentNum="0" segmentsExpected="0" segmentationDuration="2702700">
 <SegmentationUpid name="ATSC" segmentationUpidType="11" segmentationUpidFormat="hexbinary" segmentationUpidLength="12">{'TSID': 241, 'reserved': 3, 'end_of_day': 23, 'unique_for': 511, 'content_id': 'human012'}</SegmentationUpid>
 </SegmentationDescriptor>
</SpliceInfoSection>
<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095" xmlns="http://www.scte.org/schemas/35">
 <TimeSignal>
 <SpliceTime ptsTime="2832024813"/>
 </TimeSignal>
 <SegmentationDescriptor segmentationMessage="Provider Placement Opportunity End" segmentationEventId="1207959725" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="53" segmentNum="2" segmentsExpected="0">
 <DeliveryRestrictions webDeliveryAllowedFlag="true" noRegionalBlackoutFlag="true" archiveAllowedFlag="true" deviceRestrictions="3"/>
 <SegmentationUpid name="AiringID" segmentationUpidType="8" segmentationUpidFormat="hexbinary" segmentationUpidLength="8">0x2cb2d79d</SegmentationUpid>
 </SegmentationDescriptor>
 <SegmentationDescriptor segmentationMessage="Program End" segmentationEventId="1207959590" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="17" segmentNum="0" segmentsExpected="0">
 <DeliveryRestrictions webDeliveryAllowedFlag="true" noRegionalBlackoutFlag="true" archiveAllowedFlag="true" deviceRestrictions="3"/>
 <SegmentationUpid name="AiringID" segmentationUpidType="8" segmentationUpidFormat="hexbinary" segmentationUpidLength="8">0x2cb2d79d</SegmentationUpid>
 </SegmentationDescriptor>
 <SegmentationDescriptor segmentationMessage="Program Start" segmentationEventId="1207959591" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="16" segmentNum="0" segmentsExpected="0">
 <DeliveryRestrictions webDeliveryAllowedFlag="true" noRegionalBlackoutFlag="true" archiveAllowedFlag="true" deviceRestrictions="3"/>
 <SegmentationUpid name="AiringID" segmentationUpidType="8" segmentationUpidFormat="hexbinary" segmentationUpidLength="8">0x2cb2d7b3</SegmentationUpid>
 </SegmentationDescriptor>
</SpliceInfoSection>
0x9 : SIGNAL:3-sQ4NgFT0OjPsG4WqUQow
<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095" xmlns="http://www.scte.org/schemas/35">
 <TimeSignal>
 <SpliceTime ptsTime="5400000"/>
 </TimeSignal>
 <SegmentationDescriptor segmentationMessage="Provider Placement Opportunity End" segmentationEventId="1644168586" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="53" segmentNum="0" segmentsExpected="0">
 <SegmentationUpid name="ADI" segmentationUpidType="9" segmentationUpidFormat="hexbinary" segmentationUpidLength="29">SIGNAL:3-sQ4NgFT0OjPsG4WqUQow</SegmentationUpid>
 </SegmentationDescriptor>
</SpliceInfoSection>
0xa : 0x1478f85ae100b0685b8fb1c8
<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095" xmlns="http://www.scte.org/schemas/35">
 <TimeSignal>
 <SpliceTime ptsTime="0"/>
 </TimeSignal>
 <SegmentationDescriptor segmentationMessage="Program Start" segmentationEventId="3" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="16" segmentNum="0" segmentsExpected="0" segmentationDuration="2702700">
 <SegmentationUpid name="EIDR" segmentationUpidType="10" segmentationUpidFormat="hexbinary" segmentationUpidLength="12">0x1478f85ae100b0685b8fb1c8</SegmentationUpid>
 </SegmentationDescriptor>
</SpliceInfoSection>
0xf : urn:uuid:aa85bbb6-5c43-4b6a-bebb-ee3b13eb7999
<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095" xmlns="http://www.scte.org/schemas/35">
 <TimeSignal>
 <SpliceTime ptsTime="0"/>
 </TimeSignal>
 <SegmentationDescriptor segmentationMessage="Program Start" segmentationEventId="10" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="16" segmentNum="0" segmentsExpected="0" segmentationDuration="2700000">
 <SegmentationUpid name="URI" segmentationUpidType="15" segmentationUpidFormat="hexbinary" segmentationUpidLength="45">urn:uuid:aa85bbb6-5c43-4b6a-bebb-ee3b13eb7999</SegmentationUpid>
 </SegmentationDescriptor>
</SpliceInfoSection>

1 reply

futzu Oct 2, 2024
Maintainer Author

@wabiloo A lot of the UPIDS are working, upids that are dictionaries don't work yet when creating xml,
everything should be working.

futzu
Oct 2, 2024
Maintainer Author

@davemevans we still need to do the other Splice Commands, Splice Null, Private, and BandwidthReservatiuon.
You want to do the xml parts?

5 replies

davemevans Oct 2, 2024

Yeah, happy to take a look at this over the next few days.

davemevans Oct 3, 2024

Looking at PrivateCommand, decode doesn't currently read the private_bytes so I'm working on that, but there's nothing within the command itself to say how many private_bytes there are.

I note that the other command calculate their own length and set the splice_command_length based on what they read, rather than pass in the signalled splice_command_length. I assume this is to deal with the legacy length of 0xFFF.

I'm a bit stuck working out what to do here as I need a length to know when to stop consuming private_bytes, but clearly can't always trust the splice_command_length. Got any ideas?

futzu Oct 3, 2024
Maintainer Author

@davemevans First, lets talk comment nodes.

I tried doing Comment as a subclass last too, my problem with it is the init,
if we subclass then a comment node can have a value , children, an attributes,
and that 's a problem for me .

However, when I saw how you did it in Node.mk() and checking the instance, that works,
so I don't think it should be a subclass.

you have mk() already, you need an init , repr, and set depth.

 def __init__(self,comment):
 self.comment = comment
 self.depth= None
 def __repr__(self):
 return self.mk()
 def set_depth(self):
 """
 set_depth is used to format
 tabs in output
 """
 if not self.depth:
 self.depth = 0

I would MUCH rather have it as a separate class, I absolutely agree with you there,
make it it's own class,

futzu Oct 3, 2024
Maintainer Author

@davemevans the Command length is calculated and then set in the Info Section because a lot o people put bad values in the info section, and this way if something changes, when I re-encode, values are verified.

The reason it's not done in private, probably because nobody ever uses it.

the Private Command doesn't use bitn, it's just raw bytes,
Since it calls super().init it has the command_length var.
You can add it to deccode, just not encode.

 def decode(self): 
 """
 PrivateCommand.decode method
 """
 self.identifier = int.from_bytes(
 self.bites[0:3], byteorder="big"
 ) # 3 bytes = 24 bits
 self.command_length= len(self.bites) # For Command length it goes here 
 self.bites = self.bites[3:]

futzu Oct 3, 2024
Maintainer Author

Bacl in the Node.mk()
take out the else, you don't need it.

 if isinstance(obj, Comment):
 return obj.mk(obj)
 new_attrs = mk_xml_attrs(obj.attrs)
 rendrd = f"{ndent}<{obj.name}{new_attrs}>"
 if obj.value:
 return f"{rendrd}{obj.value}</{obj.name}>\n"
 rendrd = f"{rendrd}\n"
 for child in obj.children:
 rendrd += obj.mk(child)
 if obj.children:
 return f"{rendrd}{ndent}</{obj.name}>\n"
 return rendrd.replace(">", "/>")

futzu
Oct 3, 2024
Maintainer Author

Here's an updated private command class

 class PrivateCommand(SpliceCommand):
 """
 Table 13 - private_command
 """
 def __init__(self, bites=None):
 super().__init__(bites)
 self.command_type = 255
 self.name = "Private Command"
 self.identifier = None
 self.private_bytes = None
 def decode(self):
 """
 PrivateCommand.decode method
 """
 self.identifier = int.from_bytes(
 self.bites[0:3], byteorder="big"
 ) # 3 bytes = 24 bits
 self.command_length = len(self.bites)
 self.private_bytes = self.bites[3:]
 
 def encode(self, nbin=None):
 """
 encode private command
 """
 nbin = self._chk_nbin(nbin)
 self._chk_var(int, nbin.add_int, "identifier", 24) # 3 bytes = 24 bits
 nbin.add_bites(self.private_bytes)
 return nbin.bites

2 replies

futzu Oct 3, 2024
Maintainer Author

anything threefive specific uses 'bites' with an i , early on I had some name collisions .

davemevans Oct 3, 2024

Unfortunately this doesn't work because when decode is called self.bites contains the remainder of the cue unparsed up til that point ie it has the descriptor loop length, descriptors and crc.

futzu
Oct 3, 2024
Maintainer Author

I need cue.py, where are you at on the comment class?

1 reply

futzu Oct 3, 2024
Maintainer Author

I'm just going to accept the pull request and update he comment class.

futzu
Oct 3, 2024
Maintainer Author

@davemevans , going forward, if you see a problem bring it up and let's decide the plan before you do a pull request.

6 replies

futzu Oct 3, 2024
Maintainer Author

@davemevans
Hold up on any changes, I got to figure something out

 def xml(self, binary=False):
 """
 xml returns a threefive.Node instance
 which can be edited as needed or printed.
 """
 if binary:
 sig_attrs = {"xmlns": "https://scte.org/schemas/35"}
 sig_node = Node("Signal", attrs=sig_attrs)
 bin_node = Node("Binary", value=self.encode())
 sig_node.add_child(bin_node)
 return sig_node
 sis = self.info_section.xml()
 self.decode() <--- Having this here breaks loading from json and converting to xml

without that, the comments don't work.

futzu Oct 3, 2024
Maintainer Author

I have no idea why.

futzu Oct 3, 2024
Maintainer Author

I also added xml to the threefive cli tool.

threefive xml '/DBMAAAAAsrbAP/wBQb+zTXUKAA2AhlDVUVJABWWDH+DCAgAAAAALfOq1zQAAAAAAhlDVUVJAAA3Qn+DCAgAAAAALfOq1zQAAAAAAhmCQA=='

it also works on streams

threefive xml video.ts

futzu Oct 3, 2024
Maintainer Author

and json to xml

threefive '/DBMAAAAAsrbAP/wBQb+zTXUKAA2AhlDVUVJABWWDH+DCAgAAAAALfOq1zQAAAAAAhlDVUVJAAA3Qn+DCAgAAAAALfOq1zQAAAAAAhmCQA==' 2>&1 | ./threefive encode xml
<SpliceInfoSection xmlns="https://scte.org/schemas/35" ptsAdjustment="183003" protocolVersion="0" sapType="3" tier="4095">
 <TimeSignal>
 <SpliceTime ptsTime="3442857000"/>
 </TimeSignal>
 <!-- Provider Placement Opportunity Start -->
 <SegmentationDescriptor segmentationEventId="1414668" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="52" segmentNum="0" segmentsExpected="0" subSegmentNum="0" subSegmentsExpected="0">
 <DeliveryRestrictions webDeliveryAllowedFlag="false" noRegionalBlackoutFlag="false" archiveAllowedFlag="false" deviceRestrictions="3"/>
 <!-- UPID: AiringID -->
 <SegmentationUpid segmentationUpidType="8" segmentationUpidFormat="hexbinary" segmentationUpidLength="8">2df3aad7</SegmentationUpid>
 </SegmentationDescriptor>
 <!-- Provider Placement Opportunity Start -->
 <SegmentationDescriptor segmentationEventId="14146" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="52" segmentNum="0" segmentsExpected="0" subSegmentNum="0" subSegmentsExpected="0">
 <DeliveryRestrictions webDeliveryAllowedFlag="false" noRegionalBlackoutFlag="false" archiveAllowedFlag="false" deviceRestrictions="3"/>
 <!-- UPID: AiringID -->
 <SegmentationUpid segmentationUpidType="8" segmentationUpidFormat="hexbinary" segmentationUpidLength="8">2df3aad7</SegmentationUpid>
 </SegmentationDescriptor>
</SpliceInfoSection>

futzu Oct 3, 2024
Maintainer Author

@wabiloo command line xml to Cue

xemel.txt

a@fu:~/build/SCTE35_threefive$ cat xemel.txt | threefive encode
{
 "info_section": {
 "table_id": "0xfc",
 "section_syntax_indicator": false,
 "private": false,
 "sap_type": "0x03",
 "sap_details": "No Sap Type",
 "section_length": 76,
 "protocol_version": 0,
 "encrypted_packet": false,
 "encryption_algorithm": 0,
 "pts_adjustment": 2.033367,
 "cw_index": "0x0",
 "tier": "0xfff",
 "splice_command_length": 5,
 "splice_command_type": 6,
 "descriptor_loop_length": 54,
 "crc": "0x2198240"
 },
 "command": {
 "command_length": 5,
 "command_type": 6,
 "name": "Time Signal",
 "time_specified_flag": true,
 "pts_time": 38253.966667
 },
 "descriptors": [
 {
 "tag": 2,
 "descriptor_length": 25,
 "name": "Segmentation Descriptor",
 "identifier": "CUEI",
 "segmentation_event_id": "0x15960c",
 "segmentation_event_cancel_indicator": false,
 "segmentation_event_id_compliance_indicator": true,
 "program_segmentation_flag": true,
 "segmentation_duration_flag": false,
 "delivery_not_restricted_flag": false,
 "web_delivery_allowed_flag": false,
 "no_regional_blackout_flag": false,
 "archive_allowed_flag": false,
 "device_restrictions": "No Restrictions",
 "segmentation_upid_type": 8,
 "segmentation_upid_length": 8,
 "segmentation_upid": "2df3aad7",
 "segmentation_type_id": 52,
 "segment_num": 0,
 "segments_expected": 0,
 "sub_segment_num": 0,
 "sub_segments_expected": 0
 },
 {
 "tag": 2,
 "descriptor_length": 25,
 "name": "Segmentation Descriptor",
 "identifier": "CUEI",
 "segmentation_event_id": "0x3742",
 "segmentation_event_cancel_indicator": false,
 "segmentation_event_id_compliance_indicator": true,
 "program_segmentation_flag": true,
 "segmentation_duration_flag": false,
 "delivery_not_restricted_flag": false,
 "web_delivery_allowed_flag": false,
 "no_regional_blackout_flag": false,
 "archive_allowed_flag": false,
 "device_restrictions": "No Restrictions",
 "segmentation_upid_type": 8,
 "segmentation_upid_length": 8,
 "segmentation_upid": "2df3aad7",
 "segmentation_type_id": 52,
 "segment_num": 0,
 "segments_expected": 0,
 "sub_segment_num": 0,
 "sub_segments_expected": 0
 }
 ]
}

futzu
Oct 3, 2024
Maintainer Author

@davemevans We've added a little over 400 lines of code, xml.py is almost 300 of them. I it's a new ability, it should require code.
We haven't had to make many changes to the rest of threefive, it's mostly new code. That's a strong indicator that we're doing it well.

1 reply

futzu Oct 3, 2024
Maintainer Author

If it's a method is only called by it's own instance, prefix it with an underscore and that way it doesn't show up in the help.

class XmlParser(builtins.object)
 | Methods defined here:
 | 
 | __init__(self)
 | 
 | chk_node_list(self, node)
 | chk_node_list is used to track open xml nodes
 | 
 | mk_active(self, node)
 | mk_active sets self.active to the current node name.
 | 
 | mk_attrs(self, node)
 | mk_attrs parses the current node for attributes
 | and stores them in self.stuff[self.active]
 | 
 | mk_value(self, value, stuff)
 | mk_value, if the xml node has a value, write it to self.stuff
 | 
 | <name>value</name>
 | 
 | parse(self, exemel, descriptor_parse=False)
 | parse parses an xml string for a SCTE-35 Cue.
 | 
 | ----------------------------------------------------------------------

The only that I should have there is parse, the rest I need to prefix with an understorre.
This is what I try to target with my comments.
I always have python shell open and I use help extensively,.

futzu
Oct 3, 2024
Maintainer Author

Current Cyclomatic Complexity Score

a@fu:~/build/SCTE35_threefive$ radon cc -sa threefive
threefive/packetdata.py
 C 8:0 PacketData - A (3)
 M 21:4 PacketData.mk_pcr - A (2)
 M 31:4 PacketData.mk_pts - A (2)
 M 15:4 PacketData.__init__ - A (1)
threefive/upids.py
 C 151:0 Eidr - A (3)
 C 201:0 Mid - A (3)
 M 242:4 Mid.xml - A (3)
 C 303:0 Umid - A (3)
 C 23:0 Upid - A (2)
 M 43:4 Upid.encode - A (2)
 C 63:0 NoUpid - A (2)
 C 91:0 AirId - A (2)
 C 121:0 Atsc - A (2)
 M 156:4 Eidr.decode - A (2)
 M 170:4 Eidr.encode - A (2)
 C 182:0 Isan - A (2)
 M 206:4 Mid.decode - A (2)
 M 230:4 Mid.encode - A (2)
 C 262:0 Mpu - A (2)
 M 280:4 Mpu.decode - A (2)
 M 308:4 Umid.decode - A (2)
 M 320:4 Umid.encode - A (2)
 M 28:4 Upid.__init__ - A (1)
 M 36:4 Upid.decode - A (1)
 M 51:4 Upid.xml - A (1)
 M 68:4 NoUpid.decode - A (1)
 M 74:4 NoUpid.encode - A (1)
 M 80:4 NoUpid.xml - A (1)
 M 96:4 AirId.decode - A (1)
 M 103:4 AirId.encode - A (1)
 M 109:4 AirId.xml - A (1)
 M 126:4 Atsc.decode - A (1)
 M 140:4 Atsc.encode - A (1)
 M 187:4 Isan.decode - A (1)
 M 194:4 Isan.encode - A (1)
 M 267:4 Mpu._decode_adfr - A (1)
 M 292:4 Mpu.encode - A (1)
threefive/decode.py
 F 27:0 _read_stuff - A (3)
 F 43:0 decode - A (3)
threefive/encode.py
 F 49:0 mk_splice_insert - A (3)
 F 24:0 mk_time_signal - A (2)
 F 12:0 mk_splice_null - A (1)
threefive/smoketest.py
 F 33:0 smoke - A (4)
 F 24:0 _decode_test - A (2)
threefive/stream.py
 M 544:4 Stream._parse_scte35 - B (8)
 M 568:4 Stream._parse_sdt - B (7)
 M 332:4 Stream.show - B (6)
 M 470:4 Stream._parse_tables - B (6)
 M 625:4 Stream._parse_pmt - B (6)
 M 220:4 Stream._find_start - A (5)
 M 265:4 Stream.decode_fu - A (5)
 M 278:4 Stream.decode_next - A (5)
 M 352:4 Stream.show_pts - A (5)
 M 497:4 Stream._parse - A (5)
 M 250:4 Stream.decode - A (4)
 M 317:4 Stream.proxy - A (4)
 M 431:4 Stream._parse_pts - A (4)
 M 604:4 Stream._parse_pat - A (4)
 C 39:0 ProgramInfo - A (3)
 M 53:4 ProgramInfo.show - A (3)
 C 116:0 Stream - A (3)
 M 405:4 Stream._unpad - A (3)
 M 421:4 Stream._parse_cc - A (3)
 M 514:4 Stream._same_as_last - A (3)
 M 677:4 Stream._set_scte35_pids - A (3)
 C 78:0 Pids - A (2)
 C 96:0 Maps - A (2)
 M 136:4 Stream.__init__ - A (2)
 M 214:4 Stream._split_by_idx - A (2)
 M 244:4 Stream._mk_pkts - A (2)
 M 309:4 Stream.decode_start_time - A (2)
 M 370:4 Stream.pid2prgm - A (2)
 M 380:4 Stream.pid2pts - A (2)
 M 390:4 Stream.pid2pcr - A (2)
 M 400:4 Stream._unpad_afc - A (2)
 M 449:4 Stream._parse_pcr - A (2)
 M 459:4 Stream._parse_payload - A (2)
 M 487:4 Stream._parse_info - A (2)
 M 509:4 Stream._chk_partial - A (2)
 M 521:4 Stream._section_incomplete - A (2)
 M 528:4 Stream._parse_cue - A (2)
 M 536:4 Stream._strip_scte35_pes - A (2)
 M 652:4 Stream._parse_program_streams - A (2)
 F 15:0 no_op - A (1)
 F 23:0 show_cue - A (1)
 F 31:0 show_cue_stderr - A (1)
 M 46:4 ProgramInfo.__init__ - A (1)
 M 86:4 Pids.__init__ - A (1)
 M 106:4 Maps.__init__ - A (1)
 M 160:4 Stream.__repr__ - A (1)
 M 164:4 Stream.as_90k - A (1)
 M 171:4 Stream._pusi_flag - A (1)
 M 175:4 Stream._afc_flag - A (1)
 M 179:4 Stream._pcr_flag - A (1)
 M 183:4 Stream._spi_flag - A (1)
 M 187:4 Stream._pts_flag - A (1)
 M 192:4 Stream._parse_length - A (1)
 M 199:4 Stream._parse_pid - A (1)
 M 207:4 Stream._parse_program - A (1)
 M 238:4 Stream.iter_pkts - A (1)
 M 292:4 Stream.decode_program - A (1)
 M 300:4 Stream.decode_pids - A (1)
 M 364:4 Stream.pts - A (1)
 M 414:4 Stream._mk_packet_data - A (1)
 M 667:4 Stream._parse_stream_type - A (1)
threefive/commands.py
 M 203:4 SpliceInsert.decode - A (5)
 M 229:4 SpliceInsert.encode - A (5)
 M 263:4 SpliceInsert.xml - A (5)
 C 172:0 SpliceInsert - A (4)
 M 143:4 TimeSignal._encode_splice_time - A (3)
 M 295:4 SpliceInsert.from_xml - A (3)
 C 10:0 SpliceCommand - A (2)
 C 41:0 BandwidthReservation - A (2)
 C 57:0 PrivateCommand - A (2)
 C 90:0 SpliceNull - A (2)
 C 101:0 TimeSignal - A (2)
 M 131:4 TimeSignal._splice_time - A (2)
 M 157:4 TimeSignal.xml - A (2)
 M 167:4 TimeSignal.from_xml - A (2)
 M 15:4 SpliceCommand.__init__ - A (1)
 M 21:4 SpliceCommand.decode - A (1)
 M 26:4 SpliceCommand._set_len - A (1)
 M 33:4 SpliceCommand.encode - A (1)
 M 46:4 BandwidthReservation.__init__ - A (1)
 M 51:4 BandwidthReservation.decode - A (1)
 M 62:4 PrivateCommand.__init__ - A (1)
 M 69:4 PrivateCommand.decode - A (1)
 M 80:4 PrivateCommand.encode - A (1)
 M 95:4 SpliceNull.__init__ - A (1)
 M 106:4 TimeSignal.__init__ - A (1)
 M 113:4 TimeSignal.decode - A (1)
 M 122:4 TimeSignal.encode - A (1)
 M 177:4 SpliceInsert.__init__ - A (1)
 M 194:4 SpliceInsert.decode_break_duration - A (1)
 M 254:4 SpliceInsert.encode_break_duration - A (1)
threefive/xml.py
 M 130:4 Node.mk - B (6)
 F 26:0 un_xml - A (5)
 M 217:4 XmlParser._mk_attrs - A (5)
 F 42:0 iter_attrs - A (4)
 F 17:0 un_camel - A (3)
 F 53:0 val2xml - A (3)
 C 82:0 Node - A (3)
 M 116:4 Node.set_depth - A (3)
 C 166:0 Comment - A (3)
 C 174:0 XmlParser - A (3)
 M 190:4 XmlParser._chk_node_list - A (3)
 M 239:4 XmlParser._parse_nodes - A (3)
 M 246:4 XmlParser._parse_most - A (3)
 F 64:0 key2xml - A (2)
 F 74:0 mk_xml_attrs - A (2)
 M 104:4 Node.__init__ - A (2)
 M 167:4 Comment.mk - A (2)
 M 199:4 XmlParser._mk_value - A (2)
 M 228:4 XmlParser.parse - A (2)
 M 264:4 XmlParser._parse_descriptor - A (2)
 F 8:0 t2s - A (1)
 M 113:4 Node.__repr__ - A (1)
 M 126:4 Node._get_indent - A (1)
 M 153:4 Node.add_child - A (1)
 M 159:4 Node.add_comment - A (1)
 M 186:4 XmlParser.__init__ - A (1)
 M 210:4 XmlParser._mk_active - A (1)
threefive/stuff.py
 F 8:0 print2 - A (1)
threefive/cue.py
 M 144:4 Cue._mk_bits - B (7)
 M 103:4 Cue.get - A (5)
 M 412:4 Cue.xml - A (5)
 M 319:4 Cue.load_descriptors - A (4)
 M 334:4 Cue.load - A (4)
 M 366:4 Cue._xml_splice_command - A (4)
 C 25:0 Cue - A (3)
 M 89:4 Cue._descriptor_loop - A (3)
 M 280:4 Cue._unloop_descriptors - A (3)
 M 306:4 Cue.load_command - A (3)
 M 374:4 Cue._xml_splice_descriptor - A (3)
 M 387:4 Cue._xml_event_signal - A (3)
 M 393:4 Cue.from_xml - A (3)
 M 55:4 Cue.__init__ - A (2)
 M 74:4 Cue.decode - A (2)
 M 121:4 Cue.get_descriptors - A (2)
 M 136:4 Cue.fix_bad_b64 - A (2)
 M 173:4 Cue._mk_descriptors - A (2)
 M 198:4 Cue._set_splice_command - A (2)
 M 296:4 Cue.load_info_section - A (2)
 M 361:4 Cue._xml_splice_info_section - A (2)
 M 69:4 Cue.__repr__ - A (1)
 M 128:4 Cue.get_json - A (1)
 M 187:4 Cue.mk_info_section - A (1)
 M 213:4 Cue.show - A (1)
 M 221:4 Cue.encode - A (1)
 M 245:4 Cue.encode_as_int - A (1)
 M 252:4 Cue.encode2int - A (1)
 M 258:4 Cue.encode_as_hex - A (1)
 M 265:4 Cue.encode2hex - A (1)
 M 272:4 Cue._encode_crc - A (1)
threefive/base.py
 M 139:4 SCTE35Base.load - A (5)
 M 26:4 SCTE35Base._chk_var - A (3)
 M 87:4 SCTE35Base.kv_clean - A (3)
 C 9:0 SCTE35Base - A (2)
 M 21:4 SCTE35Base._chk_nbin - A (2)
 M 54:4 SCTE35Base.as_hms - A (2)
 M 105:4 SCTE35Base.has - A (2)
 M 126:4 SCTE35Base.idxsplit - A (2)
 M 17:4 SCTE35Base.__repr__ - A (1)
 M 40:4 SCTE35Base.as_90k - A (1)
 M 47:4 SCTE35Base.as_ticks - A (1)
 M 68:4 SCTE35Base.fix_hex - A (1)
 M 74:4 SCTE35Base.get - A (1)
 M 80:4 SCTE35Base.get_json - A (1)
 M 113:4 SCTE35Base.xml - A (1)
threefive/segment.py
 M 119:4 Segment.decode - B (6)
 M 57:4 Segment.__init__ - A (4)
 C 16:0 Segment - A (3)
 M 109:4 Segment.show_cue - A (2)
 M 77:4 Segment.__repr__ - A (1)
 M 80:4 Segment._mk_tmp - A (1)
 M 84:4 Segment._aes_get_key - A (1)
 M 88:4 Segment._aes_decrypt - A (1)
 M 95:4 Segment._add_cue - A (1)
 M 102:4 Segment.shushed - A (1)
threefive/bitn.py
 C 8:0 BitBin - A (2)
 M 29:4 BitBin.as_int - A (2)
 M 47:4 BitBin.as_charset - A (2)
 C 93:0 NBin - A (2)
 M 127:4 NBin.add_int - A (2)
 M 145:4 NBin.add_hex - A (2)
 M 168:4 NBin.reserve - A (2)
 M 185:4 NBin.zeroed - A (2)
 M 16:4 BitBin.__init__ - A (1)
 M 21:4 BitBin.as_90k - A (1)
 M 39:4 BitBin.as_hex - A (1)
 M 62:4 BitBin.as_bytes - A (1)
 M 71:4 BitBin.as_flag - A (1)
 M 77:4 BitBin.forward - A (1)
 M 84:4 BitBin.negative_shift - A (1)
 M 101:4 NBin.__init__ - A (1)
 M 106:4 NBin.nbits2bites - A (1)
 M 117:4 NBin.add_bites - A (1)
 M 136:4 NBin.add_90k - A (1)
 M 159:4 NBin.add_flag - A (1)
 M 179:4 NBin.forward - A (1)
threefive/descriptors.py
 M 469:4 SegmentationDescriptor.xml - B (6)
 M 516:4 SegmentationDescriptor.from_xml - A (5)
 M 69:4 SpliceDescriptor.encode - A (4)
 F 12:0 k_by_v - A (3)
 C 22:0 SpliceDescriptor - A (3)
 C 299:0 SegmentationDescriptor - A (3)
 M 367:4 SegmentationDescriptor._decode_segmentation - A (3)
 M 380:4 SegmentationDescriptor._chk_sub_segments - A (3)
 M 458:4 SegmentationDescriptor._encode_segments - A (3)
 F 551:0 splice_descriptor - A (2)
 M 39:4 SpliceDescriptor.parse_tag_and_len - A (2)
 M 50:4 SpliceDescriptor.parse_id - A (2)
 M 61:4 SpliceDescriptor.decode - A (2)
 C 92:0 DVBDASDescriptor - A (2)
 M 111:4 DVBDASDescriptor.decode - A (2)
 C 142:0 AvailDescriptor - A (2)
 M 176:4 AvailDescriptor.from_xml - A (2)
 C 184:0 DtmfDescriptor - A (2)
 M 207:4 DtmfDescriptor.encode - A (2)
 M 234:4 DtmfDescriptor.from_xml - A (2)
 C 244:0 TimeDescriptor - A (2)
 M 291:4 TimeDescriptor.from_xml - A (2)
 M 342:4 SegmentationDescriptor.decode - A (2)
 M 355:4 SegmentationDescriptor._decode_flags - A (2)
 M 389:4 SegmentationDescriptor._decode_segments - A (2)
 M 401:4 SegmentationDescriptor.encode - A (2)
 M 419:4 SegmentationDescriptor._encode_flags - A (2)
 M 432:4 SegmentationDescriptor.mk_the_upid - A (2)
 M 446:4 SegmentationDescriptor._encode_segmentation - A (2)
 M 29:4 SpliceDescriptor.__init__ - A (1)
 M 83:4 SpliceDescriptor._encode_id - A (1)
 M 97:4 DVBDASDescriptor.__init__ - A (1)
 M 128:4 DVBDASDescriptor.encode - A (1)
 M 147:4 AvailDescriptor.__init__ - A (1)
 M 154:4 AvailDescriptor.decode - A (1)
 M 161:4 AvailDescriptor.encode - A (1)
 M 169:4 AvailDescriptor.xml - A (1)
 M 189:4 DtmfDescriptor.__init__ - A (1)
 M 198:4 DtmfDescriptor.decode - A (1)
 M 221:4 DtmfDescriptor.xml - A (1)
 M 249:4 TimeDescriptor.__init__ - A (1)
 M 258:4 TimeDescriptor.decode - A (1)
 M 267:4 TimeDescriptor.encode - A (1)
 M 277:4 TimeDescriptor.xml - A (1)
 M 315:4 SegmentationDescriptor.__init__ - A (1)
threefive/streamtypes.py
 F 105:0 add_stream_types - A (2)
 F 116:0 mk_streamtype_map - A (2)
 F 98:0 hex_literal - A (1)
threefive/crc.py
 F 16:0 _bytecrc - A (2)
 F 25:0 _mk_table - A (2)
 F 31:0 crc32 - A (2)
 F 44:0 crc32hex - A (1)
threefive/section.py
 M 116:4 SpliceInfoSection._encode_encrypted - A (3)
 M 150:4 SpliceInfoSection._encode_splice_command - A (3)
 M 196:4 SpliceInfoSection.from_xml - A (3)
 C 20:0 SpliceInfoSection - A (2)
 M 45:4 SpliceInfoSection.decode - A (2)
 M 91:4 SpliceInfoSection._encode_sap - A (2)
 M 100:4 SpliceInfoSection._encode_section_length - A (2)
 M 108:4 SpliceInfoSection._encode_protocol_version - A (2)
 M 134:4 SpliceInfoSection._encode_cw_index - A (2)
 M 142:4 SpliceInfoSection._encode_tier - A (2)
 M 26:4 SpliceInfoSection.__init__ - A (1)
 M 70:4 SpliceInfoSection._encode_table_id - A (1)
 M 77:4 SpliceInfoSection._encode_section_syntax_indicator - A (1)
 M 84:4 SpliceInfoSection._encode_private_flag - A (1)
 M 128:4 SpliceInfoSection._encode_pts_adjustment - A (1)
 M 162:4 SpliceInfoSection.encode - A (1)
 M 182:4 SpliceInfoSection.xml - A (1)
309 blocks (classes, functions, methods) analyzed.
Average complexity: A (2.087378640776699) <------Boom goes the dynamite.,

a@fu:~/build/SCTE35_threefive$

That's a sweet score

python 3.11 scored : A (3.076061694485527) . We beat python.

0 replies

futzu
Oct 4, 2024
Maintainer Author

4 replies

futzu Oct 4, 2024
Maintainer Author

On Hex: You're right about the attributes , but wrong about the values. we can use hex in the xml value, like for UPIDs.

xsd: Schema says unsignedByte in an attribute cannot be hex or octal

davemevans Oct 4, 2024

Correct.

For the UPID value, what's valid will depend on the format specified. Currently this is hexbinary, so my understanding is that it cannot have 0x (https://www.w3.org/TR/xmlschema11-2/#hexBinary), so text would be needed which is what I think Table 22 examples imply they are encoded as.

There are a few examples in other specs (SCTE224, ESAM), open source projects (scte35-go) and various services outputting/ingesting this stuff (eg AWS Media Services), but you are right that there are no canonical examples in any spec, other than Table 22.

futzu Oct 4, 2024
Maintainer Author

@davemevans I don't think that format going to used much , but let's do text for safety.
Any upids , like ATSC, that have Dict for the values, if we use NBin we can pack them together , just like the encoding and get bytes back and well return those as text hex, sound good?

futzu Oct 4, 2024
Maintainer Author

@davemevans , the only upids without a text representation in table 22 are ATSC, and type 1.
type 1 is deprecated but I still see them a lot , they are usually ASCII we can use that as is,
and I'l repack ATSC.

futzu
Oct 4, 2024
Maintainer Author

We need a hard target for the next release. I think Dash is not clear, the info is spread out every, and I'm tired of guessing. What we can do , fairly easily, is meet the SCTE-35 XML spec, so let's finish that up, cool? It's on the way to DASH anyway. If there is something you want to work on, just call it. If you got a question, just ask it. Everybody Cue, InfoSection, all the SpliceCommands, all the Descriptors, and all the UPIDS need a xml() method that returns a Node Instance and a from_xml(stuff) that loads from the dict stuff.
I'm going to be in Cue for a minute setting up _xml_splice_command

0 replies

futzu
Oct 4, 2024
Maintainer Author

@davemevans they mention timestamps in ID3tags for TimeDescriptors, I've just about finished that.

1 reply

futzu Oct 4, 2024
Maintainer Author

I'm just about done with encryption too.

futzu
Oct 7, 2024
Maintainer Author

upids with verified working xml()

0x0 : No Upid
0x1 : Deprecated
0x3 : AdID
0x4 : UMID
0x6 : ISAN
0x7 : TID
0x8 : AiringID
0x9 : ADI
0xa : EIDR
0xb : ATSC
0xc : MPU
0xd : MID
0xe : ADS Info
0xf : URI

1 reply

davemevans Oct 7, 2024

This is looking awesome - almost there!

One thing I noticed is that MID didn't quite work properly since it dumps the contained upids directly with incorrect attrs, whereas it should call the underlying upid.xml(). That's an easy enough fix - can PR that later.

When calling Cue.from_xml, I found that when you have a MID in XML and want to from_xml it, only one of them ends up in the JSON and some values are overwritten. This will need special handling in the XML parser to ensure a list of segmentation upids attached to the descriptor. I'm wasn't really sure on the way forward here as I'm not that experienced with XMLParser and the codes looks pretty gnarly 😄. Here's an example XML to demonstrate - see that the ADI is passed to the EIDR encoder:

<SpliceInfoSection xmlns="http://www.scte.org/schemas/35" ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
 <TimeSignal>
 <SpliceTime ptsTime="4294967296"/>
 </TimeSignal>
 <!-- Distributor Placement Opportunity End -->
 <SegmentationDescriptor segmentationEventId="2" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="55" segmentNum="1" segmentsExpected="3">
 <!-- UPID: MID -->
 <SegmentationUpid upidType="10" name="EIDR">0x14778be5e3f6000000000000</SegmentationUpid>
 <SegmentationUpid upidType="10" name="EIDR">0x14778be5e3f6000000000000</SegmentationUpid>
 <SegmentationUpid upidType="9" name="ADI">SIGNAL:Ly9EMGxKR0hFZUtpMHdCUVZnRUFnZz1</SegmentationUpid>
 </SegmentationDescriptor>
</SpliceInfoSection>

futzu
Oct 7, 2024
Maintainer Author

@davemevans , you got to shorten these names up , unescape_special_chars, you sound like a damn dot Net programmer :)

unescape and escape or something is fine by me , I mean most of my vars are named "stuff" or "data",

6 replies

futzu Oct 7, 2024
Maintainer Author

After we do this release, I want to fold XmlParser into Node.
XmlParser parses xml and makes a dictionary,
but it should be making Node instances

Look at these methods, it all lines up
 def mk_active(self, node): #Node.name
 def mk_value(self, value, stuff): # Node.value
 def mk_attrs(self, node): # Node.attrs

instead passing the stuff dict to SpliceInfoSection, we should pass a Node instance.

SpliceInfoSections currently loads stuff['SpliceInfoSection"],
it just as easily load Node.attrs from a Node instance, it's the same thing.

Does that make sense?

davemevans Oct 8, 2024

Yeah, makes total sense - love it.

https://github.com/futzu/cuei/blob/0a62872963adccb57004e29d06718dd13c0b4fa4/commands.go

futzu Oct 8, 2024
Maintainer Author

Have you ever look at cuei, my other SCTE-35 parser in Go?
It's done very differently from threefive. I was paid to do it but that wasn't the issue, it was strong typing in Go, it broke my dot notation. Since a Command can be several different types,, as well as Upids and Descriptors. I made a struct for a SpliceCommand, type that had all the variables for all the SpliceCommands. To return a Splice Command that was a SpliceInsert, I marshaled the data into a SpliceInsert struct.

I feel like was the best way to do it.
The company that paid me to do cuei, didn't get the marshal thing so I put it back

I feel like that's the approach we should be using.
I think we have xml working but it's not right.

However this OO, we need more classes.

Every xml Node that we use , should have a class.

XmlSpliceInsert
XmlTimeSignal
etc...

This not just displaying the data in a different format, this is different data.
When We see a "<SpliceInsert" in xml we should load the chunk from there to " "
into a XmlSplixceInsert Node

I think XmlSpliceInsert and SpliceInsert should marshal into each other .
The logic is with the object.

futzu Oct 8, 2024
Maintainer Author

The Xml Types in the xsd should be classes in threefive.

futzu Oct 8, 2024
Maintainer Author

This is why we aren't using etree.
image

futzu
Oct 7, 2024
Maintainer Author

@davemevans I'm going to do release on Wednesday I think.as long as what is working is stable, even if some of the UPIDs are not complete.
There is a lot o new code, and only about half is xml related. I want to get it to the people so we can get the bugs out.
Wednesday is not hard date, we'll see how it goes.

1 reply

futzu Oct 7, 2024
Maintainer Author

Did you see fixsix, it works pretty well, but the code is kind of clunky.
I tried to compress the cli help down a little , here's what it looks like now....

image

futzu
Oct 8, 2024
Maintainer Author

Upids: with a working from_xml() and xml()

 0x0 : No Upid
 0x1 : Deprecated
 0x2: Deprecated
 0x3 : AdID
 0x4 : UMID
 0x6 : ISAN
 0x7 : TID
 0x8 : AiringID
 0x9 : ADI
 0xa : EIDR
 0xe : ADS Info
 0xf : URI

UPIDS with just a working xml()

 0xb : ATSC
 0xc : MPU
 0xd : MID

I am okay with that for this release.

5 replies

futzu Oct 8, 2024
Maintainer Author

I have 45 GB worth of mpegts I use for testing. They are from ABC, Disney, CBS, CNN, MsNBC, NBC, NewMax and a few more sources.

They contain: 665 upids:

8 No Upid ,
7 AiringID,
640 type 1 (Deprecated)

davemevans Oct 9, 2024

Worth noting many of the from_xml only appear to work because the XML output from threefive has the non-standard segmentationUpidLength attribute included.

I would agree that this is ok for this release, but in the long term this would need addressing. Maybe worth opening a tracking issue.

futzu Oct 9, 2024
Maintainer Author

0,3,4,7,9,10,14,15 all work without length.
1 and 8 don't though, and they are the most used.

futzu Oct 9, 2024
Maintainer Author

Now, 0,1,2,3,4,5,6,7,8,9,10,14,15 all work without length

futzu Oct 10, 2024
Maintainer Author

"Worth noting many of the from_xml only appear to work because the XML output from threefive has the non-standard segmentationUpidLength attribute included." said Dave.

You got in my head with that one, I added length to the upid_map for all the upids with fixed sizes, and for the ones that are variable length, I am grabbing the length after we load the xml and call encode. Now, we don't need length in xml, and I think I took them all out.

futzu
Oct 8, 2024
Maintainer Author

Howw about you tune xml.py ?

cyclomatic complexity needs to be closer to 2 , do what you can, don't let it make you crazy.

pylint score needs to be over 9.75

python3 -mpip install radon
a@fu:~/build/SCTE35_threefive$ radon cc -sa threefive/xml.py

threefive/xml.py
 M 160:4 Node.mk - B (6)
 F 26:0 un_xml - A (5)
 M 248:4 XmlParser.mk_attrs - A (5)
 F 42:0 iter_attrs - A (4)
 M 271:4 XmlParser._parse_nodes - A (4)
 F 17:0 un_camel - A (3)
 F 53:0 val2xml - A (3)
 F 90:0 escape - A (3)
 F 101:0 unescape - A (3)
 C 112:0 Node - A (3)
 M 146:4 Node.set_depth - A (3)
 C 195:0 Comment - A (3)
 C 204:0 XmlParser - A (3)
 M 220:4 XmlParser.chk_node_list - A (3)
 M 280:4 XmlParser._parse_most - A (3)
 F 64:0 key2xml - A (2)
 F 74:0 mk_xml_attrs - A (2)
 M 134:4 Node.__init__ - A (2)
 M 196:4 Comment.mk - A (2)
 M 229:4 XmlParser.mk_value - A (2)
 M 260:4 XmlParser.parse - A (2)
 M 299:4 XmlParser._parse_descriptor - A (2)
 F 8:0 t2s - A (1)
 M 143:4 Node.__repr__ - A (1)
 M 156:4 Node.get_indent - A (1)
 M 183:4 Node.add_child - A (1)
 M 189:4 Node.add_comment - A (1)
 M 216:4 XmlParser.__init__ - A (1)
 M 241:4 XmlParser.mk_active - A (1)
 M 315:4 XmlParser._skip_comment - A (1)
30 blocks (classes, functions, methods) analyzed.
Average complexity: A (2.533333333333333)

a@fu:~/build/SCTE35_threefive$ pylint threefive/xml.py

************* Module threefive.xml
threefive/xml.py:134:4: W0102: Dangerous default value {} as argument (dangerous-default-value) # <---- I don't care about these
threefive/xml.py:156:4: C0116: Missing function or method docstring (missing-function-docstring)
threefive/xml.py:195:0: C0115: Missing class docstring (missing-class-docstring)
threefive/xml.py:204:0: C0115: Missing class docstring (missing-class-docstring)
threefive/xml.py:248:4: R1710: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements)
threefive/xml.py:308:8: W0702: No exception type(s) specified (bare-except) # <---- I don't care about these
------------------------------------------------------------------
Your code has been rated at 9.61/10 (previous run: 9.42/10, +0.19)

5 replies

davemevans Oct 8, 2024

Yes, happy to take a crack at this. But it won't be til tomorrow morning now.

futzu Oct 8, 2024
Maintainer Author

no rush.

futzu Oct 9, 2024
Maintainer Author

@davemevans in XmlParser.mkattrs

Just call unescape on the whole node,

 def mk_attrs(self, node):
 """
 mk_attrs parses the current node for attributes
 and stores them in self.stuff[self.active]
 """
 if "<!--" not in unescape(node): # <---- Call unescape on on the whole node
 attrs = [x for x in node.split(" ") if "=" in x]
 parsed = {
 x.split('="')[0]: x.split('="')[1].split('"')[0] # <----- You're call it here and in mk_value
 for x in attrs
 }
 it = iter_attrs(parsed)
 return it

It's way more efficient to call it on the whole thing.

davemevans Oct 9, 2024

It would be more efficient, but parsing would then fail for attributes that contain " because eg """ would be unescaped to """ which would not be split correctly. Probably not an issue for schema-compliant attributes, but anything in ##any could contain this construct.

futzu Oct 9, 2024
Maintainer Author

i got a hex fix for section.py that cleans up so other stuff I did earlier too. in xml.py val2xml will handle hex to int conversion,

futzu
Oct 10, 2024
Maintainer Author

@davemevans when you have a minute, Read the README and and see if the xml stuff makes sense, please.

10 replies

davemevans Oct 11, 2024

Yeah, that's annoying. The spec is clear that you derive the length after converting the XML representation to bitstream format, so the measuring should be done after encoding back to the native format.

This is fine for now, but probably a more complete solution for parsing from XML is needed for the future. Worth opening an issue to cover this and come to it later. I have a list of other things that need addressing down the line - where would be the best place to log these - GH issues? This discussion? New discussion?

futzu Oct 15, 2024
Maintainer Author

I got all the UPID stuff sorted, everything is working.
I don't see me doing further xml development, I asked everyone I've worked for in the last couple of years and nobody cares about DASH, and I don't see a need to invest anymore time in it. threefive is SCTE-35 specific, it's not a DASH parser.

The way to use threefive with DASH:

parse the mpd
when you get to an event node, pass it to threefive for parsing.

 <EventStream
 schemeIdUri="urn:scte:scte35:2014:xml+bin"
 timescale="1000">
 <!-- 2024年09月10日T05:33:47.520000Z - 2024年09月10日T05:34:25.920000Z -->
 <Event
 presentationTime="1725946427520"
 duration="38400"
 id="14268737">
 <Signal
 xmlns="http://www.scte.org/schemas/35/2016">
 <Binary>/DAgAAAAAAAAAP/wDwUA2blBf//+ADS8AMAAAAAAAORhJCQ=</Binary>
 </Signal>
 </Event>
 <!-- 2024年09月10日T05:39:50.400000Z - 2024年09月10日T05:40:28.800000Z -->
 </EventStream>

from threefive import Cue
cue = Cue()
cue.load(data)

That's all xml parsing threefive needs to do , you have the Cue instance cue, and all the SCTE-35 decoded.

cue.show()

{
 "info_section": {
 "table_id": "0xfc",
 "section_syntax_indicator": false,
 "private": false,
 "sap_type": "0x03",
 "sap_details": "No Sap Type",
 "section_length": 32,
 "protocol_version": 0,
 "encrypted_packet": false,
 "encryption_algorithm": 0,
 "pts_adjustment": 0.0,
 "cw_index": "0x00",
 "tier": "0x0fff",
 "splice_command_length": 15,
 "splice_command_type": 5,
 "descriptor_loop_length": 0,
 "crc": "0x961b236d"
 },
 "command": {
 "command_length": 15,
 "command_type": 5,
 "name": "Splice Insert",
 "break_auto_return": true,
 "break_duration": 38.4,
 "splice_event_id": 14268737,
 "splice_event_cancel_indicator": false,
 "out_of_network_indicator": true,
 "program_splice_flag": true,
 "duration_flag": true,
 "splice_immediate_flag": true,
 "event_id_compliance_flag": true,
 "unique_program_id": 49152,
 "avail_num": 0,
 "avails_expected": 0
 },
 "descriptors": [],
 "dash_data": {
 "EventStream": {
 "scheme_id_uri": "urn:scte:scte35:2014:xml+bin",
 "timescale": 1000
 },
 "Event": {
 "presentation_time": 1725946427520,
 "duration": 0.426667,
 "id": 14268737
 },
 "Signal": {
 "xmlns": "http://www.scte.org/schemas/35/2016"
 },
 "Binary": {
 "binary": "/DAgAAAAAAAAAP/wDwUA2blBf//+ADS8AMAAAAAAAORhJCQ="
 }
 }
}

davemevans Oct 15, 2024

XML representation of cues is part of SCTE35 so I think it is reasonable for parsing from and serialising to a SpliceInfoSection to be in scope for a SCTE35-specific parser.

Regardless, thanks for your efforts on this up to this point - it's genuinely much appreciated.

futzu Oct 15, 2024
Maintainer Author

@davemevans I'm not following you man, how is it not doing that?

>>>> from threefive import Cue
>>>> more_xml = """<Period start="PT444334H55M0.010S" id="break-1">
.... <EventStream timescale="90000" schemeIdUri="urn:scte:scte35:2013:xml">
.... <Event duration="540">
.... <scte35:SpliceInfoSection protocolVersion="0" ptsAdjustment="140429" tier="4095">
.... <scte35:SpliceInsert spliceEventId="11" spliceEventCancelIndicator="false" outOfNetworkIndicator="true" spliceImmediateFlag="true" uniqueProgramId="1"availNum="1" availsExpected="1">
.... <scte35:Program><scte35:SpliceTime ptsTime="5272624400"/></scte35:Program>
.... <scte35:BreakDuration autoReturn="true" duration="9450000"/>
.... </scte35:SpliceInsert>
.... </scte35:SpliceInfoSection>
.... </Event>
.... </EventStream>
.... </Period>
.... """
>>>> cue=Cue()
>>>> cue.load(more_xml)
True
>>>> print(cue.xml())
<SpliceInfoSection xmlns="https://scte.org/schemas/35" ptsAdjustment="140429" protocolVersion="0" sapType="3" tier="4095">
 <SpliceInsert spliceEventId="11" spliceEventCancelIndicator="false" spliceImmediateFlag="true" eventIdComplianceFlag="true" availNum="1" availsExpected="true" outOfNetworkIndicator="true" uniqueProgramId="1">
 <Program>
 <SpliceTime ptsTime="5272624400"/>
 </Program>
 <BreakDuration autoReturn="true" duration="9450000"/>
 </SpliceInsert>
</SpliceInfoSection>

futzu Oct 15, 2024
Maintainer Author

I can even modify the cue

>>>> ad=AvailDescriptor()
>>>> ad.provider_avail_id = 0x1234567
>>>> cue.descriptors.append(ad)
>>>> cue.encode()
'/DBHAAGRZOeYAP/wBQb+jzHOMAAxAiVDVUVJAAAAAX+/ARRtc25iY19FUDA0MzExMjIxMDU1NzYBAQAAAAhDVUVJASNFZ/rzHLQ='
>>>> print(cue.xml())
<SpliceInfoSection xmlns="https://scte.org/schemas/35" ptsAdjustment="6734276504" protocolVersion="0" sapType="3" tier="4095">
 <TimeSignal>
 <SpliceTime ptsTime="2402405936"/>
 </TimeSignal>
 <!-- Distributor Placement Opportunity Start -->
 <SegmentationDescriptor segmentationEventId="1" segmentationEventCancelIndicator="false" segmentationEventIdComplianceIndicator="true" segmentationTypeId="54" segmentNum="1" segmentsExpected="1" subSegmentNum="0" subSegmentsExpected="0">
 <!-- UPID: Deprecated -->
 <SegmentationUpid segmentationUpidType="1" segmentationUpidFormat="hexbinary">msnbc_EP043112210557</SegmentationUpid>
 </SegmentationDescriptor>
 <AvailDescriptor providerAvailId="19088743"/>
</SpliceInfoSection>
>>>>

print it as json

>>>> cue.show()
{
 "info_section": {
 "table_id": "0xfc",
 "section_syntax_indicator": false,
 "private": false,
 "sap_type": "0x03",
 "sap_details": "No Sap Type",
 "section_length": 71,
 "protocol_version": 0,
 "encrypted_packet": false,
 "encryption_algorithm": 0,
 "pts_adjustment": 74825.294489,
 "cw_index": "0x0",
 "tier": 4095,
 "splice_command_length": 5,
 "splice_command_type": 6,
 "descriptor_loop_length": 49,
 "crc": "0xfaf31cb4"
 },
 "command": {
 "command_length": 5,
 "command_type": 6,
 "name": "Time Signal",
 "time_specified_flag": true,
 "pts_time": 26693.399289
 },
 "descriptors": [
 {
 "tag": 2,
 "descriptor_length": 37,
 "name": "Segmentation Descriptor",
 "identifier": "CUEI",
 "segmentation_event_id": "0x1",
 "segmentation_event_cancel_indicator": false,
 "segmentation_event_id_compliance_indicator": true,
 "program_segmentation_flag": true,
 "segmentation_duration_flag": false,
 "delivery_not_restricted_flag": true,
 "segmentation_duration": 60.033333,
 "segmentation_message": "Distributor Placement Opportunity Start",
 "segmentation_upid_type": 1,
 "segmentation_upid_type_name": "Deprecated",
 "segmentation_upid_length": 20,
 "segmentation_upid": "msnbc_EP043112210557",
 "segmentation_type_id": 54,
 "segment_num": 1,
 "segments_expected": 1,
 "sub_segment_num": 0,
 "sub_segments_expected": 0
 },
 {
 "tag": 0,
 "descriptor_length": 8,
 "name": "Avail Descriptor",
 "identifier": "CUEI",
 "provider_avail_id": 19088743
 }
 ]
}
>>>>

the cli tool can convert from xml to base64,bytes, int, hex, json and vice versa. It will output xml when you parse mpegts.

What does it not do ?

futzu
Oct 16, 2024
Maintainer Author

@davemevans @wabiloo I can parse mpd filles over the network and convert the data to cues, no problem.

a@fu:~/build/SCTE35_threefive$ pypy3 mpdp.py https://demo.unified-streaming.com/k8s/live/stable/scte35-no-splicing.isml/.mpd
{
 "info_section": {
 "table_id": "0xfc",
 "section_syntax_indicator": false,
 "private": false,
 "sap_type": "0x03",
 "sap_details": "No Sap Type",
 "section_length": 32,
 "protocol_version": 0,
 "encrypted_packet": false,
 "encryption_algorithm": 0,
 "pts_adjustment": 0.0,
 "cw_index": "0x00",
 "tier": "0x0fff",
 "splice_command_length": 15,
 "splice_command_type": 5,
 "descriptor_loop_length": 0,
 "crc": "0xe4612424"
 },
 "command": {
 "command_length": 15,
 "command_type": 5,
 "name": "Splice Insert",
 "break_auto_return": true,
 "break_duration": 38.4,
 "splice_event_id": 14294827,
 "splice_event_cancel_indicator": false,
 "out_of_network_indicator": true,
 "program_splice_flag": true,
 "duration_flag": true,
 "splice_immediate_flag": true,
 "event_id_compliance_flag": true,
 "unique_program_id": 49152,
 "avail_num": 0,
 "avails_expected": 0
 },
 "descriptors": []
}
{
 "info_section": {
 "table_id": "0xfc",
 "section_syntax_indicator": false,
 "private": false,
 "sap_type": "0x03",
 "sap_details": "No Sap Type",
 "section_length": 32,
 "protocol_version": 0,
 "encrypted_packet": false,
 "encryption_algorithm": 0,
 "pts_adjustment": 0.0,
 "cw_index": "0x00",
 "tier": "0x0fff",
 "splice_command_length": 15,
 "splice_command_type": 5,
 "descriptor_loop_length": 0,
 "crc": "0xe4612424"
 },
 "command": {
 "command_length": 15,
 "command_type": 5,
 "name": "Splice Insert",
 "break_auto_return": true,
 "break_duration": 38.4,
 "splice_event_id": 14294828,
 "splice_event_cancel_indicator": false,
 "out_of_network_indicator": true,
 "program_splice_flag": true,
 "duration_flag": true,
 "splice_immediate_flag": true,
 "event_id_compliance_flag": true,
 "unique_program_id": 49152,
 "avail_num": 0,
 "avails_expected": 0
 },
 "descriptors": []
}
{
 "info_section": {
 "table_id": "0xfc",
 "section_syntax_indicator": false,
 "private": false,
 "sap_type": "0x03",
 "sap_details": "No Sap Type",
 "section_length": 32,
 "protocol_version": 0,
 "encrypted_packet": false,
 "encryption_algorithm": 0,
 "pts_adjustment": 0.0,
 "cw_index": "0x00",
 "tier": "0x0fff",
 "splice_command_length": 15,
 "splice_command_type": 5,
 "descriptor_loop_length": 0,
 "crc": "0xe4612424"
 },
 "command": {
 "command_length": 15,
 "command_type": 5,
 "name": "Splice Insert",
 "break_auto_return": true,
 "break_duration": 38.4,
 "splice_event_id": 14294829,
 "splice_event_cancel_indicator": false,
 "out_of_network_indicator": true,
 "program_splice_flag": true,
 "duration_flag": true,
 "splice_immediate_flag": true,
 "event_id_compliance_flag": true,
 "unique_program_id": 49152,
 "avail_num": 0,
 "avails_expected": 0
 },
 "descriptors": []
}
{
 "info_section": {
 "table_id": "0xfc",
 "section_syntax_indicator": false,
 "private": false,
 "sap_type": "0x03",
 "sap_details": "No Sap Type",
 "section_length": 32,
 "protocol_version": 0,
 "encrypted_packet": false,
 "encryption_algorithm": 0,
 "pts_adjustment": 0.0,
 "cw_index": "0x00",
 "tier": "0x0fff",
 "splice_command_length": 15,
 "splice_command_type": 5,
 "descriptor_loop_length": 0,
 "crc": "0xe4612424"
 },
 "command": {
 "command_length": 15,
 "command_type": 5,
 "name": "Splice Insert",
 "break_auto_return": true,
 "break_duration": 38.4,
 "splice_event_id": 14294830,
 "splice_event_cancel_indicator": false,
 "out_of_network_indicator": true,
 "program_splice_flag": true,
 "duration_flag": true,
 "splice_immediate_flag": true,
 "event_id_compliance_flag": true,
 "unique_program_id": 49152,
 "avail_num": 0,
 "avails_expected": 0
 },
 "descriptors": []
}
{
 "info_section": {
 "table_id": "0xfc",
 "section_syntax_indicator": false,
 "private": false,
 "sap_type": "0x03",
 "sap_details": "No Sap Type",
 "section_length": 32,
 "protocol_version": 0,
 "encrypted_packet": false,
 "encryption_algorithm": 0,
 "pts_adjustment": 0.0,
 "cw_index": "0x00",
 "tier": "0x0fff",
 "splice_command_length": 15,
 "splice_command_type": 5,
 "descriptor_loop_length": 0,
 "crc": "0xe4612424"
 },
 "command": {
 "command_length": 15,
 "command_type": 5,
 "name": "Splice Insert",
 "break_auto_return": true,
 "break_duration": 38.4,
 "splice_event_id": 14294831,
 "splice_event_cancel_indicator": false,
 "out_of_network_indicator": true,
 "program_splice_flag": true,
 "duration_flag": true,
 "splice_immediate_flag": true,
 "event_id_compliance_flag": true,
 "unique_program_id": 49152,
 "avail_num": 0,
 "avails_expected": 0
 },
 "descriptors": []

0 replies

futzu
Oct 16, 2024
Maintainer Author

You can convert them to cues, and the convert the cues back to xml

}
a@fu:~/build/SCTE35_threefive$ pypy3 mpdp.py https://demo.unified-streaming.com/k8s/live/stable/scte35-no-splicing.isml/.mpd
{
 "info_section": {
 "table_id": "0xfc",
 "section_syntax_indicator": false,
 "private": false,
 "sap_type": "0x03",
 "sap_details": "No Sap Type",
 "section_length": 32,
 "protocol_version": 0,
 "encrypted_packet": false,
 "encryption_algorithm": 0,
 "pts_adjustment": 0.0,
 "cw_index": "0x00",
 "tier": "0x0fff",
 "splice_command_length": 15,
 "splice_command_type": 5,
 "descriptor_loop_length": 0,
 "crc": "0xe4612424"
 },
 "command": {
 "command_length": 15,
 "command_type": 5,
 "name": "Splice Insert",
 "break_auto_return": true,
 "break_duration": 38.4,
 "splice_event_id": 14294830,
 "splice_event_cancel_indicator": false,
 "out_of_network_indicator": true,
 "program_splice_flag": true,
 "duration_flag": true,
 "splice_immediate_flag": true,
 "event_id_compliance_flag": true,
 "unique_program_id": 49152,
 "avail_num": 0,
 "avails_expected": 0
 },
 "descriptors": []
}
<SpliceInfoSection xmlns="https://scte.org/schemas/35" ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
 <SpliceInsert spliceEventId="14294830" spliceEventCancelIndicator="false" spliceImmediateFlag="true" eventIdComplianceFlag="true" availNum="0" availsExpected="0" outOfNetworkIndicator="true" uniqueProgramId="49152">
 <BreakDuration autoReturn="true" duration="3456000"/>
 </SpliceInsert>
</SpliceInfoSection>
{
 "info_section": {
 "table_id": "0xfc",
 "section_syntax_indicator": false,
 "private": false,
 "sap_type": "0x03",
 "sap_details": "No Sap Type",
 "section_length": 32,
 "protocol_version": 0,
 "encrypted_packet": false,
 "encryption_algorithm": 0,
 "pts_adjustment": 0.0,
 "cw_index": "0x00",
 "tier": "0x0fff",
 "splice_command_length": 15,
 "splice_command_type": 5,
 "descriptor_loop_length": 0,
 "crc": "0xe4612424"
 },
 "command": {
 "command_length": 15,
 "command_type": 5,
 "name": "Splice Insert",
 "break_auto_return": true,
 "break_duration": 38.4,
 "splice_event_id": 14294831,
 "splice_event_cancel_indicator": false,
 "out_of_network_indicator": true,
 "program_splice_flag": true,
 "duration_flag": true,
 "splice_immediate_flag": true,
 "event_id_compliance_flag": true,
 "unique_program_id": 49152,
 "avail_num": 0,
 "avails_expected": 0
 },
 "descriptors": []
}
<SpliceInfoSection xmlns="https://scte.org/schemas/35" ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
 <SpliceInsert spliceEventId="14294831" spliceEventCancelIndicator="false" spliceImmediateFlag="true" eventIdComplianceFlag="true" availNum="0" availsExpected="0" outOfNetworkIndicator="true" uniqueProgramId="49152">
 <BreakDuration autoReturn="true" duration="3456000"/>
 </SpliceInsert>
</SpliceInfoSection>
{
 "info_section": {
 "table_id": "0xfc",
 "section_syntax_indicator": false,
 "private": false,
 "sap_type": "0x03",
 "sap_details": "No Sap Type",
 "section_length": 32,
 "protocol_version": 0,
 "encrypted_packet": false,
 "encryption_algorithm": 0,
 "pts_adjustment": 0.0,
 "cw_index": "0x00",
 "tier": "0x0fff",
 "splice_command_length": 15,
 "splice_command_type": 5,
 "descriptor_loop_length": 0,
 "crc": "0xe4612424"
 },
 "command": {
 "command_length": 15,
 "command_type": 5,
 "name": "Splice Insert",
 "break_auto_return": true,
 "break_duration": 38.4,
 "splice_event_id": 14294832,
 "splice_event_cancel_indicator": false,
 "out_of_network_indicator": true,
 "program_splice_flag": true,
 "duration_flag": true,
 "splice_immediate_flag": true,
 "event_id_compliance_flag": true,
 "unique_program_id": 49152,
 "avail_num": 0,
 "avails_expected": 0
 },
 "descriptors": []
}
<SpliceInfoSection xmlns="https://scte.org/schemas/35" ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
 <SpliceInsert spliceEventId="14294832" spliceEventCancelIndicator="false" spliceImmediateFlag="true" eventIdComplianceFlag="true" availNum="0" availsExpected="0" outOfNetworkIndicator="true" uniqueProgramId="49152">
 <BreakDuration autoReturn="true" duration="3456000"/>
 </SpliceInsert>
</SpliceInfoSection>
{
 "info_section": {
 "table_id": "0xfc",
 "section_syntax_indicator": false,
 "private": false,
 "sap_type": "0x03",
 "sap_details": "No Sap Type",
 "section_length": 32,
 "protocol_version": 0,
 "encrypted_packet": false,
 "encryption_algorithm": 0,
 "pts_adjustment": 0.0,
 "cw_index": "0x00",
 "tier": "0x0fff",
 "splice_command_length": 15,
 "splice_command_type": 5,
 "descriptor_loop_length": 0,
 "crc": "0xe4612424"
 },
 "command": {
 "command_length": 15,
 "command_type": 5,
 "name": "Splice Insert",
 "break_auto_return": true,
 "break_duration": 38.4,
 "splice_event_id": 14294833,
 "splice_event_cancel_indicator": false,
 "out_of_network_indicator": true,
 "program_splice_flag": true,
 "duration_flag": true,
 "splice_immediate_flag": true,
 "event_id_compliance_flag": true,
 "unique_program_id": 49152,
 "avail_num": 0,
 "avails_expected": 0
 },
 "descriptors": []
}
<SpliceInfoSection xmlns="https://scte.org/schemas/35" ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
 <SpliceInsert spliceEventId="14294833" spliceEventCancelIndicator="false" spliceImmediateFlag="true" eventIdComplianceFlag="true" availNum="0" availsExpected="0" outOfNetworkIndicator="true" uniqueProgramId="49152">
 <BreakDuration autoReturn="true" duration="3456000"/>
 </SpliceInsert>
</SpliceInfoSection>
{
 "info_section": {
 "table_id": "0xfc",
 "section_syntax_indicator": false,
 "private": false,
 "sap_type": "0x03",
 "sap_details": "No Sap Type",
 "section_length": 32,
 "protocol_version": 0,
 "encrypted_packet": false,
 "encryption_algorithm": 0,
 "pts_adjustment": 0.0,
 "cw_index": "0x00",
 "tier": "0x0fff",
 "splice_command_length": 15,
 "splice_command_type": 5,
 "descriptor_loop_length": 0,
 "crc": "0xe4612424"
 },
 "command": {
 "command_length": 15,
 "command_type": 5,
 "name": "Splice Insert",
 "break_auto_return": true,
 "break_duration": 38.4,
 "splice_event_id": 14294834,
 "splice_event_cancel_indicator": false,
 "out_of_network_indicator": true,
 "program_splice_flag": true,
 "duration_flag": true,
 "splice_immediate_flag": true,
 "event_id_compliance_flag": true,
 "unique_program_id": 49152,
 "avail_num": 0,
 "avails_expected": 0
 },
 "descriptors": []
}
<SpliceInfoSection xmlns="https://scte.org/schemas/35" ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
 <SpliceInsert spliceEventId="14294834" spliceEventCancelIndicator="false" spliceImmediateFlag="true" eventIdComplianceFlag="true" availNum="0" availsExpected="0" outOfNetworkIndicator="true" uniqueProgramId="49152">
 <BreakDuration autoReturn="true" duration="3456000"/>
 </SpliceInsert>
</SpliceInfoSection>
a@fu:~/build/SCTE35_threefive$

0 replies

wabiloo
Nov 5, 2024

For info, I found a small issue with the xml encode functionality, when using UPIDs - #123

0 replies

wabiloo
Nov 6, 2024

Starting a little thread about namespace.
@futzu I refer to your question/comment in #123, namely:

@wabiloo , I know SuperDave is against using the scte35 namespace, what about you? Everything I see uses it, so I'm wondering if we should.

Namespaces are one of the best things in XML, and yet one of the most misunderstood ones...

First issue: there is more than one namespace for SCTE35. The spec (SCTE35_2023r1) says it should be "http://www.scte.org/schemas/35"
Screenshot 2024年11月06日 at 19 50 13
However in the field - aside from all sort of invalid garbage - I often see sub-versions of it (eg. "http://www.scte.org/schemas/35/2016")

Therefore in my (idealistic) view, when converting to xml (with cue.xml()):

The default should be to place the canonical namespace on the root element, eg.

<SpliceInfoSection xmlns="https://scte.org/schemas/35" ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
 <SpliceInsert spliceEventId="14294834" spliceEventCancelIndicator="false" spliceImmediateFlag="true" eventIdComplianceFlag="true" availNum="0" availsExpected="0" outOfNetworkIndicator="true" uniqueProgramId="49152">
 <BreakDuration autoReturn="true" duration="3456000"/>
 </SpliceInsert>
</SpliceInfoSection>

It should be possible to provide a desired namespace (eg. cue.xml(namespace='http://www.scte.org/schemas/35/2016')) - with no need for any validation of it

<SpliceInfoSection xmlns="https://scte.org/schemas/35/2016" ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
 <SpliceInsert spliceEventId="14294834" spliceEventCancelIndicator="false" spliceImmediateFlag="true" eventIdComplianceFlag="true" availNum="0" availsExpected="0" outOfNetworkIndicator="true" uniqueProgramId="49152">
 <BreakDuration autoReturn="true" duration="3456000"/>
 </SpliceInsert>
</SpliceInfoSection>

it should also be possible to get it without a namespace (eg. cue.xml(namespace=None))

<SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
 <SpliceInsert spliceEventId="14294834" spliceEventCancelIndicator="false" spliceImmediateFlag="true" eventIdComplianceFlag="true" availNum="0" availsExpected="0" outOfNetworkIndicator="true" uniqueProgramId="49152">
 <BreakDuration autoReturn="true" duration="3456000"/>
 </SpliceInsert>
</SpliceInfoSection>

Also fine by me if case 3 (no namespace) is the default, leaving the user to use the namespace argument to do case 1

After that - as @davemevans said - it fundamentally becomes the responsibility of the developer who wants to take that payload and insert it into a larger document (say: in an MPEG-DASH presentation aka MPD) to do the "right" thing, which is to say not treat it as if it's a text file, but instead use a proper XML writer, which will ensure that it's inserted in the correct way, either as:

<MPD>
<!-- ... --->
 <EventStream timescale="90000" schemeIdUri="urn:scte:scte35:2013:xml">
 <Event>
 <SpliceInfoSection xmlns="https://scte.org/schemas/35" ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
 <SpliceInsert spliceEventId="14294834" spliceEventCancelIndicator="false" spliceImmediateFlag="true" eventIdComplianceFlag="true" availNum="0" availsExpected="0" outOfNetworkIndicator="true" uniqueProgramId="49152">
 <BreakDuration autoReturn="true" duration="3456000"/>
 </SpliceInsert>
 </SpliceInfoSection>
 </Event>
 </EventStream>
<!--... --->
</MPD>

or as

<MPD xmlns:scte35="https://scte.org/schemas/35" >
<!-- ... --->
 <EventStream timescale="90000" schemeIdUri="urn:scte:scte35:2013:xml">
 <Event>
 <scte35:SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
 <scte35:SpliceInsert spliceEventId="14294834" spliceEventCancelIndicator="false" spliceImmediateFlag="true" eventIdComplianceFlag="true" availNum="0" availsExpected="0" outOfNetworkIndicator="true" uniqueProgramId="49152">
 <scte35:BreakDuration autoReturn="true" duration="3456000"/>
 </scte35:SpliceInsert>
 </scte35:SpliceInfoSection>
 </Event>
 </EventStream>
<!--... --->
</MPD>

or any other form that is valid XML and applying namespaces in the correct way.

Now, if you wanted to be extra-friendly to the people who don't want to have to understand how XML works and want to use is as a text file, you could offer to add a prefix, with something like cue.xml(ns_prefix="scte35"), which would generate

<scte35:SpliceInfoSection xmlns:scte35="https://scte.org/schemas/35" ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
 <scte35:SpliceInsert spliceEventId="14294834" spliceEventCancelIndicator="false" spliceImmediateFlag="true" eventIdComplianceFlag="true" availNum="0" availsExpected="0" outOfNetworkIndicator="true" uniqueProgramId="49152">
 <scte35:BreakDuration autoReturn="true" duration="3456000"/>
 </scte35:SpliceInsert>
</scte35:SpliceInfoSection>

and as before allowing the user to also specify a non-default prefix, and/or non-default namespace (to match whatever garbage their packager does), such as "normal" ones: cue.xml(namespace=None, ns_prefix="SCTE35"):

<SCTE35:SpliceInfoSection ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
 <SCTE35:SpliceInsert spliceEventId="14294834" spliceEventCancelIndicator="false" spliceImmediateFlag="true" eventIdComplianceFlag="true" availNum="0" availsExpected="0" outOfNetworkIndicator="true" uniqueProgramId="49152">
 <SCTE35:BreakDuration autoReturn="true" duration="3456000"/>
 </SCTE35:SpliceInsert>
</SCTE35:SpliceInfoSection>

or funky ones (don't laugh, I've seen something not far from that in the real world): cue.xml(namespace="ns1", ns_prefix="toto"):

<toto:SpliceInfoSection xmlns:toto="ns1" ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
 <toto:SpliceInsert spliceEventId="14294834" spliceEventCancelIndicator="false" spliceImmediateFlag="true" eventIdComplianceFlag="true" availNum="0" availsExpected="0" outOfNetworkIndicator="true" uniqueProgramId="49152">
 <toto:BreakDuration autoReturn="true" duration="3456000"/>
 </toto:SpliceInsert>
</toto:SpliceInfoSection>

That's my thoughts on converting to XML. Now, when it comes to parsing from XML, it gets much harder... because there is a lot of junk out there, with all those various namespaces, invalid ones (looking at you MediaKind!), and it would be quite a limitation to want to be normative in this instance.

Want an example (again, real life)?

<EventStream schemeIdUri="urn:scte:scte35:2014:xml+bin" timescale="90000">
 <Event id="289535">
 <scte35:Signal>
 <Binary>
/DBeAAAAAAAAAP/wBQb+On0tBgBIAhRDVUVJAARqC3//AAAUmXAAADAKEAIfQ1VFSQAEav9/vwwQQURGUgEzoQE0PioEagWwGAIAAAIPQ1VFSQAEagp/vwAAMQkQuJbGZg==</Binary>
 <scte35:SpliceInfoSection scte35Point="155399966445254">
 <scte35:SegmentationDescriptor segmentationEventId="289291" segmentationTypeId="48" segmentationDuration="PT15S">AhRDVUVJAARqC3__AAAUmXAAADAKEA==</scte35:SegmentationDescriptor>
 </scte35:SpliceInfoSection>
 </scte35:Signal>
 </Event>
</EventStream>

3 replies

wabiloo Nov 6, 2024

And another random one that I had a 2 months' fight with a client about:

 <EventStream schemeIdUri="urn:scte:scte35:2013:xml" timescale="10000000" xmlns:scte35="ns2"> 
 <Event presentationTime="0" duration="0"> 
 <scte35:SpliceInfoSection scte35:ptsAdjustment="0"> 
 <scte35:TimeSignal> 
 <scte35:SpliceTime scte35:ptsTime="38019600"/> 
 </scte35:TimeSignal> 
 <scte35:SegmentationDescriptor scte35:segmentationEventId="1" scte35:segmentationEventCancelIndicator="false" scte35:segmentationDuration="0" scte35:segmentationTypeId="52" scte35:segmentNum="0" scte35:segmentsExpected="0" scte35:subSegmentNum="152" scte35:subSegmentsExpected="190"> 
 <scte35:SegmentationUpid scte35:segmentationUpidType="9" scte35:segmentationUpidFormat="hexbinary">505245564945573A6361626C654C6162732D66696C6D732E636F6D</scte35:SegmentationUpid> 
 </scte35:SegmentationDescriptor> 
 </scte35:SpliceInfoSection> 
 <scte35:Ext scte35:availType="52"/> 
 </Event>
 </EventStream>

I took offence with the use of the prefix on the attributes (which is not necessary at best, as implicit, and in my reading of it goes against the SCTE35 XSD). Yet, the SCTE Workgroup 5 - contacted by the customer - ruled that:

A prefix (while not strictly required) can be added to either the element or attribute without violating the standard.

davemevans Nov 6, 2024

Excellent summary @wabiloo 👌 This seems like a great set of feature ideas. I would still prefer option (1) to be the default.

For the record, I love namespaces. It's just that, as stated at the top, they are so often misunderstood.

wabiloo Nov 7, 2024

I'd be happy to (try and) contribute that to the code @futzu - in the coming days if I manage to carve out the time

futzu
Nov 19, 2024
Maintainer Author

1 reply

wabiloo Nov 20, 2024

XML Namespaces are often URLs by convention. These URLs do not necessarily return an actual document...

futzu
Nov 19, 2024
Maintainer Author

<EventStream schemeIdUri="urn:scte:scte35:2014:xml+bin" timescale="90000">
 <Event id="289535">
 <scte35:Signal>
 <Binary>
/DBeAAAAAAAAAP/wBQb+On0tBgBIAhRDVUVJAARqC3//AAAUmXAAADAKEAIfQ1VFSQAEav9/vwwQQURGUgEzoQE0PioEagWwGAIAAAIPQ1VFSQAEagp/vwAAMQkQuJbGZg==</Binary>
 <scte35:SpliceInfoSection scte35Point="155399966445254">
 <scte35:SegmentationDescriptor segmentationEventId="289291" segmentationTypeId="48" segmentationDuration="PT15S">AhRDVUVJAARqC3__AAAUmXAAADAKEA==</scte35:SegmentationDescriptor>
 </scte35:SpliceInfoSection>
 </scte35:Signal>
 </Event>
</EventStream>

That is explicitly not allowed by 214,

You cannot have two messages in one event.
Segmentation Descriptors do not have a value..
what the hell is a sctePoint? I haven't seen that before.

@wabiloo
I have to ask, do you work for Disney? I did some work for Disney, and that xml has a Disney feel to it. :)

4 replies

wabiloo Nov 20, 2024

Hi @futzu, no, not working for Disney. Various other dinosaur clients though (and some nice ones)

wabiloo Nov 20, 2024

This is not a case of 2 messages in 1 event, but 1 event with 2 representations (XML and binary) of the same message.
I agree wholeheartedly. It confuses things. This is likely the UPID information, didn't try to look into it too much
Neither have I. But 214 does allow random attributes to be added
Screenshot 2024年11月20日 at 13 40 36

I don't think there's much need to spend time on this one, it was just as an example of things seen in the field.

futzu Nov 21, 2024
Maintainer Author

It allows random attributes but not random nodes.
They say only one Cue per Event.
I hope not too many folks are using that format.

futzu Nov 21, 2024
Maintainer Author

This is the whole problem, so much of the spec is a not really clear.
I can't find the 2013 and 2014 schemas DASH references , I also can't find the SCTE214 schema.
I am going to that committee, at least that way I'll know what's really going on.

futzu
Nov 19, 2024
Maintainer Author

7 replies

futzu Nov 21, 2024
Maintainer Author

that's out just now in v2.4.97.

wabiloo Nov 21, 2024

Cool. And how do I set the xmlns:cat to be something else? (for example "https://scte.org/schemas/35/2016")

I'm guessing something like the following?

x.attrs['xmlns'] = "https://scte.org/schemas/35/2106"
x.set_ns(ns='cat', attrs=False)
x

On my phone, can't try by myself

futzu Nov 21, 2024
Maintainer Author

Just set it.
I use the binary here just to limit output.

>>>> x=cue.xml(binary=True)
>>>> x.attrs
{'xmlns': 'https://scte.org/schemas/35'}
>>>> x.attrs['xmlns']="whatever"
>>>> x.attrs
{'xmlns': 'whatever'}
>>>>

I really don't think anybody is going to be changing that except you,
and why would you for SCTE-35?

I'm just curious.

wabiloo Nov 21, 2024

To match what my ~~(削除) dinosaurs (削除ここまで)~~ clients are doing (when I don't get a chance to get them to correct their ways)

futzu Nov 21, 2024
Maintainer Author

Once you change the xmlns, it's no longer SCTE-35.
I'll leave what's there in there ,
If you want to expand it further, I'll accept a patch from you.

futzu
Nov 22, 2024
Maintainer Author

that being said, I just came with a better way to do it,
just in time namespaces, the namespace isn't applied until it's rendered.

@fu:~/build/SCTE35_threefive$ pypy3
Python 3.9.16 (7.3.11+dfsg-2+deb12u2, May 20 2024, 22:08:06)
[PyPy 7.3.11 with GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>> with open('xml.xml','r') as ex: exemel=ex.read() # Read in the xml from a file 
>>>> from threefive import Cue
>>>> cue=Cue(exemel)
>>>> x=cue.xml()
>>>> x.set_ns() # <-------------- clear the "scte35" namespace
>>>> x
<SpliceInfoSection xmlns="https://scte.org/schemas/35" ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
 <SpliceInsert spliceEventId="1" spliceEventCancelIndicator="false" spliceImmediateFlag="false" eventIdComplianceFlag="true" availNum="0" availsExpected="0" outOfNetworkIndicator="false" uniqueProgramId="1">
 <Program>
 <SpliceTime ptsTime="10800000"/>
 </Program>
 </SpliceInsert>
</SpliceInfoSection>
>>>> x.set_ns("toto") # <--------- set namespace
>>>> x.set_ans("fufu") # <----------set attribute namespace
>>>> x
<toto:SpliceInfoSection fufu:PtsAdjustment="0" fufu:ProtocolVersion="0" fufu:SapType="3" fufu:tier="4095" xmlns:fufu="https://scte.org/schemas/35">
 <toto:SpliceInsert fufu:SpliceEventId="1" fufu:SpliceEventCancelIndicator="false" fufu:SpliceImmediateFlag="false" fufu:EventIdComplianceFlag="true" fufu:AvailNum="0" fufu:AvailsExpected="0" fufu:OutOfNetworkIndicator="false" fufu:UniqueProgramId="1">
 <toto:Program>
 <toto:SpliceTime fufu:ptsTime="10800000"/>
 </Program>
 </SpliceInsert>
</SpliceInfoSection>
>>>> x.set_ns() # <-------- clear namespace
>>>> x
<SpliceInfoSection fufu:PtsAdjustment="0" fufu:ProtocolVersion="0" fufu:SapType="3" fufu:tier="4095" xmlns:fufu="https://scte.org/schemas/35">
 <SpliceInsert fufu:SpliceEventId="1" fufu:SpliceEventCancelIndicator="false" fufu:SpliceImmediateFlag="false" fufu:EventIdComplianceFlag="true" fufu:AvailNum="0" fufu:AvailsExpected="0" fufu:OutOfNetworkIndicator="false" fufu:UniqueProgramId="1">
 <Program>
 <SpliceTime fufu:ptsTime="10800000"/>
 </Program>
 </SpliceInsert>
</SpliceInfoSection>
>>>> x.set_ans() # <------------- clear attribute namespace
>>>> x
<SpliceInfoSection xmlns="https://scte.org/schemas/35" ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
 <SpliceInsert spliceEventId="1" spliceEventCancelIndicator="false" spliceImmediateFlag="false" eventIdComplianceFlag="true" availNum="0" availsExpected="0" outOfNetworkIndicator="false" uniqueProgramId="1">
 <Program>
 <SpliceTime ptsTime="10800000"/>
 </Program>
 </SpliceInsert>
</SpliceInfoSection>
>>>>

7 replies

wabiloo Nov 22, 2024

Nice!

This seems to be missing quotes though:
<SpliceInfoSection xmlns=https://iodisco.com/scte35

futzu Nov 22, 2024
Maintainer Author

shit :)

futzu Nov 23, 2024
Maintainer Author

>>>> y.namespace.prefix_all()
>>>> y
<scte35:SpliceInfoSection xmlns:scte35="https://scte.org/schemas/35" scte35:PtsAdjustment="0" scte35:ProtocolVersion="0" scte35:SapType="3" scte35:tier="4095">
 <scte35:SpliceInsert scte35:SpliceEventId="1" scte35:SpliceEventCancelIndicator="false" scte35:SpliceImmediateFlag="false" scte35:EventIdComplianceFlag="true" scte35:AvailNum="0" scte35:AvailsExpected="false" scte35:OutOfNetworkIndicator="false" scte35:UniqueProgramId="1">
 <scte35:Program>
 <scte35:SpliceTime scte35:ptsTime="10800000"/>
 </scte35:Program>
 <scte35:BreakDuration scte35:AutoReturn="true" scte35:duration="2971107"/>
 </scte35:SpliceInsert>
</scte35:SpliceInfoSection>
>>>> y.namespace.prefix_all(False)
>>>> 
>>>> 
>>>> 
>>>> y
<scte35:SpliceInfoSection xmlns:scte35="https://scte.org/schemas/35" ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
 <scte35:SpliceInsert spliceEventId="1" spliceEventCancelIndicator="false" spliceImmediateFlag="false" eventIdComplianceFlag="true" availNum="0" availsExpected="false" outOfNetworkIndicator="false" uniqueProgramId="1">
 <scte35:Program>
 <scte35:SpliceTime ptsTime="10800000"/>
 </scte35:Program>
 <scte35:BreakDuration autoReturn="true" duration="2971107"/>
 </scte35:SpliceInsert>
</scte35:SpliceInfoSection>
>>>> y.namespace.xmlns()
'xmlns:scte35="https://scte.org/schemas/35"'
>>>> y.namespace.uri
'https://scte.org/schemas/35'
>>>> y.namespace.uri="https://iodisco.com/scte35"
>>>> y.namespace.ns
'scte35'
>>>> y.namespace.ns="iodisco"
>>>> 
>>>> 
>>>> 
>>>> y
<iodisco:SpliceInfoSection xmlns:iodisco="https://iodisco.com/scte35" ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
 <iodisco:SpliceInsert spliceEventId="1" spliceEventCancelIndicator="false" spliceImmediateFlag="false" eventIdComplianceFlag="true" availNum="0" availsExpected="false" outOfNetworkIndicator="false" uniqueProgramId="1">
 <iodisco:Program>
 <iodisco:SpliceTime ptsTime="10800000"/>
 </iodisco:Program>
 <iodisco:BreakDuration autoReturn="true" duration="2971107"/>
 </iodisco:SpliceInsert>
</iodisco:SpliceInfoSection>
>>>> y.namespace.prefix_all()
>>>> 
>>>> 
>>>> 
>>>> 
>>>> y
<iodisco:SpliceInfoSection xmlns:iodisco="https://iodisco.com/scte35" iodisco:PtsAdjustment="0" iodisco:ProtocolVersion="0" iodisco:SapType="3" iodisco:tier="4095">
 <iodisco:SpliceInsert iodisco:SpliceEventId="1" iodisco:SpliceEventCancelIndicator="false" iodisco:SpliceImmediateFlag="false" iodisco:EventIdComplianceFlag="true" iodisco:AvailNum="0" iodisco:AvailsExpected="false" iodisco:OutOfNetworkIndicator="false" iodisco:UniqueProgramId="1">
 <iodisco:Program>
 <iodisco:SpliceTime iodisco:ptsTime="10800000"/>
 </iodisco:Program>
 <iodisco:BreakDuration iodisco:AutoReturn="true" iodisco:duration="2971107"/>
 </iodisco:SpliceInsert>
</iodisco:SpliceInfoSection>
>>>> y.namespace.ns="gonzo"
>>>> 
>>>> 
>>>> 
>>>> 
>>>> y
<gonzo:SpliceInfoSection xmlns:gonzo="https://iodisco.com/scte35" gonzo:PtsAdjustment="0" gonzo:ProtocolVersion="0" gonzo:SapType="3" gonzo:tier="4095">
 <gonzo:SpliceInsert gonzo:SpliceEventId="1" gonzo:SpliceEventCancelIndicator="false" gonzo:SpliceImmediateFlag="false" gonzo:EventIdComplianceFlag="true" gonzo:AvailNum="0" gonzo:AvailsExpected="false" gonzo:OutOfNetworkIndicator="false" gonzo:UniqueProgramId="1">
 <gonzo:Program>
 <gonzo:SpliceTime gonzo:ptsTime="10800000"/>
 </gonzo:Program>
 <gonzo:BreakDuration gonzo:AutoReturn="true" gonzo:duration="2971107"/>
 </gonzo:SpliceInsert>
</gonzo:SpliceInfoSection>
>>>> y.namespace.prefix_all(False)
>>>> y
<gonzo:SpliceInfoSection xmlns:gonzo="https://iodisco.com/scte35" ptsAdjustment="0" protocolVersion="0" sapType="3" tier="4095">
 <gonzo:SpliceInsert spliceEventId="1" spliceEventCancelIndicator="false" spliceImmediateFlag="false" eventIdComplianceFlag="true" availNum="0" availsExpected="false" outOfNetworkIndicator="false" uniqueProgramId="1">
 <gonzo:Program>
 <gonzo:SpliceTime ptsTime="10800000"/>
 </gonzo:Program>
 <gonzo:BreakDuration autoReturn="true" duration="2971107"/>
 </gonzo:SpliceInsert>
</gonzo:SpliceInfoSection>

wabiloo Nov 23, 2024

Looking great.
Just a suggestion: prefix_all() is not that obvious. Suggestion: prefix(elements=True, attrs=False)

Or because i can't think of a realistic use case in which one would prefix the attributes but not the elements: prefix_attrs(True)