XML Representation #75

Thanks! I did spot your comment and put a heart on it.

It's working well for us and we will likely take it to production.

I'm not convinced about the correctness of a few bits around segmentation UPIDs because the spec doesn't really give enough information or examples and there's not many complex public examples out there. It is also missing some marker types (because we didn't need them), though they'd be fairly straightforward to add.

Mainly though it feels a bit hacked on - as above it'd be really nice to have some unified way of writing any format of output, but I couldn't see a way to achieve that.

The key thing for me was that threefive already had full support for everything needed and was easy and intuitive to use, so the extensions were pretty much just writing out some strings with the right names 🎉

This looks much neater. I'll try to find some time this week to make these changes.

Not yet, but that's a good shout and I will do and fix any issues.

name to camel case

def to_cc(string):
 # string =string[0].upper()+ string[1:]
 while "_" in string:
 idx = string.index("_")
 n= string.replace("_","",1)
 o= n[:idx]+n[idx].upper()+n[idx+1:]
 string=o
 return string

>>>> a = "some_name_for_something"
>>>> to_cc(a)
'someNameForSomething'
>>>> 

better way

def to_cc(string):
 new_string = string.title().replace("_","")
 return new_string[0].lower()+new_string[1:]

a = "hey_you_there"
to_cc(a)
'heyYouThere'

I'm trying to think of the easiest way to convert to xml, some way to batch process it like

def val2xml(val):
 """
 val2xmlconvert val for xml
 """
 if isinstance(val,bool):
 return bool2xml(val)
 if isinstance(val,(int,float)):
 return num2xml(val)
 if isinstance(val,str):
 if val.lower()[:2]="0x":
 return hex2xml(val)
 else:
 return val
def key2xml(string):
 """
 key2xml convert name to camel case
 """
 new_string = string.title().replace("_","")
 return new_string[0].lower()+new_string[1:]
def mk_xml(name,obj):
 new_obj ={key2xml(k):val2xml(v) for k,v in vars(obj).items()}
 return ET.Element(name,new_obj)
 
 
si =SpliceInsert()
elem = mk_xml("spliceInsert", si)

I believe the schema allows us to add extra stuff so we really don't have trim extra stuff

Ah, found it: master...davemevans:scte35-threefive:XML.
Pretty far behind the HEAD though... 😢

Dave's a pretty cool guy,
For Dash you want to read SCTE-214, which describes SCTE-35 in DASH,
https://wagtail-prod-storage.s3.amazonaws.com/documents/SCTE_214-1_2024.pdf .

Sorry man, I just don't like XML.

Adrian

Yep, sorry, been rather swamped with work and family matters.

When the scte payload in the XML is base64, that is indeed what I'm doing already.
When it isn't, I am handling it without using threefive at this stage, though I am planning to attempt to perform the conversion into a Cue object - so that I only have one mechanism to interact with in the rest of the code.
However, I have no idea when I'll get to that part. Will keep you updated

@davemevans
I got it smoothed out a bit for dash, but this just parsing, it's kind of slick and works okay with dash.
https://github.com/futzu/SCTE35_threefive/blob/master/dash.md
https://github.com/futzu/SCTE35_threefive/blob/master/threefive/dash.py

Hi @futzu, I'm about to bring this into my tool.
However I'm confused now. For parsing an <Event ...> node into a Cue, should I use your dashscte35 repo, or the threefive.dash2cue function in threefive?
I'm guessing I should wait for v2.4.71 to be released?

I'm folding into threefive, wait for 2.4.71, I'm almost done.
What I need to finish is rendering Cues as xml. so it works round trip.
A cue can be made into xml and a cue can be made from xml.
I committed threefive.xml last night,that is most of heavy lifting for rendering xml and I'm about to commit updates for TimeSignal and SpliceInsert Commands now. I still have to do SegmentationDescriptor and upids.

A week from now at most I would say.
There are several improvements to the Dash stuff.
I added a method for parsing an mpd file directly, that returns a list of Cues
works like this:

from threefive import DashSCTE35
ds = DashSCTE35()
mpd = 'fu.mpd' # can be local file or http/https
cues = ds.parse_mpd(mpd) # Cues is a list of Cue instances

@wabiloo What's probably going to happen is it will work like JSON and python dictionaries, you'll just call Cue.load() and pass it the Event xml and it load the data into the Cue instance.
Last night I did Cue.xml() to output xml from a Cue and it works fairly well.

a@fu:~$ pypy3
Python 3.9.16 (7.3.11+dfsg-2+deb12u2, May 20 2024, 22:08:06)
[PyPy 7.3.11 with GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>> from threefive import Cue
>>>> cue=Cue('/DA2AAHOR/nwAAAABQb+QZYoVAAgAh5DVUVJSAAAbX/PAAEo0GwICAAAAAAt86rXNAAAAAAfqUZ6')
>>>> cue.decode()
True
>>>> cue.xml()
<scte35:SpliceInfoSection ptsAdjustment="86175.453689" protocolVersion="0" tier="0">
 <scte35:TimeSignal>
 <scte35:SpliceTime ptsTime="12226.2196"/>
 </scte35:TimeSignal>
</scte35:SpliceInfoSection>

Hi again,
Looks fantastic. For my needs, cue-to-xml is not relevant, but still one quick comment / question:
In your last message, I see you're adding scte35 namespace prefixes to the XML elements.
It might be good to make this optional (in case it's to be inserted into a node that changes the default namespace). Otherwise, XML parsers might choke on the string as well (if there is no namespace definition first)

@wabiloo , if you want to play with it , just clone the repo. if you need it installed , just type "make install" it will install it as v2.4.70

@wabiloo , this is not going to be stable right out the box, This is just a first draft. I don't currently even use Dash.
You need to verify everything before using any of this in production. threefive is solid, but this xml stuff is not.

I'm not going to have from_xml without to_xml

I was only letting you know for context (as I'll be unlikely to test that side of things - not having a use case for it)

this is not going to be stable right out the box

Naturally! Noted, and understood. I'll inform you if I see issues.

@wabiloo
What bugs me is all the conflicting info, it's not as much of a problem from xml as to xml. The spec Upid Xml stuff doesn't make any sense, and it doesn't at all match what I've found in the wild.

Check this out though, I added "dash_data" to the Cue JSON output for the EventStream, Event and Signal Xml nodes.
Because that duration is authoritative, and the timescale and stuff Let me show you.

 {
 "info_section": {
 "table_id": "0xfc",
 "section_syntax_indicator": false,
 "private": false,
 "sap_type": "0x03",
 "sap_details": "No Sap Type",
 "section_length": 37,
 "protocol_version": 0,
 "encrypted_packet": false,
 "encryption_algorithm": 0,
 "pts_adjustment": 2.036278,
 "cw_index": "0x0",
 "tier": "0xfff",
 "splice_command_length": 20,
 "splice_command_type": 5,
 "descriptor_loop_length": 0,
 "crc": "0x67d7213a"
 },
 "command": {
 "command_length": 20,
 "command_type": 5,
 "name": "Splice Insert",
 "time_specified_flag": true,
 "pts_time": 87064.166667,
 "break_auto_return": true,
 "break_duration": 105.0,
 "splice_event_id": 99,
 "splice_event_cancel_indicator": false,
 "out_of_network_indicator": true,
 "program_splice_flag": true,
 "duration_flag": true,
 "splice_immediate_flag": false,
 "event_id_compliance_flag": true,
 "unique_program_id": 1,
 "avail_num": 1,
 "avails_expected": true,
 "auto_return": true,
 "duration": 105.0
 },
 "descriptors": [],
 "dash_data": { # <------ dash data 
 "EventStream": {
 "timescale": 90000, 
 "scheme_id_uri": "urn:scte:scte35:2013:xml"
 },
 "Event": {
 "duration": 10500.0
 }
 }
}

What do you think?

@wabiloo if you clone the repo and do a make install for 2.4.70, you should be pretty good loading the xml with Cue.load(exemel).

I updated https://github.com/futzu/SCTE35-threefive/blob/master/dash.md with the current directions.
It does the info section, Time Signal, SpliceInsert, Segmentation Descriptor and UPIDS. I think the only thing I'm missing is avail descriptors.

I dropped the dash.py all together.
threefive.xml.XmlParser is a drop in replacement for DashSCTE35.

I also set repr to render xml in threefive.xml.Node.
You can now just print the node
I updated https://github.com/futzu/SCTE35-threefive/blob/master/dash.md

Created #103 which addresses the above and a few other bits.

Not working:

• multiple descriptors of the same type in a XML spliceinfosection (eg more than one segmentationdescriptor)
• UPID parsing doesn't really work. Dumping is closer but not right (as noted in the code). This is the hardest bit I think, particularly due to the below ...
• Probably a bunch of other stuff (very few good quality test cases available)

• I'm working toward a full MPD parse keep that in mind.

• The namespace thing is easy enough to add. I kind of like XmlParser, I was able to implement it in less code that I needed using expat.

• Dash doesn't support multiple descriptors in a info section, Does it?
  * If it does, we'll just make a descriptor list in the stuff dictionary when the xml is parsed.
  * I'll add namespace and multiple descriptors tonight.

• Dumping the UPIDs I just did to get started, I haven't figured out the actual UPID rules, that's why I'm doing that way. The rules don't make sense,
  and what I see in the wild doesn't match at all.
  This is the picture with the spec, it is legal to add any attributes we want
  image

The format identifier only applies to MPU UPIDS and segmentationUpidFormat is something new , but that is easy to handle the way I split the UPIDS, I can do that.

• One thing to note, I dropped Splice Schedule a while back, I've never seen one in the wild, I don't care to support it at all.
• I would dump DTMF too, except that NBC does still have them even though they told me don't really use them.

image

Dash doesn't support multiple descriptors in a info section, Does it?

This is SCTE35 related rather than DASH. Descriptors has maxOccurs="unbounded in the schema, so you can have more than one of the same type. As you say, not difficult to fix.

Dumping the UPIDs I just did to get started, I haven't figured out the actual UPID rules, that's why I'm doing that way. The rules don't make sense, and what I see in the wild doesn't match at all.

Yeah. This is the bit that could really do with some examples. We'll get there.
On MPUs specifically, from examples I have seen it seems you are supposed to break them into multiple SegmentationUpids but that's not 100% clear to me.

If you do decide to get on the SCTE committee (which sounds like an awesome opportunity!), you should definitely push for some canonical examples for all features in either SCTE35 or SCTE67.

Just for your context, we are actually using threefive, with XML dumping, in a SCTE224 context, rather than DASH. But it doesn't really make any difference to what this library ought to do 😄

I need to clean up XmlParser, it's a bit messy.
I realize it's kind of silly to do an xml parser with so many available, but I've really lost faith in the python guys, too many changes just so people can put their name on something. I've tried to talk to them, but they banned me from the site. :)

I was digging throw the DASH repos and all their code only supports SpliceInsert, so we are pulling ahead.

Do you currently use DASH? I don't, I'm kind of learning as I go.

I didn't you were in London, I use to date a Geordie, I spent a good bit of time in Newcastle.

I've got a few minor improvements to go in, but I'm seeing an exception when testing after rebasing the latest:

File "threefive/xml.py", line 220, in mk_descriptor
 sub_data=data[:data.index(f'</{tag}>')+len(tag)+1]

Is it due to self-closing tags (eg <AvailDescriptor providerAvailId="555"/>) or something?

Yes, we do have DASH as an output from our products so I know a bit, and have contributed to a few DASH projects.

Never been to Newcastle, but I gather it's cold. Much warmer here in the south of England 😄

Damn it, yeah it's the self closing tag. I got a fix for it , hold on.

try it now.

I just removed parse_descriptor from XmlParser and folded it into parse, that's much better.
Now we're cooking with gas. :)

That looks good. One thing I'm not sure about is if hexbinary is supposed to be 0x-prefixed or not.
Will have a think about it next week.

I always use 0x for hex values. It is super important that things stay consistent in threefive,

Let me show you how most people use upids.
msnbc:

 "segmentation_upid_type": 1, <-----------------------type 1
 "segmentation_upid_type_name": "Deprecated", <--------- Type 1 was deprecated a few years ago, most still use it.
 "segmentation_upid": "msnbc_EP043112210557", <--- this is not valid at all
 

• ABC

 "segmentation_upid_type": 1, <-------------------------type 1
 "segmentation_upid_type_name": "Deprecated", <--------------- Deprecated
 "segmentation_upid_length": 0, <-----------------------0 length
 "segmentation_upid": "", <--- empty upid
 
 }

cnn

 "segmentation_upid_type_name": "AiringID", <--- this is actually correct 
 "segmentation_upid_length": 8,
 "segmentation_upid": "0x2df3aad7",

99% of the upids I've seen are one of those three.

Specs are great , but threefive MUST be CDN compatible,
Amazon and Akamai are two important ones, and amazon makes
several mistakes with regular SCTE-35, like the base64 encoding,
they do it wrong. The method Cue.fix_bad_b64() is there just for amazon.

that's just the name of attribute that is supposed to be present on UPID xml that is not present in regular SCTE-35 it's in the xsd

image

https://demo.unified-streaming.com/k8s/live/stable/#!/DASH
Really, I expect to see more Binary in xml than the SpliceInfoSection xml.
The SpliceInfoSection xml is the old version.
https://github.com/Elfocrash/azure-docs/blob/b023db78f5a6652eb240e4d0e31d66c3effd6105/articles/media-services/media-services-specifications-live-timed-metadata.md?plain=1#L560

I feel like I'm coming as really bitchy, please don't take it the wrong way, you're doing a great job.

there is no 2023 xsd, I don't know why. That's from the 2022.

We're going to support the non-binary, absolutely, my point was that upids are rarely used correctly, so I don't want you to drive yourself crazy with them, and just match what is returned in regular scte-35 for all except the MID and MPU.

Amazon not even using format_identifier, I dont see what it could possibly do for anyone, set them all to hexbinary is fine.

upid logic should be upid.py, not descriptors.py

I mention cyclomatic complexity and using maps in the review
here's a good example

 def _xml_splice_descriptor(self,stuff):
 if "SegmentationDescriptor" in stuff:
 segdes=SegmentationDescriptor()
 segdes.from_xml(stuff)
 self.descriptors.append(segdes)
 if "AvailDescriptor" in stuff:
 availdes=AvailDescriptor()
 availdes.from_xml(stuff)
 self.descriptors.append(availdes)
 if "DTMFDescriptor" in stuff:
 dtmfdes=DtmfDescriptor()
 dtmfdes.from_xml(stuff)
 self.descriptors.append(dtmfdes)
 if "TimeDescriptor" in stuff:
 timedes=TimeDescriptor()
 timedes.from_xml(stuff)
 self.descriptors.append(timedes)

• that's a five in cyclomatic complexity, and 18 lines of code.

 def _xml_splice_descriptor(self,stuff):
 dmap={"SegmentationDescriptor" : SegmentationDescriptor,
 "AvailDescriptor": AvailDescriptor,
 "DTMFDescriptor": DtmfDescriptor,
 "DTMFDescriptor": TimeDescriptor,}
 for dname in dmap.keys():
 if dname in stuff:
 dscptr = dmap[dname]()
 dscptr.from_xml(stuff)
 self.descriptors.append(dscptr)

• That is a three in complexity and ten lines of code.

Conditionals Cause Catastrophes

avoid them whenever possible.

Quick question: For MID, multiple <SegmentationUpid>'s need to be returned. How can MID.xml() support this when xml() methods are supposed to return a single Node (to be passed to add_child)?

Also, on the value representation, worth looking at scte35-go as that has some opinions - segmentation_upid.go.