We discovered over the weekend that S4M has the ability to schedule events from 2 consecutive broadcast dates in the same playlist. This resulted in some of Saturday’s schedule being broadcast on Sunday. Obviously, this resulted in an impact to the bottom line and we needed to prevent this from happening again. Our solution was to have me modify the application that converts the S4M H-series formatted playlist to a D-series one so that if more than one dldate node is found in the H-series playlist, to flag the playlist and not translate it for our playout servers.
The bottom line is this: don’t assume S4M will only generate playlists for a single broadcast day.
Due to the switchover currently happening in moving all channels to HD, sometimes trafficking systems like S4M may have been told to produce a playlist that contains an event which is HD media on a SD channel. Usually operations will catch this and substitute the HD content with the SD version. If they don’t then the playout server would crash on attempting to play the HD content on the SD channel. Some broadcasters may have a single playout server broadcasting for multiple channels, so crashing a server would essentially bring down all those channels too. Finally, there are usually backup servers which are hooked up as dual support, so the final chain result would bring down the backups also.
How do we prevent this? Well, in the S4M case, it does track whether an event is in HD or SD, but the vanilla install of S4M doesn’t pass this field through to the playlist. So we weighed the cost of asking S4M to make the code change to add the <highdefinition> node to the dlevent node such as:
Against the cost of several channels not airing for some downtime. Presented with this tradeoff, the executives went ahead and asked S4M to make the code change. Now, in my broadcaster’s particular case, S4M does not produce a playlist that our playout servers can understand, since S4M naturally writes in H-series format but our playout servers are DALs, and need D-series playlists. Therefore we decided that the translator I wrote to convert H-series to D-series playlists be modified to identify events that are HD (have a value of 1), and check to see if the channel is HD or not. If there is HD content scheduled for a non-HD channel, then operations will be alerted by adding HD to the column where the dlhnumber normally goes. My broadcaster uses dlhnumber to store the tape number of the content to play out, so since this content won’t be played out, we decided to substitute that value with a flag indicating HD content has been scheduled for a SD channel. Using the event id in the playlist, operational staff can check back to see what content was intended to be played and manually add in the SD version.
After the DST time change on March 7, 2009, we discovered that certain events in the playlist generated from S4M were out of chronological order. To compensate for this, I made a change to the playlist translator so when it converted the S4M H-class file to the DAL playout server format, all the events would be sorted in chronological order, from 6AM to 5:59AM the following day (broadcast time).
Although this post is more specific to EMC Documentum, I want to cover off this information as it is difficult to locate on the web, plus Documentum is a system of great value to broadcasters.
Documentum is an Enterprise Content Management (ECM) system which stores unstructured data as objects in its repository. These data include images, music, movies and any type of electronic file or record you can think of. Each of these objects in Documentum has its own unique and persistant identifier, known as the r_object_id. Even different versions of the same content have their own unique r_object_id.
What is not advertised much on the web is what the r_object_id means. Yes, it is an unique persistent identifier, but it also tells the type of Documentum object it represents, as well as the repository where the object is located.
The r_object_id is a 16 digit hexadecimal number. For example 0900055080007f1c is a valid r_object_id.
The first 2 digits, 09 in this example, are the tag, which indicate the type of Documentum object. Obviously, because Documentum supports custom types which inherit from the Documentum super type, there can be an immense number of types in the repository. Therefore, the tag only identifies the Documentum super type that the object is of or what it is inherited from. The tags and what Documentum super types they refer to are listed below: 09 dm_document 08 dm_sysobject, dm_script, dm_query 06 dmr_content 05 dmr_containment 41 dm_note 0c dm_cabinet 0b dm_folder 11 dm_user 12 dm_group 18 dm_router 19 dm_registered 27 dm_format 0d dm_assembly 3a dm_location
The other parts of the r_object_id, the repository id and the unique id generated by the server, are self explanatory. This information is covered in the Technical Fundamentals of Documentum course and manual, but oftentimes we throw those manuals away or lose them. :)
What is no touch deployment? A no touch deployment application exists in a directory underneath IIS’ wwwroot, and is called via an URL and run in the context of the caller’s machine. The advantage of no touch deployment applications is that they do not have to be installed on every user’s PC; a single desktop client application can be deployed to a local intranet server for multiple users yet be easily maintained and updated like a web application. This is ideal in many ways as it allows a development team to deploy rich thick client functionality to the masses without requiring the burden of a large IT rollout project.
Problem: No touch deployment applications use the .NET framework’s configuration system differently than regular desktop applications. Since the application is served via IIS, IIS determines which configuration file to associate with the application. This is done by appending the “.config” extension to the end of the calling URL and sending that file back to the user’s machine. For example, if the user called the application with the URL http://intranetserver/myapp.exe, IIS would send the configuration file on this URL: http://intranetserver/myapp.exe.config.
If you don’t believe me create a no-touch deployment application that uses the System.Configuration namespace. It will fail. Then create a no-touch deployment application that will accept query string arguments and call a web service. Proceed to call it via the URL. It will also fail.
However, this can be easily resolved by creating a custom configuration handler and configuring it within IIS. The drawback is that part of this configuration involves allowing anonymous access to the no touch deployment application. In a secure IT environment, this is not allowed. The broadcast industry is one of those that is closely regulated by the government, and thus most broadcasting corporations have lots of IT security regulations enforced upon them.
So the problem really is, how do you override the default configuration system in the no-touch deployment application so you can call web services within it without allowing anonymous users access?
Solution: The solution is still relatively simple. Strip off the query string (?silent=true) from the URL and set the APP_LAUNCH_URL variable of the application’s current domain (AppDomain.CurrentDomain) to the trimmed URL. Then construct the configuration URL using this URL so it is now http://intranetserver/myapp.exe.config and set the APP_CONFIG_FILE variable of AppDomain.CurrentDomain to this value. This can be done using AppDomain.CurrentDomain.SetData(“APP_LAUNCH_URL”, newURL) etc…
Broadcast No Touch Deployment Handler
Description: This abstract class provides the basic functionality required to have a functioning no touch deployment application in a secure IT environment. The constructor will have the URL processing features required to avoid the configuration system problems a no-touch deployment application experiences in a secure IT environment, and there is an abstract method to implement that processes query string arguments.
Functional How-To: Just create your own implementation of this abstract class in your application and implement the process query string method yourself. Then instantiate an instance of this class at the beginning of your application, preferably near the top of your Main() function in the Program.cs file.
When to use it: In the broadcasting technology environment, there are strict controls on who can access what, as well as the number of executables that can exist in a production environment. In one particular case, to drive a process, a no touch deployment executable had to double as a thick client as well as a web service client called by a scheduled task. Since in both cases the application would be called via an URL, the way to distinguish when to start it as a thick client and when to start it as a silent web service client was via arguments sent via an URL query string. Since the IT environment did not support anonymous access, we could not build a custom configuration handler for IIS. At first we tried to disable calls to the .NET framework configuration system, but soon found out that the SoapHttpClientProtocol and WebRequest classes of the System.Net namespace all initialized the configuration system by default. After much experimentation, I found this approach to be the best.
Every piece of TV content has an associated cuesheet indicating at what times different segments, such as commercials, voiceovers or show segments should be broadcast. These cuesheets would be fed into a broadcast trafficking system so the broadcast server would know what content to play at what timecodes.
In my current television broadcasting company, EMC Documentum is used to store the cuesheet information for an episode version by representing episodes as objects in the Documentum repository, and attaching the cuesheet data as attributes of the objects. I wrote a WDK (Documentum’s Web Development Kit) customization that displayed the cuesheet information as a panel available upon clicking the “i” icon of an object. Once the “i” icon was clicked, the customization would do some basic business logic checking to determine if a cuesheet for this episode should be allowed into S4M (some regions were not on Documentum yet so could not take advantage of our cuesheet customizations), and set the value of a true/false flag in the java class behind the JSP page. Then the user would be allowed to make changes to the cuesheet information. Afterwards, when the user wanted to save their changes, they could click on the OK button. This would trigger the onCommitChanges() method, which would check the value of the flag first before continuing. If the flag was set to not print the cuesheet, then the cuesheet file would not have been written to the monitored directory for S4M to import.
Over a weekend, this change was released to production, and seemed to work fine until suddenly cuesheets stopped being written to the monitored directory. After some careful examination of the code, I realized the flag was not being reset upon each view of the cuesheet WDK page. I then set the flag to reset upon every view of the cuesheet page, which resolved the issue. The lesson learnt from this is that values set to members of the WDK code-behind class would be remembered throughout the life of the WDK session. Therefore class members should be set back to their default values upon every view of the WDK component, by adding initialization calls to the initControls() method.
TV Broadcasters divide their broadcast time into segments indicated on cuesheets. These cuesheets contain information detailing what content to display at specific times. Each of these information bits contain the type of event to broadcast (show segment or commercial), as well as the timecodes when they start and end. The timecodes are a little different than regular timestamps, as in addition to hours, minutes and seconds, they contain the number of frames, as a second can be divided into 30 frames, and a frame is the smallest unit of time possible for TV broadcasting.
Each broadcast trafficking system has their own cuesheet format; S4M uses XML. However, the information contained in these cuesheets is generally the same, regardless of the type of trafficking system used. Usually this is unimportant as most television broadcasters only use one trafficking system, but what happens when a television station buys another? There is the opportunity to inherit a new trafficking system, and thus the opportunity to share information between the systems to provide visibility to the executives.
So what is the best way to store cuesheet information so it can be imported by different broadcast trafficking systems and be visible to the executives? I recommend storing the data in an ECM (Enterprise Content Management) system and providing customizations to export it in different trafficking systems’ formats.
For my particular scenario, EMC Documentum was chosen because of its rich media features and completeness of ECM features (I’m sure that my being a big fan of Documentum had nothing to do with it.) What we decided was to create a taxonomy in the repository that organized television shows by seasons and episode versions, and store the cuesheet information as metadata of the episode version objects. I’ll blog more on the taxonomy structure in the future; for now, I would like to focus on the cuesheets.
In the S4M cuesheets are 2 node types: SEC and EEC. These timecodes are computed values from the VO Squeeze TC in attribute of the episode version, and they represent the end credits of a show. On Jan 15, 2009 we discovered a bug in S4M that would schedule programs with SEC and EEC timecodes incorrectly. It was placing the end credits as separate segments on the playlist when it should in fact be one segment.
Of course an urgent fix was requested from S4M, but what happens in the meantime? Master Control cannot handle the end credits properly so we needed a workaround immediately before we could generate new playlists. What we decided was to not print out the SEC and EEC timecodes until the fix was provided.
The implementation of the fix was relatively easy. I already made a WDK (Documentum's Web Development Kit) customization in Documentum to print the cuesheets, so I just added a configuration value to the component’s XML configuration file to determine whether to print SEC and EEC timecodes. If the value was present, the timecodes would be printed. Anything else would default to hiding the timecodes.
Therefore in production we currently have the value set to not print the end credits for S4M cuesheets. Once S4M releases a fix, I will release a new configuration file with the value set to print the end credits. Configuration files are insignificant work so the turnaround time will be under 30 minutes.
Here is an interesting facet of the S4M system in use by certain broadcasters I would like to detail below.
Playlists S4M can write playlists in a format known as H-class, which is a XML format listing events as “dlevent” XML nodes. Each event node contains a start time which is stored as the value of the “dlstart” child node. In addition, the total duration of which the event runs is stored as the “dlduration_hmsf” child node.
Events There are 2 main classes of events: primary and secondary. Primary usually refers to a program or commercial segment. Secondary events run in parallel to primary events, and can be voiceovers, snipes or commercial secondary events.
Anchor End When writing H-class playlists, S4M sometimes writes secondary events that have a “dlstart” time that occurs after the primary event it is supposed to run within ends. This is a glitch with the S4M trafficking system but what this means is that the secondary event runs near the end of the primary event. The exact actual time the secondary event runs at can be determined by taking the difference of the primary event’s end time and the secondary event’s start time, and subtracting this offset from the primary event’s end time to determine the true “dlstart” time of this secondary event. To put this in the form of mathematical equations:
Primary event end time = primary event start time (dlstart) + primary event duration (dlduration_hmsf)
Now, determine if this secondary event is an anchor end or not: If (Secondary event start time (dlstart) > Primary event end time) then This is an anchor end.
Offset = Secondary event start time (dlstart) – Primary event end time Secondary event offset time = Primary event end time – Offset (calculated above)