First public test version of EPG2MEI. There's a readme.txt in there which should explain everything, but the important thing to point out is that this version TEMPORARILY allows you to specify the output file. Ultimately this will be fixed as EPG.mei, but since there are no TAPs out there that support this (or the API extensions) at the moment, you can choose the output file (e.g. Freeview.mei or MyStuff.mei). *Be careful to disable any TAP that might currently be generating that file.*

_______________
Original message:

A while back I did a TAP to dump the EIT (EPG) info to a file so I could check things like whether [Sl] was working. This progressed to an almost-MEI format output which made things easier to compare to eit2mei's output. It was a bit of a system hog and also incredibly slow (2mins to dump a full EPG) because it was a quick and sloppy hack... which was also hard-coded to 5.13.40.

I've just tarted it up a bit and now it's pretty generic, even with native, language-based genre text, although I have taken the opportunity to add various tweaks, based on observations about the Freeview data. It also takes 12s to do the dumping, and that's with loads of nice Idling. It occurs to me that this now might be a good starting point for at least kicking off a discussion about what is and isn't wanted in a EPG2MEI TAP, and how some of the the main issues about timeliness and updates might be handled.

For example, in cooperation with something like SecCache, a TAP could monitor the native collection and do a main EPG dump when the majority of the data is present. Then there could be a much smaller update file containing changes, created on an interval, or those changes could be posted to TAPs as a new type of event? Finally, those critical Now&Next transitions... Going to need a good deal of input from developers who might actually want to make use of this sort of new stuff.

Then there are all the important features that have already been discussed in the past and implemented in eit2mei. Which things do people feel would still be important?

MS really assumes that it can always have the entire epg data in memory at any time, and it uses this internal copy a lot. Where it gets this copy from doesn't really matter, but it does assume that this copy will not change except when it knows about it, and when it decides that it safe to do so. It can look for updated information anywhere, and as often as deemed appropriate, or even be told that there is an update available. Making it use a copy that it is not in control of, ie one which may change without MS knowing, or when it is not ready, could be a real pain. Of course, if it was update driven then MS could update its own copy when it knows that it is safe, but I'm not sure if you are working towards just having one in-memory copy of the epg data?

Like all things, it would be something I would not know how bad it is until I got stuck in. With the current state of MS 6.0 it is not something I would want to embark on at the moment! The current method of relying on ei2tmei for epg data and crid info is working well in 6.0

Obviously, for MS any new method would have to capture the crid data for the series stuff.

I think that for the other stuff, you might have to start by outlining what might be possible for the N&N info etc. I guess you would be looking to combine the Extend kind of functionality in here too?

MyStuff currently works just with a bulk update (from eit2mei or elsewhere) -- i.e. you dump the existing EPG data and suck in a completely new one. What are your thoughts on breaking this down a bit? Reducing the overhead and allowing more immediate responses to changes? As above, this could be via small update files that you get notified about, or maybe by being sent the individual changes as they happen. I imagine this would be a bit of sea-change, requiring you to be able to modify/add/delete with your EPG data store.

As for Now&Next, that could be restricted to current recording streams (a la [Ab] and Extend), and that may be best communicated by a new type of TAP event. The complete set of Now&Next transitions can be quite large (as witnessed by the logging versions of AccurateBM/AccurateBMExtend), so that kind of thing should maybe be a small update file.

Currently my prototype does some (probably Freeview-specific) stuff to extract the full programme title, the production year, and the episode (or sub-) title and number (and it follows eit2mei in not actually removing these extra bits from the description text). Are there any assumptions about the lengths (or general make up) of those MEI fields that I ought to be aware of?

I was thinking that MEI is a format that was created to give TAPs access to EPG data, so that might be the right format for delivering the messages needed for Extend-like functionality, and maybe even for small update messages. For example, a TAP might be sent an EVT_EPG_UPDATE message with a pointer to a MEI data string in param1 and maybe a count in param2.

MyStuff currently works just with a bulk update (from eit2mei or elsewhere) -- i.e. you dump the existing EPG data and suck in a completely new one. What are your thoughts on breaking this down a bit? Reducing the overhead and allowing more immediate responses to changes? As above, this could be via small update files that you get notified about, or maybe by being sent the individual changes as they happen. I imagine this would be a bit of sea-change, requiring you to be able to modify/add/delete with your EPG data store.

Yeah, but it might not be too hard. If we stick with MS having its own copy of the data, then all it needs to do is to rewrite that copy by combining its current state with the information provided in the update file/notifications. A bit of a twiddle here and there and it shouldn't be too hard to add new programmes on the end, or modify existing programmes etc.

Quote:

As for Now&Next, that could be restricted to current recording streams (a la [Ab] and Extend), and that may be best communicated by a new type of TAP event. The complete set of Now&Next transitions can be quite large (as witnessed by the logging versions of AccurateBM/AccurateBMExtend), so that kind of thing should maybe be a small update file.

To be honest, I haven't used Extend yet, and I just leave AB on to do its work, so not sure what all this would involve. The current plans for MS 6 do not include this part of the Freeview+ spec, but only the series linking part, so I have been (so far) able to ignore it all!

Quote:

Are there any assumptions about the lengths (or general make up) of those MEI fields that I ought to be aware of?

Not from MS point of view. They are all just read in and stored as text (of any length up to about 5000 chars I think).

I was thinking that MEI is a format that was created to give TAPs access to EPG data, so that might be the right format for delivering the messages needed for Extend-like functionality, and maybe even for small update messages. For example, a TAP might be sent an EVT_EPG_UPDATE message with a pointer to a MEI data string in param1 and maybe a count in param2.

If possible and relevant, it'd be very useful to give consideration to other EPG TAPs (such as EPGNavigator) that could potentially incorporate and benefit from any useful functionality that might be developed here. For example, if the terminology could be generic (MS flavour free apart from the MEI format) then non-expert developers might stand a chance.

That'll depend on feedback. Frankly, those who aren't taking part in progressing the concepts can't really complain that their (unknown) requirements aren't being taken into account, can they? And MEI format may sound MyStuff-centric but AFAIK it's the only extendible format supported by any EPG TAP.

What I meant was that apart from the mei file (which I agree ought to be the de facto epg file format), if the discussion could be readily undertsood by non-MS developers (DGB, for example, may not wish to entertain further development work) then, a la FireBird's library, others (who would definitely not complain ) could, if they wished, benefit from this work.

Current status is that EPG2MEI has progressed to using the results of the last run to help fill in any gaps in the current data (and so maintain a complete EPG), although it will drop any programmes more than 1 day old as a trade-off against the file size. Each time it is run it also generates a "changes" file, which are the changed (or completely new) programmes since the last run -- this is the bit I'm hoping will allow TAPs to choose to do an incremental update, rather than using the full file.

There are only a few significant steps left, I think: finding a mechanism to control when the scans are done (ideally, an automatic one), limiting the scan to particular LCNs (at the moment it just does them all), incorporating the MergeMEI technology, adding Extend data, and maybe worrying about satellite Toppys (use service number rather than LCN? -- how do they cope at the moment?).

Male chicken!! It's currently reporting 49 changes on LCN 727, due to the fact that the eventIDs for those programmes have changed while my Toppy was on, and so it has got both in the stored EIT data. That's the sort of duplicate that I wasn't expecting, and the system doesn't expect either! This duplication in the EIT data will constantly make it seem as if there are changes, because the merge process assumes a slot is occupied uniquely. I suspect this is going to be a pain in the neck to fix in the firmware, so a kludge in EPG2MEI may be necessary.

I think the mechanism that might work for invoking scans would be to initially monitor the rate of new events arriving and then just sit and wait for updates to imminent programmes. The former probably does not depend on something like SecCache, but the latter might.

To test out the first idea, I've done some quick tests. Sat on BBC1, from booting it took about 7mins for there to be a 20s period with no new events (and all 10,218 events loaded). But, sat on CBeebies it took 14mins to settle right down, with about 500 events still remaining to be collected (there were low rate blips at 7mins and 10mins). On ITV1 it took 17mins, but it got much closer to the complete set of events (with low rate blips at 10mins and 13mins).

I've not got [Pf] applied so I wasn't surprised to see that playback had a big effect, taking about 5mins to settle, but with only 2/3 of the data. And when playback ended it actually took a channel change to kick things back into properly mopping up the rest.

I think the difference in collection rates is due to the number of channels on a mux (more than the broadcast EIT rate on those muxes), due to the way the firmware restricts the collection (as documented in the [Pf] thread).

I think it's worth resurrecting some subtle changes, a la [Pf], to help improve things, especially now that SecCache exists to decrease the burden.

Some more observations on the native EIT collection, some of which we already knew:

The event_id is critical -- if that changes (due to ineptness, or the schedule changing to a different event) then there will be duplicate events occupying the same slot. The firmware does order them by start time and then reverse arrival order though (which may help a bit).

If the same event (by service and event_id) gets processed again (due to changes somewhere, not necessarily in this event), only the start time is checked -- if that does change (but remains on the same day) then any existing (native, unpadded) timer will be adjusted. If the duration (or day) changes then no adjustments will happen.

The event name, (short) description and extended description are collected just once. Not even if the start time is noticed to have changed does an update happen (so if you've seen any of them change then probably the event_id was the thing that really changed). This may have been with the intention of reducing processing load, but I can't see that it would make much difference really.

Certain types of event are restricted to being collected for just one service on a mux, in rotation, which probably delays a complete collection but helps to ensure coverage as well as reducing the load on the Toppy.

Playback prevents a good deal of EIT collection. (I've also seen it mess up the event ordering, causing whole channels to appear to disappear from the EPG.)

Absolute Radio (727) is one channel that seems to delight in changing the event_id of all their programmes -- it's happened again this morning during a short test. I imagine this could clog up the native EPG store if the Toppy was left on for a long time -- we're talking roughly 70 events being added on every change. I've no idea what the frequency of these changes might be, but on this update there was only one programme that actually changed, and even those changes were just fluff (i.e. it was the same programme really).

The slowness of the normal EIT collection is something I've worked around in the past by completely replacing it. That technique still works and brings in the whole EPG in 5 mins (compared to at least three times that normally, with the occasional events still trickling in 20mins later), but it's not as simple as that now, since this naive approach disables the [He] mechanism used by [Ab] and Extend...

If the firmware was better at spotting changes (and coping with them) then it would be sensible to consider loading the old data from EPG2MEI into the native EIT storage. Ho hum.