BluRay updates

Well, in the past month I've had a bit of a chance to play around with bluray playback, as well as doing an upgrade to MythTV 0.22 and a VDPAU capable graphics card.

The Upgrade

A big incentive for making the upgrade to 0.22 was to take advantage of some of the advancements that make BluRay playback better. At present time, no system is capable of software-based BluRay decoding without multithreading support, and unfortunately ffmpeg's current multithreaded implementation only works on some BluRay discs (a very small subset, based on my experiments). I was sick of having to spend an hour ripping a BluRay movie, only then to have to turn around a spend another several hours (5+ hours) transcoding to something that mythtv can handle in software.

With MythTV 0.22 and VDPAU support, I can now play nearly everything without having to do any transcoding. Full bitrate decoding is handled perfectly by the VDPAU card, and DTS-HD audio is handled by MythTV's included libraries. The only problem I've had so far was with a movie that used TrueHD for the master audio track. At present time, that results in unwatchable playback. However, I think that I'll be able to work around this without transcoding by simply extracting the core audio subtrack from within the TrueHD master track. That should work just great without adding to the ripping time. I'll post an update once I get around to testing it (such a small thing to test, but with all the stuff I've got on my MythTV todo list, small things don't necessarily get done more quickly)


Seektables needed for playback

One thing I forgot to mention before was that playing back an m2ts file doesn't work very well without building a seektable for the file. I wish that weren't the case, but luckily it's not a very difficult thing to do. A simple call to the mythtranscode program (don't worry...you aren't actually doing any transcoding) will generate the seektable for the m2ts file rather quickly. A 2 hour movie takes about 5 minutes to process on my system.

mythtranscode --mpeg2 --buildindex --allkeys --showprogress --video --verbose most --infile "$filename"

The one downside is that this leaves behind a 0 byte file that you need to delete. To deal with this, and so that I don't have to remember what parameters to use, I've written a shell script to handle this all for me. I simply call the script, pass it the filename, and it does everything.

Download the m2ts_buildindex shell script.




...click here to read more!

BluRay playback in MythTV

Recently I decided to see if I could get BluRay playback working in MythTV. I've been putting it off for quite a while because of how problematic the whole process appears to be. Lately I got bored and decided, what the heck...lets see what we can do.

BluRay player

First things first...I needed a BluRay player before doing anything. Amazon has the LiteOn iHOS-104-08 for $50 with free shipping, and it has great reviews everywhere I looked. I considered spending more and getting a burner, but at the time I ordered it was $100 more for a BluRay burner, and with BD-R media prices the way they are currently, I figured by the time they become more reasonable I can probably just buy an entire new player/burner for less than $100, and then I'd have a newer player that will hopefully be more mature. I've been burnt enough by buying DVD burners that don't work well with certain media brands or can't burn at their rated speed. I don't want to get stuck with another lemon, so I'll just wait out the burning stuff for right now.


Ripping Software

The next issue with BluRay playback is that currently there is no linux support for playing it directly off the disc. First you have to rip it to your hard drive and then do something from there. Linux has a program called DumpHD which can rip some BluRay discs, however it appears to be hit-or-miss.

I've already had enough annoyance with ripping regular DVDs in linux. Yeah, some will rip fine with one program, but others have bad sectors on disc which cause the ripper to fail. So now you have to use ddrescue to recover the bad sectors, which takes HOURS to rip a single disc. Then when I later decide I want to rip and transcode just the main title, I find that for some reason (which I still haven't figured out) the main track can't be ripped properly by programs like handbrake It comes up all garbled (I suspect it has something to do with not properly/fully removing the CSS encryption). Argh!!! Too much hassle. I want one way to do things, and I want it to work quickly and reliably every time.

For this reason, I recent purchased AnyDVD-HD for Windows. Yeah, it's sort of a mental defeat that I can't seem to do it all under linux and have it "just work" every time, but after hassling with this for so long, I don't care. I'd rather just get it working. AnyDVD-HD fits the bill there. Though it's not free, it was worth it for me. DVDs rip successfull every time, and in only about 10 minutes. It doesn't matter which copy protection they have, it handles it for me, and that (to me) is worth the cost (though I am a bit surprised that nothing equivalent has popped up under linux yet, as far as I know).

So anyway, I've already got AnyDVD-HD for Windows, and from my reading it appears to work very reliably on ripping and decrypting BluRay discs, so I figure that's my best and simplest course of action.


Ripping

Ripping the BluRay disc with AnyDVD-HD is quite simple. You insert the disc, wait for the software to analyze the disc and remove the protection, right click on the fox icon in your taskbar icon tray, and choose "Rip Video DVD to Harddisk". A window pops up where you choose the correct drive to rip from, set your output folder, and click the "Copy DVD" button. The rip process begins, and when it's done you've got a folder on the hard disk that contains the entire filesystem from the disc, with all protections removed.


Extracting the Features

Unfortunately, there's no software in linux (that I am aware of) which understands how to playback a BluRay movie (including all the menu structure and everything). So, in order to do anything, you need to extract each item individually. For some movies, apparently you can just find the largest m2ts file and copy that out, but that doesn't always work well. Many discs actually contain a ton of small m2ts files, and you need to figure out which ones are which and join the appropriate ones together in the correct order. Once again, this seems like another bit of hassle I don't want to deal with. Luckily, there's a program that can deal with this (and this time it's free).

ClownBD is a GUI that simplifies the process of extracting tracks from a ripped BluRay disc. You point it at the folder you ripped to, and it runs some backend processes (mostly eac3to), parses the output, and gives you a nice gui for dealing with the results. It's all very easy to use and quite intuitive.

So, first step is to install and setup ClownBD. That's quite simple...not much to do there. Mostly, you just need to setup some temporary folders the program can use for demuxing and remuxing the content. Then you point it at the BluRay rip folder and click the "Next" button.

Now eac3to will be run in the background, and when it is done (which only takes 5-10 seconds) you will most likely see the "Step 2" window pop up. This window contains a tree control showing each feature on the disc, and for each one it shows the length, which m2ts files it is spread across, how many chapters, which encoding codecs were used, which languages there are, etc. Your job here is to select the one feature you wish to extract. Generally this would be the main movie, and most of the time that would be the longest running feature on the disc. However, that's not always the case. Some movies have a longer feature with directors commentary or something. So in these cases, what you'd want to do is lookup the running length of the movie on the movie case or disc label (if it's not there, check amazon, imdb, or on the netflix sleeve if this is a netflix rental). Then just pick the feature that is closest in length to this figure and click "Next".

Note, that I mentioned you will *most likely* see this dialog. However, if your disc only contains a single feature, ClownBD will intelligently skip the step of giving you a dialog with only 1 option to choose from, so you will go straight to step 3.

In Step 3, you will now be focusing on a single feature of the film, and it will show you all the info about that feature. This time, you choose which components of that feature you wish to rip. So far, in my experience you only have a single video track, so that's a no brainer. However, the audio and subtitles sections usually give you several options. For each option it will tell you the contents, such as audio type (DTS Master Audio, AC3, etc), language, bitrate, etc.

Unfortunately, there is no way to preview the contents of each selection. You just have to rip it and see what you end up with. However, for each section, ClownBD preselects a few options that it thinks you will want, and most of the time it guesses correctly. So, unless you want an alternate language, I'd normally just rip it with the default selections. I've ripped the alternate tracks for a few movies. I tried ripping the AC3 track thinking I'd get AC3 audio, but it just turned out to be a director's commentary audio track. So far in my experience, the defualt track (which is usually DTS Master Audio) is the one you want.

Now that you have your selection, you have a few choices to make. First is the output Audio format. I tried leaving it unconverted, but that didn't work for my in mythtv. Instead, I check the AC3 option and that seems to work fine. The FPS and Stream boxes I just leave at default. The Movie Output Format I haven't experimented with. I just set it to m2ts. I'm not sure how that would be different from ts, and I have no idea what output you'd get from BluRay or BluRay+ISO. I just pick m2ts the first time and it worked perfectly, so I've stuck with it.

Now that you've chosen all your options, you click next and it will begin a long processes of demuxing and remuxing the content. When it is done, you will have a file in you remuxing temp folder called something like demuxer.m2ts and that is your extracted feature.


Playing

Ideally, you should just be able to copy that file over to your mythtv system and start playing back, and indeed that does work perfectly in some cases. However, in other's there may be one last gotcha in your way...decoding performance.

There are 2 ways to decode...hardware and software. At the moment, the only hardware option in linux is to use a VDPAU capable video card and VDPAU enabled software (mythtv 0.22 can do it, as well as 0.21 if you apply a certain set up backported patches). With this, the video should play back smooth as butter. However, I currently don't have a VDPAU card, so I've no experience with this area yet.

The other way to decode is in software, which means using a very fast CPU. If your CPU isn't multi-core, you can forget about it right away. There's no way it will be fast enough. Even if you do have a multi-core processor, it still may not be fast enough if it's a lower end one. Finally, even if it is multi-core and fast enough to play back some movies, there will be a final problem with other movies. MythTV uses ffmpeg to do the decoding, and unfortunately ffmpeg currently has a serious limitation in the multithreading department. It can only multithread h264 video when it is encoded utilizing slices. Not all movies do this. Some do, some don't, and some only do it for some chapters of the movie. If the movie isn't encoded with slices, playback decoding will happen on a single CPU core (while the others sit idle) and you will get pausing every few seconds while the CPU catches up with playback (unless your CPU is fast enough to decode in real time on a single core...however, I'm not sure if any current CPU is fast enough...the only one that MIGHT be able to do it is the very fastest models of the Core i7 chip, and even that it might need overclocking, and even THEN it still might be just a tad too slow).

So, the short of it is that software decoding isn't reliable. It will work for some films, and only parts of others (or not at all). So your options now are to upgrade to a VDPAU capable system, wait for better multithreading in ffmpeg (which is under development, but still buggy, and it will be a while before it is finished, and then a while longer until the updates get rolled into mythtv), or transcode to a less demanding file.

Transcoding

As I just mentioned, if you don't have VDPAU playback, you'll probably want to transcode. Even if you do have it, you still might want to transcode since a BluRay movie takes 15-40GB, and if you are going to keep a copy on your hard drive you might want it to be a bit smaller. Or, if you don't want to keep it on your hard drive but don't want to go through the re-ripping process to play it each time, another option would be to transcode it to something that will fit on a dual layer DVD+R disc.

For transcoding I've been using Handbrake under Windows (you can run it under linux too, but since I'm already running AnyDVD and ClownBD in windows, I might as well just finish up the process there, too). Handbrake has a number of option, but I can't explain most of them to you, and I have no idea what they do or which ones are better. I just pick one of the higher quality default profiles (making sure to choose one based on h264) and then tweak it a bit. First, I make sure I'm outputting to mkv format. Then, on the Picture Settings tab I change Anamorphic to "Loose" and set the width to 1280 (which equates to 720p resolution). Then, on the Video tab, you want to set a higher bitrate. For the few I've done, I just tried 6000 kbps. That gave me a file size of about 2.75 GB/hour. However, if you want to go the "burn to DVD+R DL" route, you might want to maximize quality while making sure it still fits on the disc. In that case, rather than setting an explicit bitrate, you'd want to specify a target size of slightly less than the disc can hold. Also, I've read this isn't perfect...it may go over slightly, so you might want to play it safe and just pick something like 8000 MB (disclaimer...I haven't tried this yet, so don't yell at me if you spend hours transcoding and it turns out even that is too big to fit).

With these few changes, I then proceed to transcode. This is a long process. For dual pass transcoding, it usually takes me about 2 times the length of the movie to transcode (so a 2 hour movie will transcode in about 4 hours). This is on a Core i7-920, utilizing all 4 cores. Slower processors will obviously take longer.

When this is all said and done, hopefully you will have an mkv file that works just fine. However, if you messed up and chose a wrong option, you could have a file that doesn't work (like the first time I tried it and got no sound). So before you go and invest hours of transcoding only to find it doesn't work, what you should do is experiment on transcoding a shorter clip. I'm not sure of a good way to trim down a m2ts file after the fact. Instead, what I did was go all the way back to the ClownBD step, check the box on the first screen that says "Split Output Based on Chapters", run it, and then pick one of the shortest chapter to transcode (usually you can find a 1-2 minute chapter). Just make sure to turn off that option in ClownBD when you are done experimenting.


So, in the end I end up with a 720p, 6Mbps, h264 video/AC3 audio file in mkv format. Yeah, it's not quite the quality of BluRay, but honestly I don't see any difference watching the movie, even on my 1080p TV. Maybe if I had them on 2 TVs side by side I might notice a difference, but switching back and forth between the BluRay file and the transcoded file, I see no difference. That's close enough for my satisfaction.


Conclusion

It's not perfect, but it works. I can't play straight off the BluRay disc, and I need to go through a lengthy process (though there isn't much labor involved...most just start a process, come back in an hour, start another process, etc), and I end up with a slightly lower quality result. However, it's close enough for me, and gets me results that are better than DVD. The damn DRM might not make it easy, but so far it's not an insurmountable obstacle if you are willing to put in a little effort.

Maybe in a few weeks, after I get a chance to play with some more options, I'll post an update with what I've found.

...click here to read more!

Controlling a Sharp Aquos TV via the serial port interface

Last year, I replaced my ancient CRT television with an HDTV LCD...a Sharp Aquos. When I got the TV, I was curious to find that it had a RS-232 compliant serial port onboard. I was wondering if I could setup my mythtv frontend to communicate with TV. A quick look through the user manual turned up something I found even more shocking than the existance of the serial port...the communication protocol for the TV's serial port was completely documented in the user manual, including things like port settings, command format, a list of commands, and their accepted parameters. This was exciting...it seemed like it wouldn't be too much trouble to get it all working.

Indeed, it was quite easy. I installed a Perl module called Device::SerialPort which made it a piece of cake to setup a serial port connection. In no time at all, I quickly issued my first command to the TV and had my response back. From there, it was just a simple matter of turning it into a script.

At first, I just wrote a simple standalone program to handle what I wanted. It would open a connection, send the command, and give back the response. However, the problem was that I intended to be interfacing with the TV from multiple different scripts and programs, and was concerned about cases where 2 scripts might try writing to the serial port at the same time, which would probably either fail or do something like intermix the 2 commands being written resulting in the TV getting corrupt data.

Instead, I decided the best way to handle it would be to write a simple server script. This server script would be the only script that interacts with the Aquos via serial port. All other programs would connect to the server, issue a request, and let the server handle the communication and pass back the result. If multiple clients connected to the server, the server would just handle the requests one at time, in the order received.

The end result of this effort is the Aquos Server, which is a simple perl script that handles requests via tcp/ip connections. You simply start the script running at bootup, connect to a port, issue a text command, and read back a text result. It's a very simple communication protocol. Each request is one line, and it is always responded to with a one line response.


Setting it up

First thing you will need to do is download my updated RFLibs (ver 2) library of Perl modules. Also note that even if you downloaded these libs in the last few months (since I posted the updated ones), you'll want to redownload them again. There was one remaining rare bug in the serial port communication that I was able to track down and fix. You can get the current version from here.

Then you need to download the aquosserver script. You can get that from here.

Finally, you need to make sure you have the Device::SerialPort module installed for Perl. Under Debian, you can do this using "apt-get install libdevice-serialport-perl", but the command or package name may be different for your distribution.

Once you've got it, you'll want to make sure the aquosserver script is in the same directory as the RFLibs directory. Then you may need to edit the aquosserver script and change 2 things:
1)The $listenport variable (set to 4684). This is the port that the aquose server listens for connections on. Change it to whatever you please.
2) The $serialport_device variable (set to '/dev/ttyS2'). This is the device name for the serial port that is connected to your Sharp Aquos TV.


Testing

Simply start the script running. Then, to test it out, open another ssh/telnet window and try connecting to it. The easiest program to use is nc:
nc localhost 4684

Then type "POWER" (without quotes) and hit enter. If everything went as planned, you should get back a response of either 0 or 1 (depending on if the TV is off or on). Next we want to make sure you can turn the TV on via serial port (by default, that capability was disabled on my TV). First, turn the TV on manually. Wait for it to power up completely, and then issue the "SERIALPOWERUP ENABLE" command to the aquosserver, which should give you an "OK" response.

Now you can try turning it on and off using "POWER ON", "POWER OFF", and "POWER TOGGLE". You can query the current volume with the "VOL" command, and you can adjust the volume using "VOL+" and "VOL-", or you can set it explicitly using "VOL 10". You can query the input using the "INPUT" command (which returns the input number, with input 0 being the tuner), and you can set it to input 2 by issuing "INPUT 2". The commands for muting are "MUTE", "MUTE ON", "MUTE OFF", and "MUTE TOGGLE". Finally, when you are done issuing commands, type "EXIT" and hit enter to disconnect.

This should cover most of what you want to do with the TV, but if you need to issue a command not listed here, you can issue a raw command code to the aquos server. First look up the command code and parameter in your TV user manual. You can then issue the command using the "CMD" command and pass it both the command code and the parameter value. For instance, to set the AV Mode to Game, you would use "CMD AVMD 3". Note that the parameter (3) does not need to be padded to 4 bytes as the Sharp protocol requires. My library handles the padding for you.

Also, note that this script was written around the D64U series of Aquos TVs. I've looked at a few other series and they use the same protocol and commands. However, if your TV has different protocol/commands you'll have a problem. If that's the case, let me know and I'll see if maybe I can find a way to handle other TVs in a future version.


Using

Now, how do you implement this into something useful? Well, first you need to make sure that the aquosserver script is always running in the background. You'll want to launch it from one of your init scripts. I'll let you figure out how to do that. Once it's running, you'll need to issue commands to it. You can do this from a script. For instance, if you want to power on the TV when your computer boots up, you can run a script (after the aquosserver has started up) which runs the following command:
echo -e 'POWER ON\nEXIT' | nc localhost 4684

This issues the command to turn on the TV and then disconnects from the aquosserver (without paying attention to the result).

Another thing you may want to do is issue commands in response to pressing buttons on the remote. I believe there is a way to do that via the lircrc file, but I can't instruct you on how. Myself, I do it using the latest version of my irwatch script. That will be posted about in my next blog entry, so for now, you are on your own. Stay tuned...



...click here to read more!

Updated RFLibs & new scripts coming soon

It's been a while since I posted anything here, but I have been working here and there on getting some work done on mythtv. Over the next few weeks, I'm hoping to post some updates and new scripts. A few things I've got coming include an updated version of irwatch, a new and improved alpha release of the mythimon client, and some stuff for controlling a Sharp Aquos TV via the serial port interface.


All of these scripts are based in part around my RFLibs set of perl modules. I posted a release of RFLibs last year. Since then, I've made a lot of changes to those scripts. While some of the modules only experienced minor changes or the addition of new functions only, other modules (especially the imon releated modules) experienced a considerable redesign.

As a result of these changes, if you had previously made use of my RFLibs modules in some of your own scripts, you might find that they no longer work with the new versions. I wish I could have avoided doing so, but many of the changes were necessary. I'm hoping things are in a much more stable state now, but still no guarantees.

So for now, I'm going to make the new versions of the modules available for download. However, note that if you are currently using the old mythimon (alpha 1) or irwatch (version 1) clients, you cannot use these new versions with those scripts (you need to wait for the upcoming announcment of new versions of those scripts), so don't put these in place just yet.

Download version 2 of the RFLibs perl module pack here



...click here to read more!