$Id$
Mike Isely <isely at pobox dot com>
The home location for this page is here.
This page describes the usage of the utilities included with the pvrusb2 driver distribution.
The utilities can be found in the util subdirectory. They include the following:
There is a free USB sniffer that can be run under Windows for capturing all traffic between a USB device and its driver. The home page for it is here. Using this tool we can learn about how the Windows driver operates the device. If you want to explore, download that tool and follow the related instructions.
To help with understanding the log data from the sniffer, there are two tools provided with this driver:
The first tool is usbreplayprep.pl, which will preprocess the USB sniffer data into a more compact form. This tool was originally written by Björn but I have since modified it to provide more data. The version provided here saves both directions of traffic; the previous one only saved data going in the driver-to-device direction.
The second tool is decode_log, which is compiled from the similarly named C++ source file. This program interprets the output from usbreplayprep.pl and produces a concise listing of all the commands and data moving to / from the device and the driver. (This tool can also capture and save the encoder firmware if seen as part of the log.)
The decompiler is written in C++ and must be built first. Run make in the util directory to build this tool. Note that this program has no other dependencies beyond needing C++ so it should be a straight-forward build.
To capture and decompile a session with the device when run with the Windows driver, do the following:
The decompiler also has an option for saving off the encoder firmware captured, which is useful when one has to figure out the firmware extraction details for a newer Hauppauge driver snapshot. There's more about that later on...
There are 3 ways to extract the firmware, ranging in complexity from trivial to, uh, "educational". Obviously you want to go the simplest route, but circumstances (e.g. newer Hauppauge driver package that we don't know about yet) may require a less simple way.
Regardless of the exact method used, the result will be two firmware files (pvrusb2.f1 and pvrusb2.f2) that have to be put somewhere. During its initialization the pvrusb2 driver will expect to find these files, load them into memory and then upload their contents to the pvrusb2 device. For this to work, you have to be using hotplug, and you have to place these files in a spot where hotplug can find them. The exact location may be distribution dependant. On my Debian system, viable locations include:
/lib/firmware
/usr/local/lib/firmware
/usr/lib/hotplug/firmware
However, your distribution may have things configured differently. For example, perhaps your system is configured to look in /lib/hotplug/firmware instead. With hotplug, the firmware.agent script is programmed with these locations. Try this command to find out your list of candidate firmware directories:
grep FIRMWARE_DIRS
/etc/hotplug/firmware.agentMake sure you copy the firmware files into a place on your system where hotplug can find them.
If the dual presence of fwfind.sh and fwextract.pl seem confusing, realize that they are designed to be complementary, embodying different levels of the extraction process. The lowest level is fwextract.pl - it only knows how to extract firmware data given a directory full of Windows driver files. The fwfind.sh program is the higher level. It tries to find or create that directory of Windows driver files on your behalf and then having done so, it executes fwextract.pl to do the actual extraction. What fwfind.sh adds to the picture is some extra knowledge for recognizing various Hauppauge driver packages and CDROMs and preparing the way for fwextract.pl. So, this means that while you don't strictly need fwfind.sh to extract things, it's here to make your life a little easier. You may find cases where fwfind.sh doesn't quite figure things out, but that's OK; it just means you have to expand / mount the driver files directory yourself and work directly with fwextract.pl instead to complete the extraction (i.e. the "second method").
The following sections describe each method.
Run the contributed bash program fwfind.sh. It tries a few heuristics to locate a driver package, expand it, and then it executes fwextract.pl (see next section) to perform the actual extraction. It can also search a likely mounted CDROM directory.
If fwfind.sh succeeds, then all that is left to do is copy the firmware files into a place where hotplug can find them. Basically what this program does is try to eliminate the preparation part of the normal automatic process before fwextract.pl is run.
There is a possibility that this method might not work. The fwfind.sh makes assumptions about driver package file names, CDROM mount directories, specific embedded driver files names, etc in order to guess whether or not it has found the Windows drivers. Most of the time it will probably get things right. But there are bound to be corner cases where it might not. If this happens, then use the second extraction method, described in the next section.
The fwfind.sh program was contributed by a pvrusb2 user. His name and e-mail address can be found at the top of the fwfind.sh script. Please do not ask me about issues with this script. Instead please contact the contributor (whose info is listed in the script at the top) for questions / support regarding this script. (Sorry, I am not printing his name / email address here not to be a pain, but to spare him the inevitable onslaught of junk from those scum-of-the-earth spammers out there who like harvesting addresses from web pages. However if he wishes otherwise, I'll be happy to post proper credit here.)
The fwextract.pl program is a fairly general utility that tries to handle reliable recognition and extraction of firmware data from the Hauppauge windows drivers. This program deals purely with the scanning a specified driver directory and extracting what it finds. This program only automates the extraction; you have to pick an empty spot and expand the driver package or mount the driver CDROM beforehand.
The main pvrusb2 web page gives a recipe for quick extraction using fwextract.pl. The more general process is basically this:
Note that when the term "Windows driver files" is used above, it is refering to a directory full of files that contains the drivers from Hauppauge. That unfortunately is easier defined than found because Hauppauge seems to like packaging these drivers in different ways. If you're working from a CDROM distribution, then we're refering to the root directory of the CDROM. If you are working from a download package, there might be several such packages but only one has the files you want. For example, the packages pvrusb2_23_31351.exe and pvrusb2_27_23178.zip each contain a driver set, but pvrusb2_26_23055.exe does not - instead in that case the actual drivers are in pvrusb2_inf.zip. Why did they do this? I have no idea; it just confuses. Best advice here is make a guess and if fwextract.pl doesn't spot anything then try a related package.
What this all really boils down to is just a matter of expanding all the Windows driver files into a single place and then letting fwextract.pl chew on it for a while to locate the firmware. The extract script doesn't care about file names or history; it works purely by recognizing file MD5 sums and acting on what it finds.
Assuming that fwextract.pl recognizes the driver files, it will automatically locate and extract the firmware. This program uses MD5 sums to verify that the correct file(s) are being extracted and to verify that the resulting data is correct. It is reasonably smart and can't be easily fooled by wrong files / versions. No more firmware file guesswork (yay).
It is possible that fwextract.pl might not recognize your driver package no matter what you throw at it; you'll see messages of the form "failed to locate data for writing filename" (where filename is the firmware file it tried to generate). This can happen if Hauppauge releases new drivers and we haven't updated fwextract.pl yet. If this is indeed the case, then things are somewhat more difficult - we need to manually identify the firmware data and teach the script where to find it. (Note that the same fwextract.pl handles many driver versions so we stay backwards compatible all along.) Basically what has to be done if this happens is something like this:
The second step runs a process to scan the original Windows driver files and find where the firmware (previously fetched here in step 1) is hiding. Once found, fwextract.pl will write a Perl expression to the terminal which contains information needed by the extractor for locating the firmware in the future. Send that snippet to me and I'll place it into the master copy of fwextract.pl so that other users can benefit from your hacking effort.
The first step here is obviously the big one. There are two approaches to it. One approach is that someone good at reverse-engineering starts hacking away with a hex editor to find the embedded data - which is what people have been doing all along anyway. Another approach is to capture the firmware from the hardware itself. There are two pieces of data to capture: the 8051's firmware and the encoder's firmware:
To manually capture the encoder's firmware, you can use the USB sniffer to grab it "in flight" from within Windows and then issue a command to the decompiler to write the data to a file. To do this, run decode_log with "--firmware filename" on the command line, setting filename to be "pvrusb2.f2". The decompiler will then write that file with encoder firmware data captured from the USB sniffer output (and filtered first through usbreplayprep.pl). Here's the exact procedure:
- Boot up Windows and start the USB sniffer (see the previous section about USB sniffing to learn about where to find the sniffer). Plug in the pvrusb2 device. You don't need to run anything but wait for the sniffer to collect the log data from when the driver initializes the device.
- Copy the sniffer's log data to some place where you can get at it from Linux.
- Boot Linux.
- Run this sequence (where "logfile.txt" is the file you got from the sniffer):
usbreplayprep.pl < logfile.txt | decode_log --firmware pvrusb2.f2 > /dev/nullCapturing of the 8051 firmware is entirely different. The USB sniffer doesn't see this traffic right now so we have to take a different strategy. I've added a feature to the pvrusb2 driver that makes possible the direct retrieval of the firmware data from the device. Here's the procedure:
- Boot up Windows, power cycle the device, and then plug in the pvrusb2 device.
- Reboot to Linux LEAVING THE PVRUSB2 DEVICE POWERED UP.
- Execute this sequence (substitute your device's serial number for "XXXX"):
echo "cpufw fetch" >/sys/class/pvrusb2/sn-XXXX/debugcmd cat /dev/video0 >/tmp/pvrusb2.f1 echo "cpufw done" >/sys/class/pvrusb2/sn-XXXX/debugcmdIt is imperative that you do not power cycle the device when switching from Windows to Linux. This allows the 8051 firmware to be retained, and will cause the pvrusb2 Linux driver to not attempt to reload the firmware. Thus when you run the command sequence to pull the firmware back, you'll get what had been loaded by Windows.
Your device will reset itself after the "cpufw done" debug command is sent to the driver. The file /tmp/pvrusb2.f1 is an image of the 8051's program memory. Note that this isn't the exact bit-for-bit copy of the firmware file you need; apparently the 8051 program modifies itself slightly. However it's close enough that the fwextract.pl program's --search feature can still locate the pristine firmware data using a fuzzy matching algorithm (which it will fall back to if an exact search doesn't find it).
After you've gotten pvrusb2.f1 and pvrusb2.f2 retrieved, put copies in the current directory, (if you haven't already done so) unpack the Hauppauage Windows driver package again into a empty (e.g. "win_driver") subdirectory, then run fwextract.pl --search. (If you installed from a cdrom, then mount the cdrom under Linux and use the --driverDir option to point at the mount point.) After a minute or so it will print the magic chunk of data that can then be put back inside fwextract.pl so it will now automatically understand how to extract this firmware data in the future. Send that magic chunk to me (along with a description of the driver package) and I'll ensure that the next release of fwextract.pl includes it.
You should in theory be able to use the manually extracted firmware files. However the 8051 firmware image (pvrusb2.f1) may not be exactly identical to what is really loaded, so you may really want to go back and get the real firmware out of the driver directory anyway. Do the following: