Here's a Test... and there is no wrong answer...

[Gary Miyakawa]

Ok, Your new boss has come to you and said, "xxxxx, I want you to tell me how do we PROVE that our video systems are running at xx frame rate ?"

You are allowed to add commercial hardware to this test but you may NOT modify the video systems being tested. The test(s) must be reproducible and unquestionable.

Ok, Mr. VTCTalk member, you have your goal and your guidelines, what are the solutions ?

REMEMBER, this is for FUN and NOT a life altering experience, please treat it that way!

Gary Miyakawa

[trapehzoid]

What are you trying to prove? That it can achieve a set frame rate, or that the information being displayed is infact correct?

Polycom reports a frame rate on screen diagnostics.. as well as some other systems do as well.

So my question back to you is, do you trust the equipment, or are you trying to 'externally' verify?

[Gary Miyakawa]

Simply put, I want to "prove" that, in fact, the systems are running at 15 or 30 FPS (and maybe 60 Fields if possible)..

Gary Miyakawa

[trapehzoid]

Proving they negotiate a frame rate capability is extremely simple (especially on H.323). Simply sniff the H.245 conversation and look at the Open Logical channel messages to see what video is actually being used. Look at the capability sets in the setup to look at what each endpoint is signaling it can do.

Doing it on H.320 is more complicated because you have to get the data, and then decode it. Only real easy way to do that is to use the internal diagnostics of your codec to save the data and analyse it offline. How you do that depends on your equipment.

Now if you want to know the 'actual' frame rate being used... I'd settle for what the systems are reporting. Technically you could software decode the H323 conversation, but I don't know of any 'packaged' software out there that would do that simply for you. You'd probably be doing some work to do this. Most things I know of are simply protocal analysers, not 'media' anaylzers.

Just remember there are two things.. negotiated capabilities.. which for framerate simply set the 'max' framerate to be recieved, and actual framerate, which is constantly fluctuating based on bandwidth available, the input, and how the encoder is behaving.

I can encode a video stream that can do 30fps at 284kbs with ANY video source.. but the image could still look like absolute crap because I could simply increase the quantizer to achieve that high frame rate at the sacrafice of visual clarity.

I think VTELs report 'live' framerate.. though I don't have one anymore to try. I know Polycoms do. I think ipower does, don't recall. Tandberg can, but not by the user (they have some diagnostic way of doing it). I haven't used VCONs to say if they do or not. Sony does not to my knowledge.

Just remember to use a consistant source when comparing between products to see how a system behaves. How a system behaves is totally up to the developer in terms of what he wants to proritize and what behaviors he wants. The only 'meaningful' thing that comes out of these framerate discussions is when a system can't come anywhere close to achieving the advertised framerates. Like when a system claims 30fps support, when in reality they do 12fps or so on an average video source. If you get 24-30fps over an average window, I'd consider that 'fully 30fps' capable. The rate will always drop depending on what's going on in the video if you don't have endless amounts of bandwidth (if it doesn't, they aren't paying much attention to the video itself, which leads to other problems).

Helping understand your criteria and what is 'acceptable' as a valid source would help solve this situation.

[mazzarak]

I'd just set up a call and show them the frame rate page... then if its working at said frame rate, say "Voila" and disconnect before the figure changes... ;)

[Yukikaze]

Use a packet sniffer to record all video packets during time t in seconds. Count the picture start codes psc in the recorded packets.

psc/t=fps

Like trapehzoid said you probably won't get exactly 30, especially at low bandwidths, because one of the responsibilities of the encoder is to drop the framerate to maintain adequate picture quality when the motion is too complex to be encoded at full framerate with an acceptable quantizer. This will at least tell you how many frames are encoded.

If you want to check that the *right* 30 frames were encoded as well I guess you have to generate a series of numbered test images on a pc with TV out and record the decoded stream on the receiving end using a vcr and singlestep through afterwards to make sure every frame was transmitted ok. This last method has the advantage of working on isdn as well. Ideally you'd use a video mixer(or homebrew pc software) to blend in the counter over a normal conversation.

Using simple test images like alternating black and white frames/fields as input to the encoder combined with a light sensor or oscilloscope on the receiving end should be able to measure fps automatically, however it has the problem that it isn't a natural image and won't prove that you'll get 30fps during a normal conversation. I guess it is possible to replace just part of the image with black/white though.

[Yukikaze]

Ofcourse... it is probably a mistake to focus on that single number as a measure of a codec's quality. There are so many other more important factors imho, as most codecs today have good video quality.

[Oliver Black]

I made a video in Adobe Premiere, Did upper 3/4 of screen white on odd frames, bottom 3/4 of screen with black. Center would alternate black/white allowing the eye to view a gray image in center if it was working correctly. Also added counting number to 60, as well as labeled odd/even. Basically I built two title screens, then made them alternate. If we got smooth counting, gray in the middle then it was working. If it was all black, or jerky, then something was being dropped. Analyzing a tape of the image in slow motion would show what frames would be dropped...

[trapehzoid]

Use a packet sniffer to record all video packets during time t in seconds. Count the picture start codes psc in the recorded packets.

I guess that would require decoding the media stream, not just the H.245. What sniffer do you use that can decode H.263 and H.264 effectivly like this?

[Yukikaze]

I guess that would require decoding the media stream, not just the H.245. What sniffer do you use that can decode H.263 and H.264 effectivly like this?

I don't touch H.245 at all, just look at the media stream. I used ethereal, just filter away anything non-rtp, and select an audio algorithm with a fixed rtp payload type so you can filter away that as well. When I've done similar things earlier I used homebrew software to process the sniffed packet stream afterwards(Easy to make as ethereal uses the open source libpcap library to store the packet stream).

In my application I decoded the entire video stream offline, but
if you assume that there won't be several pictures in one packet(A reasonable assumption for low-delay video conferencing) you don't need to to decode the video. The picture headers will always be at the start of the packet for H.261/H.263 so it is enough to just look at them. For H.264 the issue is slightly more complex as a change of frame isn't signalled explicitly, but is inferred from changes in NAL type or something like that(It's been a while since I looked at this and I don't have the standard in front of me, sorry). You still don't need to decode video though, because the size of each NAL is stored in the packet so you can just check the NAL type and skip over the NAL video data.

Anyway, we are talking about a day or two to make it even if you aren't familiar with the compression standards and rtp encodings. You could also use libpcap's sniffing capabilities to write a standalone app to do sniffing+fps counting in real time with a marginal increase in development time.

[Gary Miyakawa]

Thanks guys! All great ideas but most don't take into account the "possible" delays in the decode side. Also, most won't work in the H.320 world.

What do you all think about creating a "standard" (maybe bad choice of words there) video source (VCR/DVD/etc) that has a set of clocks embedded in the images that would be on the source side of the video call. And then on the recieve side of the video call we have a recording device that can be played back at a slow speed to see if there are any missing frames ? This, in my way of thinking would take into account any network traffic/delays, system encode/decode delays and protocol delays....

Thoughts ?

Gary Miyakawa

[trapehzoid]

Thanks guys! All great ideas but most don't take into account the "possible" delays in the decode side. Also, most won't work in the H.320 world.

What do you all think about creating a "standard" (maybe bad choice of words there) video source (VCR/DVD/etc) that has a set of clocks embedded in the images that would be on the source side of the video call. And then on the recieve side of the video call we have a recording device that can be played back at a slow speed to see if there are any missing frames ? This, in my way of thinking would take into account any network traffic/delays, system encode/decode delays and protocol delays....

Thoughts ?

Gary Miyakawa

I wouldn't worry about the decode side dropping stuff or not being able to display it. And as for 'delay', I guess there is a possibility of that where the decoder can't decode the frames in a consistant rate, where it slows down and then speeds up so its display 'rate' isn't the same as the rate it recieved. but I haven't heard of that being an issue...

The issue with the frame counter method is, you still need to 'offline' process the video by recording it and analysing it. This two would require a recorder that can count frames for you.. so you can compare (otherwise you don't get around your delay problem where maybe it plays all 30 frames, but not within 1 second).

As for William's point with the codian, I don't see how thats any better then simply calling into an endpoint that shows a frame counter.

[trapehzoid]

I don't touch H.245 at all, just look at the media stream. I used ethereal, just filter away anything non-rtp, and select an audio algorithm with a fixed rtp payload type so you can filter away that as well. When I've done similar things earlier I used homebrew software to process the sniffed packet stream afterwards(Easy to make as ethereal uses the open source libpcap library to store the packet stream).

In my application I decoded the entire video stream offline, but
if you assume that there won't be several pictures in one packet(A reasonable assumption for low-delay video conferencing) you don't need to to decode the video. The picture headers will always be at the start of the packet for H.261/H.263 so it is enough to just look at them. For H.264 the issue is slightly more complex as a change of frame isn't signalled explicitly, but is inferred from changes in NAL type or something like that(It's been a while since I looked at this and I don't have the standard in front of me, sorry). You still don't need to decode video though, because the size of each NAL is stored in the packet so you can just check the NAL type and skip over the NAL video data.

Anyway, we are talking about a day or two to make it even if you aren't familiar with the compression standards and rtp encodings. You could also use libpcap's sniffing capabilities to write a standalone app to do sniffing+fps counting in real time with a marginal increase in development time.


Cool. I haven't written software since university days so I'll have to pass on that side of it :) I will check into looking at ethereal's decodes for the media packets. I've never dug into the packets past their RTP sequencing before. Something to play with now :D

[DesmondC]

Hi Yukikaze,

I am currently doing a project to recover h.264 video sniffed using ethereal. Had written previous programs to tackle the H.263, but found it tougher for H.264. Could you share your knowhow on H.264, and how you would have done to recover and eventually playback the H.264 video. Many thanks in advance...

Desmond

[DesmondC]

I don't touch H.245 at all, just look at the media stream. I used ethereal, just filter away anything non-rtp, and select an audio algorithm with a fixed rtp payload type so you can filter away that as well. When I've done similar things earlier I used homebrew software to process the sniffed packet stream afterwards(Easy to make as ethereal uses the open source libpcap library to store the packet stream).

In my application I decoded the entire video stream offline, but
if you assume that there won't be several pictures in one packet(A reasonable assumption for low-delay video conferencing) you don't need to to decode the video. The picture headers will always be at the start of the packet for H.261/H.263 so it is enough to just look at them. For H.264 the issue is slightly more complex as a change of frame isn't signalled explicitly, but is inferred from changes in NAL type or something like that(It's been a while since I looked at this and I don't have the standard in front of me, sorry). You still don't need to decode video though, because the size of each NAL is stored in the packet so you can just check the NAL type and skip over the NAL video data.

Anyway, we are talking about a day or two to make it even if you aren't familiar with the compression standards and rtp encodings. You could also use libpcap's sniffing capabilities to write a standalone app to do sniffing+fps counting in real time with a marginal increase in development time.
Hi Yukikaze,

I am currently doing a project to recover h.264 video sniffed using ethereal. Had written previous programs to tackle the H.263, but found it tougher for H.264. Could you share your knowhow on H.264, and how you would have done to recover and eventually playback the H.264 video. Many thanks in advance...

Desmond

[mmagliaro]

Just to chime in... I agree, sniffing the H.26x packets with Ethereal on the receiving end is a good, easy, free way to see what your framerate is. The odds that the receiving station actually drops any frames on its own are nil, so I agree, I wouldn't worry about that.
Yes, VERY important to test with the same video source, exactly, every time. We use a DVD of a movie scene, or a VCR tape of some people (if you want "talking head" video. The content of what's being encoded and sent greatly affects the framerate, especially at bitrates below about, say, 384kbps.
Above that, the station really ought to be able to sustain a solid 30 fps rate, with maybe only occasional dips to 25 or so during higher-motion sequences.

Make sure you do this with two stations plugged directly into one LAN switch, so there are absolutely no network issues involved. Packet losses will cause the endpoints to throw out frames, request sync frames, and take other recovery measures that will put dips in the framerate, and you don't want that skewing your results.