tutorial05.java

ffmpeg开发指南

      } else {
        /* if we aren't given a pts, set it to the clock */
        pts = is->video_clock;
      }
      /* update the video clock */
      frame_delay = av_q2d(is->video_st->codec->time_base);
      /* if we are repeating a frame, adjust clock accordingly */
      frame_delay += src_frame->repeat_pict * (frame_delay * 0.5);
      is->video_clock += frame_delay;
      return pts;
    }


You'll notice we account for repeated frames in this function, too.

Now let's get our proper PTS and queue up the frame using queue_picture,
adding a new pts argument:


        // Did we get a video frame?
        if(frameFinished) {
          pts = synchronize_video(is, pFrame, pts);
          if(queue_picture(is, pFrame, pts) < 0) {
    	break;
          }
        }


The only thing that changes about queue_picture is that we save that pts value
to the VideoPicture structure that we queue up. So we have to add a pts
variable to the struct and add a line of code:


    typedef struct VideoPicture {
      ...
      double pts;
    }
    int queue_picture(VideoState *is, AVFrame *pFrame, double pts) {
      ... stuff ...
      if(vp->bmp) {
        ... convert picture ...
        vp->pts = pts;
        ... alert queue ...
      }


So now we've got pictures lining up onto our picture queue with proper PTS
values, so let's take a look at our video refreshing function. You may recall
from last time that we just faked it and put a refresh of 80ms. Well, now
we're going to find out how to actually figure it out.

Our strategy is going to be to predict the time of the next PTS by simply
measuring the time between the previous pts and this one. At the same time, we
need to sync the video to the audio. We're going to make an **audio clock**:
an internal value thatkeeps track of what position the audio we're playing is
at. It's like the digital readout on any mp3 player. Since we're synching the
video to the audio, the video thread uses this value to figure out if it's too
far ahead or too far behind.

We'll get to the implementation later; for now let's assume we have a
get_audio_clock function that will give us the time on the audio clock. Once
we have that value, though, what do we do if the video and audio are out of
sync? It would silly to simply try and leap to the correct packet through
seeking or something. Instead, we're just going to adjust the value we've
calculated for the next refresh: if the PTS is too far behind the audio time,
we double our calculated delay. if the PTS is too far ahead of the audio time,
we simply refresh as quickly as possible. Now that we have our adjusted
refresh time, or **delay**, we're going to compare that with our computer's
clock by keeping a running frame_timer. This frame timer will sum up all of
our calculated delays while playing the movie. In other words, this
frame_timer is _what time it should be when we display the next frame._ We
simply add the new delay to the frame timer, compare it to the time on our
computer's clock, and use that value to schedule the next refresh. This might
be a bit confusing, so study the code carefully:


    void video_refresh_timer(void *userdata) {

      VideoState *is = (VideoState *)userdata;
      VideoPicture *vp;
      double actual_delay, delay, sync_threshold, ref_clock, diff;

      if(is->video_st) {
        if(is->pictq_size == 0) {
          schedule_refresh(is, 1);
        } else {
          vp = &is->pictq[is->pictq_rindex];

          delay = vp->pts - is->frame_last_pts; /* the pts from last time */
          if(delay <= 0 || delay >= 1.0) {
    	/* if incorrect delay, use previous one */
    	delay = is->frame_last_delay;
          }
          /* save for next time */
          is->frame_last_delay = delay;
          is->frame_last_pts = vp->pts;

          /* update delay to sync to audio */
          ref_clock = get_audio_clock(is);
          diff = vp->pts - ref_clock;

          /* Skip or repeat the frame. Take delay into account
    	 FFPlay still doesn't "know if this is the best guess." */
          sync_threshold = (delay > AV_SYNC_THRESHOLD) ? delay :
AV_SYNC_THRESHOLD;
          if(fabs(diff) < AV_NOSYNC_THRESHOLD) {
    	if(diff <= -sync_threshold) {
    	  delay = 0;
    	} else if(diff >= sync_threshold) {
    	  delay = 2 * delay;
    	}
          }
          is->frame_timer += delay;
          /* computer the REAL delay */
          actual_delay = is->frame_timer - (av_gettime() / 1000000.0);
          if(actual_delay < 0.010) {
    	/* Really it should skip the picture instead */
    	actual_delay = 0.010;
          }
          schedule_refresh(is, (int)(actual_delay * 1000 + 0.5));
          /* show the picture! */
          video_display(is);

          /* update queue for next picture! */
          if(++is->pictq_rindex == VIDEO_PICTURE_QUEUE_SIZE) {
    	is->pictq_rindex = 0;
          }
          SDL_LockMutex(is->pictq_mutex);
          is->pictq_size--;
          SDL_CondSignal(is->pictq_cond);
          SDL_UnlockMutex(is->pictq_mutex);
        }
      } else {
        schedule_refresh(is, 100);
      }
    }


There are a few checks we make: first, we make sure that the delay between the
PTS and the previous PTS make sense. If it doesn't we just guess and use the
last delay. Next, we make sure we have a synch threshold because things are
never going to be perfectly in synch. ffplay uses 0.01 for its value. We also
make sure that the synch threshold is never smaller than the gaps in between
PTS values. Finally, we make the minimum refresh value 10 milliseconds*.*****
Really here we should skip the frame, but we're not going to bother.

We added a bunch of variables to the big struct so don't forget to check the
code. Also, don't forget to initialize the frame timer and the initial
previous frame delay in stream_component_open:


        is->frame_timer = (double)av_gettime() / 1000000.0;
        is->frame_last_delay = 40e-3;


### Synching: The Audio Clock

Now it's time for us to implement the audio clock. We can update the clock
time in our audio_decode_frame function, which is where we decode the audio.
Now, remember that we don't always process a new packet every time we call
this function, so there are two places we have to update the clock at. The
first place is where we get the new packet: we simply set the audio clock to
the packet's PTS. Then if a packet has multiple frames, we keep time the audio
play by counting the number of samples and multiplying them by the given
samples-per-second rate. So once we have the packet:


        /* if update, update the audio clock w/pts */
        if(pkt->pts != AV_NOPTS_VALUE) {
          is->audio_clock = av_q2d(is->audio_st->time_base)*pkt->pts;
        }


And once we are processing the packet:


          /* Keep audio_clock up-to-date */
          pts = is->audio_clock;
          *pts_ptr = pts;
          n = 2 * is->audio_st->codec->channels;
          is->audio_clock += (double)data_size /
    	(double)(n * is->audio_st->codec->sample_rate);


A few fine details: the template of the function has changed to include
pts_ptr, so make sure you change that. pts_ptr is a pointer we use to inform
audio_callback the pts of the audio packet. This will be used next time for
synchronizing the audio with the video.

Now we can finally implement our get_audio_clock function. It's not as simple
as getting the is->audio_clock value, thought. Notice that we set the audio
PTS every time we process it, but if you look at the audio_callback function,
it takes time to move all the data from our audio packet into our output
buffer. That means that the value in our audio clock could be too far ahead.
So we have to check how much we have left to write. Here's the complete code:



    double get_audio_clock(VideoState *is) {
      double pts;
      int hw_buf_size, bytes_per_sec, n;

      pts = is->audio_clock; /* maintained in the audio thread */
      hw_buf_size = is->audio_buf_size - is->audio_buf_index;
      bytes_per_sec = 0;
      n = is->audio_st->codec->channels * 2;
      if(is->audio_st) {
        bytes_per_sec = is->audio_st->codec->sample_rate * n;
      }
      if(bytes_per_sec) {
        pts -= (double)hw_buf_size / bytes_per_sec;
      }
      return pts;
    }


You should be able to tell why this function works by now ;)

So that's it! Go ahead and compile it:


    gcc -o tutorial05 tutorial05.c -lavutil -lavformat -lavcodec -lz
-lm`sdl-config --cflags --libs`


and **finally!** you can watch a movie on your own movie player. Next time
we'll look at audio synching, and then the tutorial after that we'll talk
about seeking.

_**>>** Synching Audio_

* * *

Function Reference
Data Reference

email:
dranger at gmail dot com
Back to dranger.com

This work is licensed under the Creative Commons Attribution-Share Alike 2.5
License. To view a copy of this license, visit
http://creativecommons.org/licenses/by-sa/2.5/ or send a letter to Creative
Commons, 543 Howard Street, 5th Floor, San Francisco, California, 94105, USA.


Code examples are based off of FFplay, Copyright (c) 2003 Fabrice Bellard, and
a tutorial by Martin Bohme.