Video Recording with MediaProjectionManager

Recording video on an Android device, as a developer, should be as easy as calling a platform-level API, possibly showing an intent for permission approval, before starting to streaming video. Unfortunately, we don’t live in an ideal world, and video recording is far more difficult than it should be.

Fortunately, starting in Android Lollipop, there is a new API (MediaProjectionManager[1]) to make recording video easier. However, there is a remarkable amount of incomplete or just plain wrong examples of how to use this API. So lets clear things up a bit.

How do it…

First get an instance of MediaProjectionManager:

// in an activity
private MediaProjectionManager mMediaProjectionManager;
mMediaProjectionManager = getSystemService(
    android.content.Context.MEDIA_PROJECTION_SERVICE);

Then create the permissions intent and show it to the user:

// in an activity
private static final int REQUEST_CODE_CAPTURE_PERM = 1234;
// …
Intent permissionIntent = mMediaProjectionManager.createScreenCaptureIntent();
startActivityForResult(permissionIntent, REQUEST_CODE_CAPTURE_PERM);

Handle the onActivityResult callback to get a MediaProjection[2]:

// in the same activity
private MediaProjection mMediaProjection;
// …
public void onActivityResult(int requestCode, int resultCode, Intent intent) {
	if (REQUEST_CODE_CAPTURE_PERM == requestCode) {
	    if (resultCode == RESULT_OK) {
            mMediaProjection = mMediaProjectionManager.getMediaProjection(resultCode, intent);
            startRecording(); // defined below
		} else {
		    // user did not grant permissions
		}
	}
}

Now create a virtual display to capture video:

// in the same activity
private static final String VIDEO_MIME_TYPE = "video/avc";
private static final int VIDEO_WIDTH = 1280;
private static final int VIDEO_HEIGHT = 720;
// …
private boolean mMuxerStarted = false;
private MediaProjection mMediaProjection;
private Surface mInputSurface;
private MediaMuxer mMuxer;
private MediaCodec mVideoEncoder;
private MediaCodec.BufferInfo mVideoBufferInfo;
private int mTrackIndex = -1;

private final Handler mDrainHandler = new Handler(Looper.getMainLooper());
private Runnable mDrainEncoderRunnable = new Runnable() {
    @Override
    public void run() {
        drainEncoder();
    }
};
// …

private void startRecording() {
    DisplayManager dm = (DisplayManager)getSystemService(Context.DISPLAY_SERVICE);
    Display defaultDisplay = dm.getDisplay(Display.DEFAULT_DISPLAY);
    if (defaultDisplay == null) {
        throw new RuntimeException("No display found.");
    }
    prepareVideoEncoder();

    try {
        mMuxer = new MediaMuxer("/sdcard/video.mp4", MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
    } catch (IOException ioe) {
        throw new RuntimeException("MediaMuxer creation failed", ioe);
    }

    // Get the display size and density.
    DisplayMetrics metrics = getResources().getDisplayMetrics();
    int screenWidth = metrics.widthPixels;
    int screenHeight = metrics.heightPixels;
    int screenDensity = metrics.densityDpi;

    // Start the video input.
    mMediaProjection.createVirtualDisplay("Recording Display", screenWidth,
            screenHeight, screenDensity, 0 /* flags */, mInputSurface,
            null /* callback */, null /* handler */);

    // Start the encoders
    drainEncoder();
}

private void prepareVideoEncoder() {
    mVideoBufferInfo = new MediaCodec.BufferInfo();
    MediaFormat format = MediaFormat.createVideoFormat(VIDEO_MIME_TYPE, VIDEO_WIDTH, VIDEO_HEIGHT);
    int frameRate = 30; // 30 fps

    // Set some required properties. The media codec may fail if these aren't defined.
    format.setInteger(MediaFormat.KEY_COLOR_FORMAT,
            MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface);
    format.setInteger(MediaFormat.KEY_BIT_RATE, 6000000); // 6Mbps
    format.setInteger(MediaFormat.KEY_FRAME_RATE, frameRate);
    format.setInteger(MediaFormat.KEY_CAPTURE_RATE, frameRate);
    format.setInteger(MediaFormat.KEY_REPEAT_PREVIOUS_FRAME_AFTER, 1000000 / frameRate);
    format.setInteger(MediaFormat.KEY_CHANNEL_COUNT, 1);
    format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1); // 1 seconds between I-frames

    // Create a MediaCodec encoder and configure it. Get a Surface we can use for recording into.
    try {
        mVideoEncoder = MediaCodec.createEncoderByType(VIDEO_MIME_TYPE);
        mVideoEncoder.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
        mInputSurface = mVideoEncoder.createInputSurface();
        mVideoEncoder.start();
    } catch (IOException e) {
        releaseEncoders();
    }
}

private boolean drainEncoder() {
    mDrainHandler.removeCallbacks(mDrainEncoderRunnable);
    while (true) {
        int bufferIndex = mVideoEncoder.dequeueOutputBuffer(mVideoBufferInfo, 0);

        if (bufferIndex == MediaCodec.INFO_TRY_AGAIN_LATER) {
            // nothing available yet
            break;
        } else if (bufferIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
            // should happen before receiving buffers, and should only happen once
            if (mTrackIndex >= 0) {
                throw new RuntimeException("format changed twice");
            }
            mTrackIndex = mMuxer.addTrack(mVideoEncoder.getOutputFormat());
            if (!mMuxerStarted && mTrackIndex >= 0) {
                mMuxer.start();
                mMuxerStarted = true;
            }
        } else if (bufferIndex < 0) {
            // not sure what's going on, ignore it
        } else {
            ByteBuffer encodedData = mVideoEncoder.getOutputBuffer(bufferIndex);
            if (encodedData == null) {
                throw new RuntimeException("couldn't fetch buffer at index " + bufferIndex);
            }

            if ((mVideoBufferInfo.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) {
                mVideoBufferInfo.size = 0;
            }

            if (mVideoBufferInfo.size != 0) {
                if (mMuxerStarted) {
                    encodedData.position(mVideoBufferInfo.offset);
                    encodedData.limit(mVideoBufferInfo.offset + mVideoBufferInfo.size);
                    mMuxer.writeSampleData(mTrackIndex, encodedData, mVideoBufferInfo);
                } else {
                    // muxer not started
                }
            }

            mVideoEncoder.releaseOutputBuffer(bufferIndex, false);

            if ((mVideoBufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
                break;
            }
        }
    }

    mDrainHandler.postDelayed(mDrainEncoderRunnable, 10);
    return false;
}

private void releaseEncoders() {
    mDrainHandler.removeCallbacks(mDrainEncoderRunnable);
    if (mMuxer != null) {
        if (mMuxerStarted) {
            mMuxer.stop();
        }
        mMuxer.release();
        mMuxer = null;
        mMuxerStarted = false;
    }
    if (mVideoEncoder != null) {
        mVideoEncoder.stop();
        mVideoEncoder.release();
        mVideoEncoder = null;
    }
    if (mInputSurface != null) {
        mInputSurface.release();
        mInputSurface = null;
    }
    if (mMediaProjection != null) {
        mMediaProjection.stop();
        mMediaProjection = null;
    }
    mVideoBufferInfo = null;
    mDrainEncoderRunnable = null;
    mTrackIndex = -1;
}

How it works…

I know, that last section is massive, so we’ll break it down. First we get the MediaProjectionManager by calling getSystemService (method available on any Context), which provides the interface for getting the video recording permission intent and then handling that intent to create a MediaProjection instance. The startActivityForResult method is used to launch the intent and onActivityResult handles the callback. This code assumes that everything is done in an activity with private member variables, but a singleton class works well too (not shown).

Once the MediaProjection instance is available, it is now possible to create the virtual display (a composite of all visible surfaces) and tell Android onto what surface you want to copy (record) the screen. The startRecord method (called as soon as the MediaProjection instance is available) first initializes the video encoder by calling prepareVideoEncoder. This is where we specify the desired output format to encode the screen video into and fetch the surface we intend to copy the screen to (mVideoEncoder.createInputSurface()). Most of the encoder configuration is pretty standard, a 1280x720 resolution at 30 frames per second.

Once the encoder is setup we create the muxer which converts the inputs into the output (a file on the sdcard in this example; your application will need permissions for this). In this case we only have one video input, so its fairly simple, but you can have multiple inputs (eg. audio from the mic or video from the camera). Calling createVirtualDisplay creates a virtual display representing the dimensions of the screen specified and outputting the screen to the surface of the encoder we created.

The drainEncoder method seems complicated, but is boilerplate code for sending a byte stream from the encoder to the muxer. It fetches an output buffer from the encoder and writes all the bytes to the muxer. This means that all the recently recorded video will be written to the file produced by the muxer. We use a handler with a ten millisecond delay to repeatedly call drainEncoder until the resources are released.

And when the recording is finished, simply call releaseEncoders to stop and release the resources. You should be able to call startRecording afterwards to begin recording again.

I hope this example provides a complete primer for using MediaProjectionManager to record video in Android Lollipop and beyond. That said, Android is often at the whim of the device manufacturer and its possible that this code may fail on certain devices. Please let me know if this is not working on your device and I’ll see if there is a way to support it.

There’s more…

While this code is a good example to show how to use MediaProjectionManager, it is simple to implement and consequently a little hacky for easy readability. Some relatively easy improvements that I recommend if you intend to productionize this are:

  • Create better logic around where you write the output. Try the SD card and if its not available, then fall back to the application package directory.
  • Separate the recording logic from the activity into a singleton class. Obviously, you will still need to use startActivity and onActivityResult in the activity, but everything else can live in a class devoted to recording.
  • Experiment with the surface size specified in createVirtualDisplay. You will probably find that a size ratio computed by the device dimensions produces better videos.
  • This is only available in Android Lollipop and beyond, so make sure everything is behind a version check.

Lastly, the muxer and encoder code works all the way back to Android Jellybean. If your application is writing directly to a surface (eg. openGL calls), then instead of using MediaProjectionManager you can manually write a second call to the input surface of the encoder anytime the main surface is written to, achieving full video recording on pre-Lollipop devices[4].

References

  1. MediaProjectionManager
  2. MediaProjection
  3. MediaMuxer
  4. Android Breakout recording on pre-L devices

Running Android Tests on a Device or Emulator

I have not been doing much web development lately, so its been difficult to come with interesting topics. However, I have been doing a lot of android development and since many engineers have to work cross discipline, I think an android article be relevant. This article will discuss how to run unit tests against your android code, directly on the android device or emulator.

Getting ready

You will need to install the