Video Recording with MediaProjectionManager

Recording video on an Android device, as a developer, should be as easy as calling a platform-level API, possibly showing an intent for permission approval, before starting to streaming video. Unfortunately, we don’t live in an ideal world, and video recording is far more difficult than it should be.

Fortunately, starting in Android Lollipop, there is a new API (MediaProjectionManager[1]) to make recording video easier. However, there is a remarkable amount of incomplete or just plain wrong examples of how to use this API. So lets clear things up a bit.

How do it…

First get an instance of MediaProjectionManager:

// in an activity
private MediaProjectionManager mMediaProjectionManager;
mMediaProjectionManager = getSystemService(
    android.content.Context.MEDIA_PROJECTION_SERVICE);

Then create the permissions intent and show it to the user:

// in an activity
private static final int REQUEST_CODE_CAPTURE_PERM = 1234;
// …
Intent permissionIntent = mMediaProjectionManager.createScreenCaptureIntent();
startActivityForResult(permissionIntent, REQUEST_CODE_CAPTURE_PERM);

Handle the onActivityResult callback to get a MediaProjection[2]:

// in the same activity
private MediaProjection mMediaProjection;
// …
public void onActivityResult(int requestCode, int resultCode, Intent intent) {
	if (REQUEST_CODE_CAPTURE_PERM == requestCode) {
	    if (resultCode == RESULT_OK) {
            mMediaProjection = mMediaProjectionManager.getMediaProjection(resultCode, intent);
            startRecording(); // defined below
		} else {
		    // user did not grant permissions
		}
	}
}

Now create a virtual display to capture video:

// in the same activity
private static final String VIDEO_MIME_TYPE = "video/avc";
private static final int VIDEO_WIDTH = 1280;
private static final int VIDEO_HEIGHT = 720;
// …
private boolean mMuxerStarted = false;
private MediaProjection mMediaProjection;
private Surface mInputSurface;
private MediaMuxer mMuxer;
private MediaCodec mVideoEncoder;
private MediaCodec.BufferInfo mVideoBufferInfo;
private int mTrackIndex = -1;

private final Handler mDrainHandler = new Handler(Looper.getMainLooper());
private Runnable mDrainEncoderRunnable = new Runnable() {
    @Override
    public void run() {
        drainEncoder();
    }
};
// …

private void startRecording() {
    DisplayManager dm = (DisplayManager)getSystemService(Context.DISPLAY_SERVICE);
    Display defaultDisplay = dm.getDisplay(Display.DEFAULT_DISPLAY);
    if (defaultDisplay == null) {
        throw new RuntimeException("No display found.");
    }
    prepareVideoEncoder();

    try {
        mMuxer = new MediaMuxer("/sdcard/video.mp4", MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
    } catch (IOException ioe) {
        throw new RuntimeException("MediaMuxer creation failed", ioe);
    }

    // Get the display size and density.
    DisplayMetrics metrics = getResources().getDisplayMetrics();
    int screenWidth = metrics.widthPixels;
    int screenHeight = metrics.heightPixels;
    int screenDensity = metrics.densityDpi;

    // Start the video input.
    mMediaProjection.createVirtualDisplay("Recording Display", screenWidth,
            screenHeight, screenDensity, 0 /* flags */, mInputSurface,
            null /* callback */, null /* handler */);

    // Start the encoders
    drainEncoder();
}

private void prepareVideoEncoder() {
    mVideoBufferInfo = new MediaCodec.BufferInfo();
    MediaFormat format = MediaFormat.createVideoFormat(VIDEO_MIME_TYPE, VIDEO_WIDTH, VIDEO_HEIGHT);
    int frameRate = 30; // 30 fps

    // Set some required properties. The media codec may fail if these aren't defined.
    format.setInteger(MediaFormat.KEY_COLOR_FORMAT,
            MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface);
    format.setInteger(MediaFormat.KEY_BIT_RATE, 6000000); // 6Mbps
    format.setInteger(MediaFormat.KEY_FRAME_RATE, frameRate);
    format.setInteger(MediaFormat.KEY_CAPTURE_RATE, frameRate);
    format.setInteger(MediaFormat.KEY_REPEAT_PREVIOUS_FRAME_AFTER, 1000000 / frameRate);
    format.setInteger(MediaFormat.KEY_CHANNEL_COUNT, 1);
    format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1); // 1 seconds between I-frames

    // Create a MediaCodec encoder and configure it. Get a Surface we can use for recording into.
    try {
        mVideoEncoder = MediaCodec.createEncoderByType(VIDEO_MIME_TYPE);
        mVideoEncoder.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
        mInputSurface = mVideoEncoder.createInputSurface();
        mVideoEncoder.start();
    } catch (IOException e) {
        releaseEncoders();
    }
}

private boolean drainEncoder() {
    mDrainHandler.removeCallbacks(mDrainEncoderRunnable);
    while (true) {
        int bufferIndex = mVideoEncoder.dequeueOutputBuffer(mVideoBufferInfo, 0);

        if (bufferIndex == MediaCodec.INFO_TRY_AGAIN_LATER) {
            // nothing available yet
            break;
        } else if (bufferIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
            // should happen before receiving buffers, and should only happen once
            if (mTrackIndex >= 0) {
                throw new RuntimeException("format changed twice");
            }
            mTrackIndex = mMuxer.addTrack(mVideoEncoder.getOutputFormat());
            if (!mMuxerStarted && mTrackIndex >= 0) {
                mMuxer.start();
                mMuxerStarted = true;
            }
        } else if (bufferIndex < 0) {
            // not sure what's going on, ignore it
        } else {
            ByteBuffer encodedData = mVideoEncoder.getOutputBuffer(bufferIndex);
            if (encodedData == null) {
                throw new RuntimeException("couldn't fetch buffer at index " + bufferIndex);
            }

            if ((mVideoBufferInfo.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) {
                mVideoBufferInfo.size = 0;
            }

            if (mVideoBufferInfo.size != 0) {
                if (mMuxerStarted) {
                    encodedData.position(mVideoBufferInfo.offset);
                    encodedData.limit(mVideoBufferInfo.offset + mVideoBufferInfo.size);
                    mMuxer.writeSampleData(mTrackIndex, encodedData, mVideoBufferInfo);
                } else {
                    // muxer not started
                }
            }

            mVideoEncoder.releaseOutputBuffer(bufferIndex, false);

            if ((mVideoBufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
                break;
            }
        }
    }

    mDrainHandler.postDelayed(mDrainEncoderRunnable, 10);
    return false;
}

private void releaseEncoders() {
    mDrainHandler.removeCallbacks(mDrainEncoderRunnable);
    if (mMuxer != null) {
        if (mMuxerStarted) {
            mMuxer.stop();
        }
        mMuxer.release();
        mMuxer = null;
        mMuxerStarted = false;
    }
    if (mVideoEncoder != null) {
        mVideoEncoder.stop();
        mVideoEncoder.release();
        mVideoEncoder = null;
    }
    if (mInputSurface != null) {
        mInputSurface.release();
        mInputSurface = null;
    }
    if (mMediaProjection != null) {
        mMediaProjection.stop();
        mMediaProjection = null;
    }
    mVideoBufferInfo = null;
    mDrainEncoderRunnable = null;
    mTrackIndex = -1;
}

How it works…

I know, that last section is massive, so we’ll break it down. First we get the MediaProjectionManager by calling getSystemService (method available on any Context), which provides the interface for getting the video recording permission intent and then handling that intent to create a MediaProjection instance. The startActivityForResult method is used to launch the intent and onActivityResult handles the callback. This code assumes that everything is done in an activity with private member variables, but a singleton class works well too (not shown).

Once the MediaProjection instance is available, it is now possible to create the virtual display (a composite of all visible surfaces) and tell Android onto what surface you want to copy (record) the screen. The startRecord method (called as soon as the MediaProjection instance is available) first initializes the video encoder by calling prepareVideoEncoder. This is where we specify the desired output format to encode the screen video into and fetch the surface we intend to copy the screen to (mVideoEncoder.createInputSurface()). Most of the encoder configuration is pretty standard, a 1280x720 resolution at 30 frames per second.

Once the encoder is setup we create the muxer which converts the inputs into the output (a file on the sdcard in this example; your application will need permissions for this). In this case we only have one video input, so its fairly simple, but you can have multiple inputs (eg. audio from the mic or video from the camera). Calling createVirtualDisplay creates a virtual display representing the dimensions of the screen specified and outputting the screen to the surface of the encoder we created.

The drainEncoder method seems complicated, but is boilerplate code for sending a byte stream from the encoder to the muxer. It fetches an output buffer from the encoder and writes all the bytes to the muxer. This means that all the recently recorded video will be written to the file produced by the muxer. We use a handler with a ten millisecond delay to repeatedly call drainEncoder until the resources are released.

And when the recording is finished, simply call releaseEncoders to stop and release the resources. You should be able to call startRecording afterwards to begin recording again.

I hope this example provides a complete primer for using MediaProjectionManager to record video in Android Lollipop and beyond. That said, Android is often at the whim of the device manufacturer and its possible that this code may fail on certain devices. Please let me know if this is not working on your device and I’ll see if there is a way to support it.

There’s more…

While this code is a good example to show how to use MediaProjectionManager, it is simple to implement and consequently a little hacky for easy readability. Some relatively easy improvements that I recommend if you intend to productionize this are:

  • Create better logic around where you write the output. Try the SD card and if its not available, then fall back to the application package directory.
  • Separate the recording logic from the activity into a singleton class. Obviously, you will still need to use startActivity and onActivityResult in the activity, but everything else can live in a class devoted to recording.
  • Experiment with the surface size specified in createVirtualDisplay. You will probably find that a size ratio computed by the device dimensions produces better videos.
  • This is only available in Android Lollipop and beyond, so make sure everything is behind a version check.

Lastly, the muxer and encoder code works all the way back to Android Jellybean. If your application is writing directly to a surface (eg. openGL calls), then instead of using MediaProjectionManager you can manually write a second call to the input surface of the encoder anytime the main surface is written to, achieving full video recording on pre-Lollipop devices[4].

References

  1. MediaProjectionManager
  2. MediaProjection
  3. MediaMuxer
  4. Android Breakout recording on pre-L devices

Karma Test Runner with QUnit in 10 Minutes

This is a ten minute primer for using Karma and QUnit to unit test JavaScript.

Getting ready

Install Karma[1], plugins, and Qunit[2].
 # Install Karma npm install karma --save-dev # Install plugins npm install karma-qunit karma-phantomjs-launcher --save-dev # Add global CLI npm install -g karma-cli # Install QUnit npm install --save-dev qunitjs 

How do it…

In your project directory, run the Karma initialization script:
 karma init ...

Object Pool Pattern in JavaScript

Now that we understand the Recycler Object for Object Pool Pattern, we can build the logic for managing the object pool. An object pool is a simple API that manages recycling and fetching recyclable objects. There are two common models for object pools, one with a fixed number of objects that errors if too many objects are requested, and the other (more flexible approach) is to use the object pool for a fixed number ...

Recycler Object for Object Pool Pattern

Creating or destroying an object is never free and JavaScript is no exception. Generally, the cost of creating/destroying an object in JIT-optimized JavaScript runtimes doesn't affect performance, but other languages will have a performance hit as well. The real culprit is the increase in your application's memory footprint (watch the memory tab in a developer tool while running the tests below for an illustration). This is why, in most cases, reusing a single object is ...

Start Using SRCSET IMG Attribute For Serving Device Specific Images

The article, JavaScript Low Resolution Image Replacer, discussed a JavaScript solution for replacing low resolution images with higher resolution ones after the page finishes loading all the initial resources (thereby reducing the load time of your pages). But what about loading different image sizes based on the resolution of the user’s device. We could write a JavaScript solution for this (and some developers have), but HTML 5 already introduces the concept of <img srcset[

JavaScript Low Resolution Image Replacer

This is a handy, yet very simple, widget I was hacking on to replace loading or low resolution images with higher resolution ones once the document has finished loading.

How do it…

The widget is built using the jQuery plugin system and here is the complete code:
 (function($) { var isLoaded = false, REPLACEMENT_CLASS = "replacement-class", REPLACEMENT_RCLASS = "replacement-rclass", REPLACEMENT_URL = "replacement-img", TIMER = 500; $(window).load(function() { isLoaded = true; }); function ...

Network Information API Polyfill

One of the many new HTML5 APIs slowly being implemented by browsers is the Network Information API[2]. It exposes information about the type of network that the connecting device is using. In theory, this allows developers to optimize content around the connection speed of the user. However, as with most HTML5 APIs it is supported only by some browsers with/without prefixes, and has a legacy implementation, so a polyfill is useful when working with ...

Detecting Object Mutations by Counting Properties

Have you ever included a library and wonder, "how much did this library add to the window object", or passed an object into a function and asked yourself, "did that function modify my object"? Instead of reading the source code, this article shows a quick trick for answering these questions.

How do it…

For starters we need a function to count the number of properties on an object:
 function fnCountProperties(o) { var ...

Using Google Play Games on the Web

As many of you know, I now work for Google on the Play Games Team. We provide APIs for game developers, implementing useful features like leaderboards and achievements, so the developer doesn't have to. While many Android developers are using our services, adoption on the web could be better, so lets take a look at how to integrate the Google Play Games Services achievements into a web game.

Getting ready

Become a Google developer ...

Site Outage Fixed - Down for 32 hours :(

I apologize that the site has been down for a couple of days. It was a perfect storm of events. For starters, Amazon told me about two weeks ago that my server was going to be terminated and that I should do something about it (they do this now and again, as they upgrade equipment). And like any good developer, I was procrastinating until the last minute (Wednesday this week). Then on Tuesday, I received ...