Struggling with the rotoscope tools in Adobe After Effects? Here’s a quick guide on how to rotoscope in AE using the masking and Roto Brush tools.
Rotoscoping can be the bane of a motion graphics artist’s existence. It’s tough, monotonous, and just generally a drag. If you’re not familiar with rotoscoping, it’s the process of animating a mask over a subject in the foreground to separate it from the background. You can use this to put objects between your foreground and background or to individually edit and manipulate the different layers. It’s almost like creating your own layers out of a single image, like you would in Photoshop.
Beginners might think rotoscoping is daunting, but it’s a lot easier than it seems. In this video, I’m going to show you two ways you can rotoscope through masking and Roto Brushing.
Rotoscoping Through Masking
Using masks to rotoscope is the OG way to do it, and if you ask any old animators, I’m sure this is the method they would teach you. It starts with selecting your video layer in AE and then selecting the pen tool. Use the pen to create an outline of your foreground object. Clicking from one place to another will create a straight line between the two points. If you click and hold, you can create a bezier curve that can trace the curved edges of your object. Follow along the entire outer edge of your object until you have a complete mask.
Now, go to your mask layer in your video, and click the keyframe stopwatch on the bottom left of the layer. From here, press Page Up and Page Down to cycle through each keyframe. As the object moves, you will have to adjust each node to fit the line again. Sometimes, if there isn’t too much movement, you won’t have to adjust the nodes on each keyframe. Don’t worry about it getting on the exact pixel — you can feather it later down the line.
Once you’ve finished your mask, play through your timeline and see if the mask matches up with the object’s outline. It’s going to look a little harsh — that’s where the feather tool comes in. Double-click the M key to access the mask options, and increase the feather. This will put a nice blur on the edges of your mask.
Using the Roto Brush Tool
For those of you who are too lazy to go through every single frame, you can take the easy way out (my preference). The Roto Brush is a tool that uses Adobe’s AI to search your video for edges, just like the magic wand tool in Photoshop. Click the Roto Brush tool in your toolbar (it looks like a little paintbrush with a man next to it). Double-click your preferred layer to open it up in a layer tab. From here, the Roto Brush will give you a small green circle to draw with. To increase the size of the circle, hold Ctrl/Cmd and drag up or down with your mouse. From here, create a rough outline of your subject.
AE will seek out the lines on your object and create a mask over it. It won’t be perfect, so you may have to go in with a smaller circle tool to clean up the edges. If the line is too big over your object, hold the Alt button and your circle will change from green to red, indicating that you are in the “subtract” mode.
After you’ve created a good outline, you can run it over with the Refine Edge tool within the Roto Brush selection tool. Go over your outline with this brush so AE can clean up the mask — right down to the last pixel. (It works extremely well with hairlines and very detailed edges.)
Now that you have a refined mask over your object, all you have to do is press the Page Up and Page Down buttons to cycle through the keyframes. Adobe’s AI will automatically seek out the changes in your outline and keyframe for you. If you need to adjust anything, just go back in with your Roto Brush tool and fix it from there. After the sequence is complete, head back to your main timeline, where you’ll find your extremely well-masked object waiting for adjustments.
Interested in the tracks we used to make this video?
- “A Breath of Fresh Air” by Vincent Tone
- “Attracted by You” by Chill Study
- “Brooklyn Drive” by Chill Study
- “Relaxing in Vermont” by Chill Study
Looking for more video tutorials? Check these out.
- The World’s First Cinematography Video Game — Cine Tracer
- How to Build Your Own Video Editing Computer
- Video Tutorial: Why You Need Lens Filters for your Drone
- Video Tutorial: How to Get Cinematic Lighting In Small Spaces
- How to Use Film Grains + FREE Film Grain Overlays
