Back to Blog

Advanced Shape Layer Techniques in After Effects

By Alex Deaton
After Effects

Clean up your workspace and rid yourself of messy precomps and alpha mattes with this professional shape layer workflow

Alpha mattes and messy precomps quickly clutter up your workspace and break as soon as you infinitely rasterize them or make them 3D. We’ll show you how to build single-layer shape “precomps” utilizing groups, merge paths, and simple path expressions so you can kiss those redundant matte layers goodbye... forever.
I’ve been a motion designer for almost ten years. Along the way I’ve picked up some After Effects workarounds that save me from daily Adobe-induced frustration migraines. One of these techniques is a shape-layer workflow I utilize in almost every project to avoid layer clutter and over-complicated matting and precomping issues. 
In this tutorial, you’ll learn:
  • How to create a clean layer workspace
  • How to utilize shape groups
  • Advanced ways to use merge paths
  • A few simple path expressions

Download the project files to follow along

To make things easier, you can download the project files we're using in this tutorial to follow along, or to practice this technique after you’re done watching.

Grab the free project files and follow along!

Download now

Advanced Shape Layer Techniques in After Effects

How to properly us shape groups in After Effects

Alpha mattes and precomps can be useful tools for increasing the complexity of an animation or combining several visual elements in a complicated design, but they make your timeline messy and introduce frustrating glitches and comp failures when infinitely rasterizing precomps or making layers 3D. Let’s work around this by taking advantage of After Effect’s shape layer tools.
In the tutorial, I'll go through how to design and animate a vase, but let's start with something a little simpler: a pair of eyes.
The first thing we'll do is open a new comp and grab the ellipse tool. We'll twirl that down to 500x500, duplicate, and name our two layers "Eye Main" and "Pupil." I'll change the color of the eye layer to white and shrink the pupil, and now we have a nice, simple eye.
To get a nice blink, I don't want to just use the dimensions as that wouldn't be realistic. Instead, I click on Path and Convert to Bezier Path, which provides me better control.
I'll duplicate this layer, rename it "Eye MASK," and pick whip the path of the mask to the main layer.
I'll highlight both the eye mask and the pupil, hit Command G, and now I've grouped those two together. I'll rename this entire group "Pupil." Now I just need to make it so the pupil itself can mask through the eye.
Select the pupil group, go to the Add drop down, and select Merge Paths.
Make sure the fill is the same as the pupil. Twirl down the Merge Paths drop down and select Intersect. Now you have a masked layer. Now if you animate the eye shape, the pupil is properly masked. Let's animate a very simple blink.
By copying keyframes and just moving a few frames forward, we can quickly set up the basic start and end points for our blink. Then we'll easy ease, adding keyframes if it looks too fast or unrealistic (not that you're really breaking the uncanny valley on something this simple).
You'll notice that there is a small sliver of white when the eye is closed. A quick fix is to add a stroke to the eye and animate it on the blink.
I took it a bit further, adding some eye movement (and an entire ghost), but you get the basic idea of how to group and what is possible. Now it's time to get more advanced.

Animating within shape groups in After Effects

Now we're going to animate this vase and give it a bit more personality.
One way of building this scene would be to break out all of the shapes on their own layers and duplicate the main shape with property links, but as you can imagine, this quickly clutters up the timeline.
Additionally, we could precomp the main shape and apply one mask as a silhouette layer at the top of the precomp, but as soon as we infinitely rasterize this precomp in the main composition, everything breaks. Instead, we're going to use our knowledge of grouping layers from earlier. This is a great time to follow along with the project files.
You'll see that we have the vase layer minus the inner shapes, with keyframes for that squishy animation. Duplicate the layer and rename it "Vase Mask." Twirl it open, pick whip the path just as before, and you can get rid of the Gradient as well.

How to use Merge Paths in After Effects

Now highlight the Secondary Shape and the Mask Shape and press “Command + G” to group these shapes together. With this new group highlighted, navigate to the “Add” dropdown on the right and select “Merge Paths”.
Twirl down the Merge Paths effect and change the dropdown within to “Intersect”.
Delete the stroke layer that was added and change the fill color to our desired color. Voila! Now you can animate the shape inside its own layer and it will be perfectly masked inside of your main shape at all times. No precomps, no alpha mattes, no mess.
You can also change the Fill to a Gradient Fill if that suits the look you’re going for. If you want to add more shapes into the mix, go ahead and duplicate the Secondary Shape group and paste in another shape or create one yourself using one of the shape tools or the pen tool. Delete the original secondary shape but keep the mask layer and everything will work just the same.  
If you want to see just how wild things can get from here, watch the full video above!

Things to keep in mind when using these advanced After Effects techniques

There are a few drawbacks to this approach that should be mentioned. For starters, you can’t make this Merge Paths trick work with strokes. The stroke will automatically close the shape where it intersects with the mask.
I work around this by simply making a fill shape that looks like the stroke I’m trying to create, but it is a less-than-perfect fix.
Additionally, you can’t apply effects to the secondary shapes independent of the original shape, such as a glow or blur, since all the layers including the original layer and the mask are on one single shape layer. Here, you’ll unfortunately have to resort to the classic matting and precomp methods, clutter and all.
Despite these drawbacks, this approach has saved me time and sanity by keeping my projects simple, compact, and iterable.

Take on an advanced After Effects course

If you’re ready to continue your education, check out Advanced Motion Methods from School of Motion. You'll learn how to structure animations according to geometric proportions found in nature, deal with complexity, create cool transitions, and learn tips from a seasoned After Effects veteran.