For beginners looking for approaches using After Effects instead of the few more specialized tools, there are some still classic interrelated resources to supplement the recent Stu Maschwitz tweet (later saved with helpful comments as a Twitter moment).

Rotoscoping:
• Isn’t the enemy.
• Use lots of shapes, with as few points as possible
• Never use bezier curves with handles
• When possible, move points in groups instead of 1 at a time
• Preview on a solid color similar to BG
• Know when bad roto (like this) is good enough pic.twitter.com/F0YYw7yi97

— Stu Maschwitz (@5tu) August 27, 2018

Rotoscoping has a long history, and can can refer to painting on frames (as in the cartoon look of A Scanner Darkly) or drawing mattes, which is now done by mostly using various digital spline types to create outlines. There are numerous articles on the web about roto, and one of the most comprehensive is The Art of Roto: 2011 by Mike Seymour, which covers the history and process, and lists of the various roto tools available in 2011.

In the early-to-mid 1990s you could use After Effects to roto but it was very easy to get chattering mattes. So, Puffin Design Commotion, a Macintosh application coming from ILM veterans, introduced a more powerful feature set with a digital gumbo of better tracking, painting, and spline tools. When Commotion added compositing and powerful keying tools (now reworked into Red Giant sets), After Effects had serious competition. Unfortunately, both Commotion and another competitor, Luminaire Paint and Effects (aka Discreet Combustion), ended up at companies (Pinnacle and Autodesk) that couldn’t or wouldn’t keep them alive.

A close look at resources on newer roto-related features in After Effects like the Roto Brush, will have to wait for another time. It’s easy to be frustrated by Roto Brush (consider the source), but Chris Meyer’s explanations here on PVC and elsewhere may still help. Learn Advanced AE Rotobrush in 20 Min by Teddy Gage may also be good for the frustrated, if not the cognoscenti.

https://www.youtube.com/watch?v=WQX_W7f4rPo

Instead of considering only new features, it might be good to look at battle-refined techniques from the recent past that can still be used. One notable figure in earlier desktop rotoscoping was Scott Stewart, who also worked at ILM and posted Confessions of a Roto Artist: Three Rules For Better Mattes (PDF), which can be found on the Internet Archive. Later, Scott Squires, the creator of Commotion, shared advice on roto in a series of posts and movies collected in Rotoscoping – The Basics at his old blog, Effects Corner. Here’s a sample:

A great general desktop VFX training set from day of yore is still available, the out-of-print Masters of Visual Effects from Matt Silverman, a Puffin Designs veteran who himself once published great tutorial DVDs on Commotion and procedual matte creation. The Masters of Visual Effects tapes covered basic compositing, keying, tracking, paint, and rotoscoping by some prominent presenters.

You can find the same series discussed by another Puffin vet, Forest Key in Masters of Visual Effects 1999 Style, who says “doing full immersive VR content requires some of the same proxy-resolution workflows that we employed in 1999 to deal with the film-video-digital steps of that era.” Here’s just the part with Scott Stewart on rotoscoping from that series, but the ones hosted by Stu Maschwitz and others are also worth ingesting.