Pi member Stuart Bonwit has been working for several years on his Mac to reproduce scenes from ballets and operas, in 3D, complete with music. Along the way he has explored a variety of tools to make this task easier. This is the latest installment in his quest.
Mimic is a 3D software application from DAZ Productions, Inc. that provides a method for easily creating and editing facial animation sequences. It can be purchased as a stand-alone application, and is also available as a plugin to NewTek's LightWave (version 7.5 or higher) or as a plugin for e-frontier’s Poser (version 3 or higher). I am reviewing the Lite stand-alone version of Mimic 3. The examples described in this review may be viewed on the Pi Web site, http://www.wap.org/journal/mimic/
The main function of the Mimic software is to create an animation of a speaking 3D figure’s facial expressions synchronized to a spoken sound track. Not advertised is its ability to sync a singing voice. This is what I am most interested in.
Mimic opens with the splash frame shown below in Figure 1,
Fig. 1 Splash Screen
followed by the screen shown in Figure 2 (note: it may open with a different character, person, or animal).
Fig. 2 Opening screen
I record a sound track (outside of the Mimic program) in which I say, “My name is Stuart Bonwit.” I click the “Load” button and select my record file. I then click the “Movie” button and, voila, Mimic plays a lip-sync movie of the cartoon character saying what I recorded. The file can be seen at the site shown above (original QuickTime 1.7 MB; QuickTime H.264 MP4, 276K). It’s as easy as that to get started.
Now, if you are reasonably observant, you will notice that the lips never completely close on the “m” sounds.
For a comparison I made a similar animation in Poser, an animation program I have used extensively. Poser allows facial expressions to be changed in a wide variety of ways using packaged expressions including those corresponding to phonemes (sounds) of speech. Basic to making this animation is precise timing of the sounds of the speech. A program called Audacity allows this by displaying the sound waves. Figure 3 illustrates this:
Figure 3 Timing of Sound
By playing the sounds one can find the location of each sound. Placing the cursor on the start of a sound gives the start time of that sound. In Figure 3 the cursor is at the start of the word “My.” At the bottom of the figure the time is indicated as 0:00:255651 min:sec. For an animation playing at 30 frames per second, this corresponds to the 0.256 sec x 30 fr/sec = 8th frame of the animation. The facial expression for “m” is then placed at the 8th frame. Actually it must be placed earlier. Try saying “my.” You close your lips a short time before any sound comes out!
Continuing this tedious process results in the My Name FQDV.mov available on the Pi site (original QuickTime, 13.8 MB; QuickTime H.264 MP4, 308K).
While the Mimic result is not quite as good as that from Poser, it is infinitely easier and faster.
The test came when I tried to use Mimic with a singer and orchestra. I am currently working on an animation of a diva singing Vissi D’Arte from Puccini’s opera Tosca. For the test I used the female figure provided with Mimic and the sound track of a diva singing the first line of the libretto of Vissi D’Arte.
The result is the Vissi Mimic.mov video on the Pi site (original QuickTime, 15.8 MB; QuickTime H.264 MP4, 1.1 MB).
Figure 4: Vissi D’Arte test.
If you ignore the fact that what looks like a sub-teen could not possibly be an opera star, you will notice that there is some correspondence between the sound track and the lip movements. So Mimic does work with singing as well as speech. You will also notice that Mimic automatically includes eye blinking. However, if you’ve ever watched a diva close up singing in an opera, you’d know these facial expressions do not correspond to those in opera. I have therefore included a segment from my animation of Vissi D’Arte, Vissi D'Arte Web Proj.mov on the Pi site (original QuickTime, 23.2 MB; QuickTime H.264 MP4, 568K).
Mimic is very good, and fast, at synchronizing facial expressions to spoken words in a sound track. It also works with a singing voice though it is not advertised to do so. This is the purpose for which I bought Mimic. However, the facial expressions it gives for a recorded operatic aria are unrealistic compared to those for a real opera singer. Therefore, my dream of automating my opera animation is dashed! I will, though, keep Mimic for some unforeseen project in the future.
Mimic is available from:
http://www.daz3d.com/i.x/software/mimic/
Mimic Lite, which I have, is $69.95. Mimic Pro is $199.95