index
3DXchange5 Pipeline test series 14: AKB48 Fortune Dance in iClone5
2013.12.20 Hirospot (Personal Site)
Trial of Motion Tracing by iClone5 Key-Frame Animation
(YouTube)
Go to other Report→
印は日本語でのレポートです

This is a traditional solution for getting Dance Motion data from Dancing Video.

AKB48 is a very charming Japanese idle group and the 'Fortune Cookie in Love'
has recorded milions hit. So many cover videos are seen in YouTube channels.
I wanted to try playing this dance motion in iClone5, without using Kinect system.
Top

By locating a 3D target model (G5 Actor) in front of a Reference Dance Video image,
G5 Actor can be manually postured to make a continuous dance motion data in iClone5.

The 'Edit Motion Layer Panel' of iClone5.5 is useful to produce and edit each postures
at each dance steps manually, the Motion Data for all keys are auto-generated as
Key-Frame Motion and the result is saved in the Motion Layer Tracks in the Timeline.

Then, iCone 'Motion Plus' tool can collect and save a part or all the Key-Frame Motion data
for the use for other models, or transfer to 3DXchange5 for further FBX exporting.

This practice is my first tryal and the traced dance video is not completed yet.

Note: This is a personal report shown for other users who might be interested in these area.

Sincere Thanks for the Following Resources that are referenced in my work.
 
Title, Autor and Copyright
Access
Notes
Music Copyright: AKB48 Japan AKB48 Official Channel （Japanese Lyrics）
Lyrics-J Yasushi Akimoto 恋するフォーチュンクッキー（Japanese）
Lyrics-E JKT48 & Fang KFC English (English Version) (AVI)
Video1 Voice & Sound:ibnu nawafil JKT48 Fortune Cookie Yang Mencinta （English Lyrics）
Video2 Dancing:かやowo Koi Suru Fortune Cookie
Video3 Dancing:yphong520 Dancing cover with China
(Note) Find the Video Source suitable for the Reference with full body, good contrast and constant camera angle.
Video4 iClone5 Tutorial
by Real;usion
Intro to Timeline
Video5 ICLONE5 Tutorial
by stuckon3d
 Sliding feet? Not on my watch:
Video6 iClone5 Tutorial
by Reallusion
Instant Kinect Motion Refinement (Human IK and Direct Puppet)
Video7 iClone5 Tutorial
by Reallusion		 Producing a Custom Pose or a Key-frame Motion

The Most Appreciation for the iClone 5.51, no more crashes when doing Timeline Editing.
Tools
Product Version
  
Notes
PMDEditor
0.1.3.9
 Add ColladaExport Plugin for DAE file export.
(For more info in Report#4)
FBX Converter

2013.3 x64

Autodesk. Convert DAE file to FBX.
3DXchange 5 Pipeline
5.5.2007.1
 5.5 Patch 8/8/2013
5.51.2209
    
5.51.2312.1
   5.51 Patch: 11/15/2013
iClone5.5 Pro
5.5.3207.1 English
Started with this version. Crashed frequently.
iClone5.51

5.51.3507.1 English

5.51 Patch 11/15/2013, Downloaded 11/30/2013
Fixed: Reduced random crashes in the 64 bit version when using Motion Puppet & Timeline Editing.

1
 Viewport Settings for Video Motion Tracing in iClone5.5
 

Selected Reference AKB Dance Motion is performed within a limited floor area.
It is preferable to set the Viewport Grid spacing and size to cover this area.

1-1
 Open Preference, Set the Grid Spacing=20, Grid Size=20.
 
Preference
2
 3D Target Model suitable for Video Motion Tracing in iClone5.5

It is preferable that the Target 3D Model to be used for the Video Motion Tracking
has similar looks and propotion with Video to simulate the body pose easier.
G5 standard Gwynn is used by modifying the Avatar Proportion and face.

Target 3D Model : G5 Actor, originated from Gwynn
2-1
 Load the G5 Standard 'Gwynn'. Open Avatar Proportion and change the proportion to look Oriental Girl.
 
1. Following modifications were made to Gwynn.
(Only Personal Preference)
1) Enlarge Head.
2) Change Facial Parts.
3) Change Hair Style.
4) Change Material color.
5) Maker Lings are attached.
2. Apply some Animation Motions for Test.


2-2

Go to Actor/Avatar, load the G5 Actor to the center of the Grid. Go to Modify, set Camera Lens=200.
 
Target Model
2-3
 Add Marker Rings to her arms. This Rings are useful to confirm the unexpected arm rotations.
 
Use Local axis for the rotation and move tools for Marker Ring setting and attach to the arm parts respectively.
Marker Rings
2-4
 Go to Set/Props/Template, drag Wall_001 on the 3D view and adjust the size for the Video screen.
 
Screen scale: X=70, Y=50, Z=90Screen
2-5
 Add the Floor_001 to make Side Grid for posing guide.
 
Vertical Grid is made by the paint tool and mapped on the Floor_001 wtih same grid size.
Use Floor instead of Wall to prevent double sided mapping.
This Prop visibility can be controlled by Scene Manager.
Side Grid
2-6
Go to Modify/Avatar and activate the Foot Contact. Heel shapes are modified to view the foot contact clearly.
 
 

3
 Making the Reference Video Screen in the iClone 3D view
 
Reference video is necessary for Motion Tracing in iClone.
Selection of Reference video is very important.
Among many AKB Fortune Dance cover videos, select
Camera angle being fixed,
Full body being clealy visible, and
Dance motion seems vivid and natural.
 
AVI

Lots of AKB Fortune Dance videos are searched in various web sites.

The Video selected and used for this test is found in the YouTube This Site.
Thank you for the excellent Dancing.

The Video was enhanced by GOM ENCODER, and saved as AVI format.
3-1
 Prepare Video with AVI format and drag the file onto the screen to map the AKB Dance Motion.
 

Initial Offset and
Tiling Vatues:

 
Offset
Tiling
U
0.000
1.000
V
0.000

1.000
3-2
 Go to Modify/Material and use Offset/Tiling Setting to get the Reference Video Actor in place.
 

Adjusted Offset and
Tiling Vatues:
 
Offset
Tiling
U
0.827
1.190
V
0.045
2.914
3-3
 Locate the G5 Actor just in front of one of the Dancing Girl with Camera Lens=200.
3-4 Confirm the Foot Contact of the Reference Model and adjust the Foot Position in the Screen.
3-5
 Go to Stage/Light/Custom/Light/Light, select Lights and apply.
3-6
 Click Mini Viewport icon (or Push F8 button) to get a Mini Viewport on the 3D view. (Optional)
3-7
 Add new camera, locate it to the Left position of the G5 Actor. This will help to confirm the Model side view.
 

Add Camera
3-8
 Observe the difference between G5 Model bone structure and the Dummy Pane to be used.
 
G Model Bone
3-9
Confirm operation of the Marker Ring at the bottom of the Upper Body for body deflection adjustment.
 


4
 Key-Frame Animation by using the 'Edit Motion Layer' Panel
 

iClone5.5 has a powerful Motion Layer Editing features
which can be used for Key-Frame Animation.
By Motion-Tracing the Target Model over the Video pose manually,
key-Frame data is recorded on the Motion Layer in the Timeline.
4-1
 Go to Animation/Motion/Modify/Motion and open 'Edit Motion Layer'.
4-2
 Locate the Edit Motion Layer Panel close to the Target G5 Actor.
4-3
 Understand the functions of Dummy pane tools in the Edit Motion Layer and operation in the 3D view.
 
Operation
4-4
 Superpose the Target 3D Model with translucent body. Motion Tracing work becomes easier.
 
Opacity Settings for Upper Body & Lower Body: Set the opacity value to about 20%, Skin to 10%
Activate Bone Edit Mode when necessary.
Edit Motion Layer
4-5
 When Finger Adjustment is needed, use Pan/Zoom/Orbit tool to get the proper Camera view.
 
Finger Adjustment
4-6
 Use Orbit/Roll, Pan and Zoom to view the G5 Actorl pose in a different angle and close up.
 
Close-up
5
 Processing Tracing Work with 'Edit Motion Layer' Panel and Timeline
 
The Video Motion is Traced by Pose to Pose basis.
Each Body Posture is to be determined in the motion flow to avoid discontinuity.
Pay attention to the body leaning for the same front look in the Video.
Think of Body Balance and correct leanings manually in the Motion Trace work.
5-1
 Push F3 key to open Timeline, open Track List/Character and Open Motion Layer under G5 Actor.
5-2
 Bring Time Slider to the target area of the video and use Zoom in/out Tool to access to the edit point.
5-3
 Observe the Video Motion flow carefully and find when and which part of the body to be moved.
5-4
 Select the key position on the timeline where the Video Actor initiats the step for body motion.
5-5
 Decide which Keying Mode to be applied between 'Select Full Body' or 'Body Part'. (Normally Full Body)
5-6
 To keep motion continuity between frames, set the center and hip angle of the G5 Body first.
5-7
 Confirm the foot position and make sure the Foot Contact at the key point.
5-8
 Select G5 Actor and use Keyboard Shortcuts A/S/D/F/G/H/J/ for Camera Moves to posture the G5 Actor.
Shortcut
A
S
D
F
G
H
J
Camera Moves to
Left
Right
Back
Front
Up
Bottom
Face
5-9
 Select Effector Point to apply the Move(Move) and/or Rotate(Rotate) tool to move the G5 Actor body part.
5-10
 Use Mouth Right Click for T/R control on the Dummy pane before Move or Rotate is starting.
 
Tracing
5-11
 Confirm the traced posture by not only moveing the Time Slider back and force between keyframes, but observe by different Camera angles.
5-12
Tips for Making Video Motion Tracing by Key-Frame Motion
 
1
 Zoom in by the Time Slider and observe the Video motion to find the right frame for editing.
2
 Move the Time Slider back and force between keyframes and compare Head Positions between G5 and Video. Adjust G5 Body center to match.
3
 Determine the side body posture from view on the Video and imagine the body posture to make the front view on the Video.
4
 Also, confirm the G5 Actor in various camera angles for unexpected Leaning and unnatural neck and body part posing.
5
Set Body Center Positions and angles first for every keyframe and pin before other body parts move.
6
To prevent unexpected IK movement, use the proper Camera angle and watch the Bone movement.
7
 Use Dummy Pane carefully as Effector Point piece of the spine area is different from G5 Character.
8
Foot Sliding tend to happen at transitional period. Remove all the T/R control in the Effector Points.
9
 Foot Sliding happens between timeframes. Use Body center of the Dummy Pane to aligne slidings.
10
 Use Vertical Grid to see a difference between frames for a side view of the 3D Target Model.
11
Mouse is not recommended for the detailed Manipuration in the Timeline, Time bar and Time Slider.
12
 Proceed Tracing little by little, save the Project periodically before losing work data by sudden trouble.
5-13
 Experiencing Wishes and solutions for the Motion Tracing from Video Image.
 
 
Items
Experiencing Wishes
1)
 Gizmo Indication

Gizmo Indication is needed when Camera view is to be changed.

Solution:Use both hands as follows.
Keep the Gizmo active and hold down 'Alt' key by left hand, then,
>Left Click and Drag the Mouse for Camera Move, Right Click for Rotate.
>Click both Left and Right button and move for Zoom.
2)
 Edit Motion Layer Bone Edit Mode selector better be located closer to Dummy Pane.
3)
 Dummy Pane Effector Point piece of the spine area does not match to G5 Character.

Solution:
Confirm the bones deflection, use pinning tool and Gizmo for correction.
3)
 Model Posture
Manipulation Tools 		Select, Move & Rotate Tool to be designed as floating near the Gizmo or
in Edit Motion Layer.
4)
 Foot Slide Fixer Foot Sliding correction by numerical manner among multiple key frames.

Partial Solution:
Remove all T/R control and use Body Center of Effector Point for adjustment over frames.
5)
 Timeline Design Display Bar, Range mark and Time Pointer be more accessible for frequent use by Mouth. Light face Letters in the Right button menus.

Partial Solution:
Slide the edge of the Timeline Panel to forward the time bar.
6)
 Key icon size

Key icon size in the sub track be more selectable for easy Mouse operation.

Solution:
1. Use Hotkeys for Character keyframe operation. Select Display bar and,
@ 'Tab' key for frame forwarding
@ 'Shift'/'Tab' for backwarding
2. Apply Flags by using the tool 'Add Flag' on Project Track. (Tutorial)
6
 Using 'Collect Clip/Add Motion-Plus' and Motion Review.
By using Collect Clip and Add Motion-Plus in the Timeline,
newly traced motion data is saved in the Motion-Plus Panel.
6-1
 Push 'Collect Clip' for the Target Model to open.
6-2
 Drag the necessary area with Mouse left button on the Collect Clip bar.
6-3
Click the Right button and select 'Add MotionPlus to Library' on the Collect Clip Bar.
 
6-4 Click OK button on the Motion Options panel and save the .iMotion data to the iClone Motion Plus file location.
 
Motion Options

iClone Motion Plus file Folder location

/Users/Public/Documents/Reallusion/Custom/Template/
iClone5 Template/iClone Template/MotionPlus/

 

 
6-5
 Go to Animation/Motion-Plus/Content Manager/Template (or Custom) and find the saved iMotionPlus file.
6-6
 Add the second G5 Actor with Visible form and apply the Clip Data to the Model.
 
Motion data can be reviewed in a visible form.

6-7
 Clips for Motion Repeat
1)
 Apply Collect Clip for the repeat area, left Click and select 'Add Motion to Libraly'.
 
2)
 This Clip is added to the Animation/Motion/Content Manaer/Custom with the new file name.
3)
 Move the Time Slider to the frame where where the repeat motion is about to start .
4)
Apply the saved Clip to the 3D Model.
5)
 The Repeat Clip is shown on the Time Line.
   
7
Applying the iMotion clip data to the other 3D Models
 

The iMotion file is applicable to the other 3D model in the same Project, however,
Motion Plus file does not show the sub track data in the original Motion Layer.
When additional editting on this first data is needed, go back to the
original Motion Layer, re-edit there and make the second Motion Plus file
is recommended.
7-1 Load the Other 3D Model and apply the same iMotion clip data from Content Manager Library.
7-2 Go to Actor/Avatar/Modify/Avatar and check on 'Foot Contact' for the added Model.
 
AddModel
7-3 Confirm that the iMotion data for the added Model is shown under the RooteNode/Motion in the Timeline.
7-4 Body part of the added Model can be modified by using Dummy Pane, however, edit on the original Clip is recommended.
Additional Editings for Motion-Plus file may genarate unexpected pose change in other frames.
Add Model
 
8
 Composing the Motion Clips for Demo Video
 
Motion Tracing work has been processed little by little.
Key-Frame Numbers for the Dance Motion are listed in the Table.
Two Clips are conbined for the Demo Video which is 60% length of the Reference Video.
8-1
 First Project includes 2 short Clips and all Key-Frames are saved as one Motion-Plus file.
8-2
 Key-Freames in the Second Project are saved as the other Motion-Plus File.
8-3
 Load a new Model and drag two Motion-Plus Files onto the Model accordingly.
 
Right-click at the duplicated clip, use 'Break' command and delete it.
Add Clip
8-4
 Keep Note the Frame Numbers and Motion Clips for repetition.
 
Frame Numbers and Motion Clips for repetition (Personal Reference only)
Frame No.
Test A
Demo
Frame
Counts/ Clip No.
The words of the Song
MP1
Long
Clip1

MP2 Long
Clip2
117
212
Start
Music Starting
 
117-1550
212-1600
1283
Intro
Intro Dancing
Start:
211
  
1550-2520
1584-2384
1600-2415
823
Clip 1
Music 1
 あなたのことが好きなのに　私にまるで興味ない 何度目かの失恋の準備　
Yeah! Yeah! Yeah!

Even though I know I am so in love with you
’m not really sure that you feel the same way too
Ready for that moment, when my hearts broken again..
Yeah! Yeah! Yeah!
 
2380-2520
2415-2550
115
Clip 2
Motion1
Yeah! Yeah! Yeah!
 
2520-3470
2500-3320
2550-3360
830
Clip 1
Music 1'
(Repeat Rhythm Motion 1)
まわりを見れば大勢の　可愛い子たちがいるんだもん 地味な花は気づいてくれない　
Yeah! Yeah! Yeah!

Then i look around me and all that i can see
So many other girls and they’re prettier than me
Ordinary flowers never get their time in the sun.. Yeahh! (3x)

 

  

3330-3470
3360-3480
87
Clip 2
Motion1'

(Repeat Motion 1)
Yeah! Yeah! Yeah!
Start:
3365


3480-4425

3400-3870
Clip3
3870-4330
Clip3’

カフェテリアに流れるMusic　ぼんやり聴い ていたら 3338-3846/(508)/ Clip 3
(Repeat Rhythm 2)
知らぬ間にリズムに合わせ　つま先から動き出す 3846-4323/(477)/ Clip 3'
(Repeat Rhythm 2')
Cafetaria airs the sound of music, listening interrupts my daydream
Then i find myself caught inside the rhythm flow
And now my body moves starting with my toes


End:
3550


  

 

 

 

4425-4935
4330-4560
(Bending)
4560-4840

止められない今の気持ち 3846-4323

カモン カモン カモン カモン ベイビー　占ってよ 4322-4590 /4590-5260 /670 music3

There is no escaping, whatever i feeling
Come on! Come on! Come on! Come on! (4x) Baby
Come and tell my future #





 

4935-6120

4840-5340
5340-5640
(Turn)


 

 

 恋するフォーチュンクッキー!　未来は　そんな悪くないよ
Hey! Hey! Hey!

(ツキを呼ぶには　笑顔を見せること ハートのフォーチュンクッキー　運勢今日よりも良くしよう)

My love’s in a Fortune Cookie And the future won’t be as bad as what i seems Hey! Hey! Hey!

Don’t you know your luck will change When you’ve got a big smile on your face In my hearts a Fortune Cookie




 

 

  
6120-6700
5630-6560

人生捨てたもんじゃないよね!!　あっと驚く奇跡が起きる!!
あなたとどこかで愛し合える予感!!

Look at my fate, make it so much better than today
Hey! Hey! Hey! Hey! Hey! Hey!

 (---------Demo Video Stops here this time-------)

Never let life get you down and don’t let it defeat you
Miracles can happen that will totally surprise you
Something tells me someday there’ll be love and it will just be you and me

 

 


 

 

 

 

 

End:
6560
6560-7200
   (Transitional Motion)    
         

7200-8050
(Not Ready)
カモン カモン カモン カモン ベイビー　
占ってよ 恋するフォーチュンクッキー　未来はそんな悪くないよ Hey! Hey! Hey!　

Come on! (4x) Baby! come and tell my future My love’s in a Fortune Cookie And the future won’t be as bad as what it seems Hey! (3x)		    

8050-9180
(Not Ready)
ツキを呼ぶには　笑顔を見せること ハートのフォーチュンクッキー　運勢今日よりも良くしよう
Hey! Hey! Hey!　Hey! Hey! Hey!

Don’t you know your luck will change When you’ve got a big smile on your face In my hearts a Fortune Cookie Look at my fate, make it so much better than today Hey! (3x) Hey! (3x)		    

9180-10000
(Not Ready)
人生捨てたもんじゃないよね　あっと驚く奇跡が起きる あなたとどこかで愛し合える予感

Never let life get you down and don’t let it defeat you
Miracles can happen that will totally surprise you
Something thells me someday there’ll be love and it will just be you and me		    
-10780
  
   

 
AKB English_v2
 
AKB song_v1
9
 Video Making
 
The Composed AKB Dance Motion Data can be applied to Bone based characters.
Following stage and models are used which have been converted to iClone format.
Only half length of Reference Video is completed and shown in my DEMO Video.
9-1
 Prepare the Stage, Models and English Song.
 

Object
Product Name
Copy Right
Notes
Stage
Sakura Stage
yuduki
  
MMD Model
HINOMOTO ONIKO/NAMANARI
HINOMOTO ONIKO Project
MMD 2nd Pack
Special Appendix Model #006
MMD Model
FARUN

SHINICHIRO IITSUKA,
Modeling:KIO

MMD 3rd Pack
Special Appendix Model #011
MMD Model
SV01 SEEU
KKUEM
Modeling:MAMAMA
MMD 3rd Pack
Special Appendix Model #008
iClone Model
3D Universe_Foxy
3D Universe
3DX5 Character Designer's Resource pack (More Info)
9-2
 Locate each Models and apply the two Motion Clips to each Models.
 
Demo
9-3
 Add small Props on the Screen and map the English Song (AVI file) to it.
9-4

Play the Music, Locate both Motion Clips to match.

9-5
 Play all the Motion and confirm the dancing forms.
9-6
 Go to Export and set the Format, Output Size, Output Fange and start Export.
 
Format
Output Size
Output Fange
Render Quality
MP4
1280 x 720
209-6700
Final Render

910
 Work Issues for the Motion Tracing from the Video Image
 
Following issues are observed when the Motion Tracing work is processed in my PC environment.
Manageable but needed strong patience for Interruptions caused by following issues:
What's wonderful that frequent crashes at Timeline Editing work has gone out in the iClone 5.51.
 
1) Following Reminder Comes out many times, but working data is saved as Project.iProject.
Project
 Rename the Project name and Re-Open the Project file.
Handle Error
Click Yes and window opens for Saving.
Closiong
2) NVIDIA Windows Kernel Mode Driver stops response frequently, but recovers soon.
DisplayDriverStop

When NVIDEA stops, wait for a while.
Working data remains and can continue after the NVIDEA recovers.

This happens even with iClone5.51.
3) iClone 5 AP stops operation many times.
APStop

When this AP Stop happens,
working data is saved as Project.iProject sometime, however,
Project data is not saved nomally.

Project

This AP stop is reduced remarkablly with iClone5.51.

4) Windows System Crash had happen some time.

All of the Last Project data had lost.
nvlddmkm.sys need to be checked out.

Rebooted PC to start iClone again.
Save the Project data frequently.

 


3DXchanege5 Pipeline test series 14
2013.12.20 Hirospot (Personal Site)