Custom Avatar Crashed and Distorted in UE5 with Livelink

ravidfebrian · June 21, 2023, 2:08pm

Hi guys,

Do any of you have experience with Livelink on Unreal Engine 5 with custom Avatar? I followed the documentation and remap all the skeleton, in my case I use 38 points. However, when I use the custom “ZED Livelink Pose” - Node that are provided in ZED Livelink project, the UE crashed and show this errors

LoginId:f694ff8d4e51ab07182e59a2ae668ace
EpicAccountId:

Unhandled Exception: EXCEPTION_ACCESS_VIOLATION reading address 0x0000000000000618

UnrealEditor_Engine!USkinnedMeshComponent::GetBoneNames() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\Components\SkinnedMeshComponent.cpp:1867]
UnrealEditor_CPP_Livelink!ULiveLinkOrientationsRemapAsset::BuildPoseFromZEDAnimationData() [C:\Users\shams\Documents\Unreal Projects\CPP_Livelink\Source\CPP_Livelink\SkeletonLiveLinkRemapAsset.cpp:207]
UnrealEditor_CPP_Livelink!FAnimNode_ZEDLiveLinkPose::Evaluate_AnyThread() [C:\Users\shams\Documents\Unreal Projects\CPP_Livelink\Source\CPP_Livelink\AnimNode_ZEDLiveLinkPose.cpp:101]
UnrealEditor_Engine!FPoseLink::Evaluate() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\Animation\AnimNodeBase.cpp:398]
UnrealEditor_Engine!FAnimInstanceProxy::EvaluateAnimationNode_WithRoot() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\Animation\AnimInstanceProxy.cpp:1389]
UnrealEditor_Engine!FAnimInstanceProxy::EvaluateAnimation_WithRoot() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\Animation\AnimInstanceProxy.cpp:1310]
UnrealEditor_Engine!UAnimInstance::ParallelEvaluateAnimation() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\Animation\AnimInstance.cpp:787]
UnrealEditor_Engine!USkeletalMeshComponent::EvaluateAnimation() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\Components\SkeletalMeshComponent.cpp:1968]
UnrealEditor_Engine!USkeletalMeshComponent::PerformAnimationProcessing() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\Components\SkeletalMeshComponent.cpp:2060]
UnrealEditor_Engine!USkeletalMeshComponent::ParallelAnimationEvaluation() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\Components\SkeletalMeshComponent.cpp:4009]
UnrealEditor_Engine!FParallelAnimationEvaluationTask::DoTask() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\Components\SkeletalMeshComponent.cpp:151]
UnrealEditor_Engine!TGraphTask<FParallelAnimationEvaluationTask>::ExecuteTask() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Async\TaskGraphInterfaces.h:1348]
UnrealEditor_Core!<lambda_13c427d0bfcf321a066cb5a2badfbc27>::operator()() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\Async\TaskGraph.cpp:2051]
UnrealEditor_Core!LowLevelTasks::TTaskDelegate<LowLevelTasks::FTask * __cdecl(bool),48>::TTaskDelegateImpl<<lambda_17c904c32264d0348d15245fba0e1bff>,0>::CallAndMove() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Async\Fundamental\TaskDelegate.h:171]
UnrealEditor_Core!LowLevelTasks::FTask::ExecuteTask() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Async\Fundamental\Task.h:656]
UnrealEditor_Core!LowLevelTasks::FScheduler::ExecuteTask() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\Async\Fundamental\Scheduler.cpp:162]
UnrealEditor_Core!LowLevelTasks::FScheduler::TryExecuteTaskFrom<&LowLevelTasks::TLocalQueueRegistry<1024>::TLocalQueue::DequeueGlobal,0>() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\Async\Fundamental\Scheduler.cpp:361]
UnrealEditor_Core!LowLevelTasks::FScheduler::WorkerMain() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\Async\Fundamental\Scheduler.cpp:402]
UnrealEditor_Core!UE::Core::Private::Function::TFunctionRefCaller<<lambda_0a548c7e497de3cc77a9e48080e1524f>,void __cdecl(void)>::Call() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Templates\Function.h:475]
UnrealEditor_Core!FThreadImpl::Run() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\HAL\Thread.cpp:67]
UnrealEditor_Core!FRunnableThreadWin::Run() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\Windows\WindowsRunnableThread.cpp:149]

When I tried using the default “Live Link Pose” - Node provided by UE, it gave me a very distorted Avatar. However, it is already tracking my movements. See Youtube.

This is my question:

Is there anyway to fix the distortion?
Also, are the skeleton poses from Livelink need to be transformed first after the remapping? How does it work?

Thank you

This is the system that I use:

ZED SDK 4.0.3
CUDA 12.0.0
UE 5.1.1

JPlou · June 21, 2023, 2:18pm

Hi @ravidfebrian, welcome to the forums.

I understand that you remade a remap asset from this part of the doc, is that correct?
Truth is the process for adding a new avatar has really evolved since the first release of the plugin, and this whole section of doc will be removed very soon. While it should work, it’s over complicated now that the retargeting works so well in UE5.

Can you follow the instructions for the Quick and easy way, which uses UE5 retargeting, and share if it solved your issue?

Best,
Jean-Loup

ravidfebrian · June 21, 2023, 8:54pm

Hi @JPlou

Yes, it is the one that I followed.

But for the Avatar to track the motion, it still needs “ZED Livelink Pose” or “Live Link Pose” node to receive the skeleton pose from the Livelink right? These nodes however need to be remapped. That’s why I followed the Custom Avatar documentation and remapped the skeleton poses with this.

Btw, I will try this tomorrow, and give you an update.

JPlou · June 22, 2023, 8:22am

@ravidfebrian

The idea behind the retargeting is that the motion from Livelink is applied to our mapped avatar, and then the avatar you want uses the motion of the former to move.

You can take a look at UE5’s doc to get a better understanding.

Best,
Jean-Loup

ravidfebrian · June 23, 2023, 10:08am

If possible I would like to make my own Avatar without depends on ZED’s Avatar. Without using retargeting is also possible right? After seeing the remapping asset from ZED’s project, there is no difference with what I have done (See screenshot).

Each skeleton pose name from Livelink are coupled with the appropriate skeleton tree of the Metahuman. What makes it different is in the use of ZED Livelink Pose Node, which crashed the UE.

I mean, what have been done in the provided UE project is no different than this documentation.

Btw, I encountered error while building this project.

JPlou · June 28, 2023, 12:25pm

@ravidfebrian
While this method should work, we don’t recommend using it as the issues can come from a number of places. For one, the skeleton needs to be in T-Pose, for example. If you still encounter issues, I encourage you to try using retargeting.

The dependence on the ZED avatar really is just about rendering, we chose a skeleton rig that matches our fitting system well. The retargeting of UE5 does the rest. Mapping it to a metahuman involves not much difference compared to any other humanoid rig. You should just have to do a retargeter and maybe adjust the IK chains, but that’s all. The whole Live Link to avatar animation pipeline should not have to be touched.

Best,
Jean-Loup