A free, multiplatform SDK for real-time facial motion capture using blendshapes, and rigid head pose in 3D space from any RGB camera, photo, or video.

Related tags

Deep Learningmachine-learningreal-timecomputer-visionanimationaravatarsfacial-recognitionface-recognitionface-detection3dface-trackingfacial-expression-recognitionfacetrackerfacial-landmarksblend-sha
Overview


mocap4face by Facemoji
mocap4face by Facemoji

mocap4face by Facemoji is a free, multiplatform SDK for real-time facial motion capture based on Facial Action Coding System or (FACS). It provides real-time FACS-derived blendshape coefficients, and rigid head pose in 3D space from any mobile camera, webcam, photo, or video enabling live animation of 3D avatars, digital characters, and more.

After fetching the input from one of the mentioned sources, mocap4face SDK produces data in ARKit-compatible blendshapes, i.e., morph targets weight values as a per-frame expression shown in the video below. Useful for, e.g., animating a 2D or 3D avatar in a way that mimics the user's facial expressions in real-time à la Apple Memoji but without the need of a hardware-based TrueDepth Camera.

With mocap4face, you can drive live avatars, build Snapchat-like lenses, AR experiences, face filters that trigger actions, VTubing apps, and more with as little energy impact and CPU/GPU use as possible. As an example, check out how the popular avatar live-streaming app REALITY is using our SDK.

Please ⭐ Star us on GitHub—it motivates us a lot!

📋 Table of Content

🤓 Tech Specs

✨ Key Features

  • 42 tracked facial expressions via blendshapes
  • Eye tracking including eye gaze vector
  • Tongue tracking
  • Light & fast, just 3MB ML model size
  • ≤ ±50° pitch, ≤ ±40° yaw and ≤ ±30° roll tracking coverage

🤳 Input

  • All RGB camera
  • Photo
  • Video

📦 Output

  • ARKit-compatible blendshapes
  • Head position and scale in 2D and 3D
  • Head rotation in world coordinates

⚡ Performance

  • 50 FPS on Pixel 4
  • 60 FPS on iPhone SE (1st gen)
  • 90 FPS on iPhone X or newer

💿 Installation

Prerequisites

  1. Get a new developer account at studio.facemoji.co
  2. Generate a unique API key for your app
  3. Paste the API key to your source code

iOS

  1. Open the sample XCode project and run the demo on your iOS device
  2. To use the SDK in your project, either use the bundled XCFramework directly or use the Swift Package manager (this repository also serves as a Swift PM repository)

Android

  1. Open the sample project in Android Studio and run the demo on your Android device
  2. Add this repository to the list of your Maven repositories in your root build.gradle, for example:
allprojects {
    repositories {
        google()
        mavenCentral()
        // Any other repositories here...

        maven {
            name = "Facemoji"
            url = uri("https://facemoji.jfrog.io/artifactory/default-maven-local/")
        }
    }
}
  1. To use the SDK in your project, add implementation 'co.facemoji:mocap4face:0.1.0' to your Gradle dependencies

🚀 Use Cases

  • Live avatar experiences
  • Snapchat-like lense
  • AR experiences
  • VTubing apps
  • Live streaming apps
  • Face filters
  • AR games with facial triggers
  • Beauty AR
  • Virtual try-on
  • Play to earn games

❤️ Links

📄 License

This library is provided under the Facemoji SDK License Agreement—see LICENSE.

The sample code in this repository is provided under the Facemoji Samples License

Notices

OSS used in mocap4face SDK:

Comments
  • Mainactivity re-opened cannot be traced

    Mainactivity re-opened cannot be traced

    Only the first open MainActivity can be tracked. when back to SplashActivity and reopen Mainactivity the track result is null.

    The test code is as follows（modify SplashAtivity）：

    override fun onResume() {
    super.onResume()
    Timer("Splash", false).schedule(5000) {
        startMainActivity()
    }
}
    bug 
    opened by cike978 9
  • can not find any documents about how to build a app using facemoji SDK

    can not find any documents about how to build a app using facemoji SDK

    i want to build a app like the offical facemoji app, but the demo code is so simple, the SDK even has no document.

    can anyone tell me how to use the SDK to build a facemoji app ?

    opened by danlanxiaohei 3
  • Fix

    Fix "artifact not found for target 'mocap4face'" error

    Issue

    • I got "artifact not found for target 'mocap4face'" error when I tried to load mocap4face Swift Package by Xcode 13.3 mocap4face_err
      • The target name should be exactly the same as the framework name (case sensitive)

    Changes

    • Change the target name from mocap4face to Mocap4Face to match the framework name
    opened by magicien 2
  • Any plans for Windows & MacOS Support?

    Any plans for Windows & MacOS Support?

    I'm guessing if this is already cross-platform on Android, Web, and iOS that this an upcoming features. Just curious since we'd love to use this everywhere if possible

    opened by aaronn 2
  • [Question]: Is this Lib compatible with react-native?

    [Question]: Is this Lib compatible with react-native?

    It would be incredibly awesome to include this library in my React-Native start-up App but I wonder whether there's React-Native compatible SDK?

    Thanks!!!

    enhancement 
    opened by shamxeed 2
  • OpenGLResizer20: setup OES texture: glError 1282: invalid operation

    OpenGLResizer20: setup OES texture: glError 1282: invalid operation

    使用 Android 版本打开 Activity 正常，关闭页面再次启用的时候出现 OpenGLResizer20: setup OES texture: glError 1282: invalid operation

    onCameraImageArkitLink(cameraTexture: OpenGLTexture)

    cameraTexture = null

    opened by pengzhenkun 1
  • Stop face tracking

    Stop face tracking

    Hello. Thanks for the great face tracking library!

    How can I stop tracking? Something like a dispose/destroy function?

    I am using this library in web (js)

    opened by Nawarius 1
  • Status of macOS build?

    Status of macOS build?

    Has any new work been done to bring this to the Mac via catalyst?

    We primarily just want to be able to use the full camera width when running under M1 catalyst. We don't need the full feature-set, but macOS forces "non-mac" frameworks to use a cropped iPhone-like camera view even on the Mac which has a much wider view.

    opened by aaronn 0
  • Problems with one eye open and the other closed

    Problems with one eye open and the other closed

    Platform Android With one eye open and the other closed, the output is eyeBlink_L 0.7 and eyeBlink_R 0.5 Expected result is eyeBlink_L 0.7 and eyeBlink_R 0

    bug 
    opened by will4621d 2
Releases(0.3.0)
Owner
Facemoji
Avatars for apps and games.
Facemoji
GitHub Repository
Official Implementation of 'UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers' ICLR 2021(spotlight)

UPDeT Official Implementation of UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers (ICLR 2021 spotlight) The

hhhusiyi 96 Dec 22, 2022
PyTorch implementation of Federated Learning with Non-IID Data, and federated learning algorithms, including FedAvg, FedProx.

Federated Learning with Non-IID Data This is an implementation of the following paper: Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, Vik

Youngjoon Lee 48 Dec 29, 2022
Code for IntraQ, PyTorch implementation of our paper under review

IntraQ: Learning Synthetic Images with Intra-Class Heterogeneity for Zero-Shot Network Quantization paper Requirements Python = 3.7.10 Pytorch == 1.7

1 Nov 19, 2021
GPU-accelerated PyTorch implementation of Zero-shot User Intent Detection via Capsule Neural Networks

GPU-accelerated PyTorch implementation of Zero-shot User Intent Detection via Capsule Neural Networks This repository implements a capsule model Inten

Joel Huang 15 Dec 24, 2022
Practical and Real-world applications of ML based on the homework of Hung-yi Lee Machine Learning Course 2021

Machine Learning Theory and Application Overview This repository is inspired by the Hung-yi Lee Machine Learning Course 2021. In that course, professo

SilenceJiang 35 Nov 22, 2022
Create time-series datacubes for supervised machine learning with ICEYE SAR images.

ICEcube is a Python library intended to help organize SAR images and annotations for supervised machine learning applications. The library generates m

ICEYE Ltd 65 Jan 03, 2023
Consecutive-Subsequence - Simple software to calculate susequence with highest sum

Simple software to calculate susequence with highest sum This repository contain

Gbadamosi Farouk 1 Jan 31, 2022
Code for SyncTwin: Treatment Effect Estimation with Longitudinal Outcomes (NeurIPS 2021)

SyncTwin: Treatment Effect Estimation with Longitudinal Outcomes (NeurIPS 2021) SyncTwin is a treatment effect estimation method tailored for observat

Zhaozhi Qian 3 Nov 03, 2022
Retinal Vessel Segmentation with Pixel-wise Adaptive Filters (ISBI 2022)

Official code of Retinal Vessel Segmentation with Pixel-wise Adaptive Filters and Consistency Training (ISBI 2022)

anonymous 14 Oct 27, 2022
A large-scale video dataset for the training and evaluation of 3D human pose estimation models

ASPset-510 (Australian Sports Pose Dataset) is a large-scale video dataset for the training and evaluation of 3D human pose estimation models. It contains 17 different amateur subjects performing 30

Aiden Nibali 25 Jun 20, 2021
Official implementation of "Membership Inference Attacks Against Self-supervised Speech Models"

Introduction Official implementation of "Membership Inference Attacks Against Self-supervised Speech Models". In this work, we demonstrate that existi

Wei-Cheng Tseng 7 Nov 01, 2022
Official Repo for ICCV2021 Paper: Learning to Regress Bodies from Images using Differentiable Semantic Rendering

[ICCV2021] Learning to Regress Bodies from Images using Differentiable Semantic Rendering Getting Started DSR has been implemented and tested on Ubunt

Sai Kumar Dwivedi 83 Nov 27, 2022
Learning from History: Modeling Temporal Knowledge Graphs with Sequential Copy-Generation Networks

CyGNet This repository reproduces the AAAI'21 paper “Learning from History: Modeling Temporal Knowledge Graphs with Sequential Copy-Generation Network

CunchaoZ 89 Jan 03, 2023
Code to run experiments in SLOE: A Faster Method for Statistical Inference in High-Dimensional Logistic Regression.

Code to run experiments in SLOE: A Faster Method for Statistical Inference in High-Dimensional Logistic Regression. Not an official Google product. Me

Google Research 27 Dec 12, 2022
A PyTorch implementation of deep-learning-based registration

DiffuseMorph Implementation A PyTorch implementation of deep-learning-based registration. Requirements OS : Ubuntu / Windows Python 3.6 PyTorch 1.4.0

24 Jan 03, 2023
Code for the ICCV'21 paper "Context-aware Scene Graph Generation with Seq2Seq Transformers"

ICCV'21 Context-aware Scene Graph Generation with Seq2Seq Transformers Authors: Yichao Lu*, Himanshu Rai*, Cheng Chang*, Boris Knyazev†, Guangwei Yu,

Layer6 Labs 37 Dec 18, 2022
Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Pytorch Lightning 1.4k Jan 01, 2023
Code for Towards Streaming Perception (ECCV 2020) :car:

sAP — Code for Towards Streaming Perception ECCV Best Paper Honorable Mention Award Feb 2021: Announcing the Streaming Perception Challenge (CVPR 2021

Martin Li 85 Dec 22, 2022
A human-readable PyTorch implementation of "Self-attention Does Not Need O(n^2) Memory"

memory_efficient_attention.pytorch A human-readable PyTorch implementation of "Self-attention Does Not Need O(n^2) Memory" (Rabe&Staats'21). def effic

Ryuichiro Hataya 7 Dec 26, 2022
Reduce end to end training time from days to hours (or hours to minutes), and energy requirements/costs by an order of magnitude using coresets and data selection.

COResets and Data Subset selection Reduce end to end training time from days to hours (or hours to minutes), and energy requirements/costs by an order

decile-team 244 Jan 09, 2023