A free, multiplatform SDK for real-time facial motion capture using blendshapes, and rigid head pose in 3D space from any RGB camera, photo, or video.

Related tags

Deep Learningmachine-learningreal-timecomputer-visionanimationaravatarsfacial-recognitionface-recognitionface-detection3dface-trackingfacial-expression-recognitionfacetrackerfacial-landmarksblend-sha
Overview


mocap4face by Facemoji
mocap4face by Facemoji

mocap4face by Facemoji is a free, multiplatform SDK for real-time facial motion capture based on Facial Action Coding System or (FACS). It provides real-time FACS-derived blendshape coefficients, and rigid head pose in 3D space from any mobile camera, webcam, photo, or video enabling live animation of 3D avatars, digital characters, and more.

After fetching the input from one of the mentioned sources, mocap4face SDK produces data in ARKit-compatible blendshapes, i.e., morph targets weight values as a per-frame expression shown in the video below. Useful for, e.g., animating a 2D or 3D avatar in a way that mimics the user's facial expressions in real-time à la Apple Memoji but without the need of a hardware-based TrueDepth Camera.

With mocap4face, you can drive live avatars, build Snapchat-like lenses, AR experiences, face filters that trigger actions, VTubing apps, and more with as little energy impact and CPU/GPU use as possible. As an example, check out how the popular avatar live-streaming app REALITY is using our SDK.

Please ⭐ Star us on GitHub—it motivates us a lot!

📋 Table of Content

🤓 Tech Specs

✨ Key Features

  • 42 tracked facial expressions via blendshapes
  • Eye tracking including eye gaze vector
  • Tongue tracking
  • Light & fast, just 3MB ML model size
  • ≤ ±50° pitch, ≤ ±40° yaw and ≤ ±30° roll tracking coverage

🤳 Input

  • All RGB camera
  • Photo
  • Video

📦 Output

  • ARKit-compatible blendshapes
  • Head position and scale in 2D and 3D
  • Head rotation in world coordinates

⚡ Performance

  • 50 FPS on Pixel 4
  • 60 FPS on iPhone SE (1st gen)
  • 90 FPS on iPhone X or newer

💿 Installation

Prerequisites

  1. Get a new developer account at studio.facemoji.co
  2. Generate a unique API key for your app
  3. Paste the API key to your source code

iOS

  1. Open the sample XCode project and run the demo on your iOS device
  2. To use the SDK in your project, either use the bundled XCFramework directly or use the Swift Package manager (this repository also serves as a Swift PM repository)

Android

  1. Open the sample project in Android Studio and run the demo on your Android device
  2. Add this repository to the list of your Maven repositories in your root build.gradle, for example:
allprojects {
    repositories {
        google()
        mavenCentral()
        // Any other repositories here...

        maven {
            name = "Facemoji"
            url = uri("https://facemoji.jfrog.io/artifactory/default-maven-local/")
        }
    }
}
  1. To use the SDK in your project, add implementation 'co.facemoji:mocap4face:0.1.0' to your Gradle dependencies

🚀 Use Cases

  • Live avatar experiences
  • Snapchat-like lense
  • AR experiences
  • VTubing apps
  • Live streaming apps
  • Face filters
  • AR games with facial triggers
  • Beauty AR
  • Virtual try-on
  • Play to earn games

❤️ Links

📄 License

This library is provided under the Facemoji SDK License Agreement—see LICENSE.

The sample code in this repository is provided under the Facemoji Samples License

Notices

OSS used in mocap4face SDK:

Comments
  • Mainactivity re-opened cannot be traced

    Mainactivity re-opened cannot be traced

    Only the first open MainActivity can be tracked. when back to SplashActivity and reopen Mainactivity the track result is null.

    The test code is as follows（modify SplashAtivity）：

    override fun onResume() {
    super.onResume()
    Timer("Splash", false).schedule(5000) {
        startMainActivity()
    }
}
    bug 
    opened by cike978 9
  • can not find any documents about how to build a app using facemoji SDK

    can not find any documents about how to build a app using facemoji SDK

    i want to build a app like the offical facemoji app, but the demo code is so simple, the SDK even has no document.

    can anyone tell me how to use the SDK to build a facemoji app ?

    opened by danlanxiaohei 3
  • Fix

    Fix "artifact not found for target 'mocap4face'" error

    Issue

    • I got "artifact not found for target 'mocap4face'" error when I tried to load mocap4face Swift Package by Xcode 13.3 mocap4face_err
      • The target name should be exactly the same as the framework name (case sensitive)

    Changes

    • Change the target name from mocap4face to Mocap4Face to match the framework name
    opened by magicien 2
  • Any plans for Windows & MacOS Support?

    Any plans for Windows & MacOS Support?

    I'm guessing if this is already cross-platform on Android, Web, and iOS that this an upcoming features. Just curious since we'd love to use this everywhere if possible

    opened by aaronn 2
  • [Question]: Is this Lib compatible with react-native?

    [Question]: Is this Lib compatible with react-native?

    It would be incredibly awesome to include this library in my React-Native start-up App but I wonder whether there's React-Native compatible SDK?

    Thanks!!!

    enhancement 
    opened by shamxeed 2
  • OpenGLResizer20: setup OES texture: glError 1282: invalid operation

    OpenGLResizer20: setup OES texture: glError 1282: invalid operation

    使用 Android 版本打开 Activity 正常，关闭页面再次启用的时候出现 OpenGLResizer20: setup OES texture: glError 1282: invalid operation

    onCameraImageArkitLink(cameraTexture: OpenGLTexture)

    cameraTexture = null

    opened by pengzhenkun 1
  • Stop face tracking

    Stop face tracking

    Hello. Thanks for the great face tracking library!

    How can I stop tracking? Something like a dispose/destroy function?

    I am using this library in web (js)

    opened by Nawarius 1
  • Status of macOS build?

    Status of macOS build?

    Has any new work been done to bring this to the Mac via catalyst?

    We primarily just want to be able to use the full camera width when running under M1 catalyst. We don't need the full feature-set, but macOS forces "non-mac" frameworks to use a cropped iPhone-like camera view even on the Mac which has a much wider view.

    opened by aaronn 0
  • Problems with one eye open and the other closed

    Problems with one eye open and the other closed

    Platform Android With one eye open and the other closed, the output is eyeBlink_L 0.7 and eyeBlink_R 0.5 Expected result is eyeBlink_L 0.7 and eyeBlink_R 0

    bug 
    opened by will4621d 2
Releases(0.3.0)
Owner
Facemoji
Avatars for apps and games.
Facemoji
GitHub Repository
This repository contains the data and code for the paper "Diverse Text Generation via Variational Encoder-Decoder Models with Gaussian Process Priors" ([email protected])

GP-VAE This repository provides datasets and code for preprocessing, training and testing models for the paper: Diverse Text Generation via Variationa

Wanyu Du 18 Dec 29, 2022
PyTorch implementation for ACL 2021 paper "Maria: A Visual Experience Powered Conversational Agent".

Maria: A Visual Experience Powered Conversational Agent This repository is the Pytorch implementation of our paper "Maria: A Visual Experience Powered

Jokie 22 Dec 12, 2022
Pytorch implementation of four neural network based domain adaptation techniques: DeepCORAL, DDC, CDAN and CDAN+E. Evaluated on benchmark dataset Office31.

Deep-Unsupervised-Domain-Adaptation Pytorch implementation of four neural network based domain adaptation techniques: DeepCORAL, DDC, CDAN and CDAN+E.

Alan Grijalva 49 Dec 20, 2022
Anti-UAV base on PaddleDetection

Paddle-Anti-UAV Anti-UAV base on PaddleDetection Background UAVs are very popular and we can see them in many public spaces, such as parks and playgro

Qingzhong Wang 2 Apr 20, 2022
Code for "Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations"

Infinitely Deep Bayesian Neural Networks with SDEs This library contains JAX and Pytorch implementations of neural ODEs and Bayesian layers for stocha

Winnie Xu 95 Nov 26, 2021
Improving Non-autoregressive Generation with Mixup Training

MIST Training MIST TRAIN_FILE=/your/path/to/train.json VALID_FILE=/your/path/to/valid.json OUTPUT_DIR=/your/path/to/save_checkpoints CACHE_DIR=/your/p

7 Nov 22, 2022
A PyTorch implementation of Mugs proposed by our paper "Mugs: A Multi-Granular Self-Supervised Learning Framework".

Mugs: A Multi-Granular Self-Supervised Learning Framework This is a PyTorch implementation of Mugs proposed by our paper "Mugs: A Multi-Granular Self-

Sea AI Lab 62 Nov 08, 2022
Code for CVPR2021 paper 'Where and What? Examining Interpretable Disentangled Representations'.

PS-SC GAN This repository contains the main code for training a PS-SC GAN (a GAN implemented with the Perceptual Simplicity and Spatial Constriction c

Xinqi/Steven Zhu 40 Dec 16, 2022
Official implementation for "Symbolic Learning to Optimize: Towards Interpretability and Scalability"

Symbolic Learning to Optimize This is the official implementation for ICLR-2022 paper "Symbolic Learning to Optimize: Towards Interpretability and Sca

VITA 8 Dec 19, 2022
Dynamic Realtime Animation Control

Our project is targeted at making an application that dynamically detects the user’s expressions and gestures and projects it onto an animation software which then renders a 2D/3D animation realtime

Harsh Avinash 10 Aug 01, 2022
This Repostory contains the pretrained DTLN-aec model for real-time acoustic echo cancellation.

This Repostory contains the pretrained DTLN-aec model for real-time acoustic echo cancellation.

Nils L. Westhausen 182 Jan 07, 2023
This demo showcase the use of onnxruntime-rs with a GPU on CUDA 11 to run Bert in a data pipeline with Rust.

Demo BERT ONNX pipeline written in rust This demo showcase the use of onnxruntime-rs with a GPU on CUDA 11 to run Bert in a data pipeline with Rust. R

Xavier Tao 14 Dec 17, 2022
WHENet - ONNX, OpenVINO, TFLite, TensorRT, EdgeTPU, CoreML, TFJS, YOLOv4/YOLOv4-tiny-3L

HeadPoseEstimation-WHENet-yolov4-onnx-openvino ONNX, OpenVINO, TFLite, TensorRT, EdgeTPU, CoreML, TFJS, YOLOv4/YOLOv4-tiny-3L 1. Usage $ git clone htt

Katsuya Hyodo 49 Sep 21, 2022
Tiny Kinetics-400 for test

Kinetics-400迷你数据集 English | 简体中文 该数据集旨在解决的问题：参照Kinetics-400数据格式，训练基于自己数据的视频理解模型。 数据集介绍 Kinetics-400是视频领域benchmark常用数据集，详细介绍可以参考其官方网站Kinetics。整个数据集包含40

38 Jan 06, 2023
Leveraging Social Influence based on Users Activity Centers for Point-of-Interest Recommendation

SUCP Leveraging Social Influence based on Users Activity Centers for Point-of-Interest Recommendation () Direct Friends (i.e., users who follow each o

Kosar 8 Nov 26, 2022
Code for the CIKM 2019 paper "DSANet: Dual Self-Attention Network for Multivariate Time Series Forecasting".

Dual Self-Attention Network for Multivariate Time Series Forecasting 20.10.26 Update: Due to the difficulty of installation and code maintenance cause

Kyon Huang 223 Dec 16, 2022
The code repository for "RCNet: Reverse Feature Pyramid and Cross-scale Shift Network for Object Detection" (ACM MM'21)

RCNet: Reverse Feature Pyramid and Cross-scale Shift Network for Object Detection (ACM MM'21) By Zhuofan Zong, Qianggang Cao, Biao Leng Introduction F

TempleX 9 Jul 30, 2022
The official implementation of A Unified Game-Theoretic Interpretation of Adversarial Robustness.

This repository is the official implementation of A Unified Game-Theoretic Interpretation of Adversarial Robustness. Requirements pip install -r requi

Jie Ren 17 Dec 12, 2022
Galileo library for large scale graph training by JD

近年来，图计算在搜索、推荐和风控等场景中获得显著的效果，但也面临超大规模异构图训练，与现有的深度学习框架Tensorflow和PyTorch结合等难题。 Galileo（伽利略）是一个图深度学习框架，具备超大规模、易使用、易扩展、高性能、双后端等优点，旨在解决超大规模图算法在工业级场景的落地难题，提

JD Galileo Team 128 Nov 29, 2022
💡 Learnergy is a Python library for energy-based machine learning models.

Learnergy: Energy-based Machine Learners Welcome to Learnergy. Did you ever reach a bottleneck in your computational experiments? Are you tired of imp

Gustavo Rosa 57 Nov 17, 2022