Nutrify - take a photo of food and learn about it

Overview

Nutrify - take a photo of food and learn about it

Work in progress.

To make this a thing, we're going to need lots of food images...

Start uploading your food images here: https://share.streamlit.io/mrdbourke/nutrify/main/food_image_collector.py

Streaming progress on Twitch/making videos about it on YouTube.

End goal: take a photo of food an learn about it (nutrition information, where it's from, recipes, etc).

Something like this (a data flywheel for food images):

Status: making a small application to collect large amounts of food images.

Comments
  • Capturing Image on mobile & page didn't refresh after upload

    Capturing Image on mobile & page didn't refresh after upload

    Hello Daniel, Shabir here first of all amazing work and I have been following this project from the start . The problem I faced was when I opened the link on my android device , and clicked on browse files I couldn't upload an image by capture , it took me to the screen to select photo from my mobile storage and also I noticed the page didn't refresh after I uploaded. I could see my image and other things as it is with the upload success message . Thanks again!

    opened by shabir15 7
  • Add a button for asking

    Add a button for asking "is this correct?"

    Every time there's a prediction, there should be an easy to use button asking "is this correct?".

    The result of tapping this button will update a database prediction to say whether or not the prediction was right or wrong.

    This will help evaluate the model(s) in the wild.

    opened by mrdbourke 4
  • `pre-commit` breaking (not letting commits happen)

    `pre-commit` breaking (not letting commits happen)

    Tried installing pre-commit (https://pre-commit.com), it looks like it works for some files but breaks others when I try to commit them.

    Because the tests don't pass they never commit.

    For example:

    flake8...................................................................Failed
    - hook id: flake8
    - exit code: 1
    
    food_image_collector.py:15:80: E501 line too long (97 > 79 characters)
    food_image_collector.py:49:80: E501 line too long (80 > 79 characters)
    food_image_collector.py:54:80: E501 line too long (82 > 79 characters)
    food_image_collector.py:59:80: E501 line too long (123 > 79 characters)
    food_image_collector.py:66:80: E501 line too long (80 > 79 characters)
    food_image_collector.py:67:80: E501 line too long (81 > 79 characters)
    food_image_collector.py:123:80: E501 line too long (114 > 79 characters)
    food_image_collector.py:129:80: E501 line too long (106 > 79 characters)
    food_image_collector.py:134:80: E501 line too long (84 > 79 characters)
    food_image_collector.py:136:80: E501 line too long (102 > 79 characters)
    food_image_collector.py:137:80: E501 line too long (84 > 79 characters)
    food_image_collector.py:144:80: E501 line too long (91 > 79 characters)
    
    mypy.....................................................................Passed
    

    See the initial setup here: https://github.com/mrdbourke/nutrify/issues/1#issuecomment-916722957

    opened by mrdbourke 4
  • Can't upload image

    Can't upload image

    Uncaught SyntaxError: "0"-prefixed octal literals are deprecated; use the "0o" prefix instead

    This may be a known issue. This is console output when accessing the webpage.

    I think this results in the upload not working (unless I'm mistaken).

    The problem is in constants.js with egg tart having an id starting with 0.

    Hope this helps. Keep up the work.

    opened by shivan-s 3
  • Github actions

    Github actions

    I heard something about CI/CD.

    Github action is good for this.

    I have no idea how it works.

    So I'm keen to tag along with this.

    Edit: The idea is you commit your code and this will automatically update the production code. No downtime, all in one automated step.

    opened by shivan-s 3
  • Add function to set random seeds and minor styling changes

    Add function to set random seeds and minor styling changes

    Noticed the TODO in evaluate.py script and thought to make your work easier. Made a function to set random seeds for random and torch module. Also made some styling changes in the script to go hand-in-hand with conventions. For example,

    # import standard python libraries
    # import from 3rd party libraries (numpy, pandas, torch etc.)
    # import from source code (code that you have written, like `utils` etc.)
    

    Hope this branch gets merged and helps in the making of this project!

    opened by ishandandekar 2
  • Build a food/not food model into the app

    Build a food/not food model into the app

    There should be a food/not food model running with the app.

    This is so the app can detect whether food is present or not and then proceed with the classification of what food is present.

    opened by mrdbourke 2
  • A domain.

    A domain.

    Was looking up some domains. https://www.nutrify.com/ - this is taken and it's a registered name. So does this mean the whole projects needs a new name? I hope not...

    opened by shivan-s 2
  • pytest, linting, code formatting, pipenv

    pytest, linting, code formatting, pipenv

    Related PR: #4

    I understand that we want simplicity, but it is also a good idea to slowly include coding best practices...

    • incorporate pytest for automated testing
    • also have pre-commit hooks for linting, (mypy, flake8) and code formating black.
    • switch to pipenv for package management instead of a requirements.txt, we can manage production packages (e.g. streamlit) and the above dev pages separately.

    Some resources:

    • https://docs.pytest.org/en/6.2.x/
    • https://shivansivakumaran.com/coding/my-tubestats-app/#Hurdle_6_how_to_test_code
    • https://ljvmiranda921.github.io/notebook/2018/06/21/precommits-using-black-and-flake8/
    • https://pipenv.pypa.io/en/latest/
    opened by shivan-s 2
  • Add model training and evaluation scripts

    Add model training and evaluation scripts

    Starting to build out functionality for Nutrify's data flywheel.

    This PR adds train.py and evaluate.py as major scripts for training a FoodVision model as well as evaluating it.

    There's also a new directory called foodvision, this is where all of Nutrify's ML related files will go (for now) when it comes to training a computer vision model for identifying foods.

    Next will be to create a script to import the "100 most wrong predictions" into Label Studio to update and improve the labels before retraining another model.

    food-vision-data-flywheel-concept@2x

    opened by mrdbourke 1
  • 'Added-Styling'

    'Added-Styling'

    Tried to make the website a little more better !. Added CSS and used Bootstrap to make the website looks more better.

    Screen Shots of the new looking Nutrify Website.

    image

    image

    opened by Md-Aquib 1
  • Create a script to import the

    Create a script to import the "100 most wrong predictions" and then export them as labelling task in Label Studio

    I've currently got a workflow for:

    evaluate.py stores the "X most wrong" predictions in a CSV/Weights & Biases Table/Artifact, so next will be to pull that information into a script such as fix_labels.py which:

    • inputs: a CSV file of "X most wrong" predictions (their labels, their images etc)
    • outputs: a Label Studio labelling task to fix/update the labels

    There could be a few options in the Label Studio interface to make the dataset better:

    1. confusing/clear - a label to state whether the image is confusing (e.g. multiple foods, lots going on, poor image) or clear (e.g. a single food with a good picture)
    2. whole_food/dish - a label to state whether the image has a single food or multiple foods in it (can use this later to differentiate between dishes and whole foods)
    3. prediction/updated class - a label which represents the updated information about the image

    food-vision-data-flywheel-concept@2x

    machine learning labelling 
    opened by mrdbourke 1
  • Create a function/script to create different models for different purposes (e.g. big, medium, small)

    Create a function/script to create different models for different purposes (e.g. big, medium, small)

    I'd like a script to create 3 different models (or more):

    1. Big - can be used for labelling/label improvements, this model will be the biggest and best but will be almost impossible to deploy properly, see here: https://huggingface.co/BAAI/EVA (this model requires a GPU with at least 24GB for batches of 16)
    2. Medium - can be used for experimenting, reviewing labels, etc
    3. Small - smallest and fastest model used for deploying (this model won't be as accurate as the others but will definitely be the fastest/capable of running on smaller devices/in the browser), see here: https://pytorch.org/vision/stable/models/generated/torchvision.models.quantization.shufflenet_v2_x2_0.html#shufflenet-v2-x2-0

    Ideas

    I could have a Python script called create_models.py which contains 3x functions (one for each model).

    Then in the train.py script, I can call the appropriate model for the job.

    Could use the "strategy" pattern in Python, see this video: https://youtu.be/WQ8bNdxREHU

    machine learning 
    opened by mrdbourke 0
  • Add a way to save to storage the images uploaded to Nutrify

    Add a way to save to storage the images uploaded to Nutrify

    Every image uploaded to Nutrify should be saved to storage (as long as the image is detected as being food).

    The image should be uploaded -> passed through food/not_food model -> saved to storage if the image if of food, then track:

    • Image UUID - use this to track information regarding different images across the database
    • Timestamp
    • Pred for image class
    opened by mrdbourke 3
  • Make the loading images function faster (this should appear on page load)

    Make the loading images function faster (this should appear on page load)

    Going to have to look into page loading times for this one.

    I'd like the upload image function to be available as soon as the page loads.

    Right now it's about a second or two behind the page itself.

    Not a huge deal but experience wise, it could be better.

    opened by mrdbourke 2
  • Database update: make it so a new food gets added via FDC ID -> information pulled into database

    Database update: make it so a new food gets added via FDC ID -> information pulled into database

    I'd like to be able to add an FDC ID to some section or a single file and that file gets watched.

    Every time a new FDC ID gets added to that file, information from the Food Data Central Database gets pulled in to be used Nutrify style.

    For example:

    • New FDC ID added to a list
    • List gets watched and update found -> triggers function
    • Function goes to FDC database and pulls information based on the ID
    • Nutrition information from that food ID is placed in the Nutrify database to be displayed on the web
    opened by mrdbourke 2
  • Model/class names not lined up + some classes are missing FDC data (

    Model/class names not lined up + some classes are missing FDC data ("Egg Tart", "Fries", "Hamimelon")

    Some classes are missing FDC data and will have to be fixed later on.

    Need a way to:

    • Know what classes a model has been trained on
    • Sync up model classes with food data (the data from the FDC)
    • Only publish models that have accompanying food data with them

    This will solve the problem of someone taking a photo of something an data not being displayed.

    Or...

    1. Create a model with X amount of classes
    2. Make dummy FDC data for the classes that don't have it yet
    3. Display information for which classes have data and which classes don't
    opened by mrdbourke 3
Releases(v0.1.3)
  • v0.1.3(Mar 14, 2022)

    Nutrify now displays macronutrient information for ~100 foods

    When you upload a photo to Nutrify, it now predicts what class of food is in the image and then displays information about that foods macronutrients.

    This happens via:

    image upload -> computer vision model -> returns class -> class index used to index on a database (hosted on Supabase) -> information from Supabase displayed on homepage

    This took a while to happen but I had to learn all about databases (Supabase uses pure PostgreSQL).

    The good thing is the app is still fast.

    A prediction + query to the database is done in ~0.2-0.5 seconds.

    That'll be the peak time I'd like to let it happen in, any longer and the experience won't be as fun.

    See the database code in the various JavaScript scripts (I'm new to JS so these are rough):

    • Constants for different food names/food data central IDs (fdc_id) live in constants.js
    • The database query and HTML update for macronutrients happen via get_data.js
    • The image upload and image classification happens via script.js
    • The connection to Supabase happens via supabaseClient.js - this uses an anon API key, that's apparently "safe to use in public", though I'm not sure what this means
    • Utility functions (all one of them) are located in utils.js

    Is this how JS projects get laid out?

    I have no idea.

    It seems to work though.

    Issues

    No nutrition information for some foods

    You might find that ~3/100 foods (like "egg tart") don't show nutrition information.

    This is because they don't exist (at least what I've found) in the Food-Central USDA Database (where the rest of the nutrition information comes from).

    Data loading is likely not optimal

    I'm a newbie to databases.

    The speed is how it is now because all of the database rows are loaded when you go to nutrify.app and then they are cached and queried when a prediction is made.

    This may not be best practice... but it works.

    Ideally, the database would be queried per image classification.

    However, I tried this and it takes 1-2 seconds per query (from where I live in Australia because the Supabase database is hosted in the US), this kind of time delay is not acceptable.

    Again, queries should happen within a maximum of 0.5 seconds.

    The model still does not predict whether there is a food present or not

    Future updates will include a food/not food model to see if there's food present in the image or not.

    Next

    • Add food/not food model + a message about "no food detected, please upload a photo of food"
    • Make app design nicer
    • Add a way to toggle different nutrients to view
    • Add a way to adjust the different amount of macronutrients per food (e.g. 100g -> 50g -> 250g)
    • Add logging? Logging for different predictions/images etc
    • Clean up the GitHub repo... seriously, what's going on here now? Perhaps it's time for an organization?
    Source code(tar.gz)
    Source code(zip)
  • v0.1.2(Jan 14, 2022)

    I've just created a new model based on EfficientNetB1-Lite (the TensorFlow JS version) that now powers Nutrify and published it to the Nutrify homepage.

    In short, Nutrify can now identify 100 total foods (previous was 78).

    The performance for each class varies quite a bit. But this will be upgraded in future models.

    Next

    Now we're up to 100 foods, the next step will be making the current model better (better data) and adding metadata to the website rather than just identifying the class of a food.

    In essence, the next few updates will focus on making the current system better rather than adding more foods.

    So the current list of foods will likely remain fixed for the next month or so.

    App updates (to come)

    • Add metadata (such as calories, protein, fat etc) for the first 100 foods (see list below) and make it display when a particular food is identified.
    • Add a model cascade for identifying food/not food (Nutrify should recognize a photo of food or not).
    • Display the top X (maybe 3?) predictions that can be clicked on and used.

    Modelling updates (to come)

    • Make training reproducible.
    • Make data traceable, e.g. what data is used where?
    • Have training and evaluation take place with results (e.g. acc, precision, recall per class)

    Updates

    The previous Nutrify model could identify the following 78 foods.

    Note: If there's more than one food in an image, Nutrify still only identifies one class, this will change with future models.

    # Previous 78 Foods Nutrify could classify
    ['Apple',
      'Artichoke',
      'BBQ sauce',
      'Bacon',
      'Bagel',
      'Banana',
      'Beef',
      'Beer',
      'Blueberries',
      'Bread',
      'Broccoli',
      'Butter',
      'Cabbage',
      'Candy',
      'Cantaloupe',
      'Carrot',
      'Cheese',
      'Chicken',
      'Chicken wings',
      'Cocktail',
      'Coconut',
      'Coffee',
      'Cookie',
      'Corn chips',
      'Cream',
      'Cucumber',
      'Doughnut',
      'Egg',
      'Fish',
      'Fries',
      'Grape',
      'Guacamole',
      'Hamburger',
      'Honey',
      'Ice cream',
      'Lemon',
      'Lime',
      'Lobster',
      'Mango',
      'Milk',
      'Muffin',
      'Mushroom',
      'Olive oil',
      'Olives',
      'Onion',
      'Orange',
      'Orange juice',
      'Pancake',
      'Pasta',
      'Pastry',
      'Pear',
      'Pepper',
      'Pineapple',
      'Pizza',
      'Pomegranate',
      'Popcorn',
      'Potato',
      'Prawns',
      'Pretzel',
      'Pumpkin',
      'Radish',
      'Rice',
      'Salad',
      'Salt',
      'Sandwich',
      'Sausages',
      'Soft drink',
      'Spinach',
      'Squid',
      'Strawberries',
      'Sushi',
      'Tea',
      'Tomato',
      'Tomato sauce',
      'Waffle',
      'Watermelon',
      'Wine',
      'Zucchini']
    

    Note: Squid and Chicken classes were removed from the above list (the Squid data was poor and Chicken as a class images are of actual chickens rather than chicken as a food).

    And now with the addition of 24 more foods:

    # 24 added foods to Nutrify
    ['Avocado',
      'Cherry',
      'Dumpling',
      'Egg tart',
      'Eggplant',
      'French fries',
      'Garlic',
      'Grapefruit',
      'Green beans',
      'Green onion',
      'Hamimelon',
      'Kiwi fruit',
      'Lettuce',
      'Meat ball',
      'Noodles',
      'Nuts',
      'Okra',
      'Papaya',
      'Peach',
      'Pie',
      'Plum',
      'Red cabbage',
      'Spring rolls',
      'Steak']
    

    This brings the total to 100 identifiable foods (1000 is the goal for the end of the 2022).

    # All current foods Nutrify can identify
    {0: 'Apple',
     1: 'Artichoke',
     2: 'Avocado',
     3: 'BBQ sauce',
     4: 'Bacon',
     5: 'Bagel',
     6: 'Banana',
     7: 'Beef',
     8: 'Beer',
     9: 'Blueberries',
     10: 'Bread',
     11: 'Broccoli',
     12: 'Butter',
     13: 'Cabbage',
     14: 'Candy',
     15: 'Cantaloupe',
     16: 'Carrot',
     17: 'Cheese',
     18: 'Cherry',
     19: 'Chicken wings',
     20: 'Cocktail',
     21: 'Coconut',
     22: 'Coffee',
     23: 'Cookie',
     24: 'Corn chips',
     25: 'Cream',
     26: 'Cucumber',
     27: 'Doughnut',
     28: 'Dumpling',
     29: 'Egg',
     30: 'Egg tart',
     31: 'Eggplant',
     32: 'Fish',
     33: 'French fries',
     34: 'Fries',
     35: 'Garlic',
     36: 'Grape',
     37: 'Grapefruit',
     38: 'Green beans',
     39: 'Green onion',
     40: 'Guacamole',
     41: 'Hamburger',
     42: 'Hamimelon',
     43: 'Honey',
     44: 'Ice cream',
     45: 'Kiwi fruit',
     46: 'Lemon',
     47: 'Lettuce',
     48: 'Lime',
     49: 'Lobster',
     50: 'Mango',
     51: 'Meat ball',
     52: 'Milk',
     53: 'Muffin',
     54: 'Mushroom',
     55: 'Noodles',
     56: 'Nuts',
     57: 'Okra',
     58: 'Olive oil',
     59: 'Olives',
     60: 'Onion',
     61: 'Orange',
     62: 'Orange juice',
     63: 'Pancake',
     64: 'Papaya',
     65: 'Pasta',
     66: 'Pastry',
     67: 'Peach',
     68: 'Pear',
     69: 'Pepper',
     70: 'Pie',
     71: 'Pineapple',
     72: 'Pizza',
     73: 'Plum',
     74: 'Pomegranate',
     75: 'Popcorn',
     76: 'Potato',
     77: 'Prawns',
     78: 'Pretzel',
     79: 'Pumpkin',
     80: 'Radish',
     81: 'Red cabbage',
     82: 'Rice',
     83: 'Salad',
     84: 'Salt',
     85: 'Sandwich',
     86: 'Sausages',
     87: 'Soft drink',
     88: 'Spinach',
     89: 'Spring rolls',
     90: 'Steak',
     91: 'Strawberries',
     92: 'Sushi',
     93: 'Tea',
     94: 'Tomato',
     95: 'Tomato sauce',
     96: 'Waffle',
     97: 'Watermelon',
     98: 'Wine',
     99: 'Zucchini'}
    

    More class-based results to come soon.

    Here's a visual update of the new classes (I've just noticed the duplicate of Fries and French Fries), this should be ironed out in the next update. 2022-01-14-nutrify-100-foods-update

    Source code(tar.gz)
    Source code(zip)
Owner
Daniel Bourke
Machine Learning Engineer live on YouTube.
Daniel Bourke
vsketch is a Python generative art toolkit for plotters

Generative plotter art environment for Python

Antoine Beyeler 380 Dec 29, 2022
GIMP script to export bitmap as GRAPHICS 4 file (aka SCREEN 5)

gimpfu-msx-gr4.py GIMP script to export bitmap as GRAPHICS 4 file (aka SCREEN 5). GRAPHICS 4 specs are: 256x212 (or 256x192); 16 color palette (from 5

Pedro de Medeiros 4 Oct 17, 2022
Generative Art Synthesizer - a python program that generates python programs that generates generative art

GAS - Generative Art Synthesizer Generative Art Synthesizer - a python program that generates python programs that generates generative art. Examples

Alexey Borsky 43 Dec 03, 2022
Combinatorial image generator for generative NFT art.

ImageGen Stitches multiple image layers together into one image. Run usage: stitch.py [-h] backgrounds_dir dinos_dir traits_dir texture_file

Dinosols NFT 19 Sep 16, 2022
Docbarcodes extracts 1D and 2D barcodes from scanned PDF documents or images. It can be used to automate extraction and processing of all kind of documents.

Intro Barcodes are being used in many documents or forms to enable machine reading capabilities and reduce manual processing effort. Simple 1D barcode

Arlind Nocaj 3 Jun 18, 2022
Create a random fluent image based on multiple colors.

FluentGenerator Create a random fluent image based on multiple colors. Navigation Example Install Update Usage In Python console FluentGenerator Fluen

1 Feb 02, 2022
Transfers a image file(.png) to an Excel file(.xlsx)

Transfers a image file(.png) to an Excel file(.xlsx)

Junu Kwon 7 Feb 11, 2022
Pythonocc nodes for Ryven

Pythonocc-nodes-for-Ryven Pythonocc nodes for Ryven Here a way to work on Pythonocc with a node editor, Ryven in that case. To get it functional you w

Tanneguy 30 Dec 18, 2022
A tool for hiding data inside of images

Stegenography-tool a tool for hiding data inside of images Quick test: do python steg-encode.py test/message.txt test/covid19.png to generate the test

luke 2 Nov 02, 2021
The InvGears workbench for FreeCAD allows the creation of gear systems

FreeCAD InvGears workbench Current version 0.1.1 Overview The InvGears workbench allows the creation of gear systems. The gear generation algorithm is

Sebastian Ernesto Garcia 8 Dec 10, 2021
An executor that wraps 3D mesh models and encodes 3D content documents to d-dimension vector.

3D Mesh Encoder An Executor that receives Documents containing point sets data in its blob attribute, with shape (N, 3) and encodes it to embeddings o

Jina AI 11 Dec 14, 2022
Collection of SVG diagrams about how UTF-8 works

Diagrams Repository of diagrams made for articles on my blog. All diagrams are created using diagrams.net. UTF-8 Licenses Copyright 2022 Seth Michael

Seth Michael Larson 24 Aug 13, 2022
Extracts random colours from an image

EXTRACT COLOURS This repo contains all the project files. Project Description A Program that extracts 10 random colours from an image and saves the rg

David .K. Danso 3 Dec 13, 2021
Art directed cropping, useful for responsive images

Art direction sets a focal point and can be used when you need multiple copies of the same Image but also in in different proportions.

Daniel 1 Aug 16, 2022
hashmask reverse lookup

ImageHashMasks Lookup Hashmask NFT index from a picture Setup pip install pillow click imagehash Usage $ python imagehashmasks.py

17 Nov 29, 2021
CropImage is a simple toolkit for image cropping, detecting and cropping main body from pictures.

CropImage is a simple toolkit for image cropping, detecting and cropping main body from pictures. Support face and saliency detection.

Haofan Wang 15 Dec 22, 2022
QR Generator using GUI with Tinker

BinCat Token System Very simple python script with GUI that generates QR codes. It don't include a QR "decription" tool. It only generate-it and thats

Hipotesi 1 Nov 06, 2021
Easy to use Python module to extract Exif metadata from digital image files.

Easy to use Python module to extract Exif metadata from digital image files.

ianaré sévi 719 Jan 05, 2023
Graphical tool to make photo collage posters

PhotoCollage Graphical tool to make photo collage posters PhotoCollage allows you to create photo collage posters. It assembles the input photographs

Adrien Vergé 350 Jan 02, 2023
Tweet2Image - Convert tweets to Instagram-friendly images.

Convert tweets to Instagram-friendly images. How to use If you want to use this repository as a submodule, don't forget to put the fonts d

Janu Lingeswaran 1 Mar 11, 2022