A reusable Django model field for storing ad-hoc JSON data

Overview

jsonfield

https://circleci.com/gh/rpkilby/jsonfield.svg?style=shield

jsonfield is a reusable model field that allows you to store validated JSON, automatically handling serialization to and from the database. To use, add jsonfield.JSONField to one of your models.

Note: django.contrib.postgres now supports PostgreSQL's jsonb type, which includes extended querying capabilities. If you're an end user of PostgreSQL and want full-featured JSON support, then it is recommended that you use the built-in JSONField. However, jsonfield is still useful when your app needs to be database-agnostic, or when the built-in JSONField's extended querying is not being leveraged. e.g., a configuration field.

Requirements

jsonfield aims to support all current versions of Django, however the explicity tested versions are:

  • Python: 3.6, 3.7, 3.8
  • Django: 2.2, 3.0

Installation

pip install jsonfield

Usage

from django.db import models
from jsonfield import JSONField

class MyModel(models.Model):
    json = JSONField()

Querying

As stated above, JSONField is not intended to provide extended querying capabilities. That said, you may perform the same basic lookups provided by regular text fields (e.g., exact or regex lookups). Since values are stored as serialized JSON, it is highly recommended that you test your queries to ensure the expected results are returned.

Handling null values

A model field's null argument typically controls whether null values may be stored in its column by setting a not-null constraint. However, because JSONField serializes its values (including nulls), this option instead controls how null values are persisted. If null=True, then nulls are not serialized and are stored as a null value in the database. If null=False, then the null is instead stored in its serialized form.

This in turn affects how null values may be queried. Both fields support exact matching:

MyModel.objects.filter(json=None)

However, if you want to use the isnull lookup, you must set null=True.

class MyModel(models.Model):
    json = JSONField(null=True)

MyModel.objects.filter(json__isnull=True)

Note that as JSONField.null does not prevent nulls from being stored, achieving this must instead be handled with a validator.

Advanced Usage

By default python deserializes json into dict objects. This behavior differs from the standard json behavior because python dicts do not have ordered keys. To overcome this limitation and keep the sort order of OrderedDict keys the deserialisation can be adjusted on model initialisation:

import collections

class MyModel(models.Model):
    json = JSONField(load_kwargs={'object_pairs_hook': collections.OrderedDict})

Other Fields

jsonfield.JSONCharField

Subclasses models.CharField instead of models.TextField.

Running the tests

The test suite requires tox.

$ pip install tox

Then, run the tox command, which will run all test jobs.

$ tox

Or, to test just one job (for example Django 2.0 on Python 3.6):

$ tox -e py36-django20

Release Process

  • Update changelog
  • Update package version in setup.py
  • Check supported versions in setup.py and readme
  • Create git tag for version
  • Upload release to PyPI test server
  • Upload release to official PyPI server
$ pip install -U pip setuptools wheel twine
$ rm -rf dist/ build/
$ python setup.py sdist bdist_wheel
$ twine upload -r test dist/*
$ twine upload dist/*

Changes

Take a look at the changelog.

Comments
  • Problems withs PostGres 9.3, JSON Field and Query Distinct

    Problems withs PostGres 9.3, JSON Field and Query Distinct

    As mentionned in #47, using ".distinct()", which is used by the admin panel creating users but also in several other cases in django processes, triggers the same bug as #47 describes.

    A workaround has been found by @mkhattab : The work around for this in the Django admin is to subclass QuerySet and Manager and override the distinct method [to remove the JSON fields from it].

    What would be the definite fix ?

    opened by ewjoachim 20
  • Can not store ordinary Python strings in JSONfield

    Can not store ordinary Python strings in JSONfield

    Since version 0.9.4 I am not able to store ordinary Python strings in a JSONfield. Up to version 0.9.2 this worked without problems. Did I miss something? From my point of view, it should not matter, if I store ordinary strings, lists or dicts in a JSONfield.

    opened by jrief 20
  • AttributeError in Creator class

    AttributeError in Creator class

    I updated to Django 1.11 and I get AttributeError from subclassing.py:33.

    In Django 1.11 (django/db/models/options.py:890) the get method of the Creator class is implicitly called. The code in subclassing.py is a copy/paste from Django and seems to be <1.7 and Django's code has updated since then. Simply copying the code from Django 1.7 seems to solve my problems.

    opened by Majsvaffla 16
  • JSONField not deserialized when used from inside a polymorphic inherited model

    JSONField not deserialized when used from inside a polymorphic inherited model

    Hi,

    I have models like this:

    class BaseFeed(PolymorphicModel):
        errors = JSONField(default=list, blank=True)
    
    
    class RssAtomFeed(BaseFeed):
        pass
    

    Note: polymorphic is here. Say I do:

    f1 = BaseFeed.objects.create()
    f2 = RssAtomFeed.objects.create()
    

    Then I got:

    f1.errors
    []
    
    f2.errors
    u'[]'
    

    And this is pretty annoying because it will obviously fail in many places because all my code expects a list.

    I have read #92, but I didn't find any obvious/easy clue on how to solve this issue.

    Any other hint I can try?

    For the record, my models are much complex than this. You can view the base feed here and the rss/atom feed here. I've tried with or without default, with or without load_kwargs={'object_pairs_hook': collections.OrderedDict}, this doesn't change anything.

    I've tried the other JSONField implementation and the problem doesn't occur, even after a south migration (I'm on Django 1.6).

    regards,

    opened by Karmak23 11
  • Problem when iterating if default value is '

    Problem when iterating if default value is '""' on postgresql-9.3

    In [1]: from my_app.models import MyModel

    In [2]: [fo for fo in MyModel.objects.all()]

    ValidationError Traceback (most recent call last) in () ----> 1 [fo for fo in MyModel.objects.all()]

    /home/florent/.virtualenvs//local/lib/python2.7/site-packages/django/db/models/query.pyc in iter(self) 94 - Responsible for turning the rows into model objects. 95 """ ---> 96 self._fetch_all() 97 return iter(self._result_cache) 98

    /home/florent/.virtualenvs//local/lib/python2.7/site-packages/django/db/models/query.pyc in _fetch_all(self) 852 def _fetch_all(self): 853 if self._result_cache is None: --> 854 self._result_cache = list(self.iterator()) 855 if self._prefetch_related_lookups and not self._prefetch_done: 856 self._prefetch_related_objects()

    /home/florent/.virtualenvs//local/lib/python2.7/site-packages/django/db/models/query.pyc in iterator(self) 228 obj = model_cls(*_dict(zip(init_list, row_data))) 229 else: --> 230 obj = model(_row_data) 231 232 # Store the source database of the object

    /home/florent/.virtualenvs//local/lib/python2.7/site-packages/django/db/models/base.pyc in init(self, _args, *_kwargs) 345 # without changing the logic. 346 for val, field in zip(args, fields_iter): --> 347 setattr(self, field.attname, val) 348 else: 349 # Slower, kwargs-ready version.

    /home/florent/Work//django-jsonfield/jsonfield/subclassing.pyc in set(self, obj, value) 39 # we can definitively tell if a value has already been deserialized 40 # More: https://github.com/bradjasper/django-jsonfield/issues/33 ---> 41 obj.dict[self.field.name] = self.field.pre_init(value, obj) 42 43

    /home/florent/Work//django-jsonfield/jsonfield/fields.pyc in pre_init(self, value, obj) 75 return json.loads(value, **self.load_kwargs) 76 except ValueError: ---> 77 raise ValidationError(_("Enter valid JSON")) 78 79 return value

    ValidationError: [u'Enter valid JSON']

    opened by toopy 11
  • Serialize ''(empty) string to empty database value

    Serialize ''(empty) string to empty database value

    Uses ""(empty) string in database to store empty json ([]).

    You can use empty value as default for your table fields. You will have no overhead in database if you have no data.

    opened by Romamo 11
  • empty string or null in db

    empty string or null in db

    Hello,

    Wouldn't it be nice to gracefully handle empty string or NULL when reading from db? It could default to empty dictionary: {}.

    I've added the field to my model manually to the DB table and it did not work with empty string I had to update my_table set settings='{}'.

    I know I can set the '{}' via the south migrations, but it would still be a nice default behavior.

    opened by adrianandreias 9
  • DBError: could not identify an equality operator for type json when annotating a model with JSONField

    DBError: could not identify an equality operator for type json when annotating a model with JSONField

    I'm working in Django 1.5.4 with PostgreSQL 9.3, and i get this error on a query like this:

    ModelWithJsonField.objects.annotate(num=Count('field_to_count_by'))
    

    Traceback:

    Traceback (most recent call last):
      File "<console>", line 1, in <module>
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/query.py", line 93, in __repr__
        data = list(self[:REPR_OUTPUT_SIZE + 1])
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/query.py", line 108, in __len__
        self._result_cache.extend(self._iter)
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/query.py", line 317, in iterator
        for row in compiler.results_iter():
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 775, in results_iter
        for rows in self.execute_sql(MULTI):
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
        cursor.execute(sql, params)
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/backends/util.py", line 41, in execute
        return self.cursor.execute(sql, params)
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 58, in execute
        six.reraise(utils.DatabaseError, utils.DatabaseError(*tuple(e.args)), sys.exc_info()[2])
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 54, in execute
        return self.cursor.execute(query, args)
    DatabaseError: could not identify an equality operator for type json
    

    Ideas? I saw another issue with similar error, going to try and find out what causes this.

    opened by neara 9
  • JSONField fails in MySQL

    JSONField fails in MySQL

    I recently added the JSONField to one of my Models and worked fine since both my local machine runs on Postgres and my Heroku instance too, since moving it over to production I have found that everytime I try and bring up an item containg a default value for MySQL it raises a validationerror.

    class Item(models.Model): ... ... json = JSONField(default='{}')

    raise ValidationError(msg) ValidationError: [u"'' is not a valid JSON string."]

    opened by appel268576 9
  • Issues between versions

    Issues between versions

    Hi,

    We have recently updated from jsonfield2 to jsonfield and we are experiencing issues when we have a dict data being passed in get_or_create().

    The idea is we were using jsonfield2, jsonfield2==3.0.2 and when we update to latest jsonfield==3.1.0 the issue appears. In order to reproduce it you need to have a model lets say:

    class MyModel:
        field_1 = models.CharField
        field_2 = JsonField()
    
       Meta:
          unique(field_1)
    
    MyModel.objects.create(field_1='test', field_2={'foo':'bar'})
    

    And while still on jsonfiield2==3.0.2 version create an entry in the database ( we use postgres ).

    Now uninstall jsonfield2 and install latest jsonfield and try to do

    MyModel.objects.get_or_create(field_1='test', field_2={'foo':'bar'}) - It will fail with IntegrityError and you will get "duplicate key value violates unique constraint" message. This is because if you just call MyModel.objects.get(field_1='test', field_2={'foo':'bar'}) - it raises ObjectDoesNotExist, and then the above tries to call create but there is already a entry in the database with those params and there is a unique constraint in our case.

    The most important thing is that this MyModel.objects.get(field_1='test', field_2={'foo':'bar'}) raises ObjectDoesNotExist when the entry clearly exists in the table

    opened by ghost 8
  • Support polymorphic objects

    Support polymorphic objects

    Fix polymorphic object issue: https://github.com/dmkoch/django-jsonfield/issues/101

    With polymorphic objects I get a AttributeError on obj.pk. obj.id works fine. pk and id has in the tests always the same value.

    opened by ubaumann 8
  • Don't autofill fields with null

    Don't autofill fields with null

    After 92613991d76429c65bd35aebea3470c0baf26520, a blank field is automatically populated with null. This is bad for null=True fields, since those should be left empty when unpopulated. This PR preserves the behaviour from before.

    opened by quantum5 0
  • Update tests

    Update tests

    I was testing some PR that you have for jsonfield

    https://github.com/rpkilby/jsonfield/pull/262 This fixes the warnings and the name of a file that causes the circle ci test to fail.

    https://github.com/rpkilby/jsonfield/pull/261 And this one adds test for django 3.1 and 3.2

    I think it would be nice to add them to the main branch. I tried both PR together and work fine (https://app.circleci.com/pipelines/github/MaferMazu/jsonfield?branch=mfmz%2Ffix-ci-and-test-django3.2)

    Hopefully you can add them so that the library remains updated.

    Any doubt or advance I am at your service.

    Thanks for your consideration.

    opened by MaferMazu 0
  • Filter warnings from Django during capture.

    Filter warnings from Django during capture.

    Django has warnings in asyncio that are also captured in the tests. Filter warnings from django.utils.asyncio module. This was fixed in Django but not yet released : https://github.com/django/django/commit/623c8cd8f41a99f22d39b264f7eaf7244417000b .

    Fedora issue: https://bugzilla.redhat.com/show_bug.cgi?id=1962449

    opened by tirkarthi 1
  • docs: fix simple typo, explicity -> explicitly

    docs: fix simple typo, explicity -> explicitly

    There is a small typo in README.rst.

    Should read explicitly rather than explicity.

    Semi-automated pull request generated by https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md

    opened by timgates42 0
  • Not enough space to persist data

    Not enough space to persist data

    I was trying to save a big json object but i couldn't accomplish it since this field generate a Long type field on my Oracle DB. there is a solution to do it or it is neccesary apply a change.

    opened by jasonlazo 0
Owner
Ryan P Kilby
beep boop
Ryan P Kilby
Utilities to make function-based views cleaner, more efficient, and better tasting.

django-fbv Utilities to make Django function-based views cleaner, more efficient, and better tasting. 💥 📖 Complete documentation: https://django-fbv

Adam Hill 49 Dec 30, 2022
Automated image processing for Django. Currently v4.0

ImageKit is a Django app for processing images. Need a thumbnail? A black-and-white version of a user-uploaded image? ImageKit will make them for you.

Matthew Dapena-Tretter 2.1k Jan 04, 2023
Use heroicons in your Django and Jinja templates.

heroicons Use heroicons in your Django and Jinja templates. Requirements Python 3.6 to 3.9 supported. Django 2.2 to 3.2 supported. Are your tests slow

Adam Johnson 52 Dec 14, 2022
Django web apps for managing schedules.

skdue Description Skdue is a web application that makes your life easier by helping you manage your schedule. With the ability which allows you to cre

Patkamon_Awai 1 Jun 30, 2022
Django StatusPage - App to display statuspage for your services

Django StatusPage - App to display statuspage for your services

Gorlik 1 Oct 27, 2021
🌟 A social media made with Django and Python and Bulma. 🎉

Vitary A simple social media made with Django Installation 🛠️ Get the source code 💻 git clone https://github.com/foxy4096/Vitary.git Go the the dir

Aditya Priyadarshi 15 Aug 30, 2022
This is a Django app that uses numerous Google APIs such as reCAPTURE, maps and waypoints

Django project that uses Googles APIs to auto populate fields, display maps and routes for multiple waypoints

Bobby Stearman 57 Dec 03, 2022
An opinionated Django CMS setup bundled as an Aldryn Addon

Aldryn CMS |PyPI Version| An opinionated django CMS setup bundled as an Aldryn Addon. This package will auto configure django CMS including some extra

Vladimir Bezrukov 1 Nov 12, 2021
Django Pickled Model

Django Pickled Model Django pickled model provides you a model with dynamic data types. a field can store any value in any type. You can store Integer

Amir 3 Sep 14, 2022
Store model history and view/revert changes from admin site.

django-simple-history django-simple-history stores Django model state on every create/update/delete. This app supports the following combinations of D

Jazzband 1.8k Jan 08, 2023
Django REST Client API

Django REST Client API Client data provider API.

Ulysses Monteiro 1 Nov 08, 2021
An example of Django project with basic user functionality and account activation.

Simple Django Login and Registration An example of Django project with basic user functionality. Screenshots Log In Create an account Authorized page

Hussein Sarea 3 Oct 19, 2022
Django Serverless Cron - Run cron jobs easily in a serverless environment

Django Serverless Cron - Run cron jobs easily in a serverless environment

Paul Onteri 41 Dec 16, 2022
django-quill-editor makes Quill.js easy to use on Django Forms and admin sites

django-quill-editor django-quill-editor makes Quill.js easy to use on Django Forms and admin sites No configuration required for static files! The ent

lhy 139 Dec 05, 2022
📝 Sticky Notes in Django admin

django-admin-sticky-notes Share notes between superusers. Installation Install via pip: pip install django_admin_sticky_notes Put django_admin_sticky_

Dariusz Choruży 7 Oct 06, 2021
Учебное пособие по основам Django и сопутствующим технологиям

Учебный проект для закрепления основ Django Подробный разбор проекта здесь. Инструкция по запуску проекта на своей машине: Скачиваем репозиторий Устан

Stanislav Garanzha 12 Dec 30, 2022
Simple tagging for django

django-taggit This is a Jazzband project. By contributing you agree to abide by the Contributor Code of Conduct and follow the guidelines. django-tagg

Jazzband 3k Jan 02, 2023
An airlines clone website with django

abc_airlines is a clone website of an airlines system the way it works is that first you add flights to the website then the users can search flights

milad 1 Nov 16, 2021
A clone of https://virgool.io written in django

Virgool clone A clone of virgool blog written in django Installation first rename the .env.sample to .env and fill it. with docker docker-compose up -

Danial Selmipoor 7 Dec 23, 2022
Indonesia's negative news detection using gaussian naive bayes with Django+Scikir Learn

Introduction Indonesia's negative news detection using gaussian naive bayes build with Django and Scikit Learn. There is also any features, are: Input

Harifzi Ham 1 Dec 30, 2021