A reusable Django model field for storing ad-hoc JSON data

Overview

jsonfield

https://circleci.com/gh/rpkilby/jsonfield.svg?style=shield

jsonfield is a reusable model field that allows you to store validated JSON, automatically handling serialization to and from the database. To use, add jsonfield.JSONField to one of your models.

Note: django.contrib.postgres now supports PostgreSQL's jsonb type, which includes extended querying capabilities. If you're an end user of PostgreSQL and want full-featured JSON support, then it is recommended that you use the built-in JSONField. However, jsonfield is still useful when your app needs to be database-agnostic, or when the built-in JSONField's extended querying is not being leveraged. e.g., a configuration field.

Requirements

jsonfield aims to support all current versions of Django, however the explicity tested versions are:

  • Python: 3.6, 3.7, 3.8
  • Django: 2.2, 3.0

Installation

pip install jsonfield

Usage

from django.db import models
from jsonfield import JSONField

class MyModel(models.Model):
    json = JSONField()

Querying

As stated above, JSONField is not intended to provide extended querying capabilities. That said, you may perform the same basic lookups provided by regular text fields (e.g., exact or regex lookups). Since values are stored as serialized JSON, it is highly recommended that you test your queries to ensure the expected results are returned.

Handling null values

A model field's null argument typically controls whether null values may be stored in its column by setting a not-null constraint. However, because JSONField serializes its values (including nulls), this option instead controls how null values are persisted. If null=True, then nulls are not serialized and are stored as a null value in the database. If null=False, then the null is instead stored in its serialized form.

This in turn affects how null values may be queried. Both fields support exact matching:

MyModel.objects.filter(json=None)

However, if you want to use the isnull lookup, you must set null=True.

class MyModel(models.Model):
    json = JSONField(null=True)

MyModel.objects.filter(json__isnull=True)

Note that as JSONField.null does not prevent nulls from being stored, achieving this must instead be handled with a validator.

Advanced Usage

By default python deserializes json into dict objects. This behavior differs from the standard json behavior because python dicts do not have ordered keys. To overcome this limitation and keep the sort order of OrderedDict keys the deserialisation can be adjusted on model initialisation:

import collections

class MyModel(models.Model):
    json = JSONField(load_kwargs={'object_pairs_hook': collections.OrderedDict})

Other Fields

jsonfield.JSONCharField

Subclasses models.CharField instead of models.TextField.

Running the tests

The test suite requires tox.

$ pip install tox

Then, run the tox command, which will run all test jobs.

$ tox

Or, to test just one job (for example Django 2.0 on Python 3.6):

$ tox -e py36-django20

Release Process

  • Update changelog
  • Update package version in setup.py
  • Check supported versions in setup.py and readme
  • Create git tag for version
  • Upload release to PyPI test server
  • Upload release to official PyPI server
$ pip install -U pip setuptools wheel twine
$ rm -rf dist/ build/
$ python setup.py sdist bdist_wheel
$ twine upload -r test dist/*
$ twine upload dist/*

Changes

Take a look at the changelog.

Comments
  • Problems withs PostGres 9.3, JSON Field and Query Distinct

    Problems withs PostGres 9.3, JSON Field and Query Distinct

    As mentionned in #47, using ".distinct()", which is used by the admin panel creating users but also in several other cases in django processes, triggers the same bug as #47 describes.

    A workaround has been found by @mkhattab : The work around for this in the Django admin is to subclass QuerySet and Manager and override the distinct method [to remove the JSON fields from it].

    What would be the definite fix ?

    opened by ewjoachim 20
  • Can not store ordinary Python strings in JSONfield

    Can not store ordinary Python strings in JSONfield

    Since version 0.9.4 I am not able to store ordinary Python strings in a JSONfield. Up to version 0.9.2 this worked without problems. Did I miss something? From my point of view, it should not matter, if I store ordinary strings, lists or dicts in a JSONfield.

    opened by jrief 20
  • AttributeError in Creator class

    AttributeError in Creator class

    I updated to Django 1.11 and I get AttributeError from subclassing.py:33.

    In Django 1.11 (django/db/models/options.py:890) the get method of the Creator class is implicitly called. The code in subclassing.py is a copy/paste from Django and seems to be <1.7 and Django's code has updated since then. Simply copying the code from Django 1.7 seems to solve my problems.

    opened by Majsvaffla 16
  • JSONField not deserialized when used from inside a polymorphic inherited model

    JSONField not deserialized when used from inside a polymorphic inherited model

    Hi,

    I have models like this:

    class BaseFeed(PolymorphicModel):
        errors = JSONField(default=list, blank=True)
    
    
    class RssAtomFeed(BaseFeed):
        pass
    

    Note: polymorphic is here. Say I do:

    f1 = BaseFeed.objects.create()
    f2 = RssAtomFeed.objects.create()
    

    Then I got:

    f1.errors
    []
    
    f2.errors
    u'[]'
    

    And this is pretty annoying because it will obviously fail in many places because all my code expects a list.

    I have read #92, but I didn't find any obvious/easy clue on how to solve this issue.

    Any other hint I can try?

    For the record, my models are much complex than this. You can view the base feed here and the rss/atom feed here. I've tried with or without default, with or without load_kwargs={'object_pairs_hook': collections.OrderedDict}, this doesn't change anything.

    I've tried the other JSONField implementation and the problem doesn't occur, even after a south migration (I'm on Django 1.6).

    regards,

    opened by Karmak23 11
  • Problem when iterating if default value is '

    Problem when iterating if default value is '""' on postgresql-9.3

    In [1]: from my_app.models import MyModel

    In [2]: [fo for fo in MyModel.objects.all()]

    ValidationError Traceback (most recent call last) in () ----> 1 [fo for fo in MyModel.objects.all()]

    /home/florent/.virtualenvs//local/lib/python2.7/site-packages/django/db/models/query.pyc in iter(self) 94 - Responsible for turning the rows into model objects. 95 """ ---> 96 self._fetch_all() 97 return iter(self._result_cache) 98

    /home/florent/.virtualenvs//local/lib/python2.7/site-packages/django/db/models/query.pyc in _fetch_all(self) 852 def _fetch_all(self): 853 if self._result_cache is None: --> 854 self._result_cache = list(self.iterator()) 855 if self._prefetch_related_lookups and not self._prefetch_done: 856 self._prefetch_related_objects()

    /home/florent/.virtualenvs//local/lib/python2.7/site-packages/django/db/models/query.pyc in iterator(self) 228 obj = model_cls(*_dict(zip(init_list, row_data))) 229 else: --> 230 obj = model(_row_data) 231 232 # Store the source database of the object

    /home/florent/.virtualenvs//local/lib/python2.7/site-packages/django/db/models/base.pyc in init(self, _args, *_kwargs) 345 # without changing the logic. 346 for val, field in zip(args, fields_iter): --> 347 setattr(self, field.attname, val) 348 else: 349 # Slower, kwargs-ready version.

    /home/florent/Work//django-jsonfield/jsonfield/subclassing.pyc in set(self, obj, value) 39 # we can definitively tell if a value has already been deserialized 40 # More: https://github.com/bradjasper/django-jsonfield/issues/33 ---> 41 obj.dict[self.field.name] = self.field.pre_init(value, obj) 42 43

    /home/florent/Work//django-jsonfield/jsonfield/fields.pyc in pre_init(self, value, obj) 75 return json.loads(value, **self.load_kwargs) 76 except ValueError: ---> 77 raise ValidationError(_("Enter valid JSON")) 78 79 return value

    ValidationError: [u'Enter valid JSON']

    opened by toopy 11
  • Serialize ''(empty) string to empty database value

    Serialize ''(empty) string to empty database value

    Uses ""(empty) string in database to store empty json ([]).

    You can use empty value as default for your table fields. You will have no overhead in database if you have no data.

    opened by Romamo 11
  • empty string or null in db

    empty string or null in db

    Hello,

    Wouldn't it be nice to gracefully handle empty string or NULL when reading from db? It could default to empty dictionary: {}.

    I've added the field to my model manually to the DB table and it did not work with empty string I had to update my_table set settings='{}'.

    I know I can set the '{}' via the south migrations, but it would still be a nice default behavior.

    opened by adrianandreias 9
  • DBError: could not identify an equality operator for type json when annotating a model with JSONField

    DBError: could not identify an equality operator for type json when annotating a model with JSONField

    I'm working in Django 1.5.4 with PostgreSQL 9.3, and i get this error on a query like this:

    ModelWithJsonField.objects.annotate(num=Count('field_to_count_by'))
    

    Traceback:

    Traceback (most recent call last):
      File "<console>", line 1, in <module>
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/query.py", line 93, in __repr__
        data = list(self[:REPR_OUTPUT_SIZE + 1])
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/query.py", line 108, in __len__
        self._result_cache.extend(self._iter)
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/query.py", line 317, in iterator
        for row in compiler.results_iter():
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 775, in results_iter
        for rows in self.execute_sql(MULTI):
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
        cursor.execute(sql, params)
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/backends/util.py", line 41, in execute
        return self.cursor.execute(sql, params)
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 58, in execute
        six.reraise(utils.DatabaseError, utils.DatabaseError(*tuple(e.args)), sys.exc_info()[2])
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 54, in execute
        return self.cursor.execute(query, args)
    DatabaseError: could not identify an equality operator for type json
    

    Ideas? I saw another issue with similar error, going to try and find out what causes this.

    opened by neara 9
  • JSONField fails in MySQL

    JSONField fails in MySQL

    I recently added the JSONField to one of my Models and worked fine since both my local machine runs on Postgres and my Heroku instance too, since moving it over to production I have found that everytime I try and bring up an item containg a default value for MySQL it raises a validationerror.

    class Item(models.Model): ... ... json = JSONField(default='{}')

    raise ValidationError(msg) ValidationError: [u"'' is not a valid JSON string."]

    opened by appel268576 9
  • Issues between versions

    Issues between versions

    Hi,

    We have recently updated from jsonfield2 to jsonfield and we are experiencing issues when we have a dict data being passed in get_or_create().

    The idea is we were using jsonfield2, jsonfield2==3.0.2 and when we update to latest jsonfield==3.1.0 the issue appears. In order to reproduce it you need to have a model lets say:

    class MyModel:
        field_1 = models.CharField
        field_2 = JsonField()
    
       Meta:
          unique(field_1)
    
    MyModel.objects.create(field_1='test', field_2={'foo':'bar'})
    

    And while still on jsonfiield2==3.0.2 version create an entry in the database ( we use postgres ).

    Now uninstall jsonfield2 and install latest jsonfield and try to do

    MyModel.objects.get_or_create(field_1='test', field_2={'foo':'bar'}) - It will fail with IntegrityError and you will get "duplicate key value violates unique constraint" message. This is because if you just call MyModel.objects.get(field_1='test', field_2={'foo':'bar'}) - it raises ObjectDoesNotExist, and then the above tries to call create but there is already a entry in the database with those params and there is a unique constraint in our case.

    The most important thing is that this MyModel.objects.get(field_1='test', field_2={'foo':'bar'}) raises ObjectDoesNotExist when the entry clearly exists in the table

    opened by ghost 8
  • Support polymorphic objects

    Support polymorphic objects

    Fix polymorphic object issue: https://github.com/dmkoch/django-jsonfield/issues/101

    With polymorphic objects I get a AttributeError on obj.pk. obj.id works fine. pk and id has in the tests always the same value.

    opened by ubaumann 8
  • Don't autofill fields with null

    Don't autofill fields with null

    After 92613991d76429c65bd35aebea3470c0baf26520, a blank field is automatically populated with null. This is bad for null=True fields, since those should be left empty when unpopulated. This PR preserves the behaviour from before.

    opened by quantum5 0
  • Update tests

    Update tests

    I was testing some PR that you have for jsonfield

    https://github.com/rpkilby/jsonfield/pull/262 This fixes the warnings and the name of a file that causes the circle ci test to fail.

    https://github.com/rpkilby/jsonfield/pull/261 And this one adds test for django 3.1 and 3.2

    I think it would be nice to add them to the main branch. I tried both PR together and work fine (https://app.circleci.com/pipelines/github/MaferMazu/jsonfield?branch=mfmz%2Ffix-ci-and-test-django3.2)

    Hopefully you can add them so that the library remains updated.

    Any doubt or advance I am at your service.

    Thanks for your consideration.

    opened by MaferMazu 0
  • Filter warnings from Django during capture.

    Filter warnings from Django during capture.

    Django has warnings in asyncio that are also captured in the tests. Filter warnings from django.utils.asyncio module. This was fixed in Django but not yet released : https://github.com/django/django/commit/623c8cd8f41a99f22d39b264f7eaf7244417000b .

    Fedora issue: https://bugzilla.redhat.com/show_bug.cgi?id=1962449

    opened by tirkarthi 1
  • docs: fix simple typo, explicity -> explicitly

    docs: fix simple typo, explicity -> explicitly

    There is a small typo in README.rst.

    Should read explicitly rather than explicity.

    Semi-automated pull request generated by https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md

    opened by timgates42 0
  • Not enough space to persist data

    Not enough space to persist data

    I was trying to save a big json object but i couldn't accomplish it since this field generate a Long type field on my Oracle DB. there is a solution to do it or it is neccesary apply a change.

    opened by jasonlazo 0
Owner
Ryan P Kilby
beep boop
Ryan P Kilby
A Django backed for PostgreSQL using Psycopg 3

A Django backend for PostgreSQL using Psycopg 2 The backend passes the entire Django test suite, but it needs a few modifications to Django and to i

Daniele Varrazzo 42 Dec 16, 2022
Example project demonstrating using Django’s test runner with Coverage.py

Example project demonstrating using Django’s test runner with Coverage.py Set up with: python -m venv --prompt . venv source venv/bin/activate python

Adam Johnson 5 Nov 29, 2021
A Django chatbot that is capable of doing math and searching Chinese poet online. Developed with django, channels, celery and redis.

Django Channels Websocket Chatbot A Django chatbot that is capable of doing math and searching Chinese poet online. Developed with django, channels, c

Yunbo Shi 8 Oct 28, 2022
This is raw connection between redis server and django python app

Django_Redis This repository contains the code for this blogpost. Running the Application Clone the repository git clone https://github.com/xxl4tomxu9

Tom Xu 1 Sep 15, 2022
open source online judge based on Vue, Django and Docker

An onlinejudge system based on Python and Vue

Qingdao University(青岛大学) 5.2k Jan 09, 2023
Django-environ allows you to utilize 12factor inspired environment variables to configure your Django application.

Django-environ django-environ allows you to use Twelve-factor methodology to configure your Django application with environment variables. import envi

Daniele Faraglia 2.7k Jan 07, 2023
Updates redisearch instance with igdb data used for kimosabe

igdb-pdt Update RediSearch with IGDB games data in the following Format: { "game_slug": { "name": "game_name", "cover": "igdb_coverart_url",

6rotoms 0 Jul 30, 2021
MAC address Model Field & Form Field for Django apps

django-macaddress MAC Address model and form fields for Django We use netaddr to parse and validate the MAC address. The tests aren't complete yet. Pa

49 Sep 04, 2022
REST API with Django and SQLite3

REST API with Django and SQLite3

Luis Quiñones Requelme 1 Nov 07, 2021
The uncompromising Python code formatter

The Uncompromising Code Formatter “Any color you like.” Black is the uncompromising Python code formatter. By using it, you agree to cede control over

Python Software Foundation 30.7k Jan 03, 2023
A UUIDField for Django

django-uuidfield Provides a UUIDField for your Django models. Installation Install it with pip (or easy_install): pip install django-uuidfield Usage

David Cramer 265 Nov 30, 2022
Store model history and view/revert changes from admin site.

django-simple-history django-simple-history stores Django model state on every create/update/delete. This app supports the following combinations of D

Jazzband 1.8k Jan 08, 2023
A Django app to accept payments from various payment processors via Pluggable backends.

Django-Merchant Django-Merchant is a django application that enables you to use multiple payment processors from a single API. Gateways Following gate

Agiliq 997 Dec 24, 2022
Django-shared-app-isolated-databases-example - Django - Shared App & Isolated Databases

Django - Shared App & Isolated Databases An app that demonstrates the implementa

Ajai Danial 5 Jun 27, 2022
django+bootstrap5 实现的 个人博客

项目状态: 正在开发中【目前已基本可用】 项目地址: https://github.com/find456789/django_blog django_blog django+bootstrap5 实现的 个人博客 特点 文章的历史版本管理(随时回退) rss、atom markdown 评论功能

名字 3 Nov 16, 2021
Full control of form rendering in the templates.

django-floppyforms Full control of form rendering in the templates. Authors: Gregor Müllegger and many many contributors Original creator: Bruno Renié

Jazzband 811 Dec 01, 2022
Official clone of the Subversion repository.

Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design. All documentation is in the "docs" directo

Raymond Penners 3 May 06, 2022
Django Advance DumpData

Django Advance Dumpdata Django Manage Command like dumpdata but with have more feature to Output the contents of the database from given fields of a m

EhsanSafir 7 Jul 25, 2022
Django datatables and widgets, both AJAX and traditional. Display-only ModelForms.

Django datatables and widgets, both AJAX and traditional. Display-only ModelForms. ModelForms / inline formsets with AJAX submit and validation. Works with Django templates.

Dmitriy Sintsov 132 Dec 14, 2022
Declarative model lifecycle hooks, an alternative to Signals.

Django Lifecycle Hooks This project provides a @hook decorator as well as a base model and mixin to add lifecycle hooks to your Django models. Django'

Robert Singer 1k Dec 31, 2022