I have tried both default discovery as well as specifying it manually using the ROBOTS_SITEMAP_URLS setting but I am still unable to get the site map to show up on the robots.txt

Just created a new setting to enable HTTPS or HTTP on robots.txt HOST record. Will be great if somebody will review and if everything is ok merge it to the project

I have a setup of Django CMS + Gunicorn + Nginx and when the robots.txt file is displayed, it shows the scheme of the URL twice.

This is the robots file that is generated:

User-agent: * Allow: / Disallow: /admin

Host: https://www.chitwa.co.za Sitemap: https://https://www.chitwa.co.za/en/sitemap.xml

Add django 3.2 and 4.0 support

I am facing the following warning while upgrading the django to 3.2 which take the default variable for DEFAULT_AUTO_FIELD ref

web_1     | robots.Rule: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
web_1     | 	HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
web_1     | robots.Url: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
web_1     | 	HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.

So in lieu of that I am facing the error in my CI/CD that is

[TEST SUITE]./scripts/test/detect_missing_migrations.sh
Error: one or more migrations are missing
Migrations for 'robots': /opt/hostedtoolcache/Python/3.7.5/x64/lib/python3.7/site-packages/robots/migrations/0002_auto_20210923_1203.py - Alter field id on rule - Alter field id on url

Hey, It seems like pypi still has 3.0 https://pypi.python.org/pypi/django-robots

Can you guys also update pypi with 3.1.0 so it would be downlodable with pip?

The current django-robots implementation is dependent on django.contrib.sites. Wagtail uses its own Site model, meaning django-robots currently is not compatible with Wagtail. It would be a nice feature to be able to use django-robots for django applications that cannot make use of the 'sites' framework, such as Wagtail-sites.

Crawl delay is intended as an integer in most specifications (https://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive), no need to format as float

The error code is : File "/home/***/***/venv/lib/python3.9/site-packages/robots/models.py", line 4, in from django.utils.translation import ugettext_lazy as _ ImportError: cannot import name 'ugettext_lazy' from 'django.utils.translation' (/home/nour/LPM/venv/lib/python3.9/site-packages/django/utils/

We must change ugettextlazy to gettextlazy in import section to solve the problem And THX <3 .

Related to #124, a migration file now seems to be included in 5.0, but results (at least on my system) in the following error:

psycopg2.errors.ObjectNotInPrerequisiteState: sequence must have same owner as table it is linked to

When you have Django 1.5.* and you run tests, django-robots 1.1 fails with message:

File "/lib/python2.7/site-packages/robots/forms.py", line 7, in <module>
  class RuleAdminForm(forms.ModelForm):
File "/lib/python2.7/site-packages/django/forms/models.py", line 221, in __new__
  raise FieldError(message)
FieldError: Unknown field(s) (a, l, _) specified for Rule

This is related to https://github.com/jezdez/django-robots/pull/30 where __all__ was added to support Django 1.8 which causes to break in older versions.

Just pip installed this but it doesn't have the fix for fields fix in robots/forms.py.

I've worked around this by changing the entry in my requirements.txt file to -e git://github.com/jezdez/django-robots.git#egg=robots to install it from GitHub.

django-robots version 5.0 I am using above version of django-robots. When database is SQLite, things are working fine but when the database is MySQL, it gives following error while running migrate command.

Error: django.db.utils.OperationalError: (3780, "Referencing column 'rule_id' and referenced column 'id' in foreign key constraint 'robots_rule_sites_rule_id_7921a799_fk_robots_rule_id' are incompatible.")

I investigate and figure out that the BigAutoField is causing the problem.

updates:

• github.com/pre-commit/pre-commit-hooks: v4.3.0 → v4.4.0
• github.com/pycqa/isort: 5.10.1 → 5.11.4
• github.com/PyCQA/flake8: 5.0.4 → 6.0.0
• github.com/psf/black: 22.10.0 → 22.12.0

Hi!

If anyone is able to help out with maintenance, QA, writing tests, reviewing the pull request list, and triaging issues, this would be helpful! At this time I'm a bit short-handed.

It's critical we get releases tested and that any fixes on the above are future issues working correctly.

Outstanding issues, especially if you're familiar with migrations:

• #124, #125. We have a PR at #134, which look decent to me
• Same goes for #132 which as PR #133

It's not clear to me - at this time - what the ramifications for creating migrations will be at scale for users across different django versions, who may have made their own migration, etc.

fix #132

• The MaxLengthValidator has problem, that it doesn't allow / with 254 other characters (which is generated when saved previously with long pattern). But I suppose it is very minor problem not worth adding more complexity to the code.

If I try to create Url model with model_bakery without any parameters, I have got this error:

django.db.utils.DataError: value too long for type character varying(255)

The problem is, that pattern parameter (generated by model_bakery) gets lengthen in save() method by leading / and doesn't fit in the field anymore.

The solution is very simple: add default="" to the pattern field.

There is also second problem with the save method - if user enters pattern exactly 255 characters long in the admin, it will throw an exception rather than showing validation error.