Hello, I was resolving DNS records for some domains and I found an unexpected error resolving the TXT record of like.com.sa

Here is a code to reproduce the error

import asyncio
import aiodns

loop = asyncio.get_event_loop()
resolver = aiodns.DNSResolver(loop=loop)
result = loop.run_until_complete(resolver.query('like.com.sa', 'TXT'))
print(result)

Thanks

I'm getting build/temp.macosx-10.14-x86_64-3.7/_cares.c:2573:55: error: too many arguments to function call, expected 6, have 7 on pip install aiodns pip/python version: pip 19.0.3 from path/to/venv/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip (python 3.7) OS: macOS Mojave 10.14.6

Please issue a tagged release since there are bugfixes accumulated since last release, to enable distributions to distribute the updates.

https://github.com/saghul/aiodns/issues/71 , which causes Gentoo bug https://bugs.gentoo.org/692720 has been resolved in Sep 2019. Yet the last release of aiodns is from Mar 2019.

Thanks in advance.

Hi!

Question. I'm trying to fetch the name servers of a specific domain name. This works when I don't provide nameservers to the DNSResolver:

import asyncio
import aiodns

loop = asyncio.get_event_loop()
resolver = aiodns.DNSResolver(loop=loop)
f = resolver.query('test.de', 'NS')
result = loop.run_until_complete(f)
print(result)

This results in:

[ares_query_ns_result(host='dns3.iwelt-ag.de', ttl=None), ares_query_ns_result(host='dns2.iwelt-ag.net', ttl=None), ares_query_ns_result(host='dns.iwelt-ag.net', ttl=None)]

But, when I do this:

import asyncio
import aiodns

loop = asyncio.get_event_loop()
resolver = aiodns.DNSResolver(loop=loop, nameservers=['81.91.164.5'])
f = resolver.query('test.de', 'NS')
result = loop.run_until_complete(f)
print(result)

This is the result:

aiodns.error.DNSError: (1, 'DNS server returned answer with no data')

Though when I capture the packets I see that I'm getting the results back from the name server:

 sudo tcpdump -i en0 -AAA -vvv host 81.91.164.5

tcpdump: listening on en0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:40:30.518946 IP (tos 0x0, ttl 64, id 23865, offset 0, flags [none], proto UDP (17), length 53)
    192.168.178.10.55500 > f.nic.de.domain: [udp sum ok] 58234+ NS? test.de. (25)
.Y3.;/..V?. ..E..5][email protected]...
Q[.....5.!...z...........test.de.....
21:40:30.532423 IP (tos 0x0, ttl 57, id 27877, offset 0, flags [none], proto UDP (17), length 146)
    f.nic.de.domain > 192.168.178.10.55500: [udp sum ok] 58234- q: NS? test.de. 0/3/1 ns: test.de. [1d] NS dns.iwelt-ag.net., test.de. [1d] NS dns2.iwelt-ag.net., test.de. [1d] NS dns3.iwelt-ag.de. ar: dns3.iwelt-ag.de. [1d] A 85.236.41.196 (118)
..V?. .Y3.;/..E...l...9..bQ[.....
.5...~...z...........test.de.............Q....dns.iwelt-ag.net.........Q....dns2.)........Q....dns3.iwelt-ag...V......Q...U.).

Am I missing something, or is this a bug somewhere in aiodns or pycares?

Hi, I have a very similar problem to this solved issue:

import asyncio
import aiodns

loop = asyncio.get_event_loop()
resolver = aiodns.DNSResolver(loop=loop)
result = loop.run_until_complete(resolver.query('xn--cardeosapeluqueros-r0b.com', 'MX'))
print(result)

I get this error and the program hangs.

Exception ignored in: functools.partial(<function DNSResolver._callback at 0x7fe1067b2ea0>, <Future pending cb=[_run_until_complete_cb() at /usr/local/lib/python3.6/asyncio/base_events.py:176]>)
Traceback (most recent call last):
  File "/home/steven/Envs/vPandapipe/lib/python3.6/site-packages/aiodns/__init__.py", line 56, in _callback
    if fut.cancelled():
SystemError: <built-in method cancelled of _asyncio.Future object at 0x7fe106ee7828> returned a result with an error set

Looking further I found that this problem happens when the result of the query has special characters, Doing a ns lookup I found this for MX:

Non-authoritative answer:
 xn--cardeosapeluqueros-r0b.com mail exchanger = 10 carde\241osapeluqueros.com.

Querying SOA for ayesas.com. Its record also contains a special character:

Non-authoritative answer:
ayesas.com
	origin = ns2.3c1b.com
	mail addr = adm\253n.domain.com
	serial = 2015010244
	refresh = 3600
	retry = 600
	expire = 1209600
	minimum = 3600

I'm using aiodns 1.1.1 and pycares 2.3.0.

Thank you!

Hi,

I've had a few problems trying to implement asyncio and aiohttp into my script running out of sockets to perform the connection in SelectorEventLoop. I've then tried to use ProactorEventLoop on Windows that doesn't seem to not have this limitation. However when I try:

import asyncio
import aiohttp


async def getHeaders(url, session, sema):
    async with session:
        async with sema:
            try:
                async with session.head(url) as response:
                    try:
                        if "html" in response.headers["Content-Type"]:
                            return url, True
                        else:
                            return url, False
                    except:
                        return url, False
            except:
                return url, False


def removeUrlsWithoutHtml(setOfUrls, MAXitems):
    listOfUrls = list(setOfUrls)
    while(len(listOfUrls) != 0):
        blockurls = []
        print("URLS left to process: " + str(len(listOfUrls)))
        items = 0
        for num in range(0, len(listOfUrls)):
            if num < MAXitems:
                blockurls.append(listOfUrls[num - items])
                listOfUrls.remove(listOfUrls[num - items])
                items += 1
        loop = asyncio.ProactorEventLoop()
        asyncio.set_event_loop(loop)
        semaphoreHeaders = asyncio.Semaphore(50)
        session = aiohttp.ClientSession()
        data = loop.run_until_complete(asyncio.gather(*(getHeaders(url, session, semaphoreHeaders) for url in blockurls)))
        for header in data:
            if False == header[1]:
                setOfUrls.remove(header[0])

MAXitems = 10
setOfUrls = {'http://www.google.com', 'http://www.reddit.com'}
removeUrlsWithoutHtml(setOfUrls, MAXitems)

for link in list(setOfUrls):
    print(link)

Note the use of semaphore and chuncking to try to get around the selector limit issue that I face if I replace

        loop = asyncio.ProactorEventLoop()
        asyncio.set_event_loop(loop)

with: loop = asyncio.get_event_loop()

With the current configuratioon it raises:

Exception ignored in: <bound method DNSResolver._sock_state_cb of <aiodns.DNSResolver object at 0x0616F830>>
Traceback (most recent call last):
  File "USER\AppData\Local\Programs\Python\Python36-32\lib\site-packages\aiodns\__init__.py", line 85, in _sock_state_cb
    self.loop.add_reader(fd, self._handle_event, fd, READ)
  File "USER\AppData\Local\Programs\Python\Python36-32\lib\asyncio\events.py", line 453, in add_reader
    raise NotImplementedError
NotImplementedError:

Note my direction path has been manually changed to USER

Python documentation says: https://docs.python.org/3/library/asyncio-eventloops.html#asyncio.ProactorEventLoop

add_reader() and add_writer() only accept file descriptors of sockets

Is aiodns not supported with ProactorEventLoop? Is this some type of weird bug? Is aiodns fully supported on Windows?

I can provide more info, but in case you need a little bit more background I've been derived here by @asvetlov in the following stack overflow question: https://stackoverflow.com/questions/47675410/python-asyncio-aiohttp-valueerror-too-many-file-descriptors-in-select-on-win

Python 3.4.x is unable to build / install the latest version of this library due to incompatiable unicode used in the authors name/email field. Removal of this line or usage of conversion tools such as iconv fix the issue.

I'm not sure if the problem is in aiodns, pycares or even Python itself, but to start off this seems like a good place to put this, especially if it turns out that the issue is just me doing something very very wrong.

I'm trying to look up all IP addresses belonging to a certain hostname. Due to load balancing techniques different DNS servers may report different IP addresses, which means that I first need to find a list of public DNS servers, and then query each one with the hostname. From what I can tell, DNSResolver (or rather pycares) doesn't get very happy when you try to change the nameservers from different simultaneously running coroutines (not very surprising really), so separate DNSResolver instances are needed.

Here's an example of what I'm trying to do, using aiodns in conjunction with aiohttp to fetch the DNS server list (note that it may take several seconds to fetch the DNS list):

import asyncio
import json

import aiodns
import aiohttp

json_url = "http://public-dns.tk/nameservers.json"
address = "google.com"

@asyncio.coroutine
def main():
    try:
        print("Fetching DNS server list...")
        response = yield from asyncio.wait_for(
            aiohttp.request('GET', json_url), 30)
    except asyncio.TimeoutError:
        print("Error: Couldn't fetch DNS server list")
        return

    body = yield from response.read_and_close()
    dns_data = json.loads(body.decode('utf-8'))
    dns_servers = [entry['ip'] for entry in dns_data
                   if entry['state'] == 'valid'
                      and len(entry['ip'].split('.')) == 4]

    ips = set()
    i = 0
    @asyncio.coroutine
    def do_lookup(ip):
        nonlocal i
        try:
            resolver = aiodns.DNSResolver(nameservers=[ip],
                                          timeout=3, tries=2)
            ips.update((yield from resolver.query(address, 'A')))
        except Exception as e:
            print("Warning: Couldn't connect to DNS server {}"
                    .format(ip))
        i += 1
        print("Queried DNS server {}/{}".format(i, len(dns_servers)))

    print("Resolving IP addresses...")
    yield from asyncio.wait([do_lookup(server)
                             for server in dns_servers])
    print("Got IP addresses:", ips)


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

When running the above code I get the following output:

Fetching DNS server list...
Resolving IP addresses...
Warning: Couldn't connect to DNS server 85.185.171.82
Queried DNS server 1/3130
[...]
Queried DNS server 2205/3130
Queried DNS server 2206/3130
Queried DNS server 2207/3130
Queried DNS server 2208/3130
Queried DNS server 2209/3130
[...]
Warning: Couldn't connect to DNS server 207.172.11.73
Queried DNS server 3129/3130
Warning: Couldn't connect to DNS server 83.238.39.254
Queried DNS server 3130/3130
Got IP addresses: {'212.39.82.177', '74.125.239.99', [...], '173.194.124.39', '93.191.15.103'}
Segmentation fault

Backtrace:

Program received signal SIGSEGV, Segmentation fault.
visit_decref (op=0x7ffff017c370, data=0x0) at Modules/gcmodule.c:373
373     Modules/gcmodule.c: No such file or directory.
(gdb) bt
#0  visit_decref (op=0x7ffff017c370, data=0x0) at Modules/gcmodule.c:373
#1  0x00007ffff79d279b in list_traverse (o=0x7ffff2b29e48, visit=0x7ffff7a95ee0 <visit_decref>, arg=0x0)
    at Objects/listobject.c:2217
#2  0x00007ffff7a951bf in subtract_refs (containers=<optimized out>) at Modules/gcmodule.c:398
#3  collect ([email protected]=2, [email protected]=0x0, 
    [email protected]=0x0, [email protected]=1) at Modules/gcmodule.c:969
#4  0x00007ffff7a96301 in _PyGC_CollectNoFail () at Modules/gcmodule.c:1638
#5  0x00007ffff7a705f8 in PyImport_Cleanup () at Python/import.c:483
#6  0x00007ffff7a7cd16 in Py_Finalize () at Python/pythonrun.c:616
#7  0x00007ffff7a9408f in Py_Main (argc=-136464988, argv=0x0) at Modules/main.c:771
#8  0x0000000000400af6 in main ()

I'm using aiodns 0.3.0 with Python 3.4.1 running on 64-bit (Arch) Linux.

Env

Win10 19.09 x64 Python 3.8.0

Code

import aiodns
import asyncio

async def query_a(name):
    resolver = aiodns.DNSResolver()
    return await resolver.query(name, "A")

if __name__ == '__main__':
    res = asyncio.run(query_a('qq.com'), debug=True)
    pass

Error

Connected to pydev debugger (build 193.5662.61)
From cffi callback <function _sock_state_cb at 0x000001C25B3C3430>:
Traceback (most recent call last):
  File "C:\Users\test\.virtualenvs\OneForAll-qYrK1GZO\lib\site-packages\pycares\__init__.py", line 91, in _sock_state_cb
    sock_state_cb(socket_fd, readable, writable)
  File "C:\Users\test\.virtualenvs\OneForAll-qYrK1GZO\lib\site-packages\aiodns\__init__.py", line 104, in _sock_state_cb
    self.loop.add_reader(fd, self._handle_event, fd, READ)
  File "C:\Program Files\Python38\Lib\asyncio\events.py", line 501, in add_reader
    raise NotImplementedError
NotImplementedError

I don't know if this is a limitation of pycares, or if I'm missing something obvious.

Pycares seems to automatically follow CNAME entries to resolve the hostname to an IP address, but gives no indication of when it has done so. Is there any way to tell when this has happened, and if so what the CNAME was?

Thanks very much.

Not really sure if it's an issue with aiodns or pycares. I'm using aiodns to resolve ~300k domains/hour and it works great most of the time, but I'm getting "Fatal Python error: Segmentation fault" from time to time, and it doesn't seem to be related to the load.

Fatal Python error: Segmentation fault
 Current thread 0x00007f23a2ffd700 (most recent call first):
   File "/task/python3/lib/python3.7/site-packages/pycares/__init__.py", line 519 in _do_query
   File "/task/python3/lib/python3.7/site-packages/pycares/__init__.py", line 505 in query
   File "/task/python3/lib/python3.7/site-packages/aiodns/__init__.py", line 79 in query
   File "/task/batch/ResolverBatch.py", line 24 in query
   File "/usr/lib/python3.7/asyncio/events.py", line 88 in _run
   File "/usr/lib/python3.7/asyncio/base_events.py", line 1775 in _run_once
   File "/usr/lib/python3.7/asyncio/base_events.py", line 539 in run_forever
   File "/task/batch/ResolverBatch.py", line 29 in resolver_worker
   File "/usr/lib/python3.7/threading.py", line 865 in run
   File "/usr/lib/python3.7/threading.py", line 917 in _bootstrap_inner
   File "/usr/lib/python3.7/threading.py", line 885 in _bootstrap

Asyncio loop is running in a separate thread and queries are executed using asyncio.run_coroutine_threadsafe(query(domain, 'A'), loop):

loop = asyncio.get_event_loop()
resolver = DNSResolver(loop=loop, nameservers=pDNS)

async def query(name, query_type):
    return await resolver.query(name, query_type)

def resolver_worker(loop):
    """ Switch to new event loop and run forever """
    asyncio.set_event_loop(loop)
    loop.run_forever()

def start_worker():
    """ Start worker thread """
    log.info(f"Starting resolver thread")
    worker = Thread(target=resolver_worker, args=(loop, ))
    worker.setDaemon(True)
    worker.start()

def run_resolver(testing=False):
    log.info(f"Starting Resolver with nameserver {pDNS}")
    start_worker()
    queue = Queue('queue'))
    for domains in queue.recv(forever=True):
        for domain in domains:
            future = asyncio.run_coroutine_threadsafe(query(domain, 'A'), loop)
            if testing:
                return future.result()

Package versions:

        "aiodns": {
            "hashes": [
                "sha256:815fdef4607474295d68da46978a54481dd1e7be153c7d60f9e72773cd38d77d",
                "sha256:aaa5ac584f40fe778013df0aa6544bf157799bd3f608364b451840ed2c8688de"
            ],
            "index": "pypi",
            "version": "==2.0.0"
        },
        "asyncio": {
            "hashes": [
                "sha256:83360ff8bc97980e4ff25c964c7bd3923d333d177aa4f7fb736b019f26c7cb41",
                "sha256:b62c9157d36187eca799c378e572c969f0da87cd5fc42ca372d92cdb06e7e1de",
                "sha256:c46a87b48213d7464f22d9a497b9eef8c1928b68320a2fa94240f969f6fec08c",
                "sha256:c4d18b22701821de07bd6aea8b53d21449ec0ec5680645e5317062ea21817d2d"
            ],
            "index": "pypi",
            "version": "==3.4.3"
        "pycares": {
            "hashes": [
                "sha256:2ca080db265ea238dc45f997f94effb62b979a617569889e265c26a839ed6305",
                "sha256:6f79c6afb6ce603009db2042fddc2e348ad093ece9784cbe2daa809499871a23",
                "sha256:70918d06eb0603016d37092a5f2c0228509eb4e6c5a3faacb4184f6ab7be7650",
                "sha256:755187d28d24a9ea63aa2b4c0638be31d65fbf7f0ce16d41261b9f8cb55a1b99",
                "sha256:7baa4b1f2146eb8423ff8303ebde3a20fb444a60db761fba0430d104fe35ddbf",
                "sha256:90b27d4df86395f465a171386bc341098d6d47b65944df46518814ae298f6cc6",
                "sha256:9e090dd6b2afa65cb51c133883b2bf2240fd0f717b130b0048714b33fb0f47ce",
                "sha256:a11b7d63c3718775f6e805d6464cb10943780395ab042c7e5a0a7a9f612735dd",
                "sha256:b253f5dcaa0ac7076b79388a3ac80dd8f3bd979108f813baade40d3a9b8bf0bd",
                "sha256:c7f4f65e44ba35e35ad3febc844270665bba21cfb0fb7d749434e705b556e087",
                "sha256:cdb342e6a254f035bd976d95807a2184038fc088d957a5104dcaab8be602c093",
                "sha256:cf08e164f8bfb83b9fe633feb56f2754fae6baefcea663593794fa0518f8f98c",
                "sha256:df9bc694cf03673878ea8ce674082c5acd134991d64d6c306d4bd61c0c1df98f"
            ],
            "version": "==3.0.0"
        },```

The RFC1035 indicates that TXT records are "" which are to be interpreted as binary data of length at most 255, prefixed with a length character.

If I'm packing a TXT record containing a null byte, it is not correctly parsed by the library and the parsed output contains an empty buffer

aiodns version : v3.0.0 pycares version : v4.0.0

steps to reproduce :

python -c "import aiodns"

Exception :

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/aiodns/__init__.py", line 4, in <module>
    import pycares
  File "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pycares-4.0.0-py3.9-linux-aarch64.egg/pycares/__init__.py", line 2, in <module>
    from ._cares import ffi as _ffi, lib as _lib
ImportError: dlopen failed: cannot locate symbol "getservbyport_r" referenced by "/data/data/com.termux/files/usr/lib/python3.9/site-packages/pycares-4.0.0-py3.9-linux-aarch64.egg/pycares/_cares.cpython-39.so"...

Hello,

aiodns contains type hints, but code that uses aiodns does not benefit from having it typed, due to the lack of a "py.typed" file in the module root. This should be easily fixed by creating the file and adding it as package data in the setup script.

ref: https://www.python.org/dev/peps/pep-0561/

I love this library of all the ones i've tried it was the fastest easiest to use. I will say at first it took a bit to figure out how to best get all the results I wanted to reuse.

I think the examples would be super helpful for any people checking out the project and wanting something easy to use to see how they can do most anything easily.

The rdns/ptr lookups for example are glossed over on the examples but fully shown with functions for both using coroutines.

The domaindns.py has a really badass class I have been working on for my app which uses whois/rdap whois and aiodns. I have commented out the non aiodns stuff but left it in so people can see what's possible. If possible would be nice if this was included as some of these things will save people having to reinvent the wheel. For anyone else that already knows how to do this stuff it won't affect them negatively.

Keep up the great work with this library.

Examples:

From domaindns.py

print('')
print('DNS Records JSON:')
# print(json.dumps(domain_dict, default=str))
print(json.dumps(DomainInfo('google.com').domain_dict))

print('')
print('DNS Records JSON Pretty Print:')
print(json.dumps(DomainInfo('google.com').domain_dict, indent=4, sort_keys=False))

Output:

DNS Records JSON:
{"domain": "google.com", "DNS": {"SOA": {"nsname": "ns1.google.com", "hostmaster": "dns-admin.google.com", "serial": "343814713", "refresh": "900", "retry": "900", "expires": "1800", "minttl": "60", "ttl": "16"}, "NS": [["ns1.google.com", "216.239.32.10"], ["ns3.google.com", "216.239.36.10"], ["ns4.google.com", "216.239.38.10"], ["ns2.google.com", "216.239.34.10"]], "WWW": [["A", "www.google.com", "142.250.64.196"], ["A", "google.com", "142.250.64.238"], ["AAAA", "google.com", "2607:f8b0:4008:807::200e"]], "MX": [["MX", "alt3.aspmx.l.google.com", "40"], ["MX", "aspmx.l.google.com", "10"], ["MX", "alt4.aspmx.l.google.com", "50"], ["MX", "alt2.aspmx.l.google.com", "30"], ["MX", "alt1.aspmx.l.google.com", "20"]], "TXT": [["TXT", "_dmarc.google.com", "v=DMARC1; p=reject; rua=mailto:[email protected]"], ["TXT", "google.com", "v=spf1 include:_spf.google.com ~all"], ["TXT", "google.com", "globalsign-smime-dv=CDYX+XFHUw2wml6/Gb8+59BsH31KzUr6c1l2BPvqKX8="], ["TXT", "google.com", "docusign=05958488-4752-4ef2-95eb-aa7ba8a3bd0e"], ["TXT", "google.com", "docusign=1b0a6754-49b1-4db5-8540-d2c12664b289"], ["TXT", "google.com", "facebook-domain-verification=22rm551cu4k0ab0bxsw536tlds4h95"]]}}

DNS Records JSON Pretty Print:
{
    "domain": "google.com",
    "DNS": {
        "SOA": {
            "nsname": "ns1.google.com",
            "hostmaster": "dns-admin.google.com",
            "serial": "343814713",
            "refresh": "900",
            "retry": "900",
            "expires": "1800",
            "minttl": "60",
            "ttl": "15"
        },
        "NS": [
            [
                "ns2.google.com",
                "216.239.34.10"
            ],
            [
                "ns4.google.com",
                "216.239.38.10"
            ],
            [
                "ns3.google.com",
                "216.239.36.10"
            ],
            [
                "ns1.google.com",
                "216.239.32.10"
            ]
        ],
        "WWW": [
            [
                "A",
                "www.google.com",
                "142.250.64.196"
            ],
            [
                "A",
                "google.com",
                "142.250.64.238"
            ],
            [
                "AAAA",
                "google.com",
                "2607:f8b0:4008:807::200e"
            ]
        ],
        "MX": [
            [
                "MX",
                "alt1.aspmx.l.google.com",
                "20"
            ],
            [
                "MX",
                "alt2.aspmx.l.google.com",
                "30"
            ],
            [
                "MX",
                "alt4.aspmx.l.google.com",
                "50"
            ],
            [
                "MX",
                "aspmx.l.google.com",
                "10"
            ],
            [
                "MX",
                "alt3.aspmx.l.google.com",
                "40"
            ]
        ],
        "TXT": [
            [
                "TXT",
                "_dmarc.google.com",
                "v=DMARC1; p=reject; rua=mailto:mai[email protected]"
            ],
            [
                "TXT",
                "google.com",
                "facebook-domain-verification=22rm551cu4k0ab0bxsw536tlds4h95"
            ],
            [
                "TXT",
                "google.com",
                "docusign=1b0a6754-49b1-4db5-8540-d2c12664b289"
            ],
            [
                "TXT",
                "google.com",
                "docusign=05958488-4752-4ef2-95eb-aa7ba8a3bd0e"
            ],
            [
                "TXT",
                "google.com",
                "globalsign-smime-dv=CDYX+XFHUw2wml6/Gb8+59BsH31KzUr6c1l2BPvqKX8="
            ],
            [
                "TXT",
                "google.com",
                "v=spf1 include:_spf.google.com ~all"
            ]
        ]
    }
}