Thanks for publishing this.

I see the release process is fairly automated, is it possible to use CI to automatically build binaries of this, and publish them as releases? I don't have experience in Github Actions, but I do in Gitlab CI. Both containers take 3 GB, but the binaries themselves are only ~7 MB.

Another interesting build artefact would be to "bake in" the settings in curl_ff95 script in the build itself, and create a regular curl shared library. This could replace with the upstream libcurl file to use the new settings, or hacked around with LD_LIBRARY_PATH. It would need to be a different build per browser.

Now curl is built in static mode. The library itself is in lib/.libs/libcurl.a.

Safari being the second most used desktop browser according to some websites, it can be a good candidate for impersonation as well. I don't have access to a Mac right now.. Is anyone willing to share a Wireshark capture of a TLS session from Safari ? Bonus if it's HTTP/2 and if it can be decrypted as well (you can set up a local nginx with a self signed key for this).

When using libcurl and reusing the same connection, if I set the "Host:" header on the connection, and reuse it to make a request without the host header, the header is still included with the same value

<?php
putenv('CURL_IMPERSONATE=chrome98');
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://headers.cf');
curl_setopt($ch, CURLINFO_HEADER_OUT, 1);
curl_setopt( $ch, CURLOPT_ENCODING, "" );
curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, false );
curl_setopt( $ch, CURLOPT_ENCODING, "" );
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, true );
curl_setopt( $ch, CURLOPT_HTTPHEADER, ['Host: abc.com']);
curl_setopt( $ch, CURLOPT_AUTOREFERER, true );

curl_exec($ch);
print_r(curl_getinfo($ch));

//curl_reset($ch);
curl_setopt($ch, CURLOPT_URL, 'https://headers.cf');
curl_setopt( $ch, CURLOPT_HTTPHEADER, ['connection: Keep-Alive']); // i didn't set "host:" there



echo curl_exec($ch);
print_r(curl_getinfo($ch));

ON the first request, this is what is sent

GET / HTTP/1.1
Host: abc.com <--- notice this
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="98", "Google Chrome";v="98"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9

and this is sent on the second request

GET / HTTP/1.1
Host: abc.com <--- this is incorrect
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="98", "Google Chrome";v="98"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
connection: Keep-Alive

if i remove the line putenv('CURL_IMPERSONATE=chrome98');, everything works fine : first request :

GET / HTTP/1.1
Host: abc.com <-- notice this
Accept: */*
Accept-Encoding: deflate, gzip, br

second request

GET / HTTP/1.1
Host: headers.cf <--- this is correct this time
Accept: */*
Accept-Encoding: deflate, gzip, br
connection: Keep-Alive

I'm using libcurl and php for curl-impersonate, i'm setting the ciphers, the ssl version, disable NPN, but i'm still missing the extensions 27 (compress_certificate) and 17513 (extensionApplicationSettings (boringssl))

I can't set the parameters "--alps" and "--cert-compression brotli" using libcurl and PHP. The curl php extension doesn't support new parameters names (see https://github.com/php/php-src/blob/master/ext/curl/interface.c#L2306 ) What should I do to enable those 2 extensions ?

edit: calling putenv('CURL_IMPERSONATE=chrome98'); or CURL_IMPERSONATE=chrome98 php /path/to/script.php beforehand doesn't help, the two tls extensions are still not enabled.

My guess is that that both "alps" and "cert-compression" have not been yet added to the default parameters on curl-impersonate for libcurl.

Using CURL_IMPERSONATE=chrome98 curl "http://...." is correctly adding the 2 extensions however

I checked my laptop's Windows Chrome 103 headers using your socat procedure, compared to the chrome101 curl-impersonate script's results I see extra headers for Connection:, Cache-Control:, and also DNT:. That last one might be due to my personal Chrome settings, not sure. Also the sec-ch-ua: string seems to be a bit different.

Chrome 103:

GET / HTTP/1.1\r
Host: xx.xx.xx.xx:8443\r
Connection: keep-alive\r
Cache-Control: max-age=0\r
sec-ch-ua: ".Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103"\r
sec-ch-ua-mobile: ?0\r
sec-ch-ua-platform: "Windows"\r
DNT: 1\r
Upgrade-Insecure-Requests: 1\r
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36\r
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\r
Sec-Fetch-Site: none\r
Sec-Fetch-Mode: navigate\r
Sec-Fetch-User: ?1\r
Sec-Fetch-Dest: document\r
Accept-Encoding: gzip, deflate, br\r
Accept-Language: en-US,en;q=0.9\r

curl-impersonate chrome101:

GET / HTTP/1.1\r
Host: localhost:8443\r
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="101", "Google Chrome";v="101"\r
sec-ch-ua-mobile: ?0\r
sec-ch-ua-platform: "Windows"\r
Upgrade-Insecure-Requests: 1\r
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36\r
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\r
Sec-Fetch-Site: none\r
Sec-Fetch-Mode: navigate\r
Sec-Fetch-User: ?1\r
Sec-Fetch-Dest: document\r
Accept-Encoding: gzip, deflate, br\r
Accept-Language: en-US,en;q=0.9\r

Related question: I tried setting custom headers via node-libcurl in my program, by passing an array to my .get() method's 'HTTPHEADER' option, like below, but in the resulting request socat is showing the Connection: and Cache-Control: headers end up at the end, despite my array passing them first. Is this a curl bug? Anyway to override it to get the right order like Chrome 103?

let headers = [
    'Connection: keep-alive',
    'Cache-Control: max-age=0',
    'sec-ch-ua: ".Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103"',
    'sec-ch-ua-mobile: ?0',
    'sec-ch-ua-platform: "Windows"',
    'Upgrade-Insecure-Requests: 1',
    'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
    'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
    'Sec-Fetch-Site: none',
    'Sec-Fetch-Mode: navigate',
    'Sec-Fetch-User: ?1',
    'Sec-Fetch-Dest: document',
    'Accept-Encoding: gzip, deflate, br',
    'Accept-Language: en-US,en;q=0.9'
];
let response = await aCurly.get('https://localhost:8443/', { SSL_VERIFYPEER: false, SSL_VERIFYHOST: false, HTTPHEADER: headers });

socat result:

GET / HTTP/1.1\r
Host: localhost:8443\r
sec-ch-ua: ".Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103"\r
sec-ch-ua-mobile: ?0\r
sec-ch-ua-platform: "Windows"\r
Upgrade-Insecure-Requests: 1\r
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36\r
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\r
Sec-Fetch-Site: none\r
Sec-Fetch-Mode: navigate\r
Sec-Fetch-User: ?1\r
Sec-Fetch-Dest: document\r
Accept-Encoding: gzip, deflate, br\r
Accept-Language: en-US,en;q=0.9\r
Connection: keep-alive\r
Cache-Control: max-age=0\r

I tried editing the curl_chrome101 script to add these 2 headers in there, and same behavior: they end up at the bottom of the headers instead of top.

Hi, found this warning while running curl_ff95 (I guess happens on others too):

* WARNING: failed to load NSS PEM library libnsspem.so. Using OpenSSL PEM certificates will not work

Downloaded and ran the binary without installing anything else. Not sure if it affects anything.

thought this should be quite similar to build on Ubuntu, tried for hours but failed

my efforts include:

• install build prerequisites by: sudo yum groupinstall "Development Tools"
• upgrade cmake by: pip3 install cmake --upgrade
• create symbolic link for ninjia-build by: ln -s /usr/bin/ninja-build /usr/bin/ninja
• install nss by: sudo yum install libnss3.so libnssutil3.so
• add search path(/usr/lib/) for lib nss in curl-impersonate.patch#:

  -+          search_paths="/usr/lib/$host /usr/lib/$host/nss"
  ++          search_paths="/usr/lib/ /usr/lib/$host /usr/lib/$host/nss"
  

still failed when doing make firefox-build, the log suggested failed linking NSS:

configure: WARNING: Using hard-wired libraries and compilation flags for NSS.
checking if libnssckbi is in a non-standard location... /usr/lib
checking for SSL_VersionRangeSet in -lnss_static... no
configure: error: Failed linking NSS statically
Makefile:198: recipe for target 'curl-7.81.0/.firefox' failed
make: *** [curl-7.81.0/.firefox] Error 1

the related log in ./curl-7.81.0/config.log is:

...
configure:26797: checking if libnssckbi is in a non-standard location
configure:26815: result: /usr/lib
configure:26843: checking for SSL_VersionRangeSet in -lnss_static
configure:26865: gcc -o conftest -I/home/ec2-user/apps/curl-impersonate/buil
d/nss-3.75/dist/Release/../public/nss -I/home/ec2-user/apps/curl-impersonate
/build/nss-3.75/dist/Release/include/nspr -Werror-implicit-function-declarat
ion -O2 -Wno-system-headers    -I/home/ec2-user/apps/curl-impersonate/build/
brotli-1.0.9/out/installed/include -I/home/ec2-user/apps/curl-impersonate/bu
ild/nss-3.75/dist/Release/include -L/home/ec2-user/apps/curl-impersonate/bui
ld/nss-3.75/dist/Release/lib -Wl,-rpath,/usr/lib    -L/home/ec2-user/apps/cu
rl-impersonate/build/brotli-1.0.9/out/installed/lib conftest.c -lnss_static
 -Wl,--start-group -lssl -lnss_static -lpk11wrap_static -lcertdb -lcerthi -l
nsspki -lnssdev -lsoftokn_static -lfreebl_static -lnssutil -lnssb -lcryptohi
 -l:libplc4.a -l:libplds4.a -l:libnspr4.a -lsqlite -lgcm-aes-x86_c_lib -lhw-
acc-crypto-avx -lhw-acc-crypto-avx2 -lsha-x86_c_lib -lintel-gcm-wrap_c_lib -
lintel-gcm-s_lib -Wl,--end-group -pthread -ldl -lbrotlidec-static -lbrotlico
mmon-static -lz  >&5
/usr/bin/ld: cannot find -lbrotlidec-static
/usr/bin/ld: cannot find -lbrotlicommon-static
collect2: error: ld returned 1 exit status
configure:26865: $? = 1
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME "curl"
| #define PACKAGE_TARNAME "curl"
| #define PACKAGE_VERSION "-"
| #define PACKAGE_STRING "curl -"
| #define PACKAGE_BUGREPORT "a suitable curl mailing list: https://curl.se/mail/"
...

any help is appreciated

Is it possible to set custom values of Client Hello packet? Like with usage of curl_easy_impersonate set values of for example CURLOPT_SSL_CIPHER_LIST or CURLOPT_SSL_EC_CURVES

Thanks for publishing this useful tool.

Is it possible or any insights to run on Mac OS? currently it always throws error: ./curl_ff95 https://www.ha.com/c/search-results.zx ./curl_ff95: line 10: ./curl-impersonate: cannot execute binary file

1. Dockerfile :

FROM php:8.1.1-cli-buster

COPY --from=lwthiker/curl-impersonate:0.5-chrome /usr/local/bin/ /usr/local/bin/

2. docker build -t test .

3. docker run -it test bash

4. curl_chrome101

the following error will ocurrs:

/usr/bin/env: 'ash': No such file or directory

What is wrong here ?

It seems that something is potentially off with your generate_dockerfiles.sh script, as the non-Alpine images are, in fact, also Alpine:

[email protected]:~/rss-bridge/rss-bridge$ sudo docker run -it lwthiker/curl-impersonate:0.5.1-ff sh
/ # apk
apk-tools 2.12.7, compiled for x86_64.

usage: apk [<OPTIONS>...] COMMAND [<ARGUMENTS>...]

Package installation and removal:
  add        Add packages to WORLD and commit changes
  del        Remove packages from WORLD and commit changes

System maintenance:
  fix        Fix, reinstall or upgrade packages without modifying WORLD
  update     Update repository indexes
  upgrade    Install upgrades available from repositories
  cache      Manage the local package cache

Querying package information:
  info       Give detailed information about packages or repositories
  list       List packages matching a pattern or other criteria
  dot        Render dependencies as graphviz graphs
  policy     Show repository policy for packages
  search     Search for packages by name or description

Repository maintenance:
  index      Create repository index file from packages
  fetch      Download packages from global repositories to a local directory
  manifest   Show checksums of package contents
  verify     Verify package integrity and signature

Miscellaneous:
  audit      Audit system for changes
  stats      Show statistics about repositories and installations
  version    Compare package versions or perform tests on version strings

This apk has coffee making abilities.

Sir, I follow the guide: https://github.com/lwthiker/curl-impersonate/blob/main/INSTALL.md#macos to build source code on MacBook air - m2 chip. All is ok until running gmake, here is the logs:

> gmake firefox-build
.... some other logs

curl -L -o nss-3.77.tar.gz https://ftp.mozilla.org/pub/security/nss/releases/NSS_3_77_RTM/src/nss-3.77-with-nspr-4.32.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 82.2M  100 82.2M    0     0  1059k      0  0:01:19  0:01:19 --:--:-- 1226k
tar xf nss-3.77.tar.gz
# Native build, use NSS' build script.
cd nss-3.77/nss
./build.sh -o --disable-tests --static --python=python3
# Hack for macOS: Remove dynamic libraries to force the linker to use the
# static ones when linking curl.
rm -Rf /Users/ntopliu/workspace/GithubProjects/curl-impersonate/build/nss-3.77/dist/Release/lib/*.dylib
NSPR [1/5] configure ...
NSPR [2/5] make ...
NSPR [3/5] NOT building tests
NSPR [4/5] NOT running tests
NSPR [5/5] install ...
Building NSS requires an installation of gyp: https://gyp.gsrc.io/
gmake: *** [Makefile:211: /Users/ntop/workspace/GithubProjects/curl-impersonate/build/nss-3.77/dist/Release/lib/libnss_static.a] Error 3

I searched the directory nss-3.77/dist/Release/lib/ there is no libnss_static.a :

libnspr4.a     libnspr4.dylib libplc4.a      libplc4.dylib  libplds4.a     libplds4.dylib pkgconfig

I don't know what happened and how can I fix it.

I also installed the gyp.

pip3 install gyp-next
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: gyp-next in /Users/ntopliu/Library/Python/3.9/lib/python/site-packages (0.4.0)
WARNING: You are using pip version 21.2.4; however, version 22.3.1 is available.
You should consider upgrading via the '/Applications/Xcode.app/Contents/Developer/usr/bin/python3 -m pip install --upgrade pip' command.

Hi, why do we need the Accept header among the impersonated browser headers?

Also, sec-fetch-* and a few others change depending on how the request is made, so it doesn't make a lot of sense to keep them every requests. Why are they set for all requests?

Great project. I've had success in impersonating chrome 101. Are there are any API services available that offer this as a SaaS? I have tried a number of web scraping APIs and none of them quite match the capabilities of curl-impersonate.

See here:

https://groups.google.com/a/chromium.org/g/blink-dev/c/zdmNs2rTyVI/m/MAiQwQkwCAAJ

it's not clear to me if this is enabled on most installations yet, but there is a report here of some cases of encountering it:

https://github.com/refraction-networking/utls/issues/132

that project implemented a PR to address it:

https://github.com/refraction-networking/utls/pull/133

Something to keep an eye on apparently?

Hi, please help me make curl-impersonate to work via PyCurl. (I'm aware of #111)

I'm on debian bullseye, i had compile the library myself.

I had try:

export LD_LIBRARY_PATH=/usr/local/lib CURL_IMPERSONATE=ff102 test

Seems like the real libcurl.so get's loaded

export LD_PRELOAD=/usr/local/lib/libcurl-impersonate-ff.so

ImportError: pycurl: libcurl link-time ssl backends (nss) do not include compile-time ssl backend (gnutls)

cd pycurl export C_INCLUDE_PATH=$C_INCLUDE_PATH:/home/user/Documents/curl-impersonate/build/curl-7.84.0/include python3 setup.py install --curl-config=/usr/local/bin/curl-impersonate-ff-config

src/pycurl.h:33:10: fatal error: curl/curl.h: No such file or directory 33 | #include <curl/curl.h> | ^~~~~~~~~~~~~ compilation terminated. error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1

Currently no success.