浏览器提取url至Curl，Curl请求与Python request之间的转化

1. 背景

curl 是一个非常实用的、用来与服务器之间传输数据的工具。

curl支持的协议很多，包括 DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET and TFTP。curl设计为无用户交互下完成工作。

curl提供了一大堆非常有用的功能，包括代理访问、用户认证、ftp上传下载、HTTP POST、SSL连接、cookie支持、断点续传等等

使用curl的时候掌握一些小技巧可以加速工作速度。这里总结了几个技巧。

2.浏览器提取url至curl

f12- network，选择url右击，选择copy、Copy as cURL，即可复制为curl命令。

 复制出来的curl命令如下：

curl "https://www.baidu.com/" ^
  -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9" ^
  -H "Accept-Language: zh-CN,zh;q=0.9" ^
  -H "Connection: keep-alive" ^
  -H "Cookie: BIDUPSID=61EDAD335605FB5CA64D805B2003AD64; PSTM=1667138050; BAIDUID=61EDAD335605FB5C14F509C2EB71B594:FG=1; BD_UPN=12314753; BA_HECTOR=2la4a18k2581ah8k800kecvf1hlt0g21b; BAIDUID_BFESS=61EDAD335605FB5C14F509C2EB71B594:FG=1; ZFY=EsI6H59hbOhALYJbTN0ArAY0MCXO0ON2meAufYBBPjg:C; BDORZ=B490B5EBF6F3CD402E515D22BCDA1598; COOKIE_SESSION=73907_0_0_0_3_0_1_0_0_0_0_0_73908_0_2_0_1667211968_0_1667211966^%^7C2^%^230_0_1667211966^%^7C1; BD_HOME=1; H_PS_PSSID=36546_37584_36885_36789_37539_37575_26350; delPer=0; BD_CK_SAM=1; PSINO=2; H_PS_645EC=ef46AWOvUKTVSL89g^%^2FMxUiCAJn5hW2VO53FKWV0jrkHNjitkOGn7I1Ty5x0; baikeVisitId=fb52826b-9f53-43ce-8656-eaaa3d5bbe5b; shifen^[1674610128_57943^]=1667218702; BCLID=9341060554169253781; BCLID_BFESS=9341060554169253781; BDSFRCVID=mEuOJeCmHRh4nBOjVz8WhdmKOeKK0gOTHlln3wqXrK95vw0VJeC6EG0Ptf8g0KubFTPRogKK0gOTH6KF_2uxOjjg8UtVJeC6EG0Ptf8g0M5; BDSFRCVID_BFESS=mEuOJeCmHRh4nBOjVz8WhdmKOeKK0gOTHlln3wqXrK95vw0VJeC6EG0Ptf8g0KubFTPRogKK0gOTH6KF_2uxOjjg8UtVJeC6EG0Ptf8g0M5; H_BDCLCKID_SF=tJkD_I_hJKt3fP36q6_a2-F_2xQ0etJXfKj4Ll7F5lOVOModj-TIXPIZ2G7UaP3LW2O-ahkM5h7xObR1h6bbKqkXyNut-J-jQeQ-5KQN3KJmfbL9bT3v5tDz3b3N2-biWbRM2MbdJqvP_IoG2Mn8M4bb3qOpBtQmJeTxoUJ25DnJhhCGe4bK-TrLjGtHqUK; H_BDCLCKID_SF_BFESS=tJkD_I_hJKt3fP36q6_a2-F_2xQ0etJXfKj4Ll7F5lOVOModj-TIXPIZ2G7UaP3LW2O-ahkM5h7xObR1h6bbKqkXyNut-J-jQeQ-5KQN3KJmfbL9bT3v5tDz3b3N2-biWbRM2MbdJqvP_IoG2Mn8M4bb3qOpBtQmJeTxoUJ25DnJhhCGe4bK-TrLjGtHqUK; Hm_lvt_aec699bb6442ba076c8981c6dc490771=1667218705; Hm_lpvt_aec699bb6442ba076c8981c6dc490771=1667218705; WWW_ST=1667221069080" ^
  -H "Referer: https://www.baidu.com/" ^
  -H "Sec-Fetch-Dest: document" ^
  -H "Sec-Fetch-Mode: navigate" ^
  -H "Sec-Fetch-Site: same-origin" ^
  -H "Sec-Fetch-User: ?1" ^
  -H "Upgrade-Insecure-Requests: 1" ^
  -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36" ^
  -H "sec-ch-ua: ^\^"Google Chrome^\^";v=^\^"107^\^", ^\^"Chromium^\^";v=^\^"107^\^", ^\^"Not=A?Brand^\^";v=^\^"24^\^"" ^
  -H "sec-ch-ua-mobile: ?0" ^
  -H "sec-ch-ua-platform: ^\^"Windows^\^"" ^
  --compressed

3. curl命令转为python request

推荐一个在线转换的网址

Convert curl commands to code

直接输入curl命令就可以转换成功

4. python request转为curl命令

从pyhthon导入一个叫curlify的库，可以完成转化

import curlify
import requests
 
 
data = {
    "wd": "hello"
}
 
url = 'http://www.baidu.com/s'
 
response = requests.get(url=url, data=data, headers={
    'Accept': 'text/html,application/xhtml+xml',
})
 
print(curlify.to_curl(response.request))

执行结果

$ python3 a.py
curl -X GET -H 'Accept: text/html,application/xhtml+xml' -H 'Accept-Encoding: gzip, deflate' -H 'Connection: keep-alive' -H 'Content-Length: 8' -H 'Content-Type: application/x-www-form-urlencoded' -H 'User-Agent: python-requests/2.25.1' -d wd=hello http://www.baidu.com/s

5.最后

本文讲了浏览器提取url至Curl，Curl请求与Python request之间的转化的方法，如果你想知道很多curl命令有关的信息可以查看以下资料

Linux curl命令最全详解_Angel_CG的博客-CSDN博客_curl