Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

默认并未使用http2.0 #20

Open
dream2333 opened this issue Apr 19, 2023 · 7 comments
Open

默认并未使用http2.0 #20

dream2333 opened this issue Apr 19, 2023 · 7 comments

Comments

@dream2333
Copy link

如图,请求抖音,图一python,图二go,并没有发现http2协商过程,有没有一种方案可以强制设置http2.0
image
image

@wangluozhe
Copy link
Owner

可以附上代码吗?涉及私密的地方可以用*代替,但请不要忽略不写。

@dream2333
Copy link
Author

可以附上代码吗?涉及私密的地方可以用*代替,但请不要忽略不写

代码比较复杂,涉及到逆向,抖音网页版全站都是http2.0的,您可以试一下,抓包出来的结果是http1.1的

@dream2333
Copy link
Author

dream2333 commented Apr 24, 2023

可以附上代码吗?涉及私密的地方可以用*代替,但请不要忽略不写。

rawurl := "https://www.douyin.com/aweme/v1/web/aweme/post/?"
ori_params := device_platform=webapp&aid=6383&channel=channel_pc_web&sec_user_id=MS4wLjABAAAA_nwyX0DfxMKDTbjs7GpTBjbDsKnN9eszwAa7r8MthWM&max_cursor=1635588600000&locate_query=false&show_live_replay_strategy=1&count=10&publish_video_strategy_type=2&pc_client_type=1&version_code=170400&version_name=17.4.0&cookie_enabled=true&screen_width=1920&screen_height=1080&browser_language=zh-CN&browser_platform=Linux+x86_64&browser_name=Edge&browser_version=112.0.1722.34&browser_online=true&engine_name=Blink&engine_version=112.0.0.0&os_name=Linux&os_version=x86_64&cpu_core_num=12&device_memory=8&platform=PC&downlink=10&effective_type=4g&round_trip_time=150&webid=7221748043930371587&msToken=ibF7wcwyHIkkMdztr4IJ_GAr9gwU8pOqx6O0Ond4m2io0U7ezpzT2NA7TFR8EiVIM5JQUoqLLV4CFLJ3LH7vdOwPc5KCkPaL2yLqhjDrMao9sL2hgaWwZX0Y4qtPmwaNRQ==&X-Bogus=DFSzswSLFcsANCIIteg2Az9WcBjW HTTP/1.1
headers := url.NewHeaders()
headers.Set("accept", "my-app/0.0.1")
headers.Set("user-agent", "application/json, text/plain, /")
headers.Set("accept-language", "zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6")
headers.Set("cookie", "s_v_web_id=verify_lgg0huen_daX7wtLx_0tle_4eit_8ovH_Dg65iBlNNvDo; _tea_utm_cache_2018=undefined; ttwid=1%7C1w5kilF7Hm3tpF_ulVKyASmNqAIf3NFZt_E2_l9lA7I%7C1681444261%7Cbf6c415ec2fb9ad4f9d986d20beadd9e42e84f992935655545db65f45fdd3193; passport_csrf_token=fe226ac31e4cb9a8fb3778722671d2fe; passport_csrf_token_default=fe226ac31e4cb9a8fb3778722671d2fe; ttcid=a308c7ee6b7f4e37a4fdd482d4b0194e34; download_guide=%223%2F20230414%22; douyin.com; bd_ticket_guard_client_data=eyJiZC10aWNrZXQtZ3VhcmQtdmVyc2lvbiI6MiwiYmQtdGlja2V0LWd1YXJkLWl0ZXJhdGlvbi12ZXJzaW9uIjoxLCJiZC10aWNrZXQtZ3VhcmQtY2xpZW50LWNzciI6Ii0tLS0tQkVHSU4gQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tXHJcbk1JSUJEVENCdFFJQkFEQW5NUXN3Q1FZRFZRUUdFd0pEVGpFWU1CWUdBMVVFQXd3UFltUmZkR2xqYTJWMFgyZDFcclxuWVhKa01Ga3dFd1lIS29aSXpqMENBUVlJS29aSXpqMERBUWNEUWdBRUMyY1VncW1iYWJDemxXYmFuRXMzd3A5MlxyXG5yNHJLMmlLNEUyZlB3czhmUGxlV2VVWEhIWHRhSXRDODhZdEQ4M1cwNU5OZUdjck1RK09EMWhIWkRQcVc3NkFzXHJcbk1Db0dDU3FHU0liM0RRRUpEakVkTUJzd0dRWURWUjBSQkJJd0VJSU9kM2QzTG1SdmRYbHBiaTVqYjIwd0NnWUlcclxuS29aSXpqMEVBd0lEUndBd1JBSWdGMzhEb1dQSjAwcDZpeTEwU204TStqSkgvaGRMOUJkb21OZzBzZDROSk9nQ1xyXG5JRnRMRkt2Q3BIemkycW45TXFFR0JvSUtucVQxNW9NY2NEQ3ZseGMzdTl5MFxyXG4tLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS1cclxuIn0=; csrf_session_id=88788e107da38c65034fff85f99f331e; strategyABtestKey=%221681868544.997%22; __ac_nonce=0643f544d0011ca9a247c; __ac_signature=_02B4Z6wo00f01F4Nj3gAAIDBcMxlCW6Av5ReLYvAAHPD5BBoxE0z4V0Z7pI8IF-xzsFU60Yd7VZa4CapVmH.t6naIU3HYX8D5YbnPoVAGpEIFIyT0BkLZwnrXP2vC9oW5Md7skqd6tvrdAqT2a; VIDEO_FILTER_MEMO_SELECT=%7B%22expireTime%22%3A1682476749862%2C%22type%22%3A1%7D; msToken=a0B1-QoQ0DyNHbn6ovUVO3a4zMbX5NfvOva82_GHbkCTnc7_yd9iVXXhCwaITyylRZWluTgVCfj-ktwitx0yTBJLS6gKJvF9mdz0IuB2vK-UQ3PrAuvJZMdX4efkMv824A==; tt_scid=neXSttxYg6FHT7IBM58VjP9IwK8qP8gAO1h77BU5zHWoRyPGwsvrIMK1sgOOfOmh4c82; pwa2=%223%7C0%22; home_can_add_dy_2_desktop=%221%22; msToken=ibF7wcwyHIkkMdztr4IJ_GAr9gwU8pOqx6O0Ond4m2io0U7ezpzT2NA7TFR8EiVIM5JQUoqLLV4CFLJ3LH7vdOwPc5KCkPaL2yLqhjDrMao9sL2hgaWwZX0Y4qtPmwaNRQ==")
headers.Set("referer", "https://www.douyin.com/user/MS4wLjABAAAA_nwyX0DfxMKDTbjs7GpTBjbDsKnN9eszwAa7r8MthWM")
headers.Set("user-agent", ua)
req := url.NewRequest()
req.Headers = headers
// req.Proxies = "http://127.0.0.1:8888"
// req.Ja3 = "771,4865-4866-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,45-0-10-17513-16-35-18-43-51-23-13-65281-27-11-5-21,29-23-24,0"
res,err:=requests.Get(rawurl+ori_params,req)

这里是测试代码

@wangluozhe
Copy link
Owner

新版本已更新,请你更新到1.1.0以后尝试一下看看

@dream2333
Copy link
Author

dream2333 commented May 14, 2023

新版本已更新,请你更新到1.1.0以后尝试一下看看

依旧不行,这里是测试代码,TestRequest是使用requests请求,TestHTTP是使用net/HTTP请求,net/HTTP正常而requests库失败,请求的页面为抖音网页版用户已发布的视频列表页

`package main

import (
"fmt"
"github.com/wangluozhe/requests"
"github.com/wangluozhe/requests/url"
"io"
"log"
"net/http"
)

func TestRequest() {
rawurl := "https://www.douyin.com/aweme/v1/web/aweme/post/?device_platform=webapp&aid=6383&channel=channel_pc_web&sec_user_id=MS4wLjABAAAACX0jKoZARKXONKiywn2yeDglLJ8oKoU5E_iPLflzKqM&max_cursor=1676028071000&locate_query=false&show_live_replay_strategy=1&count=10&publish_video_strategy_type=2&pc_client_type=1&version_code=170400&version_name=17.4.0&cookie_enabled=true&screen_width=2560&screen_height=1440&browser_language=zh-CN&browser_platform=Win32&browser_name=Edge&browser_version=113.0.1774.42&browser_online=true&engine_name=Blink&engine_version=113.0.0.0&os_name=Windows&os_version=10&cpu_core_num=12&device_memory=8&platform=PC&downlink=10&effective_type=4g&round_trip_time=50&webid=7214076632230282789&msToken=PJkiDtDJPv_gkTg0Lxkoo2yQfJOQP_CFLO0s0MwZdVj5Bt9p8ye7i22qXvlAjX3fMeQC053OkJR8ukzJZB1TS1yeIWIskE0SPtE02jE4YcYlpwqMhLxXVV6488dR6s8=&X-Bogus=DFSzswVYQOvANamNttwrDToB6lQc"
headers := url.NewHeaders()
headers.Set("authority", "www.douyin.com")
headers.Set("accept", "application/json, text/plain, /")
headers.Set("accept-language", "zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6")
headers.Set("cookie", "ttwid=1%7Cwkw7ngkVUZ3CuQne2ijSq3yEOofa-kCE_rpwkDChoHQ%7C1679658117%7C91361174e5a36c322ebcdf8c06f4316645139f0e804fa3db01e41d154995c6ec; passport_csrf_token=18cc42990eee1bfe86924b16e55bddf1; passport_csrf_token_default=18cc42990eee1bfe86924b16e55bddf1; s_v_web_id=verify_lflzxgmo_ZGYZwYa7_rHkq_4laW_BZ7u_Ooj719O2pU4P; xgplayer_user_id=295352157608; pwa2=%223%7C0%22; download_guide=%223%2F20230507%22; __ac_nonce=0646077350040fab2831c; __ac_signature=_02B4Z6wo00f01K3IqkgAAIDBvZZPcVNWNbit6K7AAE8XeVBcDAO-s1c1w4CMvfO6J.i7i.RmoCXC3-DMQ5DsjIspI69eooizvceFOPgmJZc0M2IY23AZwmd72QWRMVEJyhoY4PApw2AYnl9M72; douyin.com; strategyABtestKey=%221684043574.82%22; VIDEO_FILTER_MEMO_SELECT=%7B%22expireTime%22%3A1684648374831%2C%22type%22%3A1%7D; csrf_session_id=9e4805e3154a3ebc1e3c7737c533f2d6; tt_scid=OrOeB1qYNNIVw1GQfhLH6Nb87zdxWZoQJbqUR-1S7BCfsWPDD5JxuGP99NA3f5z0eaba; bd_ticket_guard_client_data=eyJiZC10aWNrZXQtZ3VhcmQtdmVyc2lvbiI6MiwiYmQtdGlja2V0LWd1YXJkLWl0ZXJhdGlvbi12ZXJzaW9uIjoxLCJiZC10aWNrZXQtZ3VhcmQtY2xpZW50LWNzciI6Ii0tLS0tQkVHSU4gQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tXHJcbk1JSUJEVENCdFFJQkFEQW5NUXN3Q1FZRFZRUUdFd0pEVGpFWU1CWUdBMVVFQXd3UFltUmZkR2xqYTJWMFgyZDFcclxuWVhKa01Ga3dFd1lIS29aSXpqMENBUVlJS29aSXpqMERBUWNEUWdBRXJwUVQyTm5rYkJQeGNteXk2NTZxU29PRFxyXG5LTk51WkdTamIxR0lJWjcrZXo0NFBnQ3B0R1FsUlBSSTBGTTZ1Y1FpZmd6ZlFLRm9SaEtCSC9oSUR6VDJ5NkFzXHJcbk1Db0dDU3FHU0liM0RRRUpEakVkTUJzd0dRWURWUjBSQkJJd0VJSU9kM2QzTG1SdmRYbHBiaTVqYjIwd0NnWUlcclxuS29aSXpqMEVBd0lEUndBd1JBSWdGY2NhSittdTQyT0tsNVlNUDA2bGtYR3MwaDFLUi96VElDRGhhY01GQUhzQ1xyXG5JQ2sxWXVXc08wemlwTlRuVi9Vc3A5cDI3Wnc2QlNJRmFUcGlFYjhRK2RrVVxyXG4tLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS1cclxuIn0=; msToken=fuXeZhpRb65tedIa3jLQjkn6ZM2lxDVsB5bEkq5qGoJvql34gdEr_DduzpxOxacWDnV3YqPJn5bitDYs7LMevFoq6Tt7jWVGPimBbTKjrYC_a5rB7ccP2NthLP6smKI=; home_can_add_dy_2_desktop=%221%22; msToken=PJkiDtDJPv_gkTg0Lxkoo2yQfJOQP_CFLO0s0MwZdVj5Bt9p8ye7i22qXvlAjX3fMeQC053OkJR8ukzJZB1TS1yeIWIskE0SPtE02jE4YcYlpwqMhLxXVV6488dR6s8=")
headers.Set("referer", "https://www.douyin.com/user/MS4wLjABAAAACX0jKoZARKXONKiywn2yeDglLJ8oKoU5E_iPLflzKqM")
headers.Set("sec-ch-ua", "Microsoft Edge";v="113", "Chromium";v="113", "Not-A.Brand";v="24")
headers.Set("sec-ch-ua-mobile", "?0")
headers.Set("sec-ch-ua-platform", "Windows")
headers.Set("sec-fetch-dest", "empty")
headers.Set("sec-fetch-mode", "cors")
headers.Set("sec-fetch-site", "same-origin")
headers.Set("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36 Edg/113.0.1774.42")
req := url.NewRequest()
req.Headers = headers
res, _ := requests.Get(rawurl, req)
fmt.Println(res.Text)
}

func TestHTTP() {
client := &http.Client{}
req, _ := http.NewRequest("GET", "https://www.douyin.com/aweme/v1/web/aweme/post/?device_platform=webapp&aid=6383&channel=channel_pc_web&sec_user_id=MS4wLjABAAAACX0jKoZARKXONKiywn2yeDglLJ8oKoU5E_iPLflzKqM&max_cursor=1676028071000&locate_query=false&show_live_replay_strategy=1&count=10&publish_video_strategy_type=2&pc_client_type=1&version_code=170400&version_name=17.4.0&cookie_enabled=true&screen_width=2560&screen_height=1440&browser_language=zh-CN&browser_platform=Win32&browser_name=Edge&browser_version=113.0.1774.42&browser_online=true&engine_name=Blink&engine_version=113.0.0.0&os_name=Windows&os_version=10&cpu_core_num=12&device_memory=8&platform=PC&downlink=10&effective_type=4g&round_trip_time=50&webid=7214076632230282789&msToken=PJkiDtDJPv_gkTg0Lxkoo2yQfJOQP_CFLO0s0MwZdVj5Bt9p8ye7i22qXvlAjX3fMeQC053OkJR8ukzJZB1TS1yeIWIskE0SPtE02jE4YcYlpwqMhLxXVV6488dR6s8=&X-Bogus=DFSzswVYQOvANamNttwrDToB6lQc", nil)
req.Header.Set("authority", "www.douyin.com")
req.Header.Set("accept", "application/json, text/plain, /")
req.Header.Set("accept-language", "zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6")
req.Header.Set("cookie", "ttwid=1%7Cwkw7ngkVUZ3CuQne2ijSq3yEOofa-kCE_rpwkDChoHQ%7C1679658117%7C91361174e5a36c322ebcdf8c06f4316645139f0e804fa3db01e41d154995c6ec; passport_csrf_token=18cc42990eee1bfe86924b16e55bddf1; passport_csrf_token_default=18cc42990eee1bfe86924b16e55bddf1; s_v_web_id=verify_lflzxgmo_ZGYZwYa7_rHkq_4laW_BZ7u_Ooj719O2pU4P; xgplayer_user_id=295352157608; pwa2=%223%7C0%22; download_guide=%223%2F20230507%22; __ac_nonce=0646077350040fab2831c; __ac_signature=_02B4Z6wo00f01K3IqkgAAIDBvZZPcVNWNbit6K7AAE8XeVBcDAO-s1c1w4CMvfO6J.i7i.RmoCXC3-DMQ5DsjIspI69eooizvceFOPgmJZc0M2IY23AZwmd72QWRMVEJyhoY4PApw2AYnl9M72; douyin.com; strategyABtestKey=%221684043574.82%22; VIDEO_FILTER_MEMO_SELECT=%7B%22expireTime%22%3A1684648374831%2C%22type%22%3A1%7D; csrf_session_id=9e4805e3154a3ebc1e3c7737c533f2d6; tt_scid=OrOeB1qYNNIVw1GQfhLH6Nb87zdxWZoQJbqUR-1S7BCfsWPDD5JxuGP99NA3f5z0eaba; bd_ticket_guard_client_data=eyJiZC10aWNrZXQtZ3VhcmQtdmVyc2lvbiI6MiwiYmQtdGlja2V0LWd1YXJkLWl0ZXJhdGlvbi12ZXJzaW9uIjoxLCJiZC10aWNrZXQtZ3VhcmQtY2xpZW50LWNzciI6Ii0tLS0tQkVHSU4gQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tXHJcbk1JSUJEVENCdFFJQkFEQW5NUXN3Q1FZRFZRUUdFd0pEVGpFWU1CWUdBMVVFQXd3UFltUmZkR2xqYTJWMFgyZDFcclxuWVhKa01Ga3dFd1lIS29aSXpqMENBUVlJS29aSXpqMERBUWNEUWdBRXJwUVQyTm5rYkJQeGNteXk2NTZxU29PRFxyXG5LTk51WkdTamIxR0lJWjcrZXo0NFBnQ3B0R1FsUlBSSTBGTTZ1Y1FpZmd6ZlFLRm9SaEtCSC9oSUR6VDJ5NkFzXHJcbk1Db0dDU3FHU0liM0RRRUpEakVkTUJzd0dRWURWUjBSQkJJd0VJSU9kM2QzTG1SdmRYbHBiaTVqYjIwd0NnWUlcclxuS29aSXpqMEVBd0lEUndBd1JBSWdGY2NhSittdTQyT0tsNVlNUDA2bGtYR3MwaDFLUi96VElDRGhhY01GQUhzQ1xyXG5JQ2sxWXVXc08wemlwTlRuVi9Vc3A5cDI3Wnc2QlNJRmFUcGlFYjhRK2RrVVxyXG4tLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS1cclxuIn0=; msToken=fuXeZhpRb65tedIa3jLQjkn6ZM2lxDVsB5bEkq5qGoJvql34gdEr_DduzpxOxacWDnV3YqPJn5bitDYs7LMevFoq6Tt7jWVGPimBbTKjrYC_a5rB7ccP2NthLP6smKI=; home_can_add_dy_2_desktop=%221%22; msToken=PJkiDtDJPv_gkTg0Lxkoo2yQfJOQP_CFLO0s0MwZdVj5Bt9p8ye7i22qXvlAjX3fMeQC053OkJR8ukzJZB1TS1yeIWIskE0SPtE02jE4YcYlpwqMhLxXVV6488dR6s8=")
req.Header.Set("referer", "https://www.douyin.com/user/MS4wLjABAAAACX0jKoZARKXONKiywn2yeDglLJ8oKoU5E_iPLflzKqM")
req.Header.Set("sec-ch-ua", "Microsoft Edge";v="113", "Chromium";v="113", "Not-A.Brand";v="24")
req.Header.Set("sec-ch-ua-mobile", "?0")
req.Header.Set("sec-ch-ua-platform", "Windows")
req.Header.Set("sec-fetch-dest", "empty")
req.Header.Set("sec-fetch-mode", "cors")
req.Header.Set("sec-fetch-site", "same-origin")
req.Header.Set("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36 Edg/113.0.1774.42")
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
bodyText, _ := io.ReadAll(resp.Body)
fmt.Printf("%s\n", bodyText)
}
func main() {
TestRequest()
}
`

@jqqjj
Copy link

jqqjj commented May 14, 2023

另外,没有做到io复用,虽然代码看到有写

@meesii
Copy link

meesii commented Oct 15, 2023

最新版也不行,这个网址可以测试 https://tools.scrapfly.io/api/fp/akamai?extended=1
会返回“Bad HTTP version - only for HTTP/2.0”

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants