Welcome to Qi-U Community for programmer and developer-Open, Learning and Share

Categories

0 votes
2.2k views
in Technique[技术] by (31.9m points)

python - How do I web scrape a website behind a login?

I am trying to web scrape a landing page that requires a login to access. My current script (based off the video tutorials I found) returns a 400 error response. The referrer policy of the website is a "strict-origin-when-cross-origin" while all the videos I can find deal with a policy that is "no-referrer-when-downgrade," which is what I believe is causing the error. The following is my script, I would like to know what modifications I need to make it work:

payload = {
   'username':username,
   'password':password
   }

url = "https://audiogo.adswizz.com/"
login = "api/v2/auth/login"

headers = {
   'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36', 
   'Origin':url,
   'Accept':'application/json, text/plain, */*',
   'Accept-Encoding':'gzip, deflate, br',
   'Accept-Language':'en-US,en;q=0.9',
   'Content-Type':'application/json;charset=UTF-8'}

s = requests.session()
s.post(url=url+login, headers = headers, data = payload)

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

Please log in or register to answer this question.

1 Answer

0 votes
by (31.9m points)
等待大神答复

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome to Qi-U Community for programmer and developer-Open, Learning and Share
...