You need to send post http requests along with appropriate parameters to reach the content. This should produce results based on your current search keyword. If anyone knows what I am overlooking or missing, please let me know. I want to know why I am not able to scrape from DuckDuckGo ,but instead from Google using the same code. īut if I use Google instead of DuckDuckGo, I am able to scrape the required link. I found an answer that DuckDuckGo uses javascript for it search results and beautifulSoup can't scrape javascript, but on the other posts on StackOverflow I found that people are able to scrape the link from it results. Please look at the screenshot, the highlighted HTML syntax is what I want to scrape : I don't know why BeautifulSoup is not scraping the result containing divs. It print's the web page's HTML but not those result containing divs. I also tried find_all() with class_ argument but then it returned me an empty list. It returned me a list of some random anchor tags that is not of any use to me. # bseIndiaLink=soup.find_all('a',class_="result_a") #giving empty listįirstly I used find_all() in beautifulSoup without class_ argument. Soup=BeautifulSoup(response.text,"html.parser") Response=requests.get(duckDuckUrl,headers=headers) I want to scrape the 1st link from DuckDuckGo search result.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |