Python crawler camouflages browser_Python anti-crawler camouflages browser to crawl
For some websites in the crawler, the ip will be blocked after too many requests are set. Now the browser is simulated to crawl, that is to say, let the server realize that the access to him is a real browser rather than a machine operation Simply add the request header directly, and pass in the browser information when requesting data: Open browser–open developer mode–request any website As shown in the picture below: Find the name of the request, open it, check the headers column, find User-Agent, and copy it. Then add it to the request header The code is as follows: import requests url = ‘https://www.baidu.com’ headers ={ ‘User-Agent’:’Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko)’ ‘Chrome/65.0.3325.181 Safari/537.36’ } rq = requests. get(url=url, headers=headers) print(rq. text) Deeper camouflage browser, add multiple browser information, randomly send browser information every time a request is made, let the server know that not one browser has been accessing, (you can search user-agent on Baidu) The code is as follows: import requests import random url = ‘https://www.baidu.com’ headers_lists =( ‘Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko)’ ‘Chrome/65.0.3325.181 Safari/537.36’, ‘Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;Maxthon2.0’, ‘Opera/9.80(Android2.3.4;Linux;Operamobi/adr-1107051709;U;zh-cn)Presto/2.8.149Version/11.10’, ‘Mozilla/5.0(WindowsNT6.1;rv:2.0.1)Gecko/20100101Firefox/4.0.1’, ‘Mozilla/5.0(Android;Linuxarmv7l;rv:5.0)Gecko/Firefox/5.0fennec/5.0’,) rq = requests.get(url=url,headers={‘User-Agent’:random.choice(headers_lists)}) print(rq. text)…
Detailed explanation of VC file operation, CFile, CStdioFile_tyc13061298’s blog
In VC, if the amount of data is not large, the database is generally not used, and files are used to save data. Moreover, more file operation classes or methods are provided in VC. This article just briefly introduces a few file operations method. The first one is to introduce a VC file operation class that I think is the simplest, which can directly read the file by line, and output the string directly when outputting to the file, instead of reading and writing by byte Enter. How to use CStdioFile to read and write files in VC.txt When using this method, there may be an error situation, that is, when you use the WriteString(str) method, if the output is Chinese characters, it may become garbled. The solution is: CStdioFile.WriteString can’t handle Chinese CStdioFile’s bug, change the character set property of the project to use multi-byte character set,,,, the specific setting method is as follows: open project properties –> configuration properties –>General–>Character Set–>Change to use multi-byte character set Example methods are as follows: #include #include #define MAX_LENGTH 1024 void main() { CStdioFile stdiofile; char szPath[MAX_LENGTH] = {0}; cout << "Please enter the path of the file you want to view:"…
OpenWrt U disk partition startup settings_openwrt ubi partition_xiaomingtongxie’s blog
1. Update the router software source (note: the router needs to be connected to the Internet) opkg update 2. Add USB support opkg install kmod-usb-core #optional opkg install kmod-usb-uhci opkg install kmod-usb-storage opkg install kmod-usb2 opkg install kmod-usb-ohci #The command is lsusb 3. Add usb mount, hot swap, and boot support opkg install block-mount #Mount, hot swap, boot support Two, U disk partition Why do you want to partition the U disk? First, it is convenient to manage files. We want to start openwrt from the U disk, so run the system image on the U disk, so as to obtain more space for storing various files. For enhanced management, necessary partitions are useful. Second, create a Swap swap partition. We need to open up some space on the U disk for Swap exchange. Use Swap to increase the throughput of RAM, thereby preventing the system from hanging up when some software needs to use a large amount of RAM (for example, Transmission offline download service). So, how to partition the U disk? We need to do the following: 1. Preparation, add necessary software opkg install kmod-fs-ext4 #Add ext3 file system support opkg install fdisk #Add partition tool opkg install…
