爬虫数据如何识别?
易观方舟JS SDK 会根据数据中的 UA 信息来识别的是上报的事件是否是爬虫产生的,方便在分析时根据事件中的爬虫属性来过滤掉这部分非真实用户产生的数据。
`/(bot|crawler|spider|scrapy|dnspod|ia_archiver|jiankongbao|slurp|transcoder|networkbench|oneapm|PhantomJS|BingPreview)/i`
- 1.
- 2.
- 3.
- 4.百度移动端:Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)
- 5.
- 6.Google图片 :AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile Safari
- 7.360蜘蛛:360Spider 360搜索 Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0);
- 8.
- 9.
- 10.
- 11.
- 12.雅虎蜘蛛:Yahoo! 雅虎英文 Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
- 13.
- 14.有道蜘蛛:YoudaoBot Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )
- 15.
- 16.
- 17.瑞典 Speedy Spider: Speedy Spider Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
- 18.
- 19.
- 20.
- 21.听云爬虫:networkbench Mozilla/5.0 (Windows NT 10.0; Trident/7.0; rv: 11.0;NetworkBench/8.0.1.309-5774440-2481662) like Gecko
- 22.Alexa蜘蛛:ia_archiver ia_archiver/8.9 (Windows NT 3.1; en-US;)
- 23.
- 24.华为赛门铁克蜘蛛:HuaweiSymantecSpider HuaweiSymantecSpider/1.0+DSE-[email protected]+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/cn/IRL/spider)
- 25.七牛镜像蜘蛛:qiniu qiniu-imgstg-spider-1.0
- 26.DNSPod监控:DNSPod DNSPod-Monitor/2.0
- 27.
- 28.
- 29.即刻蜘蛛:JikeSpider
- 30.一淘网蜘蛛:EtaoSpider Mozilla/5.0 (compatible; EtaoSpider/1.0; EtaoSpider)
- 31.人工智能爬虫:crawler Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html) Gecko/2008032620
- 32.Scrapy爬虫: scrapy
- 33.监控宝:jiankongbao Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; JianKongBao Monitor 1.1)
- 34.OneAPM爬虫:OneAPM FFAgent Mozilla/5.0 (Windows NT 6.1; WOW64; rv:39.0: OneAPM FFAgent)Gecko/20100101 Firefox/39.0
- 35.PhantomJS:PhantomJS Mozilla/5.0 (Unknown; Linux x86_64)AppleWebKit/538. 1 (KHTML,like Gecko)PhantomJS/2.1.1 Safari/538.1
- 36.BingPreview:Mozilla / 5.0 + (Windows + NT + 6.1; + WOW64) + AppleWebKit / 534++(KHTML, +like + Gecko) + BingPreview / 1.0 b
最近更新 3yr ago