Hello everyone , I'm spicy .
As we all know, the previous articles of spicy bar are related to reptiles , I want to develop in this direction myself , So learn in advance about the job market for crawler Engineers , I also hope you don't step on the pit . Let's talk about what technology crawler engineers need to master, as well as the salary level and development prospect in various regions of the market .
What do reptile engineers do ？ What's the job ？ What capabilities are required ？ What skills do you need to master ？ I have been recruited by three big companies as crawler engineers , Let you know the responsibilities of reptile engineers in various industries .
Job requirements ：
2、 be familiar with Linux Common commands , be familiar with selenium、appnium And other component operations ;
3、 be familiar with MYSQL、MongoDB The use of common databases and library tables 、 Index, etc ;
4、 Be proficient in using Python Write network request and processing script , understand GO Language or willing to transfer GO Language technology stack ;
5、 Familiar with page automation testing tools and processes , Ability to analyze and locate problems ;
6、 Yes hook Experience , understand Xposed Frame first ;
7、 Experience in Internet data collection and anti crawling is preferred ;
8、 Computer related major , Bachelor degree or above ;
1. Directional data acquisition and crawling
2. Design crawler strategy and anti shielding rules , Improve the efficiency and quality of data capture
3. Development and maintenance of crawler system
1、 Bachelor degree or above , Computer related major ,3 More than years working experience in crawler development ;
2、 be familiar with html/css/xpath/ajax/xml Technology , be familiar with HTTP Transfer protocol , Proficient in web page capture principle and integration technology , Familiar with regular expressions ;
3、 Master Python Language , Rich anti climbing experience ;
4、 be familiar with MySQL、MongoDB、Redis;
5、 With data mining 、 natural language processing 、 Information retrieval 、 Machine learning background is preferred ;
6、 Experience in enterprise information data collection is preferred .
1、 Participate in the architecture design and development of distributed crawler and data acquisition system ;
2、 Use mainstream big data related technologies , Clean the captured web page data 、 Storage, etc , And continuously optimize the platform , In order to meet various crawling business needs ,;
3、 Responsible for designing crawler strategy and anti shielding rules , Improve the efficiency and quality of web crawling , Such as analysis and prediction 、 Quality judgment 、 Research on closure and anti closure, etc ;
4、 Focus on vertical domain data crawling , Capture and analyze multi platform information ;
5、 Study the optimization algorithm , Improve the stability of the crawler system 、 Extensibility , Follow up technical hot spots , Update new crawler functions in time , Continuously optimize the distributed crawler strategy .
1、 Bachelor degree or above , Computer and other related majors ,3 More than years working experience in reptile , There are e-commerce platforms, such as Amazon 、 Express experience is preferred ;
2、 Proficient in automatic acquisition and anti automatic acquisition technology , Master http The underlying protocol ;
3、 Proficient in deep crawl 、 Dynamic web page capture technology 、 Browser simulation capture technology, etc , Get information from structured and unstructured data ;
4、 Proficient in Distributed Systems 、 Multithreading 、 The cloud service , Proficient in more than one open source framework , be familiar with scrapy、 requests And one or more automatic acquisition frameworks and principles , Experience in developing automatic acquisition framework ;
5、 Good communication skills 、 Learning ability and sense of responsibility .
【 After reading this recruitment, spicy bar said that there was great pressure , There are still many things to learn , Some fans asked me how to learn reptiles , In fact, there are many online tutorials , But the system is not systematic , It's still difficult to learn by yourself , The best way is to learn from the responsibilities of recruitment positions 】
2020 Annual recruitment of reptile Engineers 1842 individual , Than 2019 The year dropped. 35%. The data comes from recruitment websites all over the world , Search only job name , The data may deviate from the objective reality due to the stability of the capture system and other factors , For reference only .
Where is the highest demand for reptile engineers in the country ？ It's better for reptile engineers to work in Shenzhen or Beijing ？ The recruitment demand of reptile engineers in Shenzhen is the highest , Occupy 20.7%, In the whole country 1. Next is Beijing 18.6%, Shanghai occupies 18.5%, Guangzhou Zhan 10.0%. Statistics depend on the public data published by various platforms , System stability affects objectivity , For reference only .
【 The first tier cities add up to more than 60% , The rest basically belong to second tier cities , What if I go to the third and fourth lines in my hometown 】
How much is the salary of reptile engineers usually a month ？ Average wage ￥ 15.8K/ month , Most people take 20K-30K,2021 Year comparison 2020 The year dropped. 12%.
The salary of reptile engineer is calculated according to educational background , College salary ￥11.4K, Undergraduate salary ￥17.4K, Master's salary ￥20.8K
How about the salary of reptile engineers this year ？2021 Average annual salary of reptile Engineer ￥15.8K,2021 The annual salary is less than 2020 year , a 2020 The year dropped. 12%.2020 Annual wage ￥18.0K,2019 Annual wage ￥18.5K,2018 Annual wage ￥18.9K,2017 Annual wage ￥17.6K,2016 Annual wage ￥15.4K,2015 Annual wage ￥15.3K,2014 Annual wage ￥16.3K. Data statistics depends on the public data released by major platforms , System stability affects objectivity , For reference only .
【 College degree is also OK , Education is the key to success , What ultimately determines your salary is your ability and career quotient 】
【 There are still many posts , It mainly depends on your city , Staying in Beijing, Shanghai and Guangzhou is the easiest stepping stone 】
If it works for you , Remember to give spicy strips three times .