Many projects of MIPO technology use reptiles to collect web data , Break through anti reptiles 、 automated testing 、 Regression testing also requires MIPO to consider building automation , To improve the efficiency of the whole team .
Because of the busy requirements and product process specification , Now the research on testing technology has accumulated a lot . But no matter what you do , It's important to do well !
Automation is mainly for team building , On the one hand, in order to provide the efficiency of the testing department , Guarantee product quality ; On the other hand , It's also to improve the testing skills of team members , Guarantee Team Good development . But anyway , Automation is a must , Otherwise, there is no efficiency guarantee and quality guarantee for tedious regression test .
The preliminary plan passed Python As a scripting language ,Selenium As web Test tool for end , At present, it is mainly based on web End to end build .
MIPO blog original :Python+Selenium2 Build an automated test environment
Python install
yum -y update
yum -y install gcc gcc-g++ python python-devel python-pip
yum -y install Xvfb firefox
pip install pyvirtualdisplay
yum -y install gcc gcc-c++ autoconf libjpeg libjpeg-devel libpng libpng-devel freetype freetype-devel libxml2 libxml2-devel zlib zlib-devel glibc glibc-devel glib2 glib2-devel bzip2 bzip2-devel ncurses ncurses-devel curl curl-devel e2fsprogs e2fsprogs-devel krb5 krb5-devel libidn libidn-devel openssl openssl-devel openldap openldap-devel nss_ldap openldap-clients openldap-servers unzip subversion
pypi Official website : https://pypi.python.org/pypi
pip Official website : https://pypi.python.org/pypi/pip/
bs4 Official website : https://pypi.python.org/pypi/beautifulsoup4/
setuptools Official website : https://pypi.python.org/pypi/setuptools/
selenium Official website :https://pypi.python.org/pypi/selenium
tornado Official website : https://pypi.python.org/pypi/tornado
tornado Official website :http://www.tornadoweb.org
tornado github: https://github.com/tornadoweb/tornado/
PyVirtualDisplay Official website : https://pypi.python.org/pypi/PyVirtualDisplay ( contain xvfb)
PyVirtualDisplay github: https://github.com/ponty/PyVirtualDisplay
firefox Old version download :http://ftp.mozilla.org/pub/firefox/releases/ ( recommend )
This section mainly records simple construction Python+Selenium The process of testing the environment , As follows :
Based on the environment :windows 7 64bit
1、 structure python development environment , Version is the latest version python2.7.5
stay python The official website chooses to download the latest windows Installation package :python-2.7.5.amd64.msi,
Pay attention to the choice 64bit Of . After installation , Need environment variables in the system path Add C:\Python27, And then on the command line .
2、SetupTools and pip Tool installation
Both of these tools belong to python Third party toolkit software , It's kind of like linux Under the installation package software , however pip Than SetupTools More powerful .
SetupTools Official explanation :Download, build, install, upgrade, and uninstall Python packages -- easily!
stay python You can find SetupTools The download , here Windows Provided only 32bit The download ,setuptools-0.6c11.win32-py2.7.exe, Double click the installation directly .
pip Official explanation :A tool for installing and managing Python packages.
cmd Go to the command line :easy_install pip Just install it online .
remarks : Note here , When installed SetupTools after , You can go to python See... In the installation directory Script Catalog , As shown in the figure below :
After this directory is generated , You need to add... To the system environment variable path:C:\Python27\Scripts, Then it can be used in the command easy_install Command to pip Online installation .
3、 install Selenium
selenium The official download :https://pypi.python.org/pypi/selenium#downloads
selenium The latest version :selenium-2.53.4.tar.gz
Because of the need to Python and Selenium Are combined , Of course Selenium Also provided based on python The implementation of the , So we need to put Selenium The package is installed to python Go to Library , For convenience python Call at development time .
stay cmd Go to the command line :pip install selenium After performing , Will automatically search for the latest selenium Version download and install , As shown in the figure below :
The above shows , It means online installation selenium success !
4、Python + Selenium Example
It can be directly here python Edit the following procedures , And save hello_selenium.py
1
2
3
4
5
6
7
8
|
from
selenium
import
webdriver
driver
=
webdriver.Firefox()
driver.get(‘https:
/
/
blog.mimvp.com’)
assert
"blog.mimvp.com Blog "
.decode(
'utf-8'
)
in
driver.title
print
driver.title
driver.close()
|
stay python Editor inside operation F5 Can run , See if the call is successful Firefox browser ...
On the basis of Python+Selenium The automation environment of has been built .
Selenium Multi browser implementation
structure Python+Selenium2 When the automated test environment is complete , You need test support python Of selenium Whether all versions of support running on different browsers , At present, we are on three most common browsers (IE,Chrome,FireFox) Test... With scripts .
1) IE browser
stay IE browser Run test script on , First you need to download IEDriverServer.exe, Put it in IE The installation directory of the browser and the same level directory , The script is as follows :
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
import
os
from
selenium
import
webdriver
from
selenium.webdriver.common.keys
import
Keys
iedriver
=
"C:Program FilesInternet ExplorerIEDriverServer.exe"
# IE driver
os.environ[
"webdriver.ie.driver"
]
=
iedriver
driver
=
webdriver.Ie(iedriver)
driver.get(
"https://blog.mimvp.com"
)
assert
"Python"
in
driver.title
elem
=
driver.find_element_by_name(
"q"
)
elem.send_keys(
"selenium"
)
elem.send_keys(Keys.RETURN)
assert
"mimvp.com"
in
driver.title
driver.close()
driver.quit()
|
2)Chrome browser
stay Chrome Run the test script on the browser , First you need to download ChromeDriver.exe, Put it in Chrome The installation directory of the browser and the same level directory , The script is as follows :
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
import
os
from
selenium
import
webdriver
from
selenium.webdriver.common.keys
import
Keys
chromedriver
=
"C:Program Files (x86)GoogleChromeApplicationchromedriver.exe"
os.environ[
"webdriver.chrome.driver"
]
=
chromedriver
driver
=
webdriver.Chrome(chromedriver)
driver.get(
"https://blog.mimvp.com"
)
assert
"Python"
in
driver.title
elem
=
driver.find_element_by_name(
"q"
)
elem.send_keys(
"selenium"
)
elem.send_keys(Keys.RETURN)
assert
"mimvp.com"
in
driver.title
driver.close()
driver.quit()
|
Pay attention to the introduction of the official website :
Chrome Driver is maintained / supported by the Chromium project iteslf. It seems that if you use new ChromeDriver() Words , It should be installed Chromium instead of Chrome, I'm too lazy to toss now , Interested children's shoes can be tested .
3) Firefox browser
stay Firefox Run the test script on the browser , As follows :
1
2
3
4
5
6
7
8
9
10
11
12
13
|
from
selenium
import
webdriver
from
selenium.webdriver.common.keys
import
Keys
driver
=
webdriver.Firefox()
driver.get(
"https://blog.mimvp.com/"
)
assert
"Python"
in
driver.title
elem
=
driver.find_element_by_name(
"q"
)
elem.send_keys(
"selenium"
)
elem.send_keys(Keys.RETURN)
assert
"mimvp.com"
in
driver.title
driver.close()
driver.quit()
|
Selenium The virtual browser silently executes ( Don't open the window )
Use selenium When opening a web page ,FireFox The browser will display... In the virtual form , Will not open in the current user form .
Application scenarios :
It's very suitable for execution on the server side ; It is very user-friendly and does not disturb the current user's work , Fabulous !
On *nix, you can run WebDriver in a headless (virtual) display to hide the browser. This can be done with Xvfb.
I personally use Python on Linux, and the PyVirtualDisplay module to handle Xvfb for me.
Code for running headless would look like this:
#!/usr/bin/env python
from pyvirtualdisplay import Display
from selenium import webdriver
display = Display(visible=0, size=(800, 600))
display.start()
# now Firefox will run in a virtual display.
# you will not see the browser.
browser = webdriver.Firefox()
browser.get('https://www.mimvp.com')
print browser.title
browser.quit()
display.stop()
Install dependencies on Debian/Ubuntu:
$ sudo apt-get install xvfb python-pip
$ sudo pip install pyvirtualdisplay
or CentOS On the system
$ yum install Xvfb python-pip // yum install xorg-x11-server-Xvfb $ yum --enablerepo=remi install firefox $ pip install pyvirtualdisplay
or AWS EC2 On the system
$ wget https://lambda-linux.io/epll-release-2015.09-1.1.ll1.noarch.rpm $ yum -y install epll-release-2015.09-1.1.ll1.noarch.rpm $ yum --enablerepo=epll install firefox-compat
download firefox, decompression , Soft link
wget http://download.firefox.com.cn/releases/firefox/45.0/en-US/Firefox-latest-x86_64.tar.bz2
tar jxvf Firefox-latest-x86_64.tar.bz2
mv Firefox-latest-x86_64 firefox
ln -s /home/ec2-user/tool-server/firefox/firefox /usr/bin/firefox
Reference resources : Announcing Firefox Browser Support for Amazon Linux
firefox More old versions to download :http://ftp.mozilla.org/pub/firefox/releases/ ( recommend )
error 1:
If you make a mistake , Update selenium To the latest version , error message :
Message: Can't load the profile. Profile Dir: %s If you specified a log_file in the FirefoxBinary constructor, check it for details.
pip install -U selenium ( Higher version CentOS 7、Firefox、Selenium, This method is useless )
error 2:
easyprocess.EasyProcessCheckInstalledError: cmd=['Xvfb', '-help']
OSError=[Errno 2] No such file or directory
Program install error!
solve :
1
|
pip
install
xvfbwrapper
|
error 3:
selenium.common.exceptions.WebDriverException: Message: 'geckodriver' executable needs to be in PATH.
This is because Firefox Higher versions need to use geckodriver To drive , No longer use Seleniu Default Firefox webdriver.
All we need to do is github download geckodriver And set its path as environment variable to solve .
solve :
1
|
brew
install
geckodriver
|
Or download and install the lower version firefox
firefox-45.0.2.tar.bz2
firefox More old versions to download :http://ftp.mozilla.org/pub/firefox/releases/ ( recommend )
Application example :
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
|
import
bs4
from
selenium
import
webdriver
from
pyvirtualdisplay
import
Display
def
spider_url(
self
, url, index, total):
print
(
"%d/%d - url: %s"
%
(index, total, url))
content
=
''
browser
=
None
table_soup
=
[]
try
:
display
=
Display(visible
=
0
, size
=
(
800
,
600
))
display.start()
# now Firefox will run in a virtual display.
# you will not see the browser.
browser
=
webdriver.Firefox()
# open FireFox browser
browser.set_page_load_timeout(
60
)
browser.get(url)
content
=
browser.find_element_by_class_name(
'table'
)
# By marking id Get the content of the web page
content
=
browser.page_source
self
.kill_firefox(browser)
display.stop()
content
=
bs4.BeautifulSoup(content, from_encoding
=
'GB18030'
)
table_soup
=
content.find(
'table'
, {
"class"
:
"table"
}).find_all(
"tr"
)
except
Exception as ex:
print
(
"error msg: "
+
str
(ex))
self
.kill_firefox(browser)
|
There is also a way of silent execution ( For reference ):
I easily managed to hide the browser window.
Just install PhantomJS. Then, change this line:
driver = webdriver.Firefox()
to:
driver = webdriver.PhantomJS()
The rest of your code won't need to be changed and no browser will open. For debugging purposes, use driver.save_screenshot('screen.png')
at different steps of your code.
summary
Pass the tests on the above three different browsers , explain selenium stay python The application of the Java The versions are the same .
because Firefox Is the default installation path ,webdriver You can find him on a regular visit , If not the default installation path of the system , You need to follow IE and Chrome Set it up the same way driver route .