Spiderkeeper 2019(1)Installation and Introduction
> python -V
Python 2.7.5
Since we want to migrate all the things to Python3, install and prepare python3
Install PYENV from the latest github
> git clone https://github.com/pyenv/pyenv.git ~/.pyenv
Add to the PATH
> vi ~/.bash_profile
PATH=$PATH:$HOME/.pyenv/bin
eval "$(pyenv init -)"
> . ~/.bash_profile
Check installation
> pyenv -v
pyenv 1.2.13-2-g0aeeb6f
Install the Python3 and Python2
> pyenv install 3.6.0
> pyenv install 2.7.10
It will take quite some time, but after the installation is fine
> pyenv versions
* system (set by /home/carl/.pyenv/version)
2.7.10
3.6.0
Globally choose 3.6.0
> pyenv global 3.6.0
> python -V
Python 3.6.0
Check all the versions, latest is 3.7.4, install some other versions
https://www.python.org/downloads/
> pyenv install 3.6.9
> pyenv install 3.7.4
Fails with ModuleNotFoundError: No module named '_ctypes'
Solution:
https://github.com/pyenv/pyenv/issues/1183
> sudo yum install libffi-devel
> pyenv install 3.7.4
> pyenv global 3.7.4
> python -V
Python 3.7.4
> pip -V
pip 19.0.3 from /home/carl/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pip (python 3.7)
Upgrade the pip
> pip install --upgrade pip
> pip -V
pip 19.2.3 from /home/carl/.pyenv/versions/3.7.4/lib/python3.7/site-packages/pip (python 3.7)
Install spider keeper
> pip install spiderkeeper
Then we are ready to start the spiderkeeper as follow:
> spiderkeeper --server=http://xxx.xx.xx.xx:6801 --server=http://xx.xx.xx.xx:7800 --port=5500 --username=username --password=password\!123
Error Message
ModuleNotFoundError: No module named 'pysqlite2'
Solution:
https://stackoverflow.com/questions/29770906/importerror-no-module-named-pysqlite2
https://github.com/jupyterhub/jupyterhub/issues/464
Install sqlite-devel and then reinstall python 3.7.4 and try again
> sudo yum install sqlite-devel
> pyenv uninstall 3.7.4
> pyenv install 3.7.4
WARNING: The Python bz2 extension was not compiled. Missing the bzip2 lib?
WARNING: The Python readline extension was not compiled. Missing the GNU readline lib?
Then it works pretty well
> spiderkeeper --server=http://xxx.xx.xx.xx:6801 --server=http://xx.xx.xx.xx:7800 --port=5500 --username=username --password=password\!123
Maybe fix the WARNING as well.
https://stackoverflow.com/questions/12806122/missing-python-bz2-module
https://github.com/pyenv/pyenv/wiki/common-build-problems
> sudo yum install bzip2-devel
> sudo yum install readline-devel
Remove the python3 in Pyenv and re-install again.
> pyenv uninstall 3.7.4
> pyenv install 3.7.4
> pip install --upgrade pip
> pip install spiderkeeper
Start the service
>spiderkeeper --server=http://xxx.xx.xx.xx:6801 --server=http://xx.xx.xx.xx:7800 --port=5500 --username=username --password=password\!123
Then I can visit the UI from here
http://centos-dev1:5500/project/manage
Then we can create project, generate egg file and upload our spider to that platform.
Some other installation command I may need
scrapy need for deploying spider
> pip install scrapy
scrapyd need for running the final spider there
> pip install scrapyd
scrapyd-client is needed for deploy or generate the egg file
> pip install scrapyd-client
References:
https://github.com/pyenv/pyenv#installation
https://www.jianshu.com/p/88ddeac92a6d
https://stackoverflow.com/questions/33321312/cannot-switch-python-with-pyenv
分享到:
相关推荐
HP SIM 7.3 installation and configuration guide
1. 文档标题和描述:文件标题为“HG4930_Installation and Interface Manual_7.14.pdf”,描述为该文档的完整名称,其中包含了 HG4930 惯性测量单元(Inertial Measurement Unit,简称IMU)的安装和接口手册,版本号...
cn_BT2019
Step by Step Oracle GoldenGate Installation and Configure Guide(一步步教你安装和配置OGG操作指南):安装前的准备、源和目标端的安装、进程配置、DDL复制支持、GOLDENGATE优化、监控GOLDENGATE、GOLDENGATE...
Installation and Licensing Documentation
Lenovo System x3750 M4 Types 8752 and 8718 Installation and Service Guide
API 686 Recommended Practice for Machinery Installation and Installation Design. second edition, 2009. 机械设备安装国际通行规范
Untangle Introduction & Installation
VSR installation and Setup Guide
1. **Installation:** - Install Python if it is not already installed. - Install PyQt4 using pip or another package manager. - Verify the installation by running a simple PyQt4 script. 2. **...
本文将基于《Cognos Installation and Configuration Guide》的内容,深入解析Cognos 8的安装与配置流程,以及在实施过程中需要注意的关键知识点。 ### 一、产品信息与版权 #### 产品信息 - **适用版本**:本指南...
在《Oracle Warehouse Builder Installation and Administration Guide 11g Release 2 (11.2)》中,用户可以找到关于如何安装、配置和管理这个工具的详细指导。 1. **安装过程**: - **系统需求**:在开始安装之前...
《Packt.Oracle.Siebel.CRM.8.Installation.And.Management》这本书深入探讨了Oracle Siebel CRM 8系统的安装、配置与管理方法,为读者提供了全面的专业指导。以下是基于标题、描述、标签以及部分内容中提炼出的关键...
Oracle Sales Analyzer Installation and Upgrade Guide Release 11 i
ARIS Server Installation and Administration Guide.pdf
总之,ARIS Server Installation and Administration Guide为管理员提供了全面的指导,涵盖了从准备安装到日常运维的所有环节,帮助用户成功地部署和管理ARIS Server,从而有效地支持企业的业务流程管理工作。
Lenovo System x3250 M4 Installation and Service Guide
Lenovo Dynamic System Analysis Installation and User's Guide
Command-Line Installation and Upgrade of VMware vCenter ServerTM 5.0
ARIS Server Installation and Administration Guide Version 9.8 – Service Release 5 是一份详细的文档,旨在指导用户正确安装和管理ARIS服务器的版本9.8及其后续发布。此指南适用于2016年7月及之后的所有版本,...