一.问题描述

最近在自学pyspark，想本地通过pycharm来编辑pyspark脚本，运行的时候，提示没有pyspark模块，于是通过pycharm安装了pyspark(最新版本)。

而且这个安装到了远程服务器上，真的坑

奈何我安装的是CDH 6.3.1版本，然后spark版本是 2.4.0-cdh6.3.1，然后最新版本是3.0.2的

新的pyspak程序，都未记录到 History Server服务上。

于是首先通过pip卸载pyspark 3.0.2版本，然后安装pyspark 2.4.0版本。
报错如下:
Command “python setup.py egg_info” failed with error code 1 in /tmp/pip-install-5otCUm/pyspark/

root@hp1 pip-10.0.1]# pip install pyspark==2.4.0
Collecting pyspark==2.4.0
  Using cached https://files.pythonhosted.org/packages/88/01/a37e827c2d80c6a754e40e99b9826d978b55254cc6c6672b5b08f2e18a7f/pyspark-2.4.0.tar.gz
    Complete output from command python setup.py egg_info:
    Could not import pypandoc - required to package PySpark
    /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'long_description_content_type'
      warnings.warn(msg)
    zip_safe flag not set; analyzing archive contents...
    pypandoc.__init__: module references __file__
    
    Installed /tmp/pip-install-5otCUm/pyspark/.eggs/pypandoc-1.5-py2.7.egg
    Searching for wheel>=0.25.0
    Reading https://pypi.python.org/simple/wheel/
    Downloading https://files.pythonhosted.org/packages/ed/46/e298a50dde405e1c202e316fa6a3015ff9288423661d7ea5e8f22f589071/wheel-0.36.2.tar.gz#sha256=e11eefd162658ea59a60a0f6c7d493a7190ea4b9a85e335b33489d9f17e0245e
    Best match: wheel 0.36.2
    Processing wheel-0.36.2.tar.gz
    Writing /tmp/easy_install-6V5vRU/wheel-0.36.2/setup.cfg
    Running wheel-0.36.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-6V5vRU/wheel-0.36.2/egg-dist-tmp-v5OZ83
    warning: no files found matching '*.dynlib' under directory 'tests'
    no previously-included directories found matching 'tests/testdata/*/build'
    no previously-included directories found matching 'tests/testdata/*/dist'
    no previously-included directories found matching 'tests/testdata/*/*.egg-info'
    warning: install_lib: 'build/lib' does not exist -- no Python modules to install
    
    zip_safe flag not set; analyzing archive contents...
    Moving UNKNOWN-0.0.0-py2.7.egg to /tmp/pip-install-5otCUm/pyspark/.eggs
    
    Installed /tmp/pip-install-5otCUm/pyspark/.eggs/UNKNOWN-0.0.0-py2.7.egg
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-5otCUm/pyspark/setup.py", line 224, in <module>
        'Programming Language :: Python :: Implementation :: PyPy']
      File "/usr/lib64/python2.7/distutils/core.py", line 112, in setup
        _setup_distribution = dist = klass(attrs)
      File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 315, in __init__
        self.fetch_build_eggs(attrs['setup_requires'])
      File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 361, in fetch_build_eggs
        replace_conflicting=True,
      File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 849, in resolve
        raise DistributionNotFound(req, requirers)
    pkg_resources.DistributionNotFound: The 'wheel>=0.25.0' distribution was not found and is required by pypandoc
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-5otCUm/pyspark/
You are using pip version 10.0.1, however version 21.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
[root@hp1 pip-10.0.1]# 

二.解决方案

2.1 升级pip

百度了一下解决方案，需要升级pip和setuptools
https://mohen.blog.csdn.net/article/details/112847570

python -m pip install --upgrade --force pip 
pip install setuptools==33.1.1

升级pip

[16:09:47] [root@hp1 ~]# python -m pip install --upgrade --force pip
[16:09:47] Collecting pip
[16:09:47]   Using cached https://files.pythonhosted.org/packages/b7/2d/ad02de84a4c9fd3b1958dc9fb72764de1aa2605a9d7e943837be6ad82337/pip-21.0.1.tar.gz
[16:09:48] Installing collected packages: pip
[16:09:48]   Found existing installation: pip 21.0.1
[16:09:48]     Uninstalling pip-21.0.1:
[16:09:48]       Successfully uninstalled pip-21.0.1
[16:09:49]   Running setup.py install for pip ... done
[16:09:49] Successfully installed pip-21.0.1
[16:09:50] [root@hp1 ~]# 

升级pip没问题，但是无论安装什么，都无法安装成功了

[16:09:50] [root@hp1 ~]# 
[16:10:00] [root@hp1 ~]# pip install setuptools==33.1.1
[16:10:01] Traceback (most recent call last):
[16:10:01]   File "/usr/bin/pip", line 11, in <module>
[16:10:01]     load_entry_point('pip==21.0.1', 'console_scripts', 'pip')()
[16:10:01]   File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 561, in load_entry_point
[16:10:01]     return get_distribution(dist).load_entry_point(group, name)
[16:10:01]   File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2627, in load_entry_point
[16:10:01]     return ep.load()
[16:10:01]   File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2287, in load
[16:10:01]     return self.resolve()
[16:10:01]   File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2293, in resolve
[16:10:01]     module = __import__(self.module_name, fromlist=['__name__'], level=0)
[16:10:01]   File "/usr/lib/python2.7/site-packages/pip/_internal/cli/main.py", line 60
[16:10:01]     sys.stderr.write(f"ERROR: {exc}")
[16:10:01]                                    ^
[16:10:01] SyntaxError: invalid syntax
[16:10:12] [root@hp1 ~]# 
[16:10:12] [root@hp1 ~]# 
[16:10:15] [root@hp1 ~]# pip install ipython                        
[16:10:15] Traceback (most recent call last):
[16:10:15]   File "/usr/bin/pip", line 11, in <module>
[16:10:15]     load_entry_point('pip==21.0.1', 'console_scripts', 'pip')()
[16:10:15]   File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 561, in load_entry_point
[16:10:15]     return get_distribution(dist).load_entry_point(group, name)
[16:10:15]   File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2627, in load_entry_point
[16:10:15]     return ep.load()
[16:10:15]   File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2287, in load
[16:10:15]     return self.resolve()
[16:10:15]   File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2293, in resolve
[16:10:15]     module = __import__(self.module_name, fromlist=['__name__'], level=0)
[16:10:15]   File "/usr/lib/python2.7/site-packages/pip/_internal/cli/main.py", line 60
[16:10:15]     sys.stderr.write(f"ERROR: {exc}")
[16:10:15]                                    ^
[16:10:15] SyntaxError: invalid syntax
[16:10:17] [root@hp1 ~]# 

2.2 yum卸载重装pip

为了修复2.1的问题，只能先通过yum卸载pip，然后重新安装pip

代码:

yum -y remove python-pip
 yum -y install python-pip

测试记录:
Your pip version is out of date, please install pip >= 9.0.1. pip 8.1.2 detected.
可以看到pip进行安装的时候提示pip的版本过期了，还是得升级pip

[16:11:01] [root@hp1 ~]# pip install ipython
[16:11:01] -bash: /usr/bin/pip: 没有那个文件或目录
[16:11:02] [root@hp1 ~]# 
[16:11:02] [root@hp1 ~]# 
[16:11:10] [root@hp1 ~]# yum -y install python-pip
[16:11:10] 已加载插件：fastestmirror
[16:11:10] Loading mirror speeds from cached hostfile
[16:11:14]  * base: mirrors.163.com
[16:11:14]  * epel: mirrors.bfsu.edu.cn
[16:11:14]  * extras: mirrors.aliyun.com
[16:11:14]  * updates: mirrors.aliyun.com
[16:11:15] https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/repodata/repomd.xml: [Errno 14]  Error 404 - The requested URL returned error: 404 Not Found
[16:11:15] 正在尝试其它镜像。
[16:11:15] To address this issue please refer to the below wiki article 
[16:11:15] 
[16:11:15] https://wiki.centos.org/yum-errors
[16:11:15] 
[16:11:15] If above article doesn't help to resolve this issue please use https://bugs.centos.org/.
[16:11:15] 
[16:11:15] 正在解决依赖关系
[16:11:15] --> 正在检查事务
[16:11:15] ---> 软件包 python2-pip.noarch.0.8.1.2-14.el7 将被 安装
[16:11:15] --> 解决依赖关系完成
[16:11:16] 
[16:11:16] 依赖关系解决
[16:11:16] 
[16:11:16] ==========================================================================================================================================================================================================
[16:11:16]  Package                                            架构                                          版本                                                  源                                           大小
[16:11:16] ==========================================================================================================================================================================================================
[16:11:16] 正在安装:
[16:11:16]  python2-pip                                        noarch                                        8.1.2-14.el7                                          epel                                        1.7 M
[16:11:16] 
[16:11:16] 事务概要
[16:11:16] ==========================================================================================================================================================================================================
[16:11:16] 安装  1 软件包
[16:11:16] 
[16:11:16] 总下载量：1.7 M
[16:11:16] 安装大小：7.2 M
[16:11:16] Downloading packages:
[16:11:33] python2-pip-8.1.2-14.el7.noarch.rpm                                                                                                                                                | 1.7 MB  00:00:17     
[16:11:33] Running transaction check
[16:11:33] Running transaction test
[16:11:33] Transaction test succeeded
[16:11:33] Running transaction
[16:11:34]   正在安装    : python2-pip-8.1.2-14.el7.noarch                                                                                                                                                       1/1 
[16:11:34]   验证中      : python2-pip-8.1.2-14.el7.noarch                                                                                                                                                       1/1 
[16:11:34] 
[16:11:34] 已安装:
[16:11:34]   python2-pip.noarch 0:8.1.2-14.el7                                                                                                                                                                       
[16:11:34] 
[16:11:34] 完毕！
[16:11:51] [root@hp1 ~]# 
[16:11:53] [root@hp1 ~]# pip install ipython      
[16:11:53] Collecting ipython
[16:11:54]   Using cached https://files.pythonhosted.org/packages/bd/59/5e4caa1b226d79508c8601888bd94af1f12ff464613daad90193c0e5fc88/ipython-7.22.0.tar.gz
[16:11:54]     Complete output from command python setup.py egg_info:
[16:11:54]     
[16:11:54]     IPython 7.17+ supports Python 3.7 and above, following NEP 29.
[16:11:54]     When using Python 2.7, please install IPython 5.x LTS Long Term Support version.
[16:11:54]     Python 3.3 and 3.4 were supported up to IPython 6.x.
[16:11:54]     Python 3.5 was supported with IPython 7.0 to 7.9.
[16:11:54]     Python 3.6 was supported with IPython up to 7.16.
[16:11:54]     
[16:11:54]     See IPython `README.rst` file for more information:
[16:11:54]     
[16:11:54]         https://github.com/ipython/ipython/blob/master/README.rst
[16:11:54]     
[16:11:54]     Python sys.version_info(major=2, minor=7, micro=5, releaselevel='final', serial=0) detected.
[16:11:54]     Your pip version is out of date, please install pip >= 9.0.1. pip 8.1.2 detected.
[16:11:54]     
[16:11:54]     
[16:11:54]     ----------------------------------------
[16:11:54] Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-NEF0D5/ipython/
[16:12:05] [root@hp1 ~]# 
[16:12:05] [root@hp1 ~]# 

2.3 通过文件的方式升级pip

pip update的方式在2.1章节已经被证实是通不过的，于是找到了一种通过文件的方式来升级pip

参考博客:
https://blog.csdn.net/qq_32670879/article/details/80654694

从 https://pypi.org/project/pip/10.0.1/#files 上下载pip-10.0.1.tar.gz文件，并上传到指定服务器。

tar -xvf pip-10.0.1.tar.gz 
cd pip-10.0.1
python setup.py build
python setup.py install
pip -version

*测试记录:

[16:14:35] [root@hp1 src]# cd pip-10.0.1
[16:14:35] [root@hp1 pip-10.0.1]# ls
[16:14:35] AUTHORS.txt  docs  LICENSE.txt  MANIFEST.in  NEWS.rst  PKG-INFO  pyproject.toml  README.rst  setup.cfg  setup.py  src
[16:15:05] [root@hp1 pip-10.0.1]# python setup.py build
[16:15:05] running build
[16:15:05] running build_py
[16:15:05] creating build
[16:15:05] creating build/lib
[16:15:05] creating build/lib/pip
[16:15:05] copying src/pip/__init__.py -> build/lib/pip
[16:15:05] copying src/pip/__main__.py -> build/lib/pip
[16:15:05] creating build/lib/pip/_internal
[16:15:05] copying src/pip/_internal/basecommand.py -> build/lib/pip/_internal
--snip--
[16:15:05] copying src/pip/_vendor/certifi/cacert.pem -> build/lib/pip/_vendor/certifi
[16:15:05] copying src/pip/_vendor/distlib/t32.exe -> build/lib/pip/_vendor/distlib
[16:15:05] copying src/pip/_vendor/distlib/t64.exe -> build/lib/pip/_vendor/distlib
[16:15:05] copying src/pip/_vendor/distlib/w32.exe -> build/lib/pip/_vendor/distlib
[16:15:05] copying src/pip/_vendor/distlib/w64.exe -> build/lib/pip/_vendor/distlib
[16:15:05] copying src/pip/_vendor/distlib/_backport/sysconfig.cfg -> build/lib/pip/_vendor/distlib/_backport
[16:15:10] [root@hp1 pip-10.0.1]# python setup.py install
[16:15:10] running install
[16:15:10] running bdist_egg
[16:15:10] running egg_info
[16:15:10] writing requirements to src/pip.egg-info/requires.txt
[16:15:10] writing src/pip.egg-info/PKG-INFO
[16:15:10] writing top-level names to src/pip.egg-info/top_level.txt
[16:15:10] writing dependency_links to src/pip.egg-info/dependency_links.txt
[16:15:10] writing entry points to src/pip.egg-info/entry_points.txt
[16:15:10] reading manifest file 'src/pip.egg-info/SOURCES.txt'
[16:15:10] reading manifest template 'MANIFEST.in'
[16:15:10] warning: no previously-included files found matching '.coveragerc'
--snip--
[16:15:12] Installing pip2.7 script to /usr/bin
[16:15:12] Installing pip2 script to /usr/bin
[16:15:12] 
[16:15:12] Installed /usr/lib/python2.7/site-packages/pip-10.0.1-py2.7.egg
[16:15:12] Processing dependencies for pip==10.0.1
[16:15:12] Finished processing dependencies for pip==10.0.1
[16:15:13] [root@hp1 pip-10.0.1]# 
[16:15:13] [root@hp1 pip-10.0.1]# 
[16:15:13] [root@hp1 pip-10.0.1]# 
[16:15:16] [root@hp1 pip-10.0.1]# pip -V
[16:15:17] pip 10.0.1 from /usr/lib/python2.7/site-packages/pip-10.0.1-py2.7.egg/pip (python 2.7)
[16:15:18] [root@hp1 pip-10.0.1]# 

现在进行pip安装没有问题了

[16:16:04] [root@hp1 pip-10.0.1]# pip install ipython 
[16:16:05] Collecting ipython
[16:16:05]   Downloading https://files.pythonhosted.org/packages/ce/2c/2849a2b37024a01a847c87d81825c0489eb22ffc6416cac009bf281ea838/ipython-5.10.0-py2-none-any.whl (760kB)
[16:16:06]     100% |████████████████████████████████| 768kB 2.0MB/s 
[16:16:06] Collecting prompt-toolkit<2.0.0,>=1.0.4 (from ipython)
[16:16:07]   Downloading 
--snip--
[16:16:13] Successfully installed backports.functools-lru-cache-1.6.3 backports.shutil-get-terminal-size-1.0.0 enum34-1.1.10 ipython-5.10.0 ipython-genutils-0.2.0 pathlib2-2.3.5 pexpect-4.8.0 pickleshare-0.7.5 prompt-toolkit-1.0.18 ptyprocess-0.7.0 pygments-2.5.2 scandir-1.10.0 simplegeneric-0.8.1 traitlets-4.3.3 wcwidth-0.2.5
[16:16:13] You are using pip version 10.0.1, however version 21.0.1 is available.
[16:16:13] You should consider upgrading via the 'pip install --upgrade pip' command.
[16:16:14] [root@hp1 pip-10.0.1]# 

2.4 再次安装pyspark

再次安装pyspark，同样的报错

[16:16:48] [root@hp1 pip-10.0.1]# pip install pyspark==2.4.0
[16:16:48] Collecting pyspark==2.4.0
[16:16:55]   Using cached https://files.pythonhosted.org/packages/88/01/a37e827c2d80c6a754e40e99b9826d978b55254cc6c6672b5b08f2e18a7f/pyspark-2.4.0.tar.gz
[16:17:04]     Complete output from command python setup.py egg_info:
[16:17:04]     Could not import pypandoc - required to package PySpark
[16:17:04]     /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'long_description_content_type'
[16:17:04]       warnings.warn(msg)
[16:17:04]     zip_safe flag not set; analyzing archive contents...
[16:17:04]     pypandoc.__init__: module references __file__
[16:17:04]     
[16:17:04]     Installed /tmp/pip-install-5otCUm/pyspark/.eggs/pypandoc-1.5-py2.7.egg
[16:17:04]     Searching for wheel>=0.25.0
[16:17:04]     Reading https://pypi.python.org/simple/wheel/
[16:17:04]     Downloading https://files.pythonhosted.org/packages/ed/46/e298a50dde405e1c202e316fa6a3015ff9288423661d7ea5e8f22f589071/wheel-0.36.2.tar.gz#sha256=e11eefd162658ea59a60a0f6c7d493a7190ea4b9a85e335b33489d9f17e0245e
[16:17:04]     Best match: wheel 0.36.2
[16:17:04]     Processing wheel-0.36.2.tar.gz
[16:17:04]     Writing /tmp/easy_install-6V5vRU/wheel-0.36.2/setup.cfg
[16:17:04]     Running wheel-0.36.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-6V5vRU/wheel-0.36.2/egg-dist-tmp-v5OZ83
[16:17:04]     warning: no files found matching '*.dynlib' under directory 'tests'
[16:17:04]     no previously-included directories found matching 'tests/testdata/*/build'
[16:17:04]     no previously-included directories found matching 'tests/testdata/*/dist'
[16:17:04]     no previously-included directories found matching 'tests/testdata/*/*.egg-info'
[16:17:04]     warning: install_lib: 'build/lib' does not exist -- no Python modules to install
[16:17:04]     
[16:17:04]     zip_safe flag not set; analyzing archive contents...
[16:17:04]     Moving UNKNOWN-0.0.0-py2.7.egg to /tmp/pip-install-5otCUm/pyspark/.eggs
[16:17:04]     
[16:17:04]     Installed /tmp/pip-install-5otCUm/pyspark/.eggs/UNKNOWN-0.0.0-py2.7.egg
[16:17:04]     Traceback (most recent call last):
[16:17:04]       File "<string>", line 1, in <module>
[16:17:04]       File "/tmp/pip-install-5otCUm/pyspark/setup.py", line 224, in <module>
[16:17:04]         'Programming Language :: Python :: Implementation :: PyPy']
[16:17:04]       File "/usr/lib64/python2.7/distutils/core.py", line 112, in setup
[16:17:04]         _setup_distribution = dist = klass(attrs)
[16:17:04]       File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 315, in __init__
[16:17:04]         self.fetch_build_eggs(attrs['setup_requires'])
[16:17:04]       File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 361, in fetch_build_eggs
[16:17:04]         replace_conflicting=True,
[16:17:04]       File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 849, in resolve
[16:17:04]         raise DistributionNotFound(req, requirers)
[16:17:04]     pkg_resources.DistributionNotFound: The 'wheel>=0.25.0' distribution was not found and is required by pypandoc
[16:17:04]     
[16:17:04]     ----------------------------------------
[16:17:05] Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-5otCUm/pyspark/
[16:17:05] You are using pip version 10.0.1, however version 21.0.1 is available.
[16:17:05] You should consider upgrading via the 'pip install --upgrade pip' command.
[16:18:03] [root@hp1 pip-10.0.1]# 

突然看到上面的报错
pkg_resources.DistributionNotFound: The ‘wheel>=0.25.0’ distribution was not found and is required by pypandoc

是不是缺少包呢，我直接通过pip安装 wheel试试。
总算大功告成

[16:19:40] [root@hp1 pip-10.0.1]# pip install wheel
[16:19:40] Collecting wheel
[16:19:41]   Downloading https://files.pythonhosted.org/packages/65/63/39d04c74222770ed1589c0eaba06c05891801219272420b40311cd60c880/wheel-0.36.2-py2.py3-none-any.whl
[16:19:41] Installing collected packages: wheel
[16:19:41] Successfully installed wheel-0.36.2
[16:19:41] You are using pip version 10.0.1, however version 21.0.1 is available.
[16:19:41] You should consider upgrading via the 'pip install --upgrade pip' command.
[16:19:44] [root@hp1 pip-10.0.1]# 
[16:19:44] [root@hp1 pip-10.0.1]# 
[16:19:47] [root@hp1 pip-10.0.1]# pip install pyspark==2.4.0
[16:19:48] Collecting pyspark==2.4.0
[16:19:54]   Using cached https://files.pythonhosted.org/packages/88/01/a37e827c2d80c6a754e40e99b9826d978b55254cc6c6672b5b08f2e18a7f/pyspark-2.4.0.tar.gz
[16:20:01] Collecting py4j==0.10.7 (from pyspark==2.4.0)
[16:20:02]   Downloading https://files.pythonhosted.org/packages/e3/53/c737818eb9a7dc32a7cd4f1396e787bd94200c3997c72c1dbe028587bd76/py4j-0.10.7-py2.py3-none-any.whl (197kB)
[16:20:03]     100% |████████████████████████████████| 204kB 951kB/s 
[16:20:03] Building wheels for collected packages: pyspark
[16:20:12]   Running setup.py bdist_wheel for pyspark ... done
[16:20:12]   Stored in directory: /root/.cache/pip/wheels/cd/54/c2/abfcc942eddeaa7101228ebd6127a30dbdf903c72db4235b23
[16:20:14] Successfully built pyspark
[16:20:14] Installing collected packages: py4j, pyspark
[16:20:14]   Found existing installation: py4j 0.10.9
[16:20:14]     Uninstalling py4j-0.10.9:
[16:20:14]       Successfully uninstalled py4j-0.10.9
[16:20:15] Successfully installed py4j-0.10.7 pyspark-2.4.0
[16:20:15] You are using pip version 10.0.1, however version 21.0.1 is available.
[16:20:15] You should consider upgrading via the 'pip install --upgrade pip' command.
[16:20:18] [root@hp1 pip-10.0.1]# 
[16:20:18] [root@hp1 pip-10.0.1]# 
[16:20:38] [root@hp1 pip-10.0.1]# 
[16:20:38] [root@hp1 pip-10.0.1]# 
[16:20:38] [root@hp1 pip-10.0.1]# 
[16:20:40] [root@hp1 pip-10.0.1]# pyspark
[16:20:40] Python 2.7.5 (default, Apr  2 2020, 13:16:51) 
[16:20:40] [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux2
[16:20:40] Type "help", "copyright", "credits" or "license" for more information.
[16:20:42] 2021-04-09 16:20:57 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[16:20:42] Setting default log level to "WARN".
[16:20:42] To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
[16:20:44] Welcome to
[16:20:44]       ____              __
[16:20:44]      / __/__  ___ _____/ /__
[16:20:44]     _\ \/ _ \/ _ `/ __/  '_/
[16:20:44]    /__ / .__/\_,_/_/ /_/\_\   version 2.4.0
[16:20:44]       /_/
[16:20:44] 
[16:20:44] Using Python version 2.7.5 (default, Apr  2 2020 13:16:51)
[16:20:44] SparkSession available as 'spark'.
[16:20:45] >>> 
[16:20:45] >>> 
[16:27:53] >>> exit;
[16:27:53] Use exit() or Ctrl-D (i.e. EOF) to exit
[16:27:55] >>> exit()
[16:34:27] [root@hp1 pip-10.0.1]#