gcr.io/getindata-images-public/jupyterlab-mlops:dev-4728c742faaacc79d6a0d0c696e9a97c987b0a4f

jupyterlab-mlops:dev-4728c742faaacc79d6a0d0c696e9a97c987b0a4f.txt

## https://github.com/getindata/jupyter-images/pull/65
## local spark and kedro verification
## [email protected]

docker run -it --network host --rm gcr.io/getindata-images-public/jupyterlab-mlops:dev-4728c742faaacc79d6a0d0c696e9a97c987b0a4f bash
Unable to find image 'gcr.io/getindata-images-public/jupyterlab-mlops:dev-4728c742faaacc79d6a0d0c696e9a97c987b0a4f' locally
dev-4728c742faaacc79d6a0d0c696e9a97c987b0a4f: Pulling from getindata-images-public/jupyterlab-mlops

Digest: sha256:311f27e69fc3f36a4fdcf5c3b2e03d8da93fb4cf50dd9a44e68551eed4ccc89c
Status: Downloaded newer image for gcr.io/getindata-images-public/jupyterlab-mlops:dev-4728c742faaacc79d6a0d0c696e9a97c987b0a4f


(base) jovyan@PF-3ASX5A:~$ ll
total 56
drwsrws--- 1 jovyan users 4096 Oct  1 08:19 ./
drwxr-xr-x 1 root   root  4096 Oct 10  2023 ../
-rw-rw-r-- 1 jovyan users  220 Jan  6  2022 .bash_logout
-rw-rw-r-- 1 jovyan users 4254 Oct  1 08:19 .bashrc
drwsrwsr-x 3 jovyan users 4096 Oct 10  2023 .cache/
drwsrwsr-x 1 jovyan users 4096 Oct 10  2023 .conda/
drwsrwsr-x 3 jovyan users 4096 Oct 10  2023 .config/
drwsrwsr-x 1 root   users 4096 Oct  1 08:19 .ipython/
drwsrws--- 2 jovyan users 4096 Oct 10  2023 .jupyter/
drwsrwsr-x 3 jovyan users 4096 Oct 10  2023 .npm/
-rw-rw-r-- 1 jovyan users  807 Jan  6  2022 .profile
-rw-rw-r-- 1 jovyan users  206 Oct  1 08:18 .wget-hsts
drwsrwsr-x 2 jovyan users 4096 Oct 10  2023 work/


(base) jovyan@PF-3ASX5A:~$ jupyter kernelspec list
Available kernels:
  python3       /opt/conda/share/jupyter/kernels/python3
  python3.11    /opt/conda/share/jupyter/kernels/python3.11


(base) jovyan@PF-3ASX5A:~$ conda env list
# conda environments:
#
base                  *  /opt/conda


(base) jovyan@PF-3ASX5A:~$ env|sort
APACHE_SPARK_VERSION=3.5.0
_CE_CONDA=
_CE_M=
CONDA_DEFAULT_ENV=base
CONDA_DIR=/opt/conda
CONDA_EXE=/opt/conda/bin/conda
CONDA_PREFIX=/opt/conda
CONDA_PROMPT_MODIFIER=(base) 
CONDA_PYTHON_EXE=/opt/conda/bin/python
CONDA_SHLVL=1
DEBIAN_FRONTEND=noninteractive
HADOOP_VERSION=3
HOME=/home/jovyan
HOSTNAME=PF-3ASX5A
JUPYTER_ENABLE_LAB=yes
JUPYTER_PORT=8888
KEDRO_DISABLE_TELEMETRY=true
KEDRO_VERSION=0.19.8
LANG=en_US.UTF-8
LANGUAGE=en_US.UTF-8
LC_ALL=en_US.UTF-8
LESSCLOSE=/usr/bin/lesspipe %s %s
LESSOPEN=| /usr/bin/lesspipe %s
MLFLOW_TRACKING_URI=http://localhost:5000
NB_GID=100
NB_UID=1000
NB_USER=jovyan
PATH=/opt/conda/bin:/opt/conda/condabin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/spark/bin
PWD=/home/jovyan
SHELL=/bin/bash
SHLVL=0
SPARK_HOME=/usr/local/spark
SPARK_OPTS=--driver-java-options=-Xms1024M --driver-java-options=-Xmx4096M --driver-java-options=-Dlog4j.logLevel=info
TERM=xterm
_=/usr/bin/env
VSCODE_VERSION=4.91.1
XDG_CACHE_HOME=/home/jovyan/.cache/
XML_CATALOG_FILES=file:///opt/conda/etc/xml/catalog file:///etc/xml/catalog


(base) jovyan@PF-3ASX5A:~$ sudo su -
root@PF-3ASX5A:~# exit
logout


(base) jovyan@PF-3ASX5A:~$ start-notebook.sh 
Entered start.sh with args: jupyter lab
Running hooks in: /usr/local/bin/start-notebook.d as uid: 1000 gid: 100
Done running hooks in: /usr/local/bin/start-notebook.d
Running hooks in: /usr/local/bin/before-notebook.d as uid: 1000 gid: 100
Sourcing shell script: /usr/local/bin/before-notebook.d/spark-config.sh
Done running hooks in: /usr/local/bin/before-notebook.d
Executing the command: jupyter lab
[I 2024-10-01 08:39:40.593 ServerApp] Package jupyterlab took 0.0000s to import
[I 2024-10-01 08:39:40.613 ServerApp] Package jupyter_lsp took 0.0184s to import
[W 2024-10-01 08:39:40.613 ServerApp] A `_jupyter_server_extension_points` function was not found in jupyter_lsp. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2024-10-01 08:39:40.615 ServerApp] Package jupyter_server_mathjax took 0.0013s to import
[I 2024-10-01 08:39:40.681 ServerApp] Package jupyter_server_proxy took 0.0651s to import
[I 2024-10-01 08:39:40.685 ServerApp] Package jupyter_server_terminals took 0.0039s to import
[I 2024-10-01 08:39:40.707 ServerApp] Package jupyterlab_git took 0.0220s to import
[I 2024-10-01 08:39:40.710 ServerApp] Package nbclassic took 0.0023s to import
[W 2024-10-01 08:39:40.712 ServerApp] A `_jupyter_server_extension_points` function was not found in nbclassic. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2024-10-01 08:39:40.712 ServerApp] Package nbdime took 0.0000s to import
[I 2024-10-01 08:39:40.713 ServerApp] Package notebook took 0.0000s to import
[I 2024-10-01 08:39:40.715 ServerApp] Package notebook_shim took 0.0000s to import
[W 2024-10-01 08:39:40.715 ServerApp] A `_jupyter_server_extension_points` function was not found in notebook_shim. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2024-10-01 08:39:40.715 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2024-10-01 08:39:40.719 ServerApp] jupyter_server_mathjax | extension was successfully linked.
[I 2024-10-01 08:39:40.719 ServerApp] jupyter_server_proxy | extension was successfully linked.
[I 2024-10-01 08:39:40.722 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2024-10-01 08:39:40.725 ServerApp] jupyterlab | extension was successfully linked.
[I 2024-10-01 08:39:40.725 ServerApp] jupyterlab_git | extension was successfully linked.
[W 2024-10-01 08:39:40.729 ServerApp] ServerApp.token config is deprecated in 2.0. Use IdentityProvider.token.
[I 2024-10-01 08:39:40.729 ServerApp] nbclassic | extension was successfully linked.
[I 2024-10-01 08:39:40.729 ServerApp] nbdime | extension was successfully linked.
[I 2024-10-01 08:39:40.734 ServerApp] notebook | extension was successfully linked.
[I 2024-10-01 08:39:40.734 ServerApp] Writing Jupyter server cookie secret to /home/jovyan/.local/share/jupyter/runtime/jupyter_cookie_secret
[I 2024-10-01 08:39:40.931 ServerApp] notebook_shim | extension was successfully linked.
[W 2024-10-01 08:39:40.945 ServerApp] All authentication is disabled.  Anyone who can connect to this server will be able to run code.
[I 2024-10-01 08:39:40.945 ServerApp] notebook_shim | extension was successfully loaded.
[I 2024-10-01 08:39:40.947 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2024-10-01 08:39:40.947 ServerApp] jupyter_server_mathjax | extension was successfully loaded.
[I 2024-10-01 08:39:40.962 ServerApp] jupyter_server_proxy | extension was successfully loaded.
[I 2024-10-01 08:39:40.963 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2024-10-01 08:39:40.965 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.11/site-packages/jupyterlab
[I 2024-10-01 08:39:40.965 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2024-10-01 08:39:40.974 LabApp] Extension Manager is 'pypi'.
[I 2024-10-01 08:39:40.976 ServerApp] jupyterlab | extension was successfully loaded.
[I 2024-10-01 08:39:40.980 ServerApp] jupyterlab_git | extension was successfully loaded.

  _   _          _      _
 | | | |_ __  __| |__ _| |_ ___
 | |_| | '_ \/ _` / _` |  _/ -_)
  \___/| .__/\__,_\__,_|\__\___|
       |_|
                                                                           
Read the migration plan to Notebook 7 to learn about the new features and the actions to take if you are using extensions.

https://jupyter-notebook.readthedocs.io/en/latest/migrate_to_notebook7.html

Please note that updating to Notebook 7 might break some of your extensions.

[I 2024-10-01 08:39:40.983 ServerApp] nbclassic | extension was successfully loaded.
[I 2024-10-01 08:39:41.024 ServerApp] nbdime | extension was successfully loaded.
[I 2024-10-01 08:39:41.027 ServerApp] notebook | extension was successfully loaded.
[I 2024-10-01 08:39:41.027 ServerApp] Serving notebooks from local directory: /home/jovyan
[I 2024-10-01 08:39:41.027 ServerApp] Jupyter Server 2.7.3 is running at:
[I 2024-10-01 08:39:41.027 ServerApp] http://PF-3ASX5A:8888/lab
[I 2024-10-01 08:39:41.027 ServerApp]     http://127.0.0.1:8888/lab
[I 2024-10-01 08:39:41.027 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2024-10-01 08:39:41.442 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server


## inside jupyterlab terminal
(base) jovyan@PF-3ASX5A:~$ kedro new -n spaceflights --tools=viz --example=yes
[10/01/24 08:40:11] INFO     Using '/opt/conda/lib/python3.11/site-packages/kedro/framework/project/rich_logging.yml' as logging configuration.

Congratulations!
Your project 'spaceflights' has been created in the directory 
/home/jovyan/spaceflights

You have selected the following project tools: ['Kedro Viz']
It has been created with an example pipeline.
## in browser open jupyterlab launcher -> kedro-viz should be available


## inside jupyterlab terminal
(base) jovyan@PF-3ASX5A:~$ pyspark
Python 3.11.6 | packaged by conda-forge | (main, Oct  3 2023, 10:40:35) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
24/10/01 08:41:09 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.5.0
      /_/

Using Python version 3.11.6 (main, Oct  3 2023 10:40:35)
Spark context Web UI available at http://PF-3ASX5A:4040
Spark context available as 'sc' (master = local[*], app id = local-1727772070117).
SparkSession available as 'spark'.
>>> 
## in browser open http://PF-3ASX5A:4040 or http://localhost:4040, pyspark UI should be available