Sudheer

EMR Spark reference

2023-05-16T00:00:00+00:00

Quick gotchas to watch out for while submitting Spark jobs on EMR cluster

EMR comes pre-installed with PySpark. Don’t install pyspark again as part of bootstrap actions or otherwise. Ref: https://stackoverflow.com/questions/63210509/com-amazon-ws-emr-hadoop-fs-emrfilesystem-not-found-on-pyspark-script-on-aws-emr
When zipping the contents of your source, ensure the zip root directly contains the source without an additional root folder. Use the following script:

cd <source_dir>
zip -r sources.zip .

To pass .env files to spark, prefix each entry with spark.yarn.appMasterEnv. like below and pass to spark via --properties-file param
```
spark.yarn.appMasterEnv.SEARCH_URL=https://search.domain.com
```
To pass JSON config files to spark use --files param

Here is a sample spark-submit with all of the above:

spark-submit --master yarn \
          --deploy-mode cluster \
          --packages org.apache.hadoop:hadoop-aws:3.3.1,com.amazonaws:aws-java-sdk-emr:1.12.468 \
          --properties-file .env \
          --files config1.json,config2.json \
          --py-files s3://bucket/path/sources.zip \
          spark_main.py arg1 arg2

Plotly Quick Reference

2022-05-06T00:00:00+00:00

Elastic Search quick reference

2021-10-17T00:00:00+00:00

Install

Install dependencies

pip install elasticsearch
pip install elasticsearch_dsl

Import

from elasticsearch import Elasticsearch
from elasticsearch_dsl import Search, Q

Connect

es_client = Elasticsearch([{"host": "es.domain.tld", "port": 9200}])

Query

search = Search(using=es_client, index="employee_index").query(
    "bool",
    must=[
        Q("range", created_date={"gte": "2021-10-01", "lt": "2021-10-18"}),
        Q("terms", department=["IT", "HR"]),
        Q("match", role="Manager")
    ]
)

terms is similar to SQL in clause. match is similar to SQL where column=value. range allows to search for a range of values with arithmetic operators.

Use Result

Check number of result records.

size = search.count()

Iterate through results

if size:
    try:
        for row in search.scan:
            print(row[col])
            print(row.to_dict())
    except StopIteration:
        pass

PySpark quick reference

2021-10-17T00:00:00+00:00

Load

j_df = spark.read.json("path_to/params.json")
p_df = spark.read.parquet("path_to/data-0000.parquet")

Load from multiple locations

p_df = spar.read.option("basePath", "~/files").parquet(*["~/files/folder1/*.parquet", "~/files/folder2/file1.parquet"])

Convert

pd_df = j_df.toPandas()

Filter

Rows

df = p_df.where("department=='HR'")
df = df.filter(df.name.isin(["Sudheer", "Sundeep"]))

Columns

p_df.drop("temp_timestamp",)

Add Column

df.withColumn("User Since", df.updated - df.created)

Group & Order

p_df.groupby("department").count().orderBy("department")

Save

p_df.write.parquet("path_to_save_folder/")
# with coalesce
p_df.coalesce(100).write.format("parquet").save("path_to_save_folder/", header="true")
# append, not overwrite
p_df.write.mode("append").parquet("save_path")
# save across partitions
p_df.write.partitionBy("department").parquet("save_path")

Matplotlib quick reference

2021-10-11T00:00:00+00:00

Create multiple plots within a fig

fig, axes = plt.subplots(no_of_rows, no_of_cols)

Flatten the axes array to ease iteration

flat_axes = np.array(axes).ravel()

Set x & y labels

axis.set(xlabel="date", ylabel="price")

Position legend

axis.legend(bbox_to_anchor=(1.01, 1), borderaxespad=0, loc="upper right")

Draw multiple graph types on the same subplot

ax2 = axis.twinx()

Fix fig size (clipping and/or overflow)

plt.tight_layout()

Save the figure

fig.savefig("plot.png", dpi=100)

Close the plot to release memory

Especially useful when many plots are needed.
plt.close(fig)

Seaborn quick reference

2021-10-11T00:00:00+00:00

Let the dataframe contain 4 columns

$ df.columns
[`time`, `current_price`, `price_change`, `margin`]

To draw time on x-axis vs rest on y-axis (multiple y columns)

sns.lineplot(
    x="time",
    y="value",
    hue="variable",
    data=pd.melt(df, ["time"])
)

Pandas quick reference

2021-10-07T00:00:00+00:00

Load

Load dataframe from a file.

df = pd.read_parquet("sample.parquet")

Update

Replace cell values

df.replace({"find1": "replace1", "find2": "replace2"})

Create/update column by applying a function on existing column

Slow. Prefer vectorized form

df["score_in_percent"] = df["score"].apply(lambda cell: cell * 100)

Rename a column

df.rename({"height": "height_in_inches"}, axis=1)

Update a cell

df.loc[index, "height_in_cms"] = df.loc[index, "height_in_inches"] * 2.54
#OR
df.iloc[row_idx, col_idx] = 100

Create new index ignoring existing

df.reset_index()

Reset an existing index/level

df.reset_index(name="counts")

Convert timestamp into pandas timestamp

pd.Timestamp(ts, unit="ms")

Rename axis

df.rename_axis("sample")

Drop duplicate rows with same name

df.drop_duplicates(["name"])

Add a row

new_df = pd.DataFrame(columns=["ID", "Name"])
row = {"ID": "2365782", "Name": "Sudheer"}
new_df.append(row, ignore_index=True)

Convert datatype

df = df.astype({"ID": "int64"})

Concat two dataframes by row

pd.concat([existing_employees_df, new_employees_df], axis=0, ignore_index=True)

Filter

Filter by column names

df.filter(items=["col1", "col4"])
# OR
df[["col1", "col4"]]

Find unique values of a column

df["height"].unique()
# Unique values with counts
df["height"].value_counts()

Filter based on condition Similar to SQL where clause.

df_tall = df[df["height_in_inches"] >= 72.]
# OR
df_tall = df.loc[df["height_in_inches"] >= 72.]

Multiple where clauses

df_big_tall = df.loc[
    (df["height_in_inches"] >= 72.) & (df["weight_in_kgs"] > 90)
]

Similar to SQL in clause

df_it_hr = df[df["department"].isin(["IT", "HR"])]

Loop

Iterate through rows

Avoid when possible

for index, row in df.iterrows():
    pass

Read

Access cell at given row index & column name

df.at[0, "height"]

Dataframe size = rows * cols

df.size

Get a column’s index

df.columns.get_loc("height")

Group & Order

Group by column(s)

df_group = df.groupby("department")

Group size, similar to SQL count(*)

df_group.size()

Sort series

df_height_sorted = df["heigh"].sort()

Sort dataframe

df.sort_values(by=["created_date"], ascending=True)

Save

df.to_parquet("path/to/save/file.parquet")

Date-Time

Get dates between a range of dates

pd.date_range("2022-01-28", "2022-02-04")

Get hours between range of dates/timestamps

pd.date_range("2022-01-28T00:00:00", "2022-01-29T08:00:00", freq="H")

Spark installation on macOS

2021-01-05T00:00:00+00:00

homebrew, Python3 is assumed to be installed. Verified on macOS BigSur 11.6

Install Java

brew install java

Install Scala

brew install scala

Install Spark

brew install apache-spark

Install Python & libs

It is assumed Python3 is installed

pip install pyspark

Setup env

echo "export JAVA_HOME=/Library/Java/JavaVirtualMachines/openjdk.jdk/Contents/Home/"  >> ~/.zshrc
echo "export JRE_HOME=/Library/Java/JavaVirtualMachines/openjdk.jdk/Contents/Home/"  >> ~/.zshrc

echo "export SPARK_HOME=/usr/local/Cellar/apache-spark/3.1.2/libexec"  >> ~/.zshrc
echo "export PATH=/usr/local/Cellar/apache-spark/3.1.2/bin:$PATH"  >> ~/.zshrc

echo "export PYSPARK_PYTHON=/Users/sudheerb/.pyenv/shims/python"  >> ~/.zshrc
echo "export PYSPARK_DRIVER_PYTHON=jupyter"  >> ~/.zshrc
echo "export PYSPARK_DRIVER_PYTHON_OPTS='notebook'"  >> ~/.zshrc
source ~/.zshrc

Verify installation

Launch pyspark from the directory containing iPython notebook

➜  pyspark
[I 18:33:57.084 NotebookApp] JupyterLab extension loaded from .../opt/anaconda3/lib/python3.7/site-packages/jupyterlab
[I 18:33:57.085 NotebookApp] JupyterLab application directory is .../opt/anaconda3/share/jupyter/lab
[I 18:33:57.090 NotebookApp] Serving notebooks from local directory: .../spark
[I 18:33:57.090 NotebookApp] The Jupyter Notebook is running at:
[I 18:33:57.090 NotebookApp] http://localhost:8888/?token=....
[I 18:33:57.090 NotebookApp]  or http://127.0.0.1:8888/?token=....
[I 18:33:57.090 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 18:33:57.127 NotebookApp] 
    
    To access the notebook, open this file in a browser:
        file:///..../Library/Jupyter/runtime/nbserver-6161-open.html
    Or copy and paste one of these URLs:
        http://localhost:8888/?token=....
     or http://127.0.0.1:8888/?token=....

Write a sample spark program & check the result

import pyspark
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('HelloSpark').getOrCreate()
df = spark.sql(''' select 'SQL' as Hello ''')
df.show()

+-----+
|Hello|
+-----+
|  SQL|
+-----+

All in one Hadoop installation on Ubuntu

2020-12-25T00:00:00+00:00

Introduction

This article describes the process of setting up a single-node aka pseudonode aka all-in-one Hadoop. Before getting started, ensure you have a Ubuntu 20.04.1 LTS running on a VM ready. Following hardware is recommended for the VM:

2 vCPUs
4GB RAM
50GB disk space

Details

Install Java

As Hadoop is Java based, installing JDK is a prerequisite.

sudo apt-get update
sudo apt-get install -y openjdk-8-jdk
echo "export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64" >> ~/.bashrc
echo "export JRE_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre" >> ~/.bashrc

Install Hadoop

wget https://mirrors.estointernet.in/apache/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz
sudo tar -xvf hadoop-3.2.1.tar.gz -C /opt/
cd /opt/
sudo mv hadoop-3.2.1 hadoop
sudo chown -R sudheer:sudheer hadoop
echo "export HADOOP_HOME=/opt/hadoop" >> ~/.bashrc
echo "export PATH=$PATH:/opt/hadoop/bin" >> ~/.bashrc
echo "export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64" >> /opt/hadoop/etc/hadoop/hadoop-env.sh
source ~/.bashrc
hadoop version

Install Spark

cd ~
wget https://mirrors.estointernet.in/apache/spark/spark-3.0.1/spark-3.0.1-bin-hadoop3.2.tgz
sudo tar -xvf spark-3.0.1-bin-hadoop3.2.tgz -C /opt/
cd /opt/
sudo mv spark-3.0.1-bin-hadoop3.2 spark
sudo chown -R sudheer:sudheer spark
echo "export SPARK_HOME=/opt/spark" >> ~/.bashrc
echo "export PATH=$PATH:/opt/spark/bin" >> ~/.bashrc
source ~/.bashrc
spark-shell --version

Configure

Here we configure Hadoop & Spark.

sudo echo '<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><configuration><property><name>fs.defaultFS</name><value>hdfs://localhost:9000</value></property></configuration>' > /opt/hadoop/etc/hadoop/core-site.xml
sudo echo '<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><configuration><property><name>dfs.datanode.data.dir</name><value>file:///opt/hadoop_tmp/hdfs/datanode</value></property><property><name>dfs.namenode.name.dir</name><value>file:///opt/hadoop_tmp/hdfs/namenode</value></property><property><name>dfs.replication</name><value>1</value></property></configuration>' > /opt/hadoop/etc/hadoop/hdfs-site.xml
sudo mkdir -p /opt/hadoop_tmp/hdfs/datanode
sudo mkdir -p /opt/hadoop_tmp/hdfs/namenode
sudo chown sudheer:sudheer -R /opt/hadoop_tmp
sudo echo '<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><configuration><property><name>mapreduce.framework.name</name><value>yarn</value></property><property><name>yarn.app.mapreduce.am.env</name><value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value></property><property><name>mapreduce.map.env</name><value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value></property><property><name>mapreduce.reduce.env</name><value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value></property></configuration>' > /opt/hadoop/etc/hadoop/mapred-site.xml
sudo echo '<?xml version="1.0"?><configuration><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property><property><name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name><value>org.apache.hadoop.mapred.ShuffleHandler</value></property></configuration>' > /opt/hadoop/etc/hadoop/yarn-site.xml

Setup SSH

This is required to communicate with Hadoop components viz. HDFS

sudo apt-get install -y openssh-server
ssh localhost
hdfs namenode -format -force
echo "export PATH=$PATH:/opt/hadoop/sbin" >> ~/.bashrc
source ~/.bashrc
ssh-keygen
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

You can create password less SSH key for development purposes. Production environments should use a strong password.

Start Hadoop & run a sample job

Verify the setup so far

start-dfs.sh && start-yarn.sh
jps
yarn jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar pi 1 50

Install MySQL

This is optional. We will import data from MySQL to Hadoop using Sqoop. You can use any other SQL database.

sudo apt-get install -y mysql-server
sudo mysql --defaults-file=/etc/mysql/debian.cnf

Install Sqoop

This is used to work with MySQL database installed above.

wget https://mirrors.estointernet.in/apache/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz
lsb_release -a
sudo tar -xvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /opt/
cd /opt/
sudo mv sqoop-1.4.7.bin__hadoop-2.6.0 sqoop
sudo chown -R sudheer:sudheer sqoop
echo "export SQOOP_HOME=/opt/sqoop" >> ~/.bashrc
echo "export PATH=$PATH:/opt/sqoop/bin" >> ~/.bashrc
source ~/.bashrc
cd /opt/sqoop/conf/
cp sqoop-env-template.sh sqoop-env.sh
echo "export HADOOP_COMMON_HOME=/opt/hadoop" >> sqoop-env.sh
echo "export HADOOP_MAPRED_HOME=/opt/hadoop" >> sqoop-env.sh
sqoop-version

Fix sqoop

The commons-lang JAR that comes prepackaged with Sqoop doesn’t work. So use commons-lang v2.6 JAR instead.

cd ~
wget https://repo1.maven.org/maven2/commons-lang/commons-lang/2.6/commons-lang-2.6.jar
cp commons-lang-2.6.jar /opt/sqoop/lib

Install MySQL driver to sqoop

Sqoop talks to MySQL through this JDBC driver.

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java_8.0.22-1ubuntu20.04_all.deb
dpkg-deb -xv mysql-connector-java_8.0.22-1ubuntu20.04_all.deb newfolder
cp newfolder/usr/share/java/mysql-connector-java-8.0.22.jar /opt/sqoop/lib

Install Hive

Hive is used to communicate with Hadoop in SQL lingo.

wget https://mirrors.estointernet.in/apache/hive/hive-3.1.2/apache-hive-3.1.2-bin.tar.gz
sudo tar -xvf apache-hive-3.1.2-bin.tar.gz -C /opt/
sudo mv /opt/apache-hive-3.1.2-bin /opt/hive
sudo chown -R sudheer:sudheer /opt/hive
echo "export HIVE_HOME=/opt/hive" >> ~/.bashrc
echo "export PATH=\$PATH:/opt/hive/bin" >> ~/.bashrc
source ~/.bashrc
cp /opt/hive/conf/hive-env.sh.template /opt/hive/conf/hive-env.sh
echo "export HADOOP_HOME=/opt/hadoop" >> /opt/hive/conf/hive-env.sh
echo "export HIVE_CONF_DIR=/opt/hive/conf" >> /opt/hive/conf/hive-env.sh
hdfs dfs -mkdir -p /user/hive/warehouse
hdfs dfs -chmod 775 /tmp
hdfs dfs -chmod 775 /user/hive/warehouse
cp /opt/hive/conf/hive-default.xml.template /opt/hive/conf/hive-site.xml
rm /opt/hive/lib/guava-19.0.jar
cp /opt/hadoop/share/hadoop/hdfs/lib/guava-27.0-jre.jar /opt/hive/lib
$HIVE_HOME/bin/schematool -initSchema -dbType derby

Summary

We have installed & verified the installation of Hadoop 3.2.1 on Ubuntu VM.

Working with Weather Dataset using CDO and NCView

2020-12-25T00:00:00+00:00

Introduction

CDO (Climate Data Operators) is a set of command line Operators to manipulate and analyse Climate data. netCDF 3/4 is one of the supported formats which we demonstrate here.

NCView is a visual browser for netCDF format files. It runs on UNIX platforms under X11.

ncdump command-line utility converts netCDF data to human-readable text form.

CDO

Extract a subset of dataset. In this case, selecting months 6-9 and writing to a file.

cdo -f nc4 -selmon,6/9 Rainfall.nc Summar_Rainfall.nc

Calculate yearly mean

cdo -f nc4 -yearmean Summar_Rainfall.nc year_mean.nc

Trend analysis / linear regression

cdo -f nc4 trend year_mean.nc trend1.nc trend2.nc

Export .nc to .tsv

cdo outputtab,year,lon,lat,value year_mean.nc > year_mean.tsv

ncdump

Dump info about a dataset (similar to pandas pd.head())

ncdump -h Rainfall.nc

Dump detailed info about specific variable

ncdump -t -v lat Rainfall.nc
ncdump -t -v time Rainfall.nc

NCView

Open ncview with a dataset.

ncview Summar_Rainfall.nc