SVIGS https://svitglobal.in Your Digital Solution Partner Thu, 12 Jun 2025 07:07:38 +0000 en-US hourly 1 https://wordpress.org/?v=7.1.2 https://svitglobal.in/wp-content/uploads/2025/06/cropped-svigs-logo-32x32.jpg SVIGS https://svitglobal.in 32 32 Transforming Open Data LakeHouses with Apache Spark https://svitglobal.in/transforming-open-data-lakehouses-with-apache-spark/ Wed, 11 Jun 2025 08:12:29 +0000 https://svitglobal.in/?p=1918 In this series of Blog articles, I am going to talk about the Modern Data Lakehouses a.k.a. Open Data Platforms doing all heavy lifts & shifts and thus shaping the data as per new requirements in terms of their formats and places!

The post Transforming Open Data LakeHouses with Apache Spark first appeared on SVIGS.

]]>

Transforming Open Data LakeHouses with Apache Spark

  • Home
  • Blog
  • Transforming Open Data LakeHouses with Apache Spark

Processing Open Table / File formats for Benchmarking Performance

In this series of Blog articles, I am going to talk about the Modern Data Lakehouses a.k.a. Open Data Platforms doing all heavy lifts & shifts and thus shaping the data as per new requirements in terms of their formats and places!

Let me start with Part 1 (This Article) with basic definitions of terms being used here for sake of clarity.

Data LakeHouses

  • Combine the best elements of Data lake & Data warehouse
  • Are built on Open Source & Open Standards
  • Simplify data estate by eliminating Silos, which impedes enterprise data value creation
  • Reduce costs and delivers Data/AI Initiatives faster!

Conceptualizing a Data Lakehouse

A Data Lakehouse is a new, open data management architecture that combines the flexibility, cost-efficiency, and scale of data lakes with the data management and ACID transactions of data warehouses, enabling business intelligence (BI) and machine learning (ML) on all data.

We plan to leverage Apache Spark for data processing, using open file and table formats such as:

Delta Lake

  • Parquet

Iceberg

  • Parquet
  • ORC
  • AVRO

Popular Big Data Format s — Open Source

Here we’re gonna exploit object storage into a scalable, high-performance, and secure data lakehouse solution.

We will be benchmarking performance of our DLH using https://www.tpc.org/tpcds/ : The Transaction Processing Performance Council (TPC) defines Transaction Processing and Database Benchmarks and delivers trusted results to the industry.

While setting up the system and during experiments, I realize that the combination of Apache Spark, Hive Metastore, and Open file/table formats on cloud object storage creates a powerful solution for data analytics that emphasizes performance, scalability, and ease of management.

  • Apache Spark provides a distributed computing engine for processing large datasets
  • Hive Metastore acts as a centralized metadata repository to manage table schemas and ensure consistency
  • By integrating cloud object storage having Open file/table data, this solution provides a scalable and fast storage backend capable of handling high-throughput data operations, making it ideal for modern data lakehouse architectures.

Next, we are gonna do Environment setup so that we can conduct our experiments using the aforementioned Open Source tech stack.

Original blog link https://shuklaashutosh.medium.com/transforming-open-data-lakehouses-with-apache-spark-1524bb343e8f

The post Transforming Open Data LakeHouses with Apache Spark first appeared on SVIGS.

]]> Setting Up Apache Spark Environment The Standalone Mode https://svitglobal.in/setting-up-apache-spark-environment-the-standalone-mode/ Wed, 11 Jun 2025 05:11:40 +0000 https://svitglobal.in/?p=1880 Server (Node) & Software Setup for Apache Spark and Hive Meta Store

The post Setting Up Apache Spark Environment The Standalone Mode first appeared on SVIGS.

]]>

Setting Up Apache Spark Environment The Standalone Mode

  • Home
  • Blog
  • Setting Up Apache Spark Environment The Standalone Mode

1. Server (Node) & Software Setup for Apache Spark and Hive Meta Store

Powering up Apache Spark Cluster for High-Scale Data Transformations

Master

  • VM for spark Master — 128 Cores / 256 GB
  • Configure Servers with Ubuntu Linux OS (Ubuntu v22.04.5)

Workers

  • 8 x Dell R740 — 96 Cores / 256 GB
  • 8 x Dell R6515 — 128 Cores / 256 GB
  • Configure Servers with Ubuntu Linux OS (Ubuntu v22.04.5)

Object Storage

  • Prepare servers for running Apache Spark & Integrating with suitable Object Storage for Data Platform

2. Steps to Download, Install and Configure Apache Spark in Standalone mode

  • Install Java Development Kit (JDK) version 8 or above
  • Download latest stable release (e.g., Spark 3.5.2) from the Apache Spark download page
  • Set Environment Variables & reload the profile
  • Edit .bashrc or /etc/profile to include Spark-related environment variables

Then,

  • Start the Master Node
$SPARK_HOME/sbin/start-master.sh

Start a Worker Node — By default, workers connect to the master at spark://<hostname>:7077

$SPARK_HOME/sbin/start-worker.sh spark://:7077

Access the Web UI — Monitor the Spark cluster from the web UI at http://<master-IP>:8080

3. Steps to Download, Install and Configure Hive Meta Store with PostgreSql

Hive Meta Store is required to store metadata into the tables and PostgreSql can be used as its backend database to ensure durability and reliability.

  • Download Hive from the Apache Hive download page
  • Edit .bashrc or /etc/profile to include Hive-related environment variables
  • Configure Hive to Use PostgreSQL — Add Postgres Connector to Hive
  • Download the Postgres Connector JAR and place it in the Hive lib directory
  • Copy hive-default.xml.template hive-site.xml located in $HIVE_HOME/conf
  • Configure the hive-site.xml file
  • Install Postgresql & start service
wget https://archive.apache.org/dist/hive/hive-3.1.3/apache-hive-3.1.3-bin.tar.gz
tar -xzf apache-hive-3.1.3-bin.tar.gz
mv apache-hive-3.1.3-bin /usr/local/hive
export HIVE_HOME=/usr/local/hive
export PATH=$PATH:$HIVE_HOME/bin
source ~/.bashrc
wget https://repo1.maven.org/maven2/org/postgresql/postgresql/42.7.0/postgresql-42.7.0.jar
cp postgresql-42.7.0.jar $HIVE_HOME/lib/
cp postgresql-42.7.0.jar $SPARK_HOME/jars

sudo apt update
sudo apt install postgresql -y
systemctl start postgresql.service
  • Create a database for the Hive Meta Store and a user with appropriate permissions:
su - postgres
psql
CREATE DATABASE hivemetastore;
CREATE USER hiveuser WITH PASSWORD 'hivepassword';
GRANT ALL PRIVILEGES ON DATABASE hivemetastore TO hiveuser;
\q

4. Download, Install and Configure Hadoop

wget https://archive.apache.org/dist/hadoop/common/hadoop-3.3.4/hadoop-3.3.4.tar.gz
tar -xzf hadoop-3.3.4.tar.gz
mv hadoop-3.3.4 /usr/local/hadoop
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
source ~/.bashrc
  • Configure Hadoop-env.sh (in Hadoop_Home) to access Object Storage.
  • Download core-site.xml and Update the variables like bucket name, s3 access key & secret key and endpoints
export HADOOP_OPTIONAL_TOOLS="hadoop-aws"
hadoop fs -ls s3a://
  • Initialize the Hive Meta Store (HMS) Schema
cd $HIVE_HOME/bin
$HIVE_HOME/bin/schematool -dbType postgres -initSchema

su - postgres
psql
postgres=# \c hivemetastore
#You are now connected to database "hivemetastore" as user "postgres".

hivemetastore=# GRANT CREATE ON SCHEMA public TO hiveuser;
#GRANT

hivemetastore=# \q

5. Connect Apache Spark to Hive Metastore

  • Copy the Hive configuration file (hive-site.xml) to the Spark configuration directory ($SPARK_HOME/conf)
  • Start Spark Shell with Hive Support
cp $HIVE_HOME/conf/hive-site.xml $SPARK_HOME/conf/
spark-shell - conf spark.sql.catalogImplementation=hive

6. Validate the Installation

  • Start the HiveServer to interact with Hive through JDBC or Beeline
  • create a test table in the Hive Meta Store
  • Open a Spark shell and run the following commands to query the Hive table:
$HIVE_HOME/bin/hive - service hiveserver2 &
/usr/local/hive/bin/hive - service metastore &
/usr/local/hive/bin/beeline -u jdbc:hive2://

-- # at Hive Prompt

>> CREATE TABLE test_table (id INT, name STRING);
>> spark.sql("SHOW TABLES").show()
>> spark.sql("SELECT * FROM test_table").show()

test_table gets created in Hive Meta Store (HMS) and this can be verified using Spark SQL Describe commands.

Original blog link https://shuklaashutosh.medium.com/setting-up-apache-spark-environment-the-standalone-mode-130b50af1a63

The post Setting Up Apache Spark Environment The Standalone Mode first appeared on SVIGS.

]]> DevOps Interview https://svitglobal.in/devops-interview/ Tue, 10 Jun 2025 11:48:01 +0000 https://svitglobal.in/?p=1869 Can you give a brief introduction about yourself?
Mr. Mahesh Chandra has over two decades of experience in delivering data solutions to various corporations including fortune 500 companies.

The post DevOps Interview first appeared on SVIGS.

]]>

DevOps Interview

Can you give a brief introduction about yourself?

Mr. Mahesh Chandra has over two decades of experience in delivering data solutions to various corporations including fortune 500 companies. This covers close to a decade’s experience working in US and UK at client location. He brings with him extensive knowledge and skills in Architecting, Project Execution & Management for Design & Development /Enhancement/Service Support of DWH/BI solutions, for clients in various sectors and geographical regions. His core skills are Data Architecture, Data Modelling, Informatica, Data Stage and SAP. Moreover, he has good command over Big Data and Google Cloud Platform technologies including DevOps.

Can you introduce your company briefly?

SVIT Global Solutions is a boutique digital transformation company that provides Cloud data solutions, DevOps and integration services to the clients. We provide end-to-end services starting from defining architecture, migration strategy, development, and support for data and analytics projects. With the help of a highly skilled and dedicated team, we add value in each step of your business process.

What is DevOps?

DevOps by the inherent nature of the capabilities can be used for the end to end process like:

  • Continuous integration of the code (Build generation)
  • Continuous deployment of bundles (Artifact deployment)
  • Configuration management of the platforms (Sync between environments)
  • Automated testing within the CI/CD pipeline
  • Security implementations as a part of overall automation
  • Setting up of platforms as a part of IAAC (Infrastructure as a code)

What are the benefits of DevOps?

  • Elimination of human errors related to deployment
  • Prevention of mis-configuration during artifact preparation
  • Validation of every deployment steps before actual deployment into various environments
  • Possibility of using deployment strategies like Blue/Green, Canary deployments, Red/Black deployments
  • Automation of repeatable activities to help with economies of scale

How does SV IT Global Solutions provide DevOps solutions?

SV IT Global Solutions adopt Agile delivery model for the successful deployment of DevOps solutions.

At the start of the application development project the team understands the requirement from the client and develops user stories as mentioned by the end user. The list of features based on these stories will serve as backlog for the development team.

The product will be developed in multiple increments by following Minimum Viable Product (MVP) approach.

Sprint Planning

Sprint planning initiates the sprint by identifying user stories to be delivered for the sprint. The plan is created by collaborative work of the entire team.

Daily Scrum

This is a non-traditional approach to application development where the team meets daily for about 15 minutes to create a plan for the day. The main objective of the team will be to remove impediments in achieving the sprint goal.

Sprint Review/Retrospection

Sprint reviews are conducted at the end of each sprint, the purpose is to review the outcome of the sprint and determine future adaptations. The team reviews the sprint outcome with its stakeholders and what has changed in the environment.

Continuous Integration

While multiple developers work on different features, code is routinely integrated into a main repository, where it is tested and integrated.

Continuous Testing

Through automated testing, released codes are tested early. The goal is to determine the impact of the changes in the code as fast as possible and whether the changes can be safely deployed into production.

Continuous Delivery

When the code is able to go through integration and testing, it is then automatically deployed into a dev/staging environment. The team makes the decision to deploy the changes into production. Thus, a minimum viable product (MVP) is made available.

The post DevOps Interview first appeared on SVIGS.

]]> Modern Data Pipeline https://svitglobal.in/modern-data-pipeline/ Tue, 10 Jun 2025 09:38:46 +0000 https://svitglobal.in/?p=1832 As organisation are adopting data-driven approach to grow and create value to their business, but the challenges in the traditional methods to get required for analysis takes days and months for business to use the data.

The post Modern Data Pipeline first appeared on SVIGS.

]]>

Modern Data Pipeline

As organisation are adopting data-driven approach to grow and create value to their business, but the challenges in the traditional methods to get required for analysis takes days and months for business to use the data.

The modern data pipelines make it easy to move data and extract value from it within hours or days. In this article we will cover definitions and examples of data pipelines, must-have features of modern data pipelines.

Data Pipeline

A data pipeline is a set of tools and processes used to automate the movement and transformation of data between a source system and a target repository.

Source systems often have different methods of processing and storing data than target systems. Therefore, data pipeline software automates the process of extracting data from many disparate source systems, transforming, combining and validating that data, and loading it into the target repository.

Building data pipelines breaks down data silos and creates 360degree view of the business. Business can then apply BI and analytics tools to create data visualisations and dashboards to derive and share actionable insights from the data.

Standard data pipelines include

A batch data pipeline periodically transfers bulk data from source to target. For example, the pipeline can run once every twelve hours. Batch pipeline can be organised to run at a specific time daily when there’s low system traffic.

Streaming data pipeline

A streaming data pipeline continually flows data from source to target while translating the data into a target format in real-time. This is suitable for data that requires continuous updating. For example, to transfer data collected from a sensor tracking traffic.

ETL

ETL (extract, transform and load) is a data integration process that includes extracting raw data from various data sources, transforming the data on a separate data processing server before loading into the target system such as data warehouse, data mart or a database for analysis or other purposes.

Data Pipeline vs. ETL

  1. ETL pipeline includes a series of processes that extracts data from a source, transform it, and load it into the target system. On the other hand, a data pipeline is a somewhat broader terminology that includes ETL pipeline as a subset. It includes a set of processing tools that transfer data from one system to another.
  2. ETL pipelines transform data before loading into the target system. But data pipelines can either transform data after loading it into the target system (ELT) or not transform it at all.
  3. ETL pipelines end after loading data into the target repository. But data pipelines can stream data, and therefore their load process can trigger processes in other systems or enable real-time reporting.

ELT

ELT stands for “extract, load, and transform” is the processes a data pipeline uses to replicate data from a source system into a target system such as a cloud data warehouse.

ELT is a modern variation of the older process of extract, transform, and load (ETL), in which transformations take place before the data is loaded.

As mentioned above ETL process requires a separate processing engines for running transformations prior to loading data into a target system. ELT, on the other hand uses the processing engines in the target system to transform data. This removal of an intermediate step helps in streamlining the data loading process.

Since ETL process transforms data prior to the loading stage.

The post Modern Data Pipeline first appeared on SVIGS.

]]>