Skip to main content

The Data Crusher

This blog is for complex data analysis problem and trying to solve in good manner.

Search This Blog

Home

Showing posts with the label sql

Posts

How to connect mysql with pyspark and make some filters

February 08, 2023

+

Get link
Facebook
X
Pinterest
Email
Other Apps

How to make balance of bank statement

February 06, 2023

sql

+

Get link
Facebook
X
Pinterest
Email
Other Apps

Find average salary of employees for each department and order employees within a department by age.

February 05, 2023

sql

+

Get link
Facebook
X
Pinterest
Email
Other Apps

Write a query to populate total_sal column value in the below format from the above input data from the table

January 29, 2023

sql

+

Get link
Facebook
X
Pinterest
Email
Other Apps

Write query to find total salary for each department and department total salary greater than 10000

January 26, 2023

sql

+

Get link
Facebook
X
Pinterest
Email
Other Apps

Weather table, need to show extra columns with nextday temps and diff in todays and nextday temps in sql

January 19, 2023

sql

+

Get link
Facebook
X
Pinterest
Email
Other Apps

Password generator from firstname with 4 latter and dob with month and day and lowercase.

January 19, 2023

sql

+

Get link
Facebook
X
Pinterest
Email
Other Apps

Lowest salary for each dept without any duplicates in sql query

January 10, 2023

sql

+

Get link
Facebook
X
Pinterest
Email
Other Apps

write query to find out next highest cost by category

January 09, 2023

sql

+

Get link
Facebook
X
Pinterest
Email
Other Apps

Page Views this month

Your can Ask me Question?

Name

Email *

Message *

Popular Post

Find Total working hours of each employee in a day through spark and scala

Find Total working hours of each employee in a day through spark and scala File name : epid.txt 1,"2020-12-30T09:00:00","IN" 1,"2020-12-30T13:00:00","OUT" 1,"2020-12-30T14:00:00","IN" 1,"2020-12-30T17:00:00","OUT" 1,"2020-12-30T17:30:00","IN" 1,"2020-12-30T20:00:00","OUT" 2,"2020-12-30T10:30:00","IN" 2,"2020-12-30T15:00:00","OUT" 2,"2020-12-30T16:30:00","IN" 2,"2020-12-30T21:30:00","IN" Solution : val schemax = new StructType() .add("eid",IntegerType,true) .add("ts",TimestampType,true) .add("event",StringType,true) val df = spark.read.format("csv").schema(schemax).load("file:///D:/epid.txt") df.show() df.printSchema() val unixtime = unix_timestamp(col("ts")) val pro = (unixtime-lag(unixtime,1).over(Window.partitionBy("eid...

How to complex string data convert into dataframe via pyspark.

How to complex string data convert into dataframe via pyspark altimetric Input: Jack xxxx ["BL", "ND"] John yyyy ["CH", "MB"] Output: +----+----+---+ | _1| _2| _3| +----+----+---+ |Jack|xxxx| BL| |Jack|xxxx| ND| |John|yyyy| CH| |John|yyyy| MB| +----+----+---+ Solve: dfjack = sc.textFile("file:////home/cloudera/jack.txt") dfjack.foreach(print) John yyyy ["CH", "MB"] Jack xxxx ["BL", "ND"] task1 = dfjack.map(lambda x : x.replace('["','')).map(lambda x:x.replace('"]','')).map(lambda x:x.replace('", "','-')).map(lambda x : x.replace(" ",",")) task1.foreach(print) Jack,xxxx,BL-ND John,yyyy,CH-MB splitdata = task1.map(lambda x:x.split(",")) splitdata.foreach(print) ['Jack', 'xxxx', 'BL-ND'] ['John', 'yyyy', 'CH-MB'] df = splitdata.toDF() df.show() +----+---...

How to make balance of bank statement

How to make balance of bank statement coforge Input: Output: Solution: create table bank (TransactionDate varchar(20),AccName varchar(20),Type varchar(20),Amount integer); insert into bank values ("2017-01-01","Anvesh","CR",60000.00),("2017-02-01","Anvesh","DB",8000.00),("2017-03-01","Anvesh","CR",8000.00),("2017-04-01","Anvesh","DB",5000.00),("2017-01-01","Nupur","CR",10000.00),("2017-02-02","Nupur","CR",8000.00),("2017-03-03","Nupur","DB",8000.00) WITH balances AS ( SELECT AccName, TransactionDate, Type, Amount, SUM(CASE WHEN Type = 'DB' THEN -Amount ELSE Amount END) OVER (PARTITION BY AccName ORDER BY TransactionDate) AS AccBalance FROM bank ) SELECT TransactionDate, AccName, Type, Amount, balance FROM balances ORDER BY AccName, TransactionDat...

Find below expected output through spark and scala

Find below expected output through spark and scala File Name : dxr.txt 11111,"2021-01-15",10 11111,"2021-01-16",15 11111,"2021-01-17",30 11112,"2021-01-15",10 11112,"2021-01-16",20 11112,"2021-01-17",30 Output Required : Solution : val schemax = new StructType() .add("sensorid",IntegerType,true) .add("timestamp",TimestampType,true) .add("values",IntegerType,true) val df = spark.read.format("csv").schema(schemax).load("file:///D:/dxr.txt") df.show() df.printSchema() val dx = df.withColumn("tx",lead("values",1).over(Window.partitionBy("sensorid") orderBy("timestamp"))) dx.show() val dr = dx.withColumn("diff",col("tx")-col("values")) .filter(col("tx").isNotNull) .drop("values","tx") .withColumnRenamed("diff","values") dr.show()

Hive Assignment

Hive Assignment incedo Input: 1) Create a database “Olympic”, then within this database create a table “player” in following structure: pid: Int, pname: String, gender: String, age: Int, height: Int, eid: Int Note: create the partition on “gender” column. 2) Create a table “event” within this database in following structure: eid: Int, ename: String, year: Int, season: String, city: String Note: create the partition on “season” column and bucketing on “year” column. 3) Write a query to insert data into two table from two text files “player.txt” and “event.txt” respectively. 4) Do the query on the following scenarios: a. List the players details who not participated in Olympic event. b. List the number of players participated in each event c. List the number of Male and Female Players participated in event d. List down number of players participated in Beijing Olympic, e. List down player details who have highest age and joined in Olympic Event. f. Find out the ...

Powered by Blogger

The Data Crusher