Spark实验报告.docx

《Spark实验报告.docx》由会员分享，可在线阅读，更多相关《Spark实验报告.docx（12页珍藏版）》请在优知文库上搜索。

1、Spark报金航1510122526SPark试验报告-、环境搭建1、卜载版本卜载地址为：2、解压和安装：解压：tar-xvfscala-2.11.4.tgz安装：mvscala-2.11.4opV3、m-.bashprofile文件增加SCA1.AHoME环境变量配置,exportexportC1.ASSPATH=.：SJAVA_HOME/jre),lib:$JAVA_HOME/lib:$JAVA_HOME.lib.aools.jarPATH=SPATHSHOMEBn:$JAVAjHoME，bin:$SCA1.A_HOMEybin马上生效source/.bash_profile4、验证SCa

2、la：scala-version5、CoPy到SIaVe机器scp/.bash_PrOfiIe:Ybashjxofile6. bspark.wget7、在master主机配置SPark：格下校的SPark-I2.0-bin-hadoop2.4.tgz解压到TOPt/即，配置环境变量SPARKJHOME# setjavaenvexportC1.ASSPATH-.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lihtTOls.jarPATH=$PATH:$HOME.，bin:$JAVA_HOME.bin:$SCA1.A_HOME/bin：SSPARK_

3、HOMEbinSHADOOPJHOMEKbin配置完成后运用source吩咐使配置生效进入SParkCOnf书目：sparkSlPA11opt$cd(sparkSlPA11spark-1.2.0-bin-hadoop2,4SIsbinconfdataec2exampleslib1.ICENSElogsNOTICEpythonREADME,mdRE1.EASESbinworksparkS1PA11spark-1.2.0-bin-hadoop2.4Scdnfspark(三)SlPA11conf)$Issaves.templatespark-env.shIog4j.properties.templa

4、teslavesspark-spark-first：修改SlaVeS文件,增加两个SlaVe节点SIPA11、S1PA222sparkS1PA11conf$vislavesS1PA11S1PA222second：配置spark-env.sh首先把sark-env.sh.temlatecopyspark-env.shvispark-env.sh文件在最下面增加：exportexportSPARKWORKERMEMORY=2ghadpHADoOP_CONF_DiR是Hadoop配置文件书目，SPARK_MASTEFUP主机IP地址,SPARKWORKER.MEMORY是WOrker运用的最大内存完

5、成配世后，将sparkHIlcopyslave机器scp-r# oplpark-l.2.0-bin-hadoop2.4:沁PV8、启动spark分布式集群并查看信息ISparkSIPA11sbin$./start-all.sh查看:(sparkSlPA11sbnSjps31233ResourceManager27201Jps30498NameNode30733SecondaryNameNode5648Worker5399Master15888JobHistoryServer假如HDFS没有启动，启动起来.# slave节点：(sparkSlPA222scala$jps20352Bootstra

6、p30737NodeManager7219Jps30482DataNode29500Bootstrap757Worker9、页面查看集群状况：进去spark集群的Web管理页面，访问WSpariiKUiteratpaZ10.M.4447:707741.ra0Mi(MM4ZVn*f*v?CC*-OUMCM*r40-GgOtMnBeCyMa。C*v*WCCMw9ICu9M*AM因为我们看到两个worker节点，因为master和slave都是WOrker节点我们进入spark的bin书Il，肩动SPark-shell限制台1SOVM14:17:30XXTOMtt.Btt2cckTcMfr*mo:S

7、rrrcxfloa3“01IV01i(B112O口2M)f*.Sltot*.al9Utaa9*MM*cXctt:V*qitDRAM.BlMttai9c4.Isc*lb3t,S72)120l4eUtaMrtMr:叼1tZU0ctMt9r1WO1414P(2OWWrt*l.frtUSZE9filMOWQOf4PVUIqoredsTV1.temtat,64Uto，M”，4t*uUUIU0JAU3(O)ISP0MNWWWUM三X*-*3ft*MI.M4*Wv一，OMtoMMA)I*OA4wPi，awzvv4aMt(qo111JoJ)88msVdsJobsStagesStorageEnvironment

8、ExecutorsExecutors(1)Memory:OOBUsed(2650MBTotal)Disk:OOBUsedExecutorIDAddressRDDBlocksMemoryUsedDiskUsedActiveTasksFailedTasksCompleteTasksTotalTaskslocalhost5740100.0B/2650MBOOB0000Spark120SPark集群环境搭建胜利了10.isfjspark-shell测试之前我们在tmp书目上住了一个README文件，我们现在就用spark读取hdfs中README.txt文件tmpPermissionOwnerGrou

9、p-W-sparksupergroupdrwxsparksupergrouphttp:drwxr-xr-xsparksupergroupHadoop.2014.取得hdfs文件:15/01/06I5IOCIS/0X/04l*,BXtf)Kif3214U913214:29:32X：25:32H29MXMFOtoc.loryflVoctXNFO39.MMMrr,XMFOat0ra9.Maaor4tor:XKFOatocagv.M3gyflt0t:eourFxcall*dVldCUrMslX.三M-2772Blockbxt4ct-0-pcOaor*dabytiM*9xystina%dais22.tx

10、w2,4.0MBIIMFOVt7.lorhMryt*t,inonloolholf49211(it22.8few(2“9MBIOftI412t22INFOcor9.BloclMn*rUrUp4tdIntoOfb：oiio4caj0jSx0OC14:29:32XNrOprk.9paxk3c%xt:CratdbrAdcatOfxc三txt11leatx12Fil!oxg.Ach.nrkr4d.RDD(Striae)Mf:/SlVAll1.txtNpf020111ttt11lal2count下READM.txt文件中文字总数,FI：三三鲁WW,:HI我们过渡README.txt包括The单词有多个sc

11、alavaxtheCount=readmeFile.filter(line=line.contains(The*)theCount:org.apache.spark.rdd.RODString)5三：rilteredRDD(3tfilterat我们算出来一共有4个The单词(sparkSlPA222hadoop-2.6.0$IsNOTICE.txtREADMspark81PA222hadoop-2.6.0)$grepTheREADME.txtwc437269_h1.1.pi.h;nAoh1.nC1-IWkSUI1UHOspark0S!PA222hadoop-2.6.0$我们通过WC也罚出来有4

12、个The单词我们再实现下HadOoPWordCoUnt功能：首先对读取的readmeFile执行以卜吩咐:calavalWordCOUnt=readmeFile.fIatMap(line=line.split().map(word=wordcount:org.apache.spark.rdd.RDD(StringrInt)=ShuffledRDD(6atred,hlip:!(.csln.notstrks11mI其次运用llect吩咐提交并执行job:00UtMois：m010CISiM/1/0(IStMoci：mocis：M104ismoeIStMi0C13:N,006ISiM12IWTO*rrh.VF4r*Co4vtt0ri*gjobrC91Xetttl?1V010IStMl/01/0C1S:M1VO)OCISiMry0tor*s