存入HDFS有好几种隐滚滑数据格式,我这里给备胡你列出一种格式的存储,灶腊sequence的 public class SeqWrite {private static final String[] data = { "a,b,c,d,e,f,g", "h,i,j,k,l,m,n", "o,p,q,r,s,t", "u,v,w,x,y,z", "0,1,2,3,4", "5,6,7,8,9" };public
⑵ sqoop导入到HDFS怎么都是java文件总结
运行一个maprece作业,该隐州作业会连接mysql数据库并读取岁禅表中的数据灶雀蔽,默认该作业会运行4个map任务来加速导入过程,每个任务都会将其导入的数据写到一个单独的文件,但所有4个文件都位于同一个目录中。
codegen:生成源代码,但并不执行导入操作
⑶ Sqoop工作原理是什么
Sqoop是一款用于把关系型数据库中的数据导入到hdfs中或者hive中的工具,当然也支持把数据从hdfs或者hive导入到关系型数据库中。
Sqoop也是基于Maprece来做的数据导入。
关于sqoop的原理
sqoop的原理比较简单,就是根据用户指定的sql或者字段参数,从数据库中读取数据导入到hive或者hdfs中。也支持基于数据库导出工具导出,不陵槐御过受限于数据库的版本。
在导出的过程中,sqoop会自动切分maprece任务。比如某个字段的主键是从1到1000,那么当设置num-mappers为2时,第一个mr任务会读取1-500的数据,第二个mr任务会读取500-1000的尺岩数据。如果是字符串还有其他的划分方法.
关明冲于架构
sqoop目前有两个大版本,第一个版本比较简单,只能使用命令行
第二个版本引入了sqoop server,统一处理连接等信息,并提供多种连接方式,还引入了权限控制,另外规范了连接的各项配置。
⑷ Sqoop的简单使用案例
(配置sqoop家目录环境变量,略)
验证Sqoop连接MySQL:
bin/sqoop list-databases --connect jdbc:mysql://hadoop01:3306/ --username root --password root
1 导入数据
在Sqoop中,“导入”概念指:从非大数据集群(RDBMS)向大数据集群(HDFS,HIVE,HBASE)中传输数据,叫做:导入,即使用import关键字。
1.1 RDBMS到HDFS
1) 确定Mysql服务开启正常
2) 在Mysql中新建一张表并插入一些数据
$ mysql -uroot -proot
create database company;
create table company.staff(id int(4) primary key not null auto_increment, name varchar(255), sex varchar(255));
insert into company.staff(name, sex) values('Thomas', 'Male');
insert into company.staff(name, sex) values('Catalina', 'FeMale');
select * from company;
3) 导入数据
(1)全部导入
$ bin/sqoop import \
--connect jdbc:mysql://hadoop01:3306/company \
--username root \
--password root \
--table staff \
--target-dir /user/company \
--delete-target-dir \
--num-mappers 1 \
--fields-terminated-by "\t"
(2)查询导入
$ bin/sqoop import \
--connect jdbc:mysql://hadoop01:3306/衡配裤company \
--username root \
--password root \
--target-dir /user/company \
--delete-target-dir \
--num-mappers 1 \
--fields-terminated-by "\t" \
--query 'select name,sex from staff where id <=1 and $CONDITIONS;'
提示:must contain '$CONDITIONS'咐简 in WHERE clause.
如果query后使用的是双引号,则$CONDITIONS前必须加转移符,防止shell识别为自己的变量。
(3)导入指定列
$ bin/sqoop import \
--connect jdbc:mysql://hadoop01:3306/company \
--username root \
--password root \
--target-dir /user/company \
--delete-target-dir \
--num-mappers 1 \
--fields-terminated-by "\t" \
--columns id,sex \
--table staff
提示:columns中如果涉及到多列,用逗号分隔,分隔时不要添加空格
(4)使用sqoop关键卖州字筛选查询导入数据
$ bin/sqoop import \
--connect jdbc:mysql://hadoop01:3306/company \
--username root \
--password root \
--target-dir /user/company \
--delete-target-dir \
--num-mappers 1 \
--fields-terminated-by "\t" \
--table staff \
--where "id=1"
1.2 RDBMS到Hive
$ bin/sqoop import \
--connect jdbc:mysql://hadoop01:3306/company \
--username root \
--password root \
--table staff \
--num-mappers 1 \
--hive-import \
--fields-terminated-by "\t" \
--hive-overwrite \
--hive-table staff_hive
提示:该过程分为两步,第一步将数据导入到HDFS,第二步将导入到HDFS的数据迁移到Hive仓库,第一步默认的临时目录是/user/atguigu/表名
1.3 RDBMS到Hbase
$ bin/sqoop import \
--connect jdbc:mysql://hadoop01:3306/company \
--username root \
--password root \
--table company \
--columns "id,name,sex" \
--column-family "info" \
--hbase-create-table \
--hbase-row-key "id" \
--hbase-table "hbase_company" \
--num-mappers 1 \
--split-by id
提示:sqoop1. 6只支持HBase1.0.1之前的版本的自动创建HBase表的功能
解决方案:手动创建HBase表
hbase> create 'hbase_company,'info'
(5) 在HBase中scan这张表得到如下内容
hbase> scan ‘hbase_company’
2、导出数据
在Sqoop中,“导出”概念指:从大数据集群(HDFS,HIVE,HBASE)向非大数据集群(RDBMS)中传输数据,叫做:导出,即使用export关键字。
2.1 HIVE/HDFS到RDBMS
$ bin/sqoop export \
--connect jdbc:mysql://hadoop01:3306/company \
--username root \
--password root \
--table staff \
--num-mappers 1 \
--export-dir /user/hive/warehouse/staff_hive \
--input-fields-terminated-by "\t"
提示:Mysql中如果表不存在,不会自动创建
3 脚本打包
使用opt格式的文件打包sqoop命令,然后执行
1) 创建一个.opt文件
$ mkdir opt
$ touch opt/job_HDFS2RDBMS.opt
2) 编写sqoop脚本
$ vi opt/job_HDFS2RDBMS.opt
export
--connect
jdbc:mysql://hadoop01:3306/company
--username
root
--password
root
--table
staff
--num-mappers
1
--export-dir
/user/hive/warehouse/staff_hive
--input-fields-terminated-by
"\t"
3) 执行该脚本
$ bin/sqoop --options-file opt/job_HDFS2RDBMS.opt
sqoop 配合集群使用:
创建连接MySQL密码存放文件 并传到HDFS中:
1.echo -n root > /usr/local/bin/sqoopPWD.pwd
2.hdfs dfs -put /usr/local/bin/sqoopPWD.pwd /user/hue/oozie/
3.sqoop job \
--create release_materiel \
-- import \
--connect jdbc:mysql://hd0:3306/throwads \
--username root \
--password-file /user/hue/oozie/sqoopPWD.pwd \
--table materiel \
--delete-target-dir \
--target-dir /gp1905/release/ods/release_materiel \
--fields-terminated-by '\001'
(注意-- import 有一个空格!!!)
⑸ sqoop2 1.99.7 java api 如何设置file format
//流读入和写入InputStreamin=null;//获取HDFS的conf//读取HDFS上的文件系统FileSystemhdfs=FileSystem.get(conf);//使用缓冲流,进行按行读取的功能BufferedReaderbuff=null;//获取日志文件的根目录Pathlistf=newPath("hdfs://10
⑹ sqoop支持java远程操作么
Sqoop(发音:skup)是一款开源的工具,主要用于在Hadoop(Hive)与传统的数据库(mysql、postgresql...)间进行数据的传递,可以将一个关系型数据库(例如 : MySQL ,Oracle ,Postgres等)中的数据导进到Hadoop的HDFS中,也可以将HDFS的数据导进到关系型数据库中。
Sqoop项目开始于2009年,最早是作为Hadoop的一个第三方模块存在,后来为了让使用者能够快速部署,也为了让开发人员能够更快速的迭代开发,Sqoop独立成为一个Apache项目。
⑺ sqoop导入到HDFS怎么都是java文件总结
如下面这个shell脚本:
#Oracle的连接字符桐粗串,其链亏中包含了Oracle的地址,SID,和端口号
CONNECTURL=jdbc:oracle:thin:@20.135.60.21:1521:DWRAC2
#使用的用户名
ORACLENAME=kkaa
#使用的密码
ORACLEPASSWORD=kkaa123
#需要从Oracle中导入的表名
oralceTableName=tt
#需要从Oracle中导入的表中的字段名
columns=AREA_ID,TEAM_NAME
#将Oracle中的数据导入到HDFS后的存放路径
hdfsPath=apps/as/hive/$oralceTableName
#执行导入逻辑。将Oracle中的局唤镇数据导入到HDFS中
sqoop import --append --connect $CONNECTURL --username $ORACLENAME --password $ORACLEPASSWORD --target-dir $hdfsPath --num-mappers 1 --table $oralceTableName --columns $columns --fields-terminated-by '\001'
执行这个脚本之后,导入程序就完成了。
⑻ sqoop Error returned by javac
运行sqoop从sqlservre抽数时报错
Text.writeString(__dataOut, ContactPerson);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1508: error: cannot find symbol
Text.writeString(__dataOut, ContactPersonTelephone);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1514: error: cannot find symbol
Text.writeString(__dataOut, ContactPersonEmail);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1520: error: cannot find symbol
Text.writeString(__dataOut, RequestorID);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1526: error: cannot find symbol
Text.writeString(__dataOut, RequestorName);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1532: error: cannot find symbol
Text.writeString(__dataOut, RequestorLogonname);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1538: error: cannot find symbol
Text.writeString(__dataOut, RequestorCountry);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1544: error: cannot find symbol
Text.writeString(__dataOut, RequestorTelephone);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1550: error: cannot find symbol
Text.writeString(__dataOut, RequestorEmail);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1556: error: cannot find symbol
Text.writeString(__dataOut, OperatorID);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1562: error: cannot find symbol
Text.writeString(__dataOut, OperatorName);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1568: error: cannot find symbol
Text.writeString(__dataOut, OperatorLogonName);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1574: error: cannot find symbol
Text.writeString(__dataOut, Country);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1580: error: cannot find symbol
Text.writeString(__dataOut, CompanyCode);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1593: error: cannot find symbol
Text.writeString(__dataOut, NonPoPurchaseCategory);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1599: error: cannot find symbol
Text.writeString(__dataOut, CategoryRemark);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1605: error: cannot find symbol
Text.writeString(__dataOut, ContractNumber);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1611: error: cannot find symbol
Text.writeString(__dataOut, Currency);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1630: error: cannot find symbol
Text.writeString(__dataOut, PayingBankAccountNumber);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1636: error: cannot find symbol
Text.writeString(__dataOut, CheckToRequestorName);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1642: error: cannot find symbol
Text.writeString(__dataOut, CheckToRequestorTelephone);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1648: error: cannot find symbol
Text.writeString(__dataOut, BankConfirmationMail);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1654: error: cannot find symbol
Text.writeString(__dataOut, BeneficiaryName);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1660: error: cannot find symbol
Text.writeString(__dataOut, BeneficiaryAddress);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1666: error: cannot find symbol
Text.writeString(__dataOut, BeneficiaryBankName);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1672: error: cannot find symbol
Text.writeString(__dataOut, BankAccountNumber);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1678: error: cannot find symbol
Text.writeString(__dataOut, BankCode);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1684: error: cannot find symbol
Text.writeString(__dataOut, TaxID);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1738: error: cannot find symbol
Text.writeString(__dataOut, PaymentfileNumber);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1751: error: cannot find symbol
Text.writeString(__dataOut, Remark);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1770: error: cannot find symbol
Text.writeString(__dataOut, PONo);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1776: error: cannot find symbol
Text.writeString(__dataOut, GEO);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1782: error: cannot find symbol
Text.writeString(__dataOut, SID);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1788: error: cannot find symbol
Text.writeString(__dataOut, UMSAttachments);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1794: error: cannot find symbol
Text.writeString(__dataOut, TaxJurisdictionReport);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1800: error: cannot find symbol
Text.writeString(__dataOut, DectableTaxCostCenter);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1806: error: cannot find symbol
Text.writeString(__dataOut, ServiceTax);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1812: error: cannot find symbol
Text.writeString(__dataOut, ServiceTaxAccount);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1818: error: cannot find symbol
Text.writeString(__dataOut, ServiceTaxCode);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1824: error: cannot find symbol
Text.writeString(__dataOut, Assignment);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1830: error: cannot find symbol
Text.writeString(__dataOut, Text);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1838: error: cannot find symbol
Text.writeString(__dataOut, ID);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1844: error: cannot find symbol
Text.writeString(__dataOut, Subject);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1850: error: cannot find symbol
Text.writeString(__dataOut, TicketNumber);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1856: error: cannot find symbol
Text.writeString(__dataOut, VendorID);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
Note: /tmp/sqoop-UsageDB/compile//QueryResult.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
100 errors
2019-06-27 19:10:40,977 ERROR [main] tool.ImportTool (ImportTool.java:run(609)) - Encountered IOException running import job: java.io.IOException: Error returned by javac
at org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:217)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:97)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
对应的是sqlserver FIle table;没办法指定--table.
网上查到说指定字符,就去检查下下字符,检查到原因时目标表中有列字段名叫Text,在抽数时将这个字段去掉了。
更改后如下
⑼ sqoop命令使用jar包,jar应该放哪个目录下
1,rmdbs to hadoop 很简单,使用一条命令 sqoop import --jar-file (你吵喊的jar包) --class-name (classname) --connect (mysql实友碰御例) --username (用户名) --password (密码) -table (表名) -hbase-table (hbase表名好岩) -column-famil...
⑽ sqoop导入到HDFS怎么都是java文件总结
您好,我看到您的问题很久没有人来回答,但是问题过期无人回答会被扣分的并且你的悬赏分也会被没收!所以我给你提几条建议:
一,你可以选择在正确的分类下去提问,这样知道你问题答案的人才会多一些,回答的人也会多些。
二,您可以到与您问题相关专业网站论坛里去看看,那里聚集悉携了许多专业人才,一定可以为你解决问题的。
三,你可以向你的网上好袭陆慧友问友打听,他们会更加真诚热心为你寻找答案的,甚至可以到相关网站直接搜索.
四,网上很多专业论坛以及知识平台,上面也有很多资料,我遇到专业性的问题总是上论坛求解决办法的。
五,将你的问题问的细一些,清楚一些!让人更加容易看懂明白是什么意思!
谢谢拍答采纳我的建议! !