导航:首页 > 编程语言 > sqoop命令转java

sqoop命令转java

发布时间:2023-03-16 10:45:35

⑴ sqoop导入到HDFS怎么都是java文件总结

存入HDFS有好几种隐滚滑数据格式,我这里给备胡你列出一种格式的存储,灶腊sequence的 public class SeqWrite {private static final String[] data = { "a,b,c,d,e,f,g", "h,i,j,k,l,m,n", "o,p,q,r,s,t", "u,v,w,x,y,z", "0,1,2,3,4", "5,6,7,8,9" };public

⑵ sqoop导入到HDFS怎么都是java文件总结

运行一个maprece作业,该隐州作业会连接mysql数据库并读取岁禅表中的数据灶雀蔽,默认该作业会运行4个map任务来加速导入过程,每个任务都会将其导入的数据写到一个单独的文件,但所有4个文件都位于同一个目录中。
codegen:生成源代码,但并不执行导入操作

⑶ Sqoop工作原理是什么

Sqoop是一款用于把关系型数据库中的数据导入到hdfs中或者hive中的工具,当然也支持把数据从hdfs或者hive导入到关系型数据库中。

Sqoop也是基于Maprece来做的数据导入。

关于sqoop的原理
sqoop的原理比较简单,就是根据用户指定的sql或者字段参数,从数据库中读取数据导入到hive或者hdfs中。也支持基于数据库导出工具导出,不陵槐御过受限于数据库的版本

在导出的过程中,sqoop会自动切分maprece任务。比如某个字段的主键是从1到1000,那么当设置num-mappers为2时,第一个mr任务会读取1-500的数据,第二个mr任务会读取500-1000的尺岩数据。如果是字符串还有其他的划分方法.

关明冲于架构
sqoop目前有两个大版本,第一个版本比较简单,只能使用命令行

第二个版本引入了sqoop server,统一处理连接等信息,并提供多种连接方式,还引入了权限控制,另外规范了连接的各项配置。

⑷ Sqoop的简单使用案例

(配置sqoop家目录环境变量,略)

验证Sqoop连接MySQL:

bin/sqoop list-databases --connect jdbc:mysql://hadoop01:3306/ --username root --password root

1 导入数据

在Sqoop中,“导入”概念指:从非大数据集群(RDBMS)向大数据集群(HDFS,HIVE,HBASE)中传输数据,叫做:导入,即使用import关键字。

  1.1 RDBMS到HDFS

1) 确定Mysql服务开启正常

2) 在Mysql中新建一张表并插入一些数据

$ mysql -uroot -proot

create database company;

create table company.staff(id int(4) primary key not null auto_increment, name varchar(255), sex varchar(255));

insert into company.staff(name, sex) values('Thomas', 'Male');

insert into company.staff(name, sex) values('Catalina', 'FeMale');

select * from company;

3) 导入数据

(1)全部导入

$ bin/sqoop import \

--connect jdbc:mysql://hadoop01:3306/company \

--username root \

--password root \

--table staff \

--target-dir /user/company \

--delete-target-dir \

--num-mappers 1 \

--fields-terminated-by "\t"

(2)查询导入

$ bin/sqoop import \

--connect jdbc:mysql://hadoop01:3306/衡配裤company \

--username root \

--password root \

--target-dir /user/company \

--delete-target-dir \

--num-mappers 1 \

--fields-terminated-by "\t" \

--query 'select name,sex from staff where id <=1 and $CONDITIONS;'

提示:must contain '$CONDITIONS'咐简 in WHERE clause.

如果query后使用的是双引号,则$CONDITIONS前必须加转移符,防止shell识别为自己的变量。

(3)导入指定列

$ bin/sqoop import \

--connect jdbc:mysql://hadoop01:3306/company \

--username root \

--password root \

--target-dir /user/company \

--delete-target-dir \

--num-mappers 1 \

--fields-terminated-by "\t" \

--columns id,sex \

--table staff

提示:columns中如果涉及到多列,用逗号分隔,分隔时不要添加空格

(4)使用sqoop关键卖州字筛选查询导入数据

$ bin/sqoop import \

--connect jdbc:mysql://hadoop01:3306/company \

--username root \

--password root \

--target-dir /user/company \

--delete-target-dir \

--num-mappers 1 \

--fields-terminated-by "\t" \

--table staff \

--where "id=1"

  1.2 RDBMS到Hive

$ bin/sqoop import \

--connect jdbc:mysql://hadoop01:3306/company \

--username root \

--password root \

--table staff \

--num-mappers 1 \

--hive-import \

--fields-terminated-by "\t" \

--hive-overwrite \

--hive-table staff_hive

提示:该过程分为两步,第一步将数据导入到HDFS,第二步将导入到HDFS的数据迁移到Hive仓库,第一步默认的临时目录是/user/atguigu/表名

  1.3 RDBMS到Hbase

$ bin/sqoop import \

--connect jdbc:mysql://hadoop01:3306/company \

--username root \

--password root \

--table company \

--columns "id,name,sex" \

--column-family "info" \

--hbase-create-table \

--hbase-row-key "id" \

--hbase-table "hbase_company" \

--num-mappers 1 \

--split-by id

提示:sqoop1.  6只支持HBase1.0.1之前的版本的自动创建HBase表的功能

解决方案:手动创建HBase表

hbase> create 'hbase_company,'info'

(5) 在HBase中scan这张表得到如下内容

hbase> scan ‘hbase_company’

  2、导出数据

在Sqoop中,“导出”概念指:从大数据集群(HDFS,HIVE,HBASE)向非大数据集群(RDBMS)中传输数据,叫做:导出,即使用export关键字。

  2.1 HIVE/HDFS到RDBMS

$ bin/sqoop export \

--connect jdbc:mysql://hadoop01:3306/company \

--username root \

--password root \

--table staff \

--num-mappers 1 \

--export-dir /user/hive/warehouse/staff_hive \

--input-fields-terminated-by "\t"

提示:Mysql中如果表不存在,不会自动创建

  3 脚本打包

使用opt格式的文件打包sqoop命令,然后执行

1) 创建一个.opt文件

$ mkdir opt

$ touch opt/job_HDFS2RDBMS.opt

2) 编写sqoop脚本

$ vi opt/job_HDFS2RDBMS.opt

export

--connect

jdbc:mysql://hadoop01:3306/company

--username

root

--password

root

--table

staff

--num-mappers

1

--export-dir

/user/hive/warehouse/staff_hive

--input-fields-terminated-by

"\t"

3) 执行该脚本

$ bin/sqoop --options-file opt/job_HDFS2RDBMS.opt

sqoop 配合集群使用:

创建连接MySQL密码存放文件 并传到HDFS中:

1.echo -n root > /usr/local/bin/sqoopPWD.pwd

2.hdfs dfs -put /usr/local/bin/sqoopPWD.pwd /user/hue/oozie/

3.sqoop job \

--create release_materiel \

-- import \

--connect jdbc:mysql://hd0:3306/throwads \

--username root \

--password-file /user/hue/oozie/sqoopPWD.pwd \

--table materiel \

--delete-target-dir \

--target-dir /gp1905/release/ods/release_materiel \

--fields-terminated-by '\001'

(注意-- import 有一个空格!!!)

⑸ sqoop2 1.99.7 java api 如何设置file format

//流读入和写入InputStreamin=null;//获取HDFS的conf//读取HDFS上的文件系统FileSystemhdfs=FileSystem.get(conf);//使用缓冲流,进行按行读取的功能BufferedReaderbuff=null;//获取日志文件的根目录Pathlistf=newPath("hdfs://10

⑹ sqoop支持java远程操作么

Sqoop(发音:skup)是一款开源的工具,主要用于在Hadoop(Hive)与传统的数据库(mysql、postgresql...)间进行数据的传递,可以将一个关系型数据库(例如 : MySQL ,Oracle ,Postgres等)中的数据导进到Hadoop的HDFS中,也可以将HDFS的数据导进到关系型数据库中。
Sqoop项目开始于2009年,最早是作为Hadoop的一个第三方模块存在,后来为了让使用者能够快速部署,也为了让开发人员能够更快速的迭代开发,Sqoop独立成为一个Apache项目。

⑺ sqoop导入到HDFS怎么都是java文件总结

如下面这个shell脚本:
#Oracle的连接字符桐粗串,其链亏中包含了Oracle的地址,SID,和端口号
CONNECTURL=jdbc:oracle:thin:@20.135.60.21:1521:DWRAC2
#使用的用户名
ORACLENAME=kkaa
#使用的密码
ORACLEPASSWORD=kkaa123
#需要从Oracle中导入的表名
oralceTableName=tt
#需要从Oracle中导入的表中的字段名
columns=AREA_ID,TEAM_NAME
#将Oracle中的数据导入到HDFS后的存放路径
hdfsPath=apps/as/hive/$oralceTableName
#执行导入逻辑。将Oracle中的局唤镇数据导入到HDFS中
sqoop import --append --connect $CONNECTURL --username $ORACLENAME --password $ORACLEPASSWORD --target-dir $hdfsPath --num-mappers 1 --table $oralceTableName --columns $columns --fields-terminated-by '\001'
执行这个脚本之后,导入程序就完成了。

⑻ sqoop Error returned by javac

运行sqoop从sqlservre抽数时报错

Text.writeString(__dataOut, ContactPerson);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1508: error: cannot find symbol
Text.writeString(__dataOut, ContactPersonTelephone);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1514: error: cannot find symbol
Text.writeString(__dataOut, ContactPersonEmail);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1520: error: cannot find symbol
Text.writeString(__dataOut, RequestorID);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1526: error: cannot find symbol
Text.writeString(__dataOut, RequestorName);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1532: error: cannot find symbol
Text.writeString(__dataOut, RequestorLogonname);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1538: error: cannot find symbol
Text.writeString(__dataOut, RequestorCountry);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1544: error: cannot find symbol
Text.writeString(__dataOut, RequestorTelephone);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1550: error: cannot find symbol
Text.writeString(__dataOut, RequestorEmail);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1556: error: cannot find symbol
Text.writeString(__dataOut, OperatorID);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1562: error: cannot find symbol
Text.writeString(__dataOut, OperatorName);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1568: error: cannot find symbol
Text.writeString(__dataOut, OperatorLogonName);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1574: error: cannot find symbol
Text.writeString(__dataOut, Country);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1580: error: cannot find symbol
Text.writeString(__dataOut, CompanyCode);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1593: error: cannot find symbol
Text.writeString(__dataOut, NonPoPurchaseCategory);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1599: error: cannot find symbol
Text.writeString(__dataOut, CategoryRemark);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1605: error: cannot find symbol
Text.writeString(__dataOut, ContractNumber);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1611: error: cannot find symbol
Text.writeString(__dataOut, Currency);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1630: error: cannot find symbol
Text.writeString(__dataOut, PayingBankAccountNumber);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1636: error: cannot find symbol
Text.writeString(__dataOut, CheckToRequestorName);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1642: error: cannot find symbol
Text.writeString(__dataOut, CheckToRequestorTelephone);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1648: error: cannot find symbol
Text.writeString(__dataOut, BankConfirmationMail);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1654: error: cannot find symbol
Text.writeString(__dataOut, BeneficiaryName);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1660: error: cannot find symbol
Text.writeString(__dataOut, BeneficiaryAddress);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1666: error: cannot find symbol
Text.writeString(__dataOut, BeneficiaryBankName);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1672: error: cannot find symbol
Text.writeString(__dataOut, BankAccountNumber);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1678: error: cannot find symbol
Text.writeString(__dataOut, BankCode);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1684: error: cannot find symbol
Text.writeString(__dataOut, TaxID);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1738: error: cannot find symbol
Text.writeString(__dataOut, PaymentfileNumber);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1751: error: cannot find symbol
Text.writeString(__dataOut, Remark);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1770: error: cannot find symbol
Text.writeString(__dataOut, PONo);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1776: error: cannot find symbol
Text.writeString(__dataOut, GEO);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1782: error: cannot find symbol
Text.writeString(__dataOut, SID);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1788: error: cannot find symbol
Text.writeString(__dataOut, UMSAttachments);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1794: error: cannot find symbol
Text.writeString(__dataOut, TaxJurisdictionReport);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1800: error: cannot find symbol
Text.writeString(__dataOut, DectableTaxCostCenter);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1806: error: cannot find symbol
Text.writeString(__dataOut, ServiceTax);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1812: error: cannot find symbol
Text.writeString(__dataOut, ServiceTaxAccount);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1818: error: cannot find symbol
Text.writeString(__dataOut, ServiceTaxCode);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1824: error: cannot find symbol
Text.writeString(__dataOut, Assignment);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1830: error: cannot find symbol
Text.writeString(__dataOut, Text);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1838: error: cannot find symbol
Text.writeString(__dataOut, ID);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1844: error: cannot find symbol
Text.writeString(__dataOut, Subject);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1850: error: cannot find symbol
Text.writeString(__dataOut, TicketNumber);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
/tmp/sqoop-UsageDB/compile//QueryResult.java:1856: error: cannot find symbol
Text.writeString(__dataOut, VendorID);
^
symbol: method writeString(DataOutput,String)
location: variable Text of type String
Note: /tmp/sqoop-UsageDB/compile//QueryResult.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
100 errors
2019-06-27 19:10:40,977 ERROR [main] tool.ImportTool (ImportTool.java:run(609)) - Encountered IOException running import job: java.io.IOException: Error returned by javac
at org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:217)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:97)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
对应的是sqlserver FIle table;没办法指定--table.
网上查到说指定字符,就去检查下下字符,检查到原因时目标表中有列字段名叫Text,在抽数时将这个字段去掉了。
更改后如下

⑼ sqoop命令使用jar包,jar应该放哪个目录下

1,rmdbs to hadoop 很简单,使用一条命令 sqoop import --jar-file (你吵喊的jar包) --class-name (classname) --connect (mysql实友碰御例) --username (用户名) --password (密码) -table (表名) -hbase-table (hbase表名好岩) -column-famil...

⑽ sqoop导入到HDFS怎么都是java文件总结

您好,我看到您的问题很久没有人来回答,但是问题过期无人回答会被扣分的并且你的悬赏分也会被没收!所以我给你提几条建议:

一,你可以选择在正确的分类下去提问,这样知道你问题答案的人才会多一些,回答的人也会多些。

二,您可以到与您问题相关专业网站论坛里去看看,那里聚集悉携了许多专业人才,一定可以为你解决问题的。

三,你可以向你的网上好袭陆慧友问友打听,他们会更加真诚热心为你寻找答案的,甚至可以到相关网站直接搜索.

四,网上很多专业论坛以及知识平台,上面也有很多资料,我遇到专业性的问题总是上论坛求解决办法的。

五,将你的问题问的细一些,清楚一些!让人更加容易看懂明白是什么意思!

谢谢拍答采纳我的建议! !

阅读全文

与sqoop命令转java相关的资料

热点内容
html包含文件代码吗 浏览:50
苹果appstore日本账号 浏览:532
解密dg加密的文件 浏览:206
gsh6什么格式文件 浏览:507
dnf85版本觉醒任务 浏览:998
范冰冰苹果百度云盘资源链接 浏览:507
数据库主机是什么系统 浏览:812
pdf表单教程 浏览:715
百度浏览器去更新安卓破解版 浏览:855
光盘内部应用程序错误 浏览:83
iphone6升级ios9步骤 浏览:873
魔力代码 浏览:497
win10打开局域网文件夹很卡 浏览:986
app收益怎么分 浏览:812
我的世界什么版本好玩 浏览:341
控制专硕如何报考编程 浏览:534
元祖在编程里是什么意思 浏览:491
小码王为什么比核桃编程贵 浏览:627
qq下载app有哪些 浏览:380
旅游app的社区模块有什么 浏览:847

友情链接