Kettle入门之四 Kettle定时任务（后台任务）

前面我们介绍了 Kettle的Spoon的转换和作业定时任务GUI设计方式以及运行，但是在实际应用中，我们需要计划任务是在服务器后台运行。

首先我们需要了解Kettle的Kitchen和Pan

Kitchen——作业(job)执行器 (命令行方式)

Pan——转换(trasform)执行器 (命令行方式)

下面我们将重点讲解经常会用到的 ***作业执行器 Kitchen.bat *** 。

1、新建一个bat文件，命名job.bat，然后编辑，输入内容如下：

注意：确保路径的正确性。

2、双击job.bat，即可运行。

我们已经建立了命令行运行的bat文件，并且已经可以正确执行我们的ETL任务了，但是现在我们在运行bat后，桌面上面会一直存在一个控制台的黑窗口，我们可以使用bat命令中的隐藏窗口的命令。

修改我们的job.bat文件

在此双击job.bat运行，我们将不会再看到控制台黑窗口继续保留在桌面上，并且等待设置的间隔时间后，我们可以看到日志文件记录。

···

2017/12/07 16:51:26 - Kitchen - Logging is at level : 基本日志

2017/12/07 16:51:26 - Kitchen - Start of run.

2017/12/07 16:51:31 - job2 - 开始执行任务

2017/12/07 16:51:31 - job2 - job2

2017/12/07 16:53:31 - job2 - 开始项[简单表同步]

2017/12/07 16:53:31 - 简单表同步 - Loading transformation from XML file [file:///D:/KettleProject/TEST2.ktr]

2017/12/07 16:53:31 - 简单表同步 - Using run configuration [Pentaho local]

2017/12/07 16:53:31 - 简单表同步 - Using legacy execution engine

2017/12/07 16:53:31 - TEST2 - 为了转换解除补丁开始 [TEST2]

2017/12/07 16:53:32 - 表输入.0 - Finished reading query, closing connection.

2017/12/07 16:53:32 - 表输入.0 - 完成处理 (I=3, O=0, R=0, W=3, U=0, E=0)

2017/12/07 16:53:32 - 插入 / 更新.0 - 完成处理 (I=3, O=0, R=3, W=3, U=1, E=0)

···

上一篇： Kettle入门之三 Kettle定时任务（GUI）

下一篇： Kettle入门之五 Kettle应用场景（增量插入和更新）

连接hive的方法：

进入hive所在的服务器，输入：hive --service hiveserver（目的：启动thrift）

打开kettle配置连接界面，输入hive所在服务器的ip、所需要的hive库、端口号（thrift默认端口为：10000）

测试连接，即可

连接hive2的方法：

Error connecting to database [Hive] : org.pentaho.di.core.exception.KettleDatabaseException:

Error occured while trying to connect to the database

Error connecting to database: (using class org.apache.hadoop.hive.jdbc.HiveDriver)

Unable to load Hive Server 2 JDBC driver for the currently active Hadoop configuration

org.pentaho.di.core.exception.KettleDatabaseException:

Error occured while trying to connect to the database

Error connecting to database: (using class org.apache.hadoop.hive.jdbc.HiveDriver)

Unable to load Hive Server 2 JDBC driver for the currently active Hadoop configuration

at org.pentaho.di.core.database.Database.normalConnect(Database.java:428)

at org.pentaho.di.core.database.Database.connect(Database.java:361)

at org.pentaho.di.core.database.Database.connect(Database.java:314)

at org.pentaho.di.core.database.Database.connect(Database.java:302)

at org.pentaho.di.core.database.DatabaseFactory.getConnectionTestReport(DatabaseFactory.java:80)

at org.pentaho.di.core.database.DatabaseMeta.testConnection(DatabaseMeta.java:2685)

at org.pentaho.di.ui.core.database.dialog.DatabaseDialog.test(DatabaseDialog.java:109)

at org.pentaho.di.ui.core.database.wizard.CreateDatabaseWizardPage2.test(CreateDatabaseWizardPage2.java:157)

at org.pentaho.di.ui.core.database.wizard.CreateDatabaseWizardPage2$3.widgetSelected(CreateDatabaseWizardPage2.java:147)

at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)

at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)

at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)

at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)

at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)

at org.eclipse.jface.window.Window.runEventLoop(Window.java:820)

at org.eclipse.jface.window.Window.open(Window.java:796)

at org.pentaho.di.ui.core.database.wizard.CreateDatabaseWizard.createAndRunDatabaseWizard(CreateDatabaseWizard.java:111)

at org.pentaho.di.ui.spoon.Spoon.createDatabaseWizard(Spoon.java:7457)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.pentaho.ui.xul.impl.AbstractXulDomContainer.invoke(AbstractXulDomContainer.java:313)

at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:157)

at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:141)

at org.pentaho.ui.xul.jface.tags.JfaceMenuitem.access$100(JfaceMenuitem.java:43)

at org.pentaho.ui.xul.jface.tags.JfaceMenuitem$1.run(JfaceMenuitem.java:106)

at org.eclipse.jface.action.Action.runWithEvent(Action.java:498)

at org.eclipse.jface.action.ActionContributionItem.handleWidgetSelection(ActionContributionItem.java:545)

at org.eclipse.jface.action.ActionContributionItem.access$2(ActionContributionItem.java:490)

at org.eclipse.jface.action.ActionContributionItem$5.handleEvent(ActionContributionItem.java:402)

at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)

at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)

at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)

at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)

at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1297)

at org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:7801)

at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:9130)

at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:638)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.pentaho.commons.launcher.Launcher.main(Launcher.java:151)

Caused by: org.pentaho.di.core.exception.KettleDatabaseException:

Error connecting to database: (using class org.apache.hadoop.hive.jdbc.HiveDriver)

Unable to load Hive Server 2 JDBC driver for the currently active Hadoop configuration

at org.pentaho.di.core.database.Database.connectUsingClass(Database.java:573)

at org.pentaho.di.core.database.Database.normalConnect(Database.java:410)

... 43 more

Caused by: java.sql.SQLException: Unable to load Hive Server 2 JDBC driver for the currently active Hadoop configuration

at org.apache.hive.jdbc.HiveDriver.getActiveDriver(HiveDriver.java:107)

at org.apache.hive.jdbc.HiveDriver.callWithActiveDriver(HiveDriver.java:121)

at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:132)

at java.sql.DriverManager.getConnection(Unknown Source)

at java.sql.DriverManager.getConnection(Unknown Source)

at org.pentaho.di.core.database.Database.connectUsingClass(Database.java:555)

... 44 more

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.apache.hive.jdbc.HiveDriver.getActiveDriver(HiveDriver.java:105)

... 49 more

Caused by: java.lang.RuntimeException: Unable to load JDBC driver of type: hive2

at org.pentaho.hadoop.shim.common.CommonHadoopShim.getJdbcDriver(CommonHadoopShim.java:108)

... 54 more

Caused by: java.lang.Exception: JDBC driver of type 'hive2' not supported

at org.pentaho.hadoop.shim.common.CommonHadoopShim.getJdbcDriver(CommonHadoopShim.java:104)

... 54 more

上述报错的解决方法如下：

1.找到%KETTLE_HOME%/plugins/pehtaho-big-data-plugin/plugin.properties文件

2.修改plugin.properties文件中的值：active.hadoop.configuration=hdp13

3.修改后重启kettle

4.配置完成后，即可连接上对应的库

如果要使用hadoop-20，则需要添加如下jar包：

hadoop-core-1.2.1.jar

hive-common-0.13.0.jar

hive-jdbc-0.13.0.jar

hive-service-0.13.0.jar

libthrift-0.9.1.jar

slf4j-api-1.7.5.jar

httpclient-4.2.5.jar

httpcore-4.2.5.jar

总结：使用hive2的好处，优化了连接、增加安全性、并行度

---