缓存检查


sparkjob.cache.names

sparkjob.cache.names (非必需)定义工作执行前需要检查的缓存,多个缓存用 ',' 区分。如果一个工作对环境里的缓存有引用,我们建议在递交工作的时候传sparkjob.cache.names这个参数。如果缓存不存在,传这个参数时检验说在工作开始前执行的,而不传是工作执行好在引用的时候动态报错的。传的报错效率会更好。

验证sparkjob.cache.names例子

BASH

curl -X POST -d 'sparkjob.configuration="{"sparkjob.access.opearations ":"Access?input=foo","sparkjob.analytic.operations":"Stats(fields = `*`, analyze_metadata = `true`, analyze_string = `true`);Result()"}", sparkjob.cache.names="foo"' 'api.exceeddata.com/analytics/v1/jobs?appName=ac&classPath=com.exceeddata.ac.job.runner.SparkJobRunner&context=mycontext&sync=true'
{
  "status": "VALIDATION FAILED",
  "result": {
    "message": "[foo] is not in cache",
    "errorClass": "java.lang.Throwable",
    "stack": ["spark.jobserver.JobManagerActor$$anonfun$spark$jobserver$JobManagerActor$$getJobFuture$4.apply(JobManagerActor.scala:212)", "scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)", "scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)", "akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)", "akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)", "scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)", "scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)", "scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)", "scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)"]
}

不验证sparkjob.cache.names例子

BASH

curl -X POST -d 'sparkjob.configuration="{"sparkjob.access.operations ":"Access?input=foo","sparkjob.analytic.operations":"Stats(fields = `*`, analyze_metadata = `true`, analyze_string = `true`);Result()"}"' '111.111.111.111:8090/jobs?appName=ac&classPath=com.exceeddata.ac.job.runner.SparkJobRunner&context=mycontext&sync=true'
{
  "status": "ERROR",
  "result": {
    "message": "Ask timed out on [Actor[akka.tcp://JobServer@127.0.0.1:52801/user/jobManager-40-a6e1-11deb9c4f6a8#2118649879]] after [10000 ms]",
    "errorClass": "akka.pattern.AskTimeoutException",
    "stack": ["akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334)", "akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)", "scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694)", "scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:691)", "akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467)", "akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419)", "akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423)", "akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)", "java.lang.Thread.run(Thread.java:745)"]
  }