Text above

Sample Client Client Server Server Test


Test Diagram U NN DN JT TT 1getNewJobId(): job-id 2getDelegToken(): deleg-token 3writeJobFiles( files, deleg-token ): void 4writeJobFiles( files, deleg-token ): void 5submitJob(): job-id 6heartbeat() : work 7refreshDelegToken( deleg-token ): void 8extractJobFiles(): data 9extractJobFiles(): data 10new() T 11exec() 12status(): void Secure Map Reduce 2 Client(c) ResourceManager(rm) HistoryServer(hs) NodeManager(nm) FileSystem(fs) NameNode(nn) DataNode(dn) Install Bootstrap 1heartbeat[?]( status ):blk-tkn-secret 2heartbeat[?](): nm-secret-key, container-secret-key Cycled daily? Job Definition 3createJob(): job-id Used to identify job-dir 4createDelegationToken(): c-nn-deleg-token 5createDelegationToken(): c-rm-deleg-token Optional, e.g. Oozie 6createDelegationToken(): c-hs-deleg-token For completed job status 7createShuffleSecret(): shuffle-secret Used to fetch data from NM 8mkdir[RPC:SASL(c,nn)]( job-dir ) loop[job-conf, splits, etc.] 9writeFile[RPC:SASL(c,nn)](): blk-id, blk-loc,blk-access-tkn 10writeBlock[DTP(blk-access-tkn)]( blk-id ): blk-data 11serializeTokens( TODO ): serialized-tokens Job Submission 12submitApplication[?]( job-dir, serialized-tokens ) Tokens also includes c-rm-deleg-token 13generateClientSecret(): client-secret For the C to talk to AM.Should this be a token? 14generateApplicationToken(): app-token For AM to talk to the RM. 15app-report{client-token{app-id}} Job Initiation 16allocateResource(): mr-container-spec{nm-token,container-token} RM allocates resourceswith itself for running AM 17startContainer[?]( mr-container-spec, serialized-tokens ) tokens: deleg, shuffle, client, app 18write( serialized-tokens ) Includes client-token, protected by OS permissions 19exec[as rm?]( job-dir, serialized-tokens-loc) ApplicationMaster(am) Provided the location of the token file via env 20register[?]( app-token ) Map Task Execution 21allocateResources[?](spec): ContainerInfo(nm-token,container-token) 22startContainer[nm-token]( spec=map,tokens byte buffer A&C,D) 23exec[as ?]() ApplicationContainer(ac) setuid? Is AC exec a setuid program? 24extractFiles? Does AC extract the job files from HDFS? 25exec[as c]() Task(t) Map Task 26readFile[RPC:SASL(c-deleg-tkn,nn)]( split-file ): blk-id, blk-loc,blk-access-tkn 27readblk[DTP(blk-access-tkn)]( blk-id ): split-blk-data 28jobStatus[?]() How does AC access AM?What does AC do now? Status Query 29statusQuery [?] () How does C know how to access AM? Reduce Task Execution 30negotateResources[?](): ContainerInfo 31launchContainer[?]( spec=reduce,tokens) 32exec[as ?]() ApplicationContainer(ac) Are different containers usedfor Map and Reduce Tasks? 33extractFiles? Does AC extract the job files from HDFS? 34exec[as c]() Task(t) Reduce Task 35writeFile[RPC:SASL(c-deleg-tkn,nn)]( reduce-file ): blk-id, blk-loc,blk-access-tkn Need to identify where c-deleg-tkncame from and when it was refreshed. 36writeblk[DTP(blk-access-tkn)]( blk-id ): reduce-blk-data 37jobStatus[?]() Job Completion

Text below

  • No labels