hadoop - Queries about YARN (failure modes, container size, practical example) -


i want ask few questions understand working of yarn:

  1. anyone can explain or refer document can failure modes in yarn (i.e. task failure, application master failure, node manager failure, resource manager failure)
  2. what container size in yarn? same slot in map reduce 1?
  3. any practical/working example of yarn ? thank you

refer hadoop definitive guide text book ... apart there lot of info in apache web site.

container size not fixed dynamically allocated based on requirement resource manager.

from developer perspective same old map-reduce work on yarn.

resourcemanager failures

in initial versions of yarn framework, resourcemanager failures meant total cluster failure, single point of failure. resourcemanager stores state of cluster, such metadata of submitted application, information on cluster resource containers, information on cluster’s general configurations, , on. therefore, if resourcemanager goes down because of hardware failure, there no way avoid manually debugging cluster , restarting resourcemanager. during time resourcemanager down, cluster unavailable, , once gets restarted, jobs need restart, half-completed jobs lose data , need restarted again. in short, restart of resourcemanager used restart running applicationmasters. latest versions of yarn address problem in 2 ways. 1 way creating active-passive resourcemanager architecture, when 1 goes down, becomes active , takes responsibility cluster. way using zookeeper resourcemanager quorum, resourcemanager state stored externally on zookeeper, , 1 resourcemanager in active state , 1 or more resourcemanagers in passive mode, waiting happen brings them active state.

applicationmaster failures when applicationmaster fails, resourcemanager starts container new applicationmaster running in application attempt. responsibility of new applicationmaster recover state of older applicationmaster, , possible when applicationmasters persist states in external location can used future reference. applicatoinmaster store state persisitant disk status till failure can recovered.

nodemanager failures if node manager fails, resourcemanager detects failure using time-out (that is, stops receiving heartbeats nodemanager). resourcemanager removes nodemanager pool of available nodemanagers. kills containers running on node & reports failure running ams. ams responsible reacting node failures, redoing work done containers running on node during fault.

container failures

container failures reported node manager resource manager , resource manager informs same application master. application restart container.


Comments

Post a Comment

Popular posts from this blog

asp.net mvc - SSO between MVCForum and Umbraco7 -

i need integrate sso between mvcforum (forum of company) , umbraco7 (manages cms company). however struggling shared authentication work. tried several ways suggested on other threads still not running expected. here did: 1. in mvcforum (forum.mywebsite.com) 1.1. machine key in web.xml: <machinekey validationkey="the same key umbraco7" decryptionkey="the same key umbraco7" validation="sha1" decryption="aes" /> 1.2. authenticate forms in web.xml <forms name="my-sso-auth" protection="all" path="/" timeout="43200" domain="forum.mywebsite.com" loginurl="/members/logon" enablecrossappredirects="true" /> 2. in umbraco7 (news.mywebsite.com) 2.1. machine key in web.xml: <machinekey validationkey="the same key mvcforum" decryptionkey="the same key mvcforum" validation="sha1" decryption="aes" /> 2.2. a
Read more

Python Tkinter keyboard using bind -

i have problem using bind option can click button in keyboard, , call function once key pressed. tried taking other codes have similar purpose , saw i'm doing pretty same, though still have problem must've missed. thankful if me this. here's code: # -*- coding: utf-8 -*- #imports tkinter import * pil import imagetk, image import time import tkmessagebox #===========================================================================================================================================# #tkinter class class app(frame): def __init__(self, master=none): frame.__init__(self, master) self.pack() #===========================================================================================================================================# #game pieces classes class board: def __init__(self, rows = 7, columns = 6, picture_height = none, picture_width = none): self.rows = rows self.columns = columns self.picture
Read more

ubuntu - Selenium Node Not Connecting to Hub, Not Opening Port -

i'm trying set demo selenium grid. have digital ocean ubuntu 14.02 droplet. (i have 2 , tried set using 2 different machines got nowhere.) i'm trying run hub , node both locally. here input both hub, node, , nmap results. https://gist.github.com/mekhami/d5bbecca5ff0de9d2dc9 so, node starting tcp connection reset , port it's supposed operate on closed. i have had no luck in number of different options , configurations work. but... of course, worked randomly once. don't know why, don't know did, know 1 brief moment, node connected , saw on grid console. tried run test , went down again , i've never seen come up. please let me know more information can provide. update: randomly came online again. don't know happened, might related timeout or time_wait or going on. this because vm/server not have enough entropy available. to resolve problem, install haveged (or other similar things). the instructions of how install haveged on ubu
Read more