Thursday, January 5, 2012

Error that occured in Hadoop and its sub-projects

1. OOZIE job failed:

Error message : ERROR is considered as FAILED for SLA
   
Cause 1 : Not able to find hadoop namenode (master), jobtracker machine.
Suppose you are running oozie, hadoop-master and job tracker on one machine  and datanode, tasktracker are running on another machine.

Your job.properties file contains following lines:
        nameNode=hdfs://localhost:9000
        jobTracker=localhost:9001
   
In above case, FS action will work fine because no map-reduce opertion is perform in FS action case. But, if you run map-reduce action then tasktracker will look hadoop-master on localhost machine becuase we have used localhost:9000 in job.properties file.
   
Solution : Used  IP of hadoop-namenode and jobtracker machine in job.properties file instead of localhost.   
   
Cause 2 : Oozie not able to find Mysql server.
Suppose I am using mysql as a metastore for hive.
Hive hive-default.xml file have following lines :
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
       
Solution : Use IP of mysql machine instead of localhost. 


2. Zookeeper server not running:
Error message: Could not find my address: zk-serevr1 in list of ZooKeeper quorum servers
   
Causes :
HBase tries to start a ZK server on some machine but that machine isn't able to find itself in the hbase.zookeeper.quorum configuration. This is a name lookup problem. 

Solution:   
Use the hostname presented in the error message instead of the value you used (zk-server1). If you have a DNS server, you can set hbase.zookeeper.dns.interface and hbase.zookeeper.dns.nameserver in hbase-site.xml to make sure it resolves to the correct FQDN.

3. Hadoop-datanode job failed or datanode not running: java.io.IOException: File ../mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
   
Cause 1: Make sure atleast one datanode is running.

Cause 2: namespaceID of master and slaves machines are not same.
If you see the error java.io.IOException: Incompatible namespaceIDs in the logs of a datanode , chances are you are affected by bug HADOOP-1212 (well, I’ve been affected by it at least).
           
Solution :               
If namespaceID of master and slaves machines are not same. Than replace the namespaceID of slaves machine with master namespaceID.
- dfs/name/current/VERSION file contains the namespaceID of master machine
- dfs/data/current/VERSION file contains the namespaceID of master machine
        
Cause 3: Datanode instance running out of space.
Solution : Free some space.

Cause 4 : You may also get this message due to permissions. May be JobTracker can not create jobtracker.info on startup.

4.    Sqoop export command failed:
Error message:
attempt_201101151840_1006_m_000001_0, Status : FAILED
java.util.NoSuchElementException
at java.util.AbstractList$Itr.next(AbstractList.java:350)
at impressions_by_zip.__loadFromFields(impressions_by_zip.java:159)
at impressions_by_zip.parse(impressions_by_zip.java:108)

   
Cause : Given field separator is not valid
Solution : Specify correct field delimeter in sqoop export command.

5. HBase regionserver not running :

Error message: 2012-01-02 13:48:49,973 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: Master rejected startup because clock is out of sync
org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server hadoop-datanode2,60020,1325492317440 has been rejected; Reported time is too far out of sync with master.  Time difference of 206141ms > max allowed of 30000ms

Solution: Clock of regionservers are not sync with master machine. Synchronized the clock of hbase master and regionserver machines.

43 comments:

Anonymous said...

/usr/local/hadoop/hadoop-0.20.203.0/bin/start-dfs.sh
starting namenode, logging to /usr/local/hadoop/hadoop-0.20.203.0/bin/../logs/hadoop-hduser-namenode-hadoop-ThinkCentre-A51.out
localhost: starting datanode, logging to /usr/local/hadoop/hadoop-0.20.203.0/bin/../logs/hadoop-hduser-datanode-hadoop-ThinkCentre-A51.out
localhost: starting secondarynamenode, logging to /usr/local/hadoop/hadoop-0.20.203.0/bin/../logs/hadoop-hduser-secondarynamenode-hadoop-ThinkCentre-A51.out
hduser@hadoop-ThinkCentre-A51:~$ jps
12799 SecondaryNameNode
12837 Jps



plz hlp us to correct this error

Ankit Jain said...

Please share the namenode and datanode logs.

Anonymous said...

ACTION[0000017-120306175650364-oozie-oozi-W@mr-node] checking action, external ID [job_201203062014_0014] status [RUNNING]
2012-03-07 22:06:23,420 INFO CallbackServlet:525 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000017-120306175650364-oozie-oozi-W] ACTION[0000017-120306175650364-oozie-oozi-W@mr-node] callback for action [0000017-120306175650364-oozie-oozi-W@mr-node]
2012-03-07 22:06:23,577 INFO MapReduceActionExecutor:525 - USER[hdfs] GROUP[users] TOKEN[] APP[map-reduce-wf] JOB[0000017-120306175650364-oozie-oozi-W] ACTION[0000017-120306175650364-oozie-oozi-W@mr-node] action completed, external ID [job_201203062014_0014]
2012-03-07 22:06:23,625 WARN MapReduceActionExecutor:528 - USER[hdfs] GROUP[users] TOKEN[] APP[map-reduce-wf] JOB[0000017-120306175650364-oozie-oozi-W] ACTION[0000017-120306175650364-oozie-oozi-W@mr-node] LauncherMapper died, check Hadoop log for job [192.168.1.123:8021:job_201203062014_0014]
2012-03-07 22:06:23,803 INFO ActionEndCommand:525 - USER[hdfs] GROUP[users] TOKEN[] APP[map-reduce-wf] JOB[0000017-120306175650364-oozie-oozi-W] ACTION[0000017-120306175650364-oozie-oozi-W@mr-node] ERROR is considered as FAILED for SLA

hi i am avinash i installed oozie using tar ball and ran oozie job from hdfs user i got error
ERROR is considered as FAILED for SLA

can u help me

Ankit Jain said...

Hi Avinash,

Which oozie action you have run?? .. Look into the hadoop jobtracker log, may you get some clue.

Renata Ghisloti Duarte de Souza Granha said...

Nice post. Thanks!

mareddyonline said...

hi ,you have gathered a valuable information on Hadoop...., and i am much impressed with the information and it is useful for Hadoop Learners.
Hadoop Training in hyderabad

Unknown said...

Uniqe informative article and of course True words, thanks for sharing. Today I see myself proud to be a hadoop professional with strong dedication and will power by blasting the obstacles. Thanks to Hadoop Training in Chennai

Unknown said...

Thanks for sharing this valuable information to our vision. You have posted a trust worthy blog keep sharing.AWS course chennai | AWS Certification in chennai | AWS Certification chennai

Unknown said...

Your posts is really helpful for me.Thanks for your wonderful post. I am very happy to read your post. VMWare Training in chennai | VMWare Training chennai | VMWare course in chennai


Unknown said...

Hi admin thanks for sharing informative article on hadoop technology. In coming years, hadoop and big data handling is going to be future of computing world. This field offer huge career prospects for talented professionals. Thus, taking Hadoop Training in Chennai will help you to enter big data technology.

Unknown said...

Whatever we gathered information from the blogs, we should implement that in practically then only we can understand that exact thing clearly,
but it’s no need to do it, because you have explained the concepts very well. It was crystal clear, keep sharing..
Websphere Training in Chennai

Mervin Parmar said...

Using big data analytics may give the companies many fruitful results, the findings can be implemented in their business decisions so as to minimize their risk and to cut the costs.
hadoop training in chennai|big data training|big data training in chennai

Unknown said...

Cloud computing is the next big thing, through cloud the users have the liberty to use a shared network. The companies can focus on core business parts rather than investing heavily on infrastucture.
cloud computing training in chennai|cloud computing courses in chennai|cloud computing training

Unknown said...

Greens Technology's the leading software Training & placement centre Chennai & ( Adyar)
base-sas training in chennai

Unknown said...

Excellent information with unique content and it is very useful to know about the information based on blogs.
android training in chennai

Unknown said...


I really enjoy it, to reading this
Informatica Training In Chennai | Hadoop Training In Chennai | Sap MM Training In Chennai

IT Tutorials said...

Thanks for your article. Its very helpful.As a beginner in hadoop ,i got depth knowlege. Thanks for your informative article. Hadoop training in chennai | Hadoop Training institute in chennai

Sivanandhana Girish said...

Usually, I never comment on blogs but yours is so convincing that I never stop myself to say something about it. keep updating regularly.

Selenium Training in Chennai
software testing selenium training
iOS Course in Chennai
iOS Training Chennai
Java training institute in chennai
Best JAVA Training institute in Chennai

Aruna Ram said...

Thanks for a marvelous posting! I definitely enjoyed reading it, you could be a great author.I like your post.
Digital Marketing Course in Bangalore
Digital Marketing Training in Bangalore
Digital Marketing Training in Tnagar
Digital Marketing Training in Nungambakkam
Digital Marketing Training in Karappakkam
Digital Marketing Training in Padur

Unknown said...

Thanks for your interesting ideas.the information's in this blog is very much useful for me to improve my knowledge.
android classes in bangalore
Android courses in Anna Nagar
Android Training in Saidapet
Android Certification Training in OMR


Anbarasan14 said...

One of the best blogs that I have read till now. Thanks for your contribution in sharing such a useful information. Waiting for your further updates.

Best IELTS Coaching Institute in Velachery
IELTS Coaching Classes in Guindy
IELTS Training Center in Madipakkam
IELTS Coaching Classes near me
IELTS Coaching in JP Nagar Bangalore
IELTS Classes in JP Nagar
IELTS Coaching Class in JP Nagar

Harish said...

Very good post. I have gained a lot of information from your blog, Keep sharing this kind of useful information

Microsoft Dynamics CRM Training Courses | CRM Training in Chennai | Microsoft Dynamics Training in Chennai | Microsoft Dynamics Training in Adyar | Microsoft Dynamics Training in Tambaram | Microsoft Dynamics Training in Velachery

LindaJasmine said...

Amazing Write-up. Great content. It shows your expertise on the subject. Thanks for Sharing.
Photoshop Classes in Chennai
Photoshop Course in Chennai
Photoshop Training in Chennai
Photo Editing Courses in Chennai
Photoshop Training Institute in Chennai
Best Place to Learn Photoshop in Chennai
Photoshop Training Classes in Chennai
Best Photoshop Training in Chennai

VenuBharath2010@gmail.com said...

Amazing Post. It shows your great understanding of the concept. Thanks for Sharing.
IELTS coaching in Chennai
IELTS Training in Chennai
IELTS coaching centre in Chennai
Best IELTS coaching in Chennai
IELTS classes in Chennai
Best IELTS coaching centres in Chennai
IELTS Centre in Chennai
IELTS Training
IELTS Course in Chennai

LindaJasmine said...

Extra-ordinary. The way you narrate the post makes it a exemplorary piece of work. Pls Keep writing.
Tableau Classes
Tableau Courses
Tableau Classroom Training
Tableau Advanced Training
Tableau Software Training
Tableau Training in Velachery
Tableau Training in Tambaram
<a href="https://www.fita.in/tableau-training-in-chennai/”>Tableau Training in Adyar</a>

jefrin said...

Best article to read very useful
Best Tableau training course in chennai

Kiruthiprabha said...
This comment has been removed by the author.
Kiruthiprabha said...

Thanks for the informative article. This is one of the best resources I have found in quite some time. Nicely written and great info. I really cannot thank you enough for sharing..
Oracle DBA Course Online Training

hari said...

very nice
inplant training in chennai
inplant training in chennai for it
suden web hosting
tunisia hosting
uruguay web hosting
Bermuda web hosting
Botswana hosting
armenia web hosting
lebanon web hosting

hari said...

good
Bermuda web hosting
Botswana hosting
armenia web hosting
lithuania shared web hosting
inplant training in chennai
inplant training in chennai for it
suden web hosting
tunisia hosting
uruguay web hosting

Alfred Avina said...

Capsule theory is an excellent concept to talk about, but you can't ignore the relation of capsule theories with Big data modernization solutions.

Cho co said...

I had to do something...QUICKLY.  I started sending emails....first to the dispute center, then to the help desk.  Eventually, I received an automated email response: Resurge Diet Supplement

talha said...

Rather than depending just on the input or sentiments from the Salesforce Consultants, it is alluring to check the conversation loads up, client bunches just as other online media outlets in discovering what really individuals are saying about the Consulting Company and its staffs and client care. Salesforce institutes in pune

Mrbk30 said...

Very Informative blog thank you for sharing. Keep sharing.

Best software training institute in Chennai. Make your career development the best by learning software courses.

power bi training in chennai
microsoft azure training in chennai
devops training in chennai

Block said...

It has been simply incredibly generous with you to provide openly
what exactly many individuals would’ve marketed for an eBook to end
up making some cash for their end, primarily given that you could
have tried it in the event you wanted.
dot net course in chennai
core java training classes in chennai
manual testing training in chennai

Anonymous said...

en son çıkan perde modelleri
Mobil Onay
Mobil odeme bozdurma
nft nasıl alınır
ankara evden eve nakliyat
TRAFİK SİGORTASİ
dedektör
KURMA WEB SİTESİ
Aşk kitapları

Anonymous said...

Smm Panel
smm panel
İs İlanlari Blog
İnstagram takipçi satın al
hirdavatciburada.com
https://www.beyazesyateknikservisi.com.tr/
SERVİS
tiktok jeton hilesi indir

Anonymous said...

üsküdar vestel klima servisi
beykoz bosch klima servisi
üsküdar bosch klima servisi
pendik samsung klima servisi
pendik mitsubishi klima servisi
çekmeköy mitsubishi klima servisi
ataşehir mitsubishi klima servisi
maltepe vestel klima servisi
kadıköy vestel klima servisi

sportsbet said...

Good content. You write beautiful things.
hacklink
hacklink
mrbahis
taksi
vbet
sportsbet
korsan taksi
mrbahis
vbet

betmatik said...

Success Write content success. Thanks.
kıbrıs bahis siteleri
deneme bonusu
kralbet
canlı poker siteleri
canlı slot siteleri
betturkey
betpark

helen said...

maraş
bursa
tokat
uşak
samsun

JQW

ilayda said...

ısparta
tunceli
ağrı
ığdır
samsun

35NE0

berkecan said...

https://saglamproxy.com
metin2 proxy
proxy satın al
knight online proxy
mobil proxy satın al
E7UN72

Post a Comment