In daily expression it's of course fine, but sometimes yes.
(1) Japanese language has complicated honorific expression, and that is not used often in daily conversation but required in some situations. Some people may feel hard to use accurate expression of honorific/humble form.
(2) There are too many kanji, and one kanji often has several reading. So when we see unfamiliar/not often used words, it is possible that we are not sure which reading we should choose, or it is even possible that it contains unfamiliar kanji.
Japanese writing kanji takes 9 years of study for Japanese people to learn 2000basic kanji. if regular japanese should know 5000 kanji I would say it is hard even for them. in the institute of languages Japanese it's ranked as very difficult.
@cuervotengu I don’t think it takes so long even though the official curriculum set by the government takes 9 years. I remember I could read and write all of 2000 常用漢字 2 years after i started learning them and it’s not rare for a child who reads can learn that fast.