How did East Asians come to be referred to as yellow-skinned? It was the result of a series of racial mappings of the world and had nothing to do with the actual colour of people’s skin. In fact, when complexion was mentioned by an early Western traveller or missionary or ambassador (and it very often wasn’t, because skin colour as a racial marker was not fully in place until the 19th century), East Asians were almost always called white, particularly during the period of first modern contact in the 16th century. And on a number of occasions, even more revealingly, the people were termed “as white as we are”. The term yellow occasionally began to appear towards the end of the 18th century and then really took hold of the Western imagination in the 19th. But by the 17th century, the Chinese and Japanese were “darkening” in published texts, gradually losing their erstwhile whiteness when it became clear they would remain unwilling to participate in European systems of trade, religion, and international relations. Calling them white, in other words, was not based on simple perception either and had less to do with pigmentation than their presumed levels of civilisation, culture, literacy, and obedience (particularly if they should become christianised).
Michael Keevak in The Chinese were white – until white men called them yellow (Scmp.com)