Answer: The yellow patch, or yellow “star of David”, set upon the clothing as a marker of Jewish identity, has its origins, not in 20th century anti-Semitism, but in 13th century Christian anti-Semitism.
In 1215 Pope Innocent III decreed that Jews living among Catholics must wear a yellow badge to distinguish themselves from the Christians.
Source: PBS Documentary from the Frontline series, titled The Longest Hatred: A Revealing History of Anti- Semitism (DVD release 2004). The whole DVD, by the way, is excellently produced and highly worth owning.
NOTE: The PBS documentary does not mention this, but Pope Innocent may have taken his decree cue from Muslim regions, where the designation of groups by distinguishing markers was widely practiced. Paul Johnson, in his 1987 book, A History of the Jews, claims that Jews were marked off, in twelfth century Baghdad, by a yellow patch (p. 204).
Below is an image of medieval Jews, with patches on their garments in the shape of Moses’s tablets of the Law, being beaten by a Christian: