Consider the situation in the United States since 2025. Even a country that has long called itself "a nation of immigrants" can completely change how it treats foreigners when the administration changes. People who were residing legally can suddenly, one day, face deportation.
This is not only about the United States. No matter how long foreigners have lived in a country or how much tax they have paid, they do not have the right to vote. For members of parliament, there may be little reason to prioritize people who do not bring in votes. Taking a cynical view, one could even feel that they are being disregarded.
Even in the United States, "the nation of immigrants," mass deportations of immigrants took place. No one has grounds to say for certain that the same will never happen in Japan.
The very fact that accepting foreigners is being "debated" is a source of anxiety for those concerned. Even without any clear ill will, they may feel "maybe we are not welcome." In Vietnam, too, there are facilities that charge foreigners a higher admission fee than Vietnamese people. That alone was enough to make me feel unwelcome. This is a trivial matter, but if so, how do foreigners feel when they hear about or experience far more serious situations?
Even if the debate over accepting foreigners reflects a minority opinion, for those on the receiving end it is not a matter of numbers. The very fact that their existence is up for debate is what makes them anxious.