The report came from three people at the meeting at the agency's office in San Francisco who spoke to the Post and said they believed he was referring to men who had transitioned to transgender women.
Two others also said Carson lamented that society did not differentiate between genders anymore.
They claimed that the comments offended many and that one woman left the meeting in protest.
A senior HUD official denied the report in a statement to the Post.
"The Secretary does not use derogatory language to refer to transgendered individuals. Any reporting to the contrary is false," the statement read.
The HUD official also said that Carson was not referring to transgender women, but to men pretending to be women.
Presidential candidate Julian Castro excoriated Carson over the report, and said it was comments like his that led to transgender persons being murdered.
"19 Black trans women have been killed this year because comments like Ben Carson's normalize violence against them," Castro tweeted.
"As HUD Secretary, I protected trans people, I didn't denigrate them," he concluded.
Here's more about Carson at HUD:
Ben Carson Holds Forum In LA To Discuss Affordable Housingwww.youtube.com