No. I think the two issues are separate.You're not considering the fact there are UK citizens claiming benefits in the EU.
After brexit, economically inactive citizens residing in another country may have to leave because the host country may refuse to provide with assistance: housing, jobs, disability benefits.
There are about 30% of 3.5 millions EU citizens that are in this category.
I don't know how many of them receiving benefits. Also, many who have jobs may have only part time jobs. The rules of benefits to those with part time jobs may change after brexit.
I guess that life will be much tougher for those with part time jobs or no jobs. Some will want to leave the UK for Germany or home. That will reduce the pressure on housing and other public services.