Do you have a Foreman in charge there? If not you should establish someone who is responsible for the card. Once you have done that, explain to them what you are finding based on the expenses that you are seeing come out and tell them that.
it does sound like they are taking advantage of you, - this happened to me as well in the past, where I sent my employees out to a job for weeks at a time and noticed that the card was almost maxed out after a few weeks - I told them the situation, and since I issued the card to the Field Supervisor, he was responsible for any over charges 100%.
Just establish an agreement with whoever has the card, and let them know that whatever you are paying them in perdiem - anything over that amount is 100% re-payable back to you and your business - as they are not authorized.
and make sure they keep receipts- if they are not even doing that, they could be spending $$ on non-per-diem stuff like clothing or other things that are not expense related.
If the keep playing around with you on this and are not taking you seriously , cut them off. - Your paying them, i am assuming either weekly or bi-weekly as a w-2 - so you really don't have to pay a per-diem by law.