Come to find out the files that were causing the issue had been de-duped and in the Get Info window it would show the actual file size, but then it would also show zero bytes on disk, meaning the file was a duplicate on the server and it only contains metadata.