I did actually test this when deciding whether to update this article or not. I ran queries with the same structure but different entities as you suggested. I found that changing the entity did not slow down the query, the compiling therefore must be for structure and not the EXACT query. I explain my decision a little more here.
You are right however and I am very keen to develop an LDBC benchmark with you. I see that they use data and query generators to get round this problem and I would definitely propose the same approach.
We have an email thread open which I would like to use for this purpose as your experience will be very valuable!
Thanks again for your feedback,