Assuming its not over lunch or dinner hour than I would be totally fine with it.

Just write on the invitation "cake and fruit to be served". or "cupcakes to be served". That way people will know.