The of stays. These days, even veteran communicators are saying and writing "couple miles from here" or "costs a couple bucks." That used to be the vernacular of tough guys in gangster movies.
Things get tricky when couple is used with words and phrases of comparison, such as more, fewer, too many, too few. Many people would say a couple of more dollars, but in that construction the of is dropped: a couple more dollars and a couple too few dollars are correct. However, if we slightly revise those phrases, of must be put back: a couple of dollars more and a couple of dollars too few are correct.