If you've never ridden a dirt bike I suggest getting a cheap lightweight one. Trailer it out to the woods and thrash it until you've learned how to ride dirt, and learned whether you like it. Honda XR250, Yamaha TTR250, or Suzuki DRZ250 are the most obvious candidates and clean ones can be had for under $2000.
Dual sports weigh 50 lb more and have a bunch of lights that will just get broken off.
The problem with getting a 650 or even a 400 right off the bat, is it is very intimidating to learn dirt riding on them. Expensive to crash, hard to pick up, and much harder to handle.
--------------------
2009 E250 Camper Van
Suzuki V-Strom 650