Pain? Nah. 6061 is nice to mill! It carries the heat off the work, it's pretty soft allowing for decent feeds and speeds but it's still reasonably stiff and holds a decent finish. You just need to respect that it will try and gum to the cutter and deal with that, and know that it will bounce a bit so super fine 0.001" spring passes do bugger all but scuff it up, that last pass needs to be a bit meatier for it to work properly.
I love working with 6061.