When you drill the frame & fork crown, make sure the new size hole is only just big enough to fit the shaft of the nut through, as the flange/head on the end isn't that much bigger. You want to make the most of that limited area to clamp with. And only drill the side the nut needs to go through; don't go all the way as you want the bolt to remain a snug fit in its hole. You'll find drilling the hole in the rear brake bridge will be at a really awkward angle, 'cos you have to come in from underneath & will quite probably find the seat tube in the way.
Just ride the bloody thing and enjoy yourself!