"Most of all I had to use my ears."
How I did it: I am a pianist by trade, so there are obviously certain aspects of this skill I already knew. The first thing I did was to go to the local music store and buy a whole bunch of music for organ ranging from the super simple to the more complex, both with and without pedals. Then I started reading through music and messing with the stops and slowly found my way around the instrument. The 3 main challenges were as follows:
1) Legato playing: The bell-like percussion sound of the piano is absolutely gorgeous, but the organ can't and shouldn't sound like that. To me it sounds the most beautiful when it sounds like a choir or a string section with very legato movement. The keys to this for me have been listening and fingering. On the piano, I can BS my way through fingering many things because I can save myself with the damper pedal if I find myself in a sticky spot. On the organ I've found I really have to plan my fingerings meticulously, and really practice those finger substitutions, especially the ones that don't come the most naturally. Also, being able to slide from one key to the next smoothly on one finger is really useful. Also, each voice has to be heard as a melody in its own right. It helps to dismantle it and play each voice individually so you can hear them when they're all put together. This helps prevent letting go of certain voices too early or too late.
2) Pedals: Oh, feet. I've been playing stocking-foot, although I might acquire a pair of organ shoes at some point. I'm also working on getting a bench that isn't so high as the organ at church, which should make it more comfortable for me to reach. In general, the hardest thing with learning to pedal has been learning how to sit without falling forward. Part of that challenge is that the bench height is wrong, but I've found a spot where I can sit up without too much strain on my lower back and still reach everything OK. That just took a lot of experimentation. A method book for new organists that I've been using has a lot of short drills for pedals and I've found that repeating those have been really useful for just physically learning my way around those keys. As far as combining pedals with hands, I've found that I have to really practice the pedal and left hand parts together, leaving out the right hand, frequently, because my left hand really wants to play the bottom line of music really really badly.
3) Registration: This is a matter of pure experimentation and playing with different combinations, and if you're playing on multiple manuals, using your ears to balance it out. That's about it. But it was harder than I initially thought it would be to find combinations I liked.
I played my first wedding today. :D
Lessons & tips: Listen, listen, listen. If you don't want to sound like a pianist playing the organ, try to sound like a choir singing.
Resources: But what do I do with my feet?: A Pianist's Guide to the Organ, by Janette Fishell
Instruction Book for Beginning Organists, by David Johnson