Hi Bluenone.
You have almost got the baudrate calculation correct, but you can't enter "1.5" into the divider register. It must be an integer.
so your options are 10MHz:SPBRG=0 20MHz:SPBRG=1 30MHz:SPBRG=2 40MHz:SPBRG=3

As to your queries:
1) The main purpose was to develop a generic servo motor driver that could use surplus servo motors . To this end it was necessary to be able to read any encoder. Particularly the Mitsubishi servos are quite cheap on the used market, and they have a lot of power for their size. So, no, it was not an academic exercise in "building" a "virtual encoder interface".

2) The JED micro JED AVR256 Single Board Computer using the ATmega2560 CPU was something I originally used to make some machine controllers, it has the big advantage that you know it will work once you turn it on! , But they are expensive at $600 , although this price would have been acceptable in the end product, the real issue was the labor cost involved in wiring up all the other interfaces to the JED board. So I made my own board after using the JED board for development. I then upgraded the first 6 machines we built, which left me with 6 of these JED boards.

I'm using BASCOM from MCS electronics Home - MCS Electronics for code writing , it has a very usable IDE , and it is a true compiler, and it allows you to seamlessly blend in assembler lines if required, it also easily supports overlaying of variables, lookup tables and working with individual bits, all very useful for bit-bashing, it does have an emulator , but I rarely use this (except for comparing speed/size of different ways of performing maths functions . I am a hardware designer so basic is much easier for me to work with.

All of the arithmetic is done as fixed point or fractional integer.

3) The Mega2560 is a bit of an overkill , whereas the mega128 is much cheaper and readily available on development boards. the Olimex AVR-MT-128 is a good example it has LCD and 5 buttons too. The other issue is that the encoder operation is very demanding of the CPU, so it can't also be used for other general purpose activities. And finally the M2560 is a 100pin fine pitch IC, so a bit finicky for soldering.

4) The motor I experimented with was a HC-KF S238
The connector on the rear of the encoder is a 9 pin square connector
I think the connector numbers have been posted before, anyway these are from my record book:
Pin 1,2 are MR and MRR this is recieve data into encoder
Pin 4,5 are MD and MDR this is TX data out of encoder
Pin 3 is battery (I didnt use)
pin 5 is +5v power to encoder
pin 8 is logic ground.
MR/MRR and MD/MDR are differential data lines.
You then connect some of these wires to your micro-controller TTL serial port:
Micro TTL serial out to MR pin
Micro TTL serial in to MDR pin
connect about 1.5v to MRR as a reference ( I use 2k/4k resistors from 5v)
connect +5 and LG to +5 and gnd on the micro

The motor connections are 1=U , 2=V , 3=W, 4=Ground/chassis 5,6 are the 24v brake release connections.
U V W are the motor phases, approx 7.3ohm per phase, nominal values are 118vat 3000rpm and 1.1A.
I used a hard-switched MOSFET bridge operating off about 70v (12v initially while I was experimenting) I think I used FAN73835's to drive the MOSFET's
For hobby/experimental purposes you could use a L6205 combo bridge to drive the motor from a 48V supply , this limits speed to ~ 1200RPM.

Hope this helps, Cheers, BobT