I know we overthink every single fruit we eat, I was so scared of lychees too! But my doctor said normal seasonal fruits in small portions are completely safe.
I am in my 3rd trimester and honestly, if I want lychees, I eat them! Just keep it to a handful so you don't get a sugar rush while feeling already heavy.
Beta, eat only 2-3 lychees at a time because they have a 'garam' tendency. It is fine to satisfy cravings but balance it with coconut water to keep body cool.
It is totally fine in moderation and doesn't cost much to add to your fruit bowl. Just don't eat too many, especially if you have to do the glucose tolerance test soon.