Having raised our two children bilingually, German (mother) and English (father) – living in Germany – I would encourage you to make the effort – speak English with your son – he will be grateful later in life!!
My experience:
For the first five years of my daughter’s life, I spoke English with her and my wife German. My daughter rewarded my efforts in that time with ca. 5 words of English. But when my sister (who speaks no German) came to visit, my daughter suddenly started speaking fluent English. When I asked her why she had not spoken English with me – she replied (in German) – “Daddy, I don’t have to with you, you understand German!”. Anyway now, she(18) is at home in both languages and cultures.
Good luck. By the way, did you also learn any Gaelic ?
TTFN, PlumD

I am also raising a bilingual child. She’s only 2 months old right now. 1/2 Brazilian and 1/2 American. I encourage my husband to speak to her only in Portuguese while I speak only English with her. I hope this tactic works.
Good luck with your baby!