Bismillahir Rahmaanir Raheem Al Jawaab Billahit-Tawfeeq
I would like to be very frank and straightforward with you. Please stop this affair immediately. What you and the brother are doing is absolutely not permitted in Islam. Not only will this haraam relationship destroy your future but also the marriage of the brother and his wife if it persists. Also, think about the child's future if the couple decided to divorce (al'iyaadhu billah).
Alhamdulillah, Allah has given you the taufiq (ability) to consult the 'ulamaa with regards to this matter. This should then make it easier for you to repent for this sin you have committed insha Allah. Make sincere repentance to Allah and a firm intention in your heart that you will not get involved in this haraam relationship anymore. You should also stop all forms of communication with the brother as he is not your mahram.
The brother may have problems/difficulties in his life but you cannot be the person whom he turns to for a solution. He should learn to turn to his Creator (ie. Allah) as only Allah can assist him and nobody else. The fact that the brother found consolation in you was just Shaithaan's way of getting both of you into an affair - which was successful.
However, you can make that change and show Shaithaan that you will not carry on with this relationship. Initially it will be difficult since you will be fighting over your nafs (desires). But do not give up! Just keep on telling yourself that if you do not stop this relationship, you will be a means of destroying other people's future and relationship. Would you like that to happen? Insha Allah, due to your sincere repentance, Allah will make things easy for you.
If the brother really wanted a future with you, then he would've proposed for marriage instead of hiding the relationship. After all, polygamy is permitted in Islam. Instead of having an affair, Allah has given a leeway for us so that a halaal bond can be created by means of polygamy. This would kill two birds with one stone - the brother's marriage is not compromised nor will you and the brother have to keep on sinning.
If marriage had not been considered by both of you, then end the relationship immediately. I cannot emphasize this further. May Allah make things easy for the both of you to lead a pure and halaal life insha Allah.