One of my GPU fan stopped working, I found a replacement fan on Ali express, but I couldn't fit the fans properly, I had to cut the cable to reroute, as it was touching the fans. After installing the fan, 5 minutes of using the PC, display freezes, but fans are spinning.
Any idea if it's worth fixing or ordering another fan?
It sounds like the fan was spinning too slow.
Your GPU overheated because the Ali Express fan was not producing enough airflow.
This might be because the the board might be designed for a 5V fan, you ordered a 12V fan and your new fan ran way too slow.
A quicker fix would be cable tie a 120mm case fan on top of the GPU's heat sink (hanging off the bottom when the GPU is installed). Plug these into your motherboard's fan headers, not the GPU's. Or buy a molex fan if you have molex connectors off your power supply.
As you are running a 760, I recommend the cheap fans (ignore the smaller fans). Pick 120mm, 12V and your connector type
This might be a sign that some other part of your GPU is actually dying, not just the fan itself. Typically a GPU should run even without a fan spinning for more than 5 minutes, since the idle temperatures should generally be low enough that the GPU should still work (in the short term). Have you monitored the temps using MSI afterburner or EVGA precision? If the freezing occurs even at low temps then you know you might have a problem with the GPU itself.
Assuming you have re-applied thermal paste, cleaned the dust off the card and made sure the heatsink is making proper contact with the card, there's not much more you can do to fix the GPU.
You could try removing all of the heatsink and shrouds from the GPU and "reflow" the solder in a oven although this would only fix issues arising from cracked or bad solder joints.
Perhaps sell the GPU as-is?
You might get a lot of money unexpectedly since people are desperate for any kind of GPU. Past listings indicate that they last sold for $80~125. Which is a lot considering the card is 8 years old now.