When the smoke chamber is not smooth, you could be experiencing excess smoke in your room and not up through that chimney. The jagged edges create turbulence and can back-up the flow. Do smoke chamber need parged?
If you have holes in your brick, you are at more of a risk. The holes do not get cleaned with a normal sweep and creosote condenses. When caught on fire, the heat can transfer to adjacent flammable area.