Patrick Swayze Brother Says Late Actor Called John Leguizamo Difficult

John Leguizamo might not remember Patrick Swayze fondly all these years later -- but he might be pleased to know the feeling was mutual ... so says PS' brother. We spoke to the late actor's sibling, Sean, who tells TMZ ... he actually recalls…