When Clint Barton came to the Carson Carnival of Traveling Wonders, the Swordsman became one of his mentors. Young Clint strove to become the Swordsman’s equal, but always fell short of his master. The Swordsman’s carnival days came to the end when Clint walked in on him stealing from the traveling attraction. The Swordsman asked if Clint would join him in a life of professional crime, but was fought instead by his student. The Swordsman defeated Clint and escaped.
The Swordsman later tried to infiltrate the Avengers as a way to get away with his crimes more easily. The Swordsman became involved in a scuffle with the Scarlet Witch and Quicksilver. He was identified by Clint who was now the heroic Hawkeye and was sent away. The Mandarin teleported Duquesne to his ship where he modified the Swordsman’s sword and made a deal with him to destroy the Avengers by again infiltrating them. Through the Mandarin’s tricks, the Avengers accepted the Swordsman who went through a trial and fought alongside them for a time. When the time came to destroy the Avengers, the Swordsman planted the bomb the Mandarin gave him, but could not bring himself to gain victory in that way. When seen trying to deactivate the bomb, he was misunderstood and was forced to flee.
The Swordsman clashed with the Avengers many times after that, including in a partnership with Powerman and the Enchantress and as part of a team of villains assembled by the Mandarin. In their clashes, Hawkeye had proved himself no longer inferior to the Swordsman and proclaimed his freedom from his old mentor. The Swordsman eventually did join the Avengers for righteous reasons and served well. During this time he had a romantic relationship with the Mantis. On one of the Avengers missions, the Swordsman was slain, dying a hero.
Years later, the Swordsman, along with other deceased Avengers, was resurrected as a slave to the Grim Reaper, who hoped to literally spread Hell to Earth. Defeating the living Avengers and raiding Avengers Mansion, the Swordsman and his team mates were restored to their right minds by the Scarlet Witch. Swordsman and his undead allies freed the Avengers and turned on the Reaper, but were again returned to the grave when the Reaper was defeated by Wonder Man.