Republican Bob Turner, a retired media executive who had called the election a repudiation of President Barack Obama, defeated Democrat David Weprin, a state assemblyman, by six points, 53 percent to 47 percent, a New York cable television station said.
"We've been asked by the people of this district to send a message to Washington and I hope they hear it laud and clear," Turner told supporters. "We're ready to say, 'Mr. President, we are on the wrong track.'"
The district, which includes parts of Brooklyn and Queens, had gone Democrat in every election since the 1920s, and Democrats outnumber Republicans three to one.
Turner's triumph, and a Republican victory in another special House election -- in Nevada -- boosted the Republican majority over Democrats in the House to 242-192.Turner, who said he entered politics because he was "fed up" with overspending in Washington, has called for deep cuts in the federal budget.
Weprin had tried to cast Turner as part of the Tea Party, which wants smaller government and lower taxes and is unpopular with many liberal New Yorkers.
Weprin appeared briefly in front of supporters but declined to concede the race. "It's not over yet," he said.
Steve Israel, chairman of the Democratic Congressional Campaign Committee, issued a statement playing down the importance of Turner's victory.
"The results ... are not reflective of what will happen in November 2012 when Democratic challengers run against Republican incumbents who voted to end Medicare and cut Social Security while protecting tax loopholes for big corporations and the ultra wealthy," he said.
Weiner, who served for seven terms, was a Democratic firebrand known for passionate speeches in support of healthcare reform and other issues dear to liberals.
Weprin had raised $500,000 for his campaign, compared with Turner's $200,000. He ran unsuccessfully in 2009 for New York City comptroller before filling the state assembly seat vacated by his brother.
In recent days, Democrats had rushed to help Weprin, who some said was a lackluster campaigner.Former President Bill Clinton recorded robo-calls urging Democrats to vote and Charles Schumer, the senior U.S. senator from New York, who used to represent the district, accompanied Weprin on the campaign trail.
In Tuesday's special election to fill a vacant House seat in Nevada, Republican Mark Amodei won in a largely rural district that has never sent a Democrat to Congress.