The new company will account for about 21 percent of the approximately 6.5 billion shares that are traded daily on exchanges, according to BATS. It would leapfrog over the Nasdaq and rank behind the New York Stock Exchange.
The combination is "an important milestone for the U.S. equities market," said BATS CEO Joe Ratterman, who will serve as CEO of the merged company.
The deal is part of a trend of consolidation among global stock exchanges. Technology has increased the speed at which trades are processed, but reduced the profit margins that exchanges can make on shares traded.
Earlier this month, regulators approved a $8 billion sale of the NYSE to Atlanta-based IntercontinentalExchange, or ICE. The NYSE's parent company was itself created by a merger of the NYSE Group and Europe's Euronext exchange company.
The NYSE has a market share of about 23 percent and the Nasdaq has a market share of 18 percent.
The four U.S. stock exchanges run by BATS and Direct Edge will continue to operate.
Both companies are privately owned. Direct Edge is owned by a consortium that includes International Securities Exchange, Knight Capital Group, Citadel Derivatives Group, The Goldman Sachs Group, and J.P. Morgan.
BATS Global Markets is owned by a group that includes Bank of America Merrill Lynch, Citigroup, Credit Suisse, Deutsche Bank and Morgan Stanley.
The combined company will use BATS technology and be headquartered in Kansas City, Mo.
The deal is expected to close in the first half of next year.