By Decade's End, Big Data Will Be Bigger Than Ever

Specialized tech knowledge will be necessary in our data-driven society.


Information technology will be one of the fastest-growing fields this decade as commerce has gone digital in industries across the board. While the world may not see the turn-of-the century staffing crunch that was feared during the transition to Y2K and euro conversion, the proliferation of data could outstrip the number of people being trained to manage it.

Teri Morisi, economist and branch chief at the Bureau of Labor Statistics, identifies computer occupations the BLS is tracking for growth. Included are computer and information research scientists; computer systems analysts; computer programmers; software applications developers; systems software developers; database administrators; network and computer systems administrators; computer support specialists; and information security analysts, Web developers, and computer network architects.

[See The 6 Best Technology Jobs of 2012.]

"All computer occupations are expected to grow faster than the average for all occupations at a rate of 14.3 percent for the 2010 to 2020 period," Morisi says. An exception is computer programming. Although it has long been a target for outsourcing, it is still expected to grow at a rate of 12 percent.

The occupation with the most job openings, software applications developers, is expected to gain 143.8 million jobs by 2020, a 27.6 percent increase, followed by systems software developers, with approximately 127,000 jobs at around a 32 percent increase. Database administrator jobs are expected to spike 30.6 percent and gain about 34,000 new positions. Computer hardware engineers, which are not classified with computer occupations, will climb 9 percent, from approximately 70,000 to 76,300 jobs. Computer and information systems managers, also not in the computer field, will grow from about 308,000 positions to approximately 364,000, an 18 percent increase.

A Ph.D. in computer science or a related field is required for most computer and information research scientist jobs. In the federal government, a bachelor's degree may be sufficient for some jobs. According to the BLS, an associate's degree could suffice for network and computer systems administrators, as well as for computer support specialists.

The term "Big Data" is generally used to describe the exponential growth of structured and unstructured data. Volume shows no sign of abating. The velocity with which it moves is approaching light speed on stock exchanges in the securities industry. There's also a much greater variety than in the past, including text, video, and audio. By some estimates, 80 percent of an organization's data is not numeric.

This data-driven sea change in the workplace has been going on at least since 1964, when IBM invented the mainframe computer, a milestone event in the conversion from manual processing of data to automation. In the mid-1960s, the New York Stock Exchange closed every Wednesday just to match buy and sell orders so trades could be settled. Trade volume spiked from 5 million shares in 1965 to 15 million in 1968. Today, in an era of high-frequency trading and 100 gig data centers, the NYSE-Euronext, as it is now known, averages in excess of 1 billion shares per day, reaching nearly 2.5 billion on June 22. Mega-volume of this kind could not have been supported by traders waving paper tickets at each other on the trading floor.

[See 10 Businesses That Will Boom in 2020.]

As data proliferates, one solution could be to put the ability to manage it—and the tools to manage it with—in the hands of more people who aren't IT pros.

"The fundamental assumption of Big Data is the amount of that data is only going to grow," said John Ginder, the lead of the big data analytics and the environmental science teams at the Ford Motor Company, during a July interview with ZDNet. If there is a shortage of database scientists, Ginder favors giving non-technical people tools to solve big-data problems themselves.

"That's the endpoint I'd love to see us move toward," he said, "but there aren't enough tools out there to enable us to do that yet."