Big Data and Regulation

The SEC's examination program has deployed a big data tool called the National Examination Analytical Tool (NEAT), which allows examiners to quickly analyze a firm's trading blotter. Once the trade blotter information has been loaded, NEAT quickly normalizes the data and provides a series of discrete tests. Examination managers report that 50 tests are already embedded in the tool, covering commissions, day trading, short sales within certain time parameters, allocations, profitability, systematically favored accounts and more. Additional tests are currently in development.

Andrew (Drew) Bowden, director of the SEC's examination program, reports that he is excited about the possibilities presented by NEAT. In an interview with this author, he highlighted several of its more important features.

First, NEAT allows examiners to study much larger sample sizes. Not long ago, SEC examiners would review a month, week or even just a day of trading, depending on the size of the firm. Now, with NEAT capability, trading samples routinely look back three or more years. This allows examiners to look for trends and patterns over longer periods of time in larger data sets.

Second, the tool tests the entire trade blotter without the intervention of subjective human factors. In the past, examiners' qualitative judgment played an important role even during quantitative analysis. Typically, examiners would sort the data and use their judgment to identify anomalies or patterns. Skilled and experienced examiners could use this approach to make significant findings ' but the output remained dependent on qualitative human factors. Now, Bowden emphasized, the same quantitative tests are applied to every trade, in the same way, by every examiner.

Third, as the examination program gains experience in NEAT's results, methodological consistency will be enhanced, because results in one examination can be compared to results in all others. Through this process, a library of “compliance signals” can be developed. Moreover, examiners can follow up on the signals to assess their value. As Bowden put it, examiners can go into the field to “test their hypotheses” and see what they have really learned from a signal.

Fourth, because NEAT tests are systematized, best-practice methodologies can be quickly deployed across the program. With a program-wide tool, developed and managed by a professional quantitative staff, new analytical approaches, or enhancements to the old, can be quickly developed, tested and deployed.

NEAT was originally developed for use in investment adviser examinations, and is now being used in other areas of the examination program as well, including with broker-dealers. It is, in Bowden's terms, a revolutionary development for examinations. SEC chair Mary Jo White agrees. She used the same term to describe NEAT ' “revolutionary” ' in testimony before Congress.

Other Big Data Programs

As important as NEAT may be at the SEC, it appears to be only the beginning. The examination program's Quantitative Analytics Unit (QAU), which developed NEAT, has 10 members with advanced degrees in mathematics, computer science, data science and financial engineering. Bowden indicates that other tools are now in development. For example, the Machine Analyzed Risk Scoring (MARS) tool is being developed to bring more quantitative rigor to the process of selecting firms for examination. Bowden described this as a strategic imperative of his program: bringing the same rigor and systematic analysis to targeting as is now deployed in the analysis of trading.

As the SEC's examination program grows increasingly sophisticated in its use of big data, we can see it begin to take on some of the dynamics of other big data practitioners. Bowden indicated that the SEC's exam program is on the hunt for data to feed its analytics. It already subscribes to commercial market data vendors for real-time validation of market prices, merger and acquisition news, and other services. In addition, data warehousing within the SEC has enhanced examiners' ability to tap into information already on file at the agency.

Another frequent benefit of big data is the ability to find new uses for previously untapped data sets. This is also playing a role at the SEC. When asked if he intended to use data from the private fund disclosure document (Form PF) for his risk targeting, Bowden replied, “Hell yes! We're going to wring it out for every valuable piece of information we can.” This is the same approach, he said, that the program is taking with every other potential data source.

Finally, as big data practitioners reach out to new sources of information, the integrity of data always takes on enhanced importance. SEC examiners in the field have reported spending more time on data-integrity issues ' and taking a much harsher view of firms who try to pass off bad data to their regulator.

The initiatives in the SEC's examination program highlight big data's promise for regulation. Not that many years ago, regulators had to worry about collecting information they couldn't use. What moral hazard, it was asked, would be created by data sitting unread deep in a government file? This author is personally familiar with regulatory proposals that were defeated with that kind of thinking. Now, with big data tools, every data set may have a role to play in identifying targets and testing their compliance.

Still, two important questions remain:

1. How will the SEC's examination program measure the success of its new tools? If the only metric of success is to increase enforcement activity, big data could end up looking and feeling like Big Brother. The SEC should give careful consideration to nonpunitive metrics. For example, examiners should be proud to report examinations in which the only outcome was to better refine the analytical tools or reduce a false positive signal.
2. How should legal and compliance practitioners respond to these new developments? As a practical matter, when SEC examiners arrive, armed with NEAT, they will not be impressed with less sophisticated compliance programs. Corporate counsel and compliance professionals must take a new look at their own processes in light of these regulatory developments.

An SEC Examination

Bowden spoke about his expectations for compliance in this area. As a practical matter, he said, based on his own past experience in private compliance, firms do not want to learn something for the first time during an SEC examination. If the SEC is going to use big data tools during examinations, than firms should make sure they have a pretty good idea of what those tools will show before examiners arrive at the door. To do that, firms need to deploy similar tools to those in the regulators' hands. In fact, Bowden said, he believes an important part of his mission is to encourage the private sector to move forward in this area. He hopes to promote compliance by “raising the bar” in regards to big data analytics.

The encouragement has begun. SEC examination requests regarding the trading blotter have grown dramatically in scale and sophistication. Firms report receiving inquiries requesting more than two-dozen fields of data about each trade. While the SEC's analytics reside in the background, one can only wonder what they will be able to do with that much information. Moreover, as firms scramble to collect the requested information, they are ' presumably ' being introduced to the new role of big data in regulation.

What should legal and compliance professionals do to respond?

1. Make sure they collect data sets that are appropriate to their business. The trading blotter is a good place to start. For some firms, this may involve collecting new data. This could be an operational challenge ' or at a minimum, an unexpected expense. However, unlike certain past regulatory initiatives ' the angry furor surrounding regulators' first venture into requesting and reading e-mails comes to mind ' the response to this initiative has been relatively matter-of-fact. Legal and compliance professionals seem to recognize the benefits of the new tools. Instead of fighting the initiative, they want in on the action.
2. Learn how to use the new tools. Providers are already stepping forward to address this need. For example, Ascendant Compliance Management (ACM) recently hosted a compliance conference dedicated to data, risk analytics and surveillance. When asked why, ACM partner and general counsel Keith Marks responded that the regulators are plainly moving in this direction and compliance must as well. Legal and compliance practitioners, he says, should listen to what the SEC expects as an appropriate standard of data review and make sure they are “right there” with the regulator.
3. Be mindful that big data tools are only that: tools. Marilyn Miles, head of consulting for National Regulatory Services, hopes regulators will remember that a positive signal in the data does not necessarily equal a violation. She also cautions that the power of the big data tools may not be needed in every business model, and that different firms may pursue different data strategies. Nonetheless, she agrees that the attention to data has been positive. Legal and compliance professionals, she suggests, are already spending more time reviewing the data created and used within their firms. At a minimum, she suggests, they should understand what their firms already have.
4. Collecting and crunching data is not enough. Marks states that the key function is to ask: why do results happen? When first entering a big data environment, many firms experience a large number of false positives. It can seem a headache to track them down and resolve them. But in reality, Marks says, the process does more than simply reduce the incidence of false positives. More importantly, it can be a valuable exercise to winnow through the data and learn where a firm's patterns differ from the norm. He suggests that this type of exploratory work gives legal and compliance professionals new perspectives on their own firms and helps get them ready for regulatory inquiries.

Conclusion

As the SEC sets out to “raise the bar” in regard to big data, it appears the regulated community is listening. There remains, however, an area where the SEC's leadership could play an even more helpful role. While the data sets of interest to the SEC quickly enter the public realm, the analytics do not. Greater disclosure of the analytical components of the new tools would be very helpful. Indeed, in a best-case scenario, regulator and regulated community alike will drive the new tools forward with an open conversation. Legal and compliance professionals want to understand the signals produced by their data so they can prevent problems, just as the SEC wants to understand the signals so it can address problems. This is an area in which mutual transparency could be very productive. The SEC can start the conversation by making its analytics public.

From an historical perspective, the present moment bears a striking resemblance to the early onset of electronic tools ' or data-processing tools, as they were first known. Before data processing, compliance was conducted with paper and pencil. The author of this column once had the opportunity to review a 1960s-era compliance tracking system. It consisted of a large index card with a handful of prelabeled lines and boxes. Compliance professionals using the system manually wrote information onto the card, where it resided for manual extraction and use. The lines were few and the boxes relatively tiny. After the development of electronic tools like automatic exception reporting, the index cards were nothing more than an historical curiosity. Today, when we look at the compliance power of big data tools, we get much the same feeling that we did when comparing an index card to a one- or two-variable exception report. The industry standard is moving again. Legal and compliance professionals must strive to keep up.

John. H. Walsh is a partner at Sutherland Asbill & Brennan. He previously served for 23 years at the U.S. Securities and Exchange Commission, where he was instrumental in creating the Office of Compliance Inspections and Examinations. This article is for informational purposes and is not intended to constitute legal advice. The views expressed by the author are the author's alone, and do not necessarily represent the views of Sutherland Asbill & Brennan or its clients.

Big data has taken the world by storm. From its origins as a technical solution for Internet search engines and online retail sales, it has spread across business, science and now government. Big data tools have shown extraordinary power to quickly sort and analyze data, both structured and unstructured. Ultimately, the power of big data resides in its ability to identify signals or patterns in vast data sets.

Historically, regulators have been information rich and resource poor. Information was available to them from disclosure documents, registration statements, visiting firms and multiple other sources. But unfortunately, the regulators often lacked the resources to sort through it all to understand what it meant. Now, big data analytics are changing the game. Recent developments at the examination program of the U.S. Securities and Exchange Commission (SEC) are a case in point.

NEAT

The SEC's examination program has deployed a big data tool called the National Examination Analytical Tool (NEAT), which allows examiners to quickly analyze a firm's trading blotter. Once the trade blotter information has been loaded, NEAT quickly normalizes the data and provides a series of discrete tests. Examination managers report that 50 tests are already embedded in the tool, covering commissions, day trading, short sales within certain time parameters, allocations, profitability, systematically favored accounts and more. Additional tests are currently in development.

Andrew (Drew) Bowden, director of the SEC's examination program, reports that he is excited about the possibilities presented by NEAT. In an interview with this author, he highlighted several of its more important features.

First, NEAT allows examiners to study much larger sample sizes. Not long ago, SEC examiners would review a month, week or even just a day of trading, depending on the size of the firm. Now, with NEAT capability, trading samples routinely look back three or more years. This allows examiners to look for trends and patterns over longer periods of time in larger data sets.

Second, the tool tests the entire trade blotter without the intervention of subjective human factors. In the past, examiners' qualitative judgment played an important role even during quantitative analysis. Typically, examiners would sort the data and use their judgment to identify anomalies or patterns. Skilled and experienced examiners could use this approach to make significant findings ' but the output remained dependent on qualitative human factors. Now, Bowden emphasized, the same quantitative tests are applied to every trade, in the same way, by every examiner.

Third, as the examination program gains experience in NEAT's results, methodological consistency will be enhanced, because results in one examination can be compared to results in all others. Through this process, a library of “compliance signals” can be developed. Moreover, examiners can follow up on the signals to assess their value. As Bowden put it, examiners can go into the field to “test their hypotheses” and see what they have really learned from a signal.

Fourth, because NEAT tests are systematized, best-practice methodologies can be quickly deployed across the program. With a program-wide tool, developed and managed by a professional quantitative staff, new analytical approaches, or enhancements to the old, can be quickly developed, tested and deployed.

NEAT was originally developed for use in investment adviser examinations, and is now being used in other areas of the examination program as well, including with broker-dealers. It is, in Bowden's terms, a revolutionary development for examinations. SEC chair Mary Jo White agrees. She used the same term to describe NEAT ' “revolutionary” ' in testimony before Congress.

Other Big Data Programs

As important as NEAT may be at the SEC, it appears to be only the beginning. The examination program's Quantitative Analytics Unit (QAU), which developed NEAT, has 10 members with advanced degrees in mathematics, computer science, data science and financial engineering. Bowden indicates that other tools are now in development. For example, the Machine Analyzed Risk Scoring (MARS) tool is being developed to bring more quantitative rigor to the process of selecting firms for examination. Bowden described this as a strategic imperative of his program: bringing the same rigor and systematic analysis to targeting as is now deployed in the analysis of trading.

As the SEC's examination program grows increasingly sophisticated in its use of big data, we can see it begin to take on some of the dynamics of other big data practitioners. Bowden indicated that the SEC's exam program is on the hunt for data to feed its analytics. It already subscribes to commercial market data vendors for real-time validation of market prices, merger and acquisition news, and other services. In addition, data warehousing within the SEC has enhanced examiners' ability to tap into information already on file at the agency.

Another frequent benefit of big data is the ability to find new uses for previously untapped data sets. This is also playing a role at the SEC. When asked if he intended to use data from the private fund disclosure document (Form PF) for his risk targeting, Bowden replied, “Hell yes! We're going to wring it out for every valuable piece of information we can.” This is the same approach, he said, that the program is taking with every other potential data source.

Finally, as big data practitioners reach out to new sources of information, the integrity of data always takes on enhanced importance. SEC examiners in the field have reported spending more time on data-integrity issues ' and taking a much harsher view of firms who try to pass off bad data to their regulator.

The initiatives in the SEC's examination program highlight big data's promise for regulation. Not that many years ago, regulators had to worry about collecting information they couldn't use. What moral hazard, it was asked, would be created by data sitting unread deep in a government file? This author is personally familiar with regulatory proposals that were defeated with that kind of thinking. Now, with big data tools, every data set may have a role to play in identifying targets and testing their compliance.

Still, two important questions remain:

1. How will the SEC's examination program measure the success of its new tools? If the only metric of success is to increase enforcement activity, big data could end up looking and feeling like Big Brother. The SEC should give careful consideration to nonpunitive metrics. For example, examiners should be proud to report examinations in which the only outcome was to better refine the analytical tools or reduce a false positive signal.
2. How should legal and compliance practitioners respond to these new developments? As a practical matter, when SEC examiners arrive, armed with NEAT, they will not be impressed with less sophisticated compliance programs. Corporate counsel and compliance professionals must take a new look at their own processes in light of these regulatory developments.

An SEC Examination

Bowden spoke about his expectations for compliance in this area. As a practical matter, he said, based on his own past experience in private compliance, firms do not want to learn something for the first time during an SEC examination. If the SEC is going to use big data tools during examinations, than firms should make sure they have a pretty good idea of what those tools will show before examiners arrive at the door. To do that, firms need to deploy similar tools to those in the regulators' hands. In fact, Bowden said, he believes an important part of his mission is to encourage the private sector to move forward in this area. He hopes to promote compliance by “raising the bar” in regards to big data analytics.

The encouragement has begun. SEC examination requests regarding the trading blotter have grown dramatically in scale and sophistication. Firms report receiving inquiries requesting more than two-dozen fields of data about each trade. While the SEC's analytics reside in the background, one can only wonder what they will be able to do with that much information. Moreover, as firms scramble to collect the requested information, they are ' presumably ' being introduced to the new role of big data in regulation.

What should legal and compliance professionals do to respond?

1. Make sure they collect data sets that are appropriate to their business. The trading blotter is a good place to start. For some firms, this may involve collecting new data. This could be an operational challenge ' or at a minimum, an unexpected expense. However, unlike certain past regulatory initiatives ' the angry furor surrounding regulators' first venture into requesting and reading e-mails comes to mind ' the response to this initiative has been relatively matter-of-fact. Legal and compliance professionals seem to recognize the benefits of the new tools. Instead of fighting the initiative, they want in on the action.
2. Learn how to use the new tools. Providers are already stepping forward to address this need. For example, Ascendant Compliance Management (ACM) recently hosted a compliance conference dedicated to data, risk analytics and surveillance. When asked why, ACM partner and general counsel Keith Marks responded that the regulators are plainly moving in this direction and compliance must as well. Legal and compliance practitioners, he says, should listen to what the SEC expects as an appropriate standard of data review and make sure they are “right there” with the regulator.
3. Be mindful that big data tools are only that: tools. Marilyn Miles, head of consulting for National Regulatory Services, hopes regulators will remember that a positive signal in the data does not necessarily equal a violation. She also cautions that the power of the big data tools may not be needed in every business model, and that different firms may pursue different data strategies. Nonetheless, she agrees that the attention to data has been positive. Legal and compliance professionals, she suggests, are already spending more time reviewing the data created and used within their firms. At a minimum, she suggests, they should understand what their firms already have.
4. Collecting and crunching data is not enough. Marks states that the key function is to ask: why do results happen? When first entering a big data environment, many firms experience a large number of false positives. It can seem a headache to track them down and resolve them. But in reality, Marks says, the process does more than simply reduce the incidence of false positives. More importantly, it can be a valuable exercise to winnow through the data and learn where a firm's patterns differ from the norm. He suggests that this type of exploratory work gives legal and compliance professionals new perspectives on their own firms and helps get them ready for regulatory inquiries.

Conclusion

As the SEC sets out to “raise the bar” in regard to big data, it appears the regulated community is listening. There remains, however, an area where the SEC's leadership could play an even more helpful role. While the data sets of interest to the SEC quickly enter the public realm, the analytics do not. Greater disclosure of the analytical components of the new tools would be very helpful. Indeed, in a best-case scenario, regulator and regulated community alike will drive the new tools forward with an open conversation. Legal and compliance professionals want to understand the signals produced by their data so they can prevent problems, just as the SEC wants to understand the signals so it can address problems. This is an area in which mutual transparency could be very productive. The SEC can start the conversation by making its analytics public.

From an historical perspective, the present moment bears a striking resemblance to the early onset of electronic tools ' or data-processing tools, as they were first known. Before data processing, compliance was conducted with paper and pencil. The author of this column once had the opportunity to review a 1960s-era compliance tracking system. It consisted of a large index card with a handful of prelabeled lines and boxes. Compliance professionals using the system manually wrote information onto the card, where it resided for manual extraction and use. The lines were few and the boxes relatively tiny. After the development of electronic tools like automatic exception reporting, the index cards were nothing more than an historical curiosity. Today, when we look at the compliance power of big data tools, we get much the same feeling that we did when comparing an index card to a one- or two-variable exception report. The industry standard is moving again. Legal and compliance professionals must strive to keep up.

John. H. Walsh is a partner at Sutherland Asbill & Brennan. He previously served for 23 years at the U.S. Securities and Exchange Commission, where he was instrumental in creating the Office of Compliance Inspections and Examinations. This article is for informational purposes and is not intended to constitute legal advice. The views expressed by the author are the author's alone, and do not necessarily represent the views of Sutherland Asbill & Brennan or its clients.