Produce Your Own Analytics or Hire a Vendor?

As colleges begin to look at using increasingly sophisticated analytic systems and see the price of partnering with a vendor, they inevitably think about using their existing institutional research (IR) and information technology (IT) resources to do the analysis themselves. But only a small number end up following this path. A recent estimate found that colleges spent about $22 million on in-house technology solutions in 2017 out of a total market of $858 million. Still, only between 1 and 3 percent of colleges have chosen this path.¹ Ivy Tech² and San Jacinto College³ are examples of community colleges that have created their own systems. Arizona State University,⁴ the University of New Mexico,⁵ and the University of Texas system⁶ are examples of public four-year colleges that have created their own in-house solutions.

By their nature, these systems are highly individualized, customized to the needs of the colleges that created them. And even these schools seek some outside help. The University of Texas system, for example, is partnering with Salesforce to create a solution for its schools. San Jacinto College is using SAS Enterprise Miner. Ivy Tech used an open source algorithm. The University System of Georgia is working to code its system in the coding language R.

So how do you know if your college should build a system in-house or partner with a vendor?

We have created a framework to help colleges think through this question. First, officials need to determine what level of experience their institutions have using data. To do that, consider the following questions:

• How integrated are your departments (IR, IT, academic advising, registrar, and financial aid)? The more connected and integrated the key departments are on campus, the more advanced the institution is in using data flexibly. Collaborative and healthy working relationships among these departments make a substantial difference in this work.
• How much experience does your college have using data to make decisions about improving student success? A college with regular meetings and committees charged with examining and making decisions based on data has a greater chance of achieving its aims.
• Does your college have a dedicated budget for the effort? Colleges with more resources to dedicate for this purpose are in a superior position to those that are trying to make it work with existing resources.
• What kinds of work has been done on data analytics so far? Colleges that have started to experiment with more sophisticated analytics have a higher capacity to use data than those just starting out.
• Can you talk to colleges like yours which are successfully using analytics about their experiences? A college that has access to a community of similar schools that have already gone through the process will be in a better position to experiment with analytics.
• Is college leadership aligned to support a culture change around using data? Having leadership on board to support the changes needed to use data will make implementation much simpler.

Once colleges have decided on their capacity for using data, they should work through the following considerations.

One of the main things that makes internal solutions attractive is their relatively low price. Even obtaining a license for an enterprise solution colleges can customize can be much cheaper than partnering with one of the large vendors. According to the Ada Center and Achieving the Dream, the cost of using a vendor can range from $30,000 to $100,000 annually, with a one-time implementation fee sometimes approaching $75,000.⁷ The cost depends on a variety of factors including the number of students in the college, the number of licenses, the number of deployed products, and the sophistication of the data integration.⁸ It is not surprising, then, that colleges would be attracted to doing this with their existing staff, particularly because existing staff will generally need to partner closely with a vendor to make any solution work. However, in-house solutions are not as low-cost as they may seem. Colleges need to take into account staff time, salaries (particularly if they need to hire new staff), and hardware needed to make an internally built solution work. This effort can be costly no matter what solution the college chooses so it is important to take all cost factors into account when deciding whether or not to hire a vendor.⁹

Another consideration for colleges is the capacity of staff and the culture of their IR and IT offices. Some IR offices are more focused on compliance reporting than using data to support student success efforts. And some IT offices concentrate on procurement and tech support rather than using data systems to help administrators serve students better. These offices need to be entrepreneurial and able to collaborate to create an internal system and make it work. They also need to focus on anticipating the next institutional needs and generating answers to those questions in user-friendly ways. In addition, the staff in these departments need to have the necessary expertise and training to be able to generate and interpret models. Finding individuals with this type of expertise can be difficult because statisticians and data scientists are in high demand in the private sector. And this understanding of data cannot just be in the IR and IT offices. Understanding how to use data needs to be a skill set across departments, including student affairs, academic affairs, financial aid, and the registrar.

This demand can lead to another problem: staff turnover. If a college relies on key staff to create a customized solution and those individuals leave, the school may not have anyone who knows how to maintain and update the internal system. To avoid this problem, colleges must ensure that the internal analytics system they are creating is not too customized, that the customizations are documented, and that other staff members are trained on how to maintain it. A sign of how difficult it is to sustain these customized solutions is the number of them that have been spun off or sold, which is done for complicated reasons including financial gain and challenges involved with maintaining a customized system in the face of staff turnover. Sometimes companies acquire promising new models to accelerate new opportunities and sometimes they acquire them to kill new competition. Some examples of spin-offs include Purdue’s Course Signals, which was licensed to Ellucian and the University of Maryland University College analytics tool, which was spun off into HelioCampus.¹⁰

A key question for colleges to consider is what they need the tool to do. If the college wants to evaluate interventions that take discrete, nimble analysis, it can be easier to keep that analysis in-house rather than trying to work through a third party. If the college wants to undertake a more sophisticated analysis with a significant visual component, the college might want to consider partnering with a vendor. It is common for colleges to experience big student success gains when these interventions are deployed that level off over time. More sophisticated algorithms, comparison data, and additional data sources can help boost student success if it hits a plateau.

It might make sense for some colleges to start with simpler, in-house analytics and build the capacity to act on the data from there. Partnering with a vendor is not a silver bullet for those with limited experience using data analytics to make decisions. Instead, colleges must do the hard work of making their institutions ready to support and act on analytic insights. Practicing with their own data analysis can help improve institutional readiness. For instance, one useful finding from analyzing the colleges’ own data is discovering all of the interventions the college already has in place and being able to assess those outcomes. To support this work, the Association for Institutional Research has created a Statement of Aspirational Practice for Institutional Research which sets a roadmap for developing analytics.¹¹ Of course, partnering with a vendor is not all or nothing. Colleges can outsource certain aspects of their predictive analytics and keep others in-house. This “smart-sourcing” strategy may be more complex to implement but may also allow colleges to have more flexible solutions.¹²