Colleges are increasingly using models to predict student behavior and intervene to change that behavior. Because of this, when projects involve partnering with a vendor, it is more important than ever to make the right choice about which vendor. In some ways, partnering with a vendor to use predictive analytics is similar to procuring any other technology product. But the complexity of the algorithms—and the predictions they produce—add another layer to the decision-making process. As we talked to a number of colleges, we heard from some whose vendors had algorithms that showed all of their students were “at risk.” And we heard from others who were told that student GPA was the most significant predictor of student success. Neither of these are useful outcomes or productive partnerships.

Currently, many colleges choose vendors based on word of mouth. Colleges should certainly talk to other schools about their vendor experiences and ask vendors for their customer retention rates. But in this era of big data, colleges have additional ethical considerations.

Ideally, predictive analytics vendors can facilitate the ethical use of data all of the way through the student life cycle. Vendors can help ensure that data are complete and integrated correctly to diminish the chances of misidentifying students. They can be transparent about their algorithms and test them for disparate impact on student populations. They can be flexible with permissions and use reasonable security protocols to help preserve student privacy and security. They can help evaluate interventions to ensure they are not harming certain groups of students. And they can train staff on the correct interpretation of data and on the dangers of implicit bias.

Not all vendors, however, take this approach. Many do the bare minimum with data integration. Some consider their models and algorithms proprietary. Some are not focused on student privacy and security. Others do not support training and evaluation.

This guide will give administrators the tools to ask the right set of questions of predictive analytics vendors and provide a sense of what kind of answers they should expect. It will focus on ensuring that vendors use predictive analytics tools like early alert systems in an ethical manner.

Partnering with a vendor will not solve all problems around data and analytics. Instead, colleges must first determine what they want out of these tools, put the capacity in place to maintain and act on the information these tools provide, and get buy-in from key members of the school community for using these tools. In 2017, we published Predictive Analytics in Higher Education:, a guide for colleges looking to properly implement these tools with ethics in mind.¹ Before colleges think about partnering with a vendor, we recommend reading our guide, as well as exploring other useful resources, such as the paper Putting Data to Work, the book Creating a Data-Informed Culture in Community Colleges, the Data Maturity Framework from the University of Chicago Center for Data Science and Public Policy, and The Ada Center, which is dedicated to helping community colleges implement technology solutions.²

In the end, no vendor is going to do the hard work for your college. Many of the stories we have heard about unhappy partnerships between colleges and vendors come from misunderstanding what vendors and their products can unilaterally change on campus.

As colleges begin to look at using increasingly sophisticated analytic systems and see the price of partnering with a vendor, they inevitably think about using their existing institutional research (IR) and information technology (IT) resources to do the analysis themselves. But only a small number end up following this path. A recent estimate found that colleges spent about $22 million on in-house technology solutions in 2017 out of a total market of $858 million. Still, only between 1 and 3 percent of colleges have chosen this path.³ Ivy Tech⁴ and San Jacinto College⁵ are examples of community colleges that have created their own systems. Arizona State University,⁶ the University of New Mexico,⁷ and the University of Texas system⁸ are examples of public four-year colleges that have created their own in-house solutions.

By their nature, these systems are highly individualized, customized to the needs of the colleges that created them. And even these schools seek some outside help. The University of Texas system, for example, is partnering with Salesforce to create a solution for its schools. San Jacinto College is using SAS Enterprise Miner. Ivy Tech used an open source algorithm. The University System of Georgia is working to code its system in the coding language R.

So how do you know if your college should build a system in-house or partner with a vendor?

We have created a framework to help colleges think through this question. First, officials need to determine what level of experience their institutions have using data. To do that, consider the following questions:

• How integrated are your departments (IR, IT, academic advising, registrar, and financial aid)? The more connected and integrated the key departments are on campus, the more advanced the institution is in using data flexibly. Collaborative and healthy working relationships among these departments make a substantial difference in this work.
• How much experience does your college have using data to make decisions about improving student success? A college with regular meetings and committees charged with examining and making decisions based on data has a greater chance of achieving its aims.
• Does your college have a dedicated budget for the effort? Colleges with more resources to dedicate for this purpose are in a superior position to those that are trying to make it work with existing resources.
• What kinds of work has been done on data analytics so far? Colleges that have started to experiment with more sophisticated analytics have a higher capacity to use data than those just starting out.
• Can you talk to colleges like yours which are successfully using analytics about their experiences? A college that has access to a community of similar schools that have already gone through the process will be in a better position to experiment with analytics.
• Is college leadership aligned to support a culture change around using data? Having leadership on board to support the changes needed to use data will make implementation much simpler.

Once colleges have decided on their capacity for using data, they should work through the following considerations.

One of the main things that makes internal solutions attractive is their relatively low price. Even obtaining a license for an enterprise solution colleges can customize can be much cheaper than partnering with one of the large vendors. According to the Ada Center and Achieving the Dream, the cost of using a vendor can range from $30,000 to $100,000 annually, with a one-time implementation fee sometimes approaching $75,000.⁹ The cost depends on a variety of factors including the number of students in the college, the number of licenses, the number of deployed products, and the sophistication of the data integration.¹⁰ It is not surprising, then, that colleges would be attracted to doing this with their existing staff, particularly because existing staff will generally need to partner closely with a vendor to make any solution work. However, in-house solutions are not as low-cost as they may seem. Colleges need to take into account staff time, salaries (particularly if they need to hire new staff), and hardware needed to make an internally built solution work. This effort can be costly no matter what solution the college chooses so it is important to take all cost factors into account when deciding whether or not to hire a vendor.¹¹

Another consideration for colleges is the capacity of staff and the culture of their IR and IT offices. Some IR offices are more focused on compliance reporting than using data to support student success efforts. And some IT offices concentrate on procurement and tech support rather than using data systems to help administrators serve students better. These offices need to be entrepreneurial and able to collaborate to create an internal system and make it work. They also need to focus on anticipating the next institutional needs and generating answers to those questions in user-friendly ways. In addition, the staff in these departments need to have the necessary expertise and training to be able to generate and interpret models. Finding individuals with this type of expertise can be difficult because statisticians and data scientists are in high demand in the private sector. And this understanding of data cannot just be in the IR and IT offices. Understanding how to use data needs to be a skill set across departments, including student affairs, academic affairs, financial aid, and the registrar.

This demand can lead to another problem: staff turnover. If a college relies on key staff to create a customized solution and those individuals leave, the school may not have anyone who knows how to maintain and update the internal system. To avoid this problem, colleges must ensure that the internal analytics system they are creating is not too customized, that the customizations are documented, and that other staff members are trained on how to maintain it. A sign of how difficult it is to sustain these customized solutions is the number of them that have been spun off or sold, which is done for complicated reasons including financial gain and challenges involved with maintaining a customized system in the face of staff turnover. Sometimes companies acquire promising new models to accelerate new opportunities and sometimes they acquire them to kill new competition. Some examples of spin-offs include Purdue’s Course Signals, which was licensed to Ellucian and the University of Maryland University College analytics tool, which was spun off into HelioCampus.¹²

A key question for colleges to consider is what they need the tool to do. If the college wants to evaluate interventions that take discrete, nimble analysis, it can be easier to keep that analysis in-house rather than trying to work through a third party. If the college wants to undertake a more sophisticated analysis with a significant visual component, the college might want to consider partnering with a vendor. It is common for colleges to experience big student success gains when these interventions are deployed that level off over time. More sophisticated algorithms, comparison data, and additional data sources can help boost student success if it hits a plateau.

It might make sense for some colleges to start with simpler, in-house analytics and build the capacity to act on the data from there. Partnering with a vendor is not a silver bullet for those with limited experience using data analytics to make decisions. Instead, colleges must do the hard work of making their institutions ready to support and act on analytic insights. Practicing with their own data analysis can help improve institutional readiness. For instance, one useful finding from analyzing the colleges’ own data is discovering all of the interventions the college already has in place and being able to assess those outcomes. To support this work, the Association for Institutional Research has created a Statement of Aspirational Practice for Institutional Research which sets a roadmap for developing analytics.¹³ Of course, partnering with a vendor is not all or nothing. Colleges can outsource certain aspects of their predictive analytics and keep others in-house. This “smart-sourcing” strategy may be more complex to implement but may also allow colleges to have more flexible solutions.¹⁴

Once a college has decided on the functionality it needs and has done the work to ensure it is ready, it may decide to partner with a vendor. To ensure the predictive analytics solution is implemented in a sustainable, ethical way, colleges should consider the following when selecting a vendor:

• Ensure that data and tools are flexible and fit the need
• Ensure transparent use of data
• Issues with predictive analytics vendor contracts
• Ensure privacy and security
• Supporting research and evaluation efforts
• Supporting staff professional development and implementation
• Return to introduction

Depending on their goals and profile, colleges have different needs when partnering with a predictive analytics vendor. Those needs will also change over time. Schools need to think through the following set of technical and ethical considerations to ensure that the vendors’ data and tools are flexible enough to meet those needs.

• Determine level of tool integration
• Make sure there is real interoperability
• Determine data needs
• Plan for the future
• Test the interface
• Make sure the tools are accessible
• Set an implementation timeline

It is important to look at the vendor landscape and decide how much integration the college should be looking for in the product. It is tempting to partner with a vendor that has all of the capabilities the college may ever need so that it would not have to go through another procurement process if it needs additional functionality. As a result, the trend in the marketplace is to build more horizontally integrated solutions. But a solution tailored to your current problem often works better. That is what colleges indicated in a recent Tyton Partners’ survey, which found that the majority of schools believe less integrated solutions perform better than solutions with more integration.¹⁵ Colleges need to assess the pros and cons of partnering with a vendor of integrated solutions.

Return to top

Vendors all say they are interoperable with major university data systems, but what that means can vary widely. Is there out-of-the-box data integration or does the vendor need to create a program interface? Creating and maintaining an interface can mean a lot of time and money.¹⁶ A good product must be able to receive data from many different sources, possibly through a robust set of application programming interfaces (APIs).¹⁷ One way to ensure that a vendor will truly integrate with a college’s data systems is to check if that company is compliant with the Learning Tools Interoperability (LTI) and the Caliper Analytics standards set out by IMS Global, a non-profit member collaborative that develops open interoperability standards.¹⁸ Another possibility is to partner with a vendor that specializes in supporting data integration. These products, like Dxtera, Jitterbit, and AcademyOne, facilitate secure information exchange between various technology tools and the institution’s data and can help support long-term integration.¹⁹

Return to top

Chances are colleges, particularly institutional research offices, will have to play a large role in cleaning college data so they are ready to put a predictive analytics model in place. Each college is unique in the way it codes and stores its data and college staff will most likely have a hands-on role in helping vendors accurately interpret institutional data. When choosing vendors, colleges should ask for a realistic picture of how much staff time this will take and ask to talk with similar colleges to check these claims.

When a college lays out a plan for using analytics, it should consider the types of data the school community is comfortable using to predict student outcomes. For instance, will the college be comfortable using location or student financial aid data or is that information too sensitive to include in these systems? When searching for a vendor, the college should make sure that it is comfortable with the types of data the system uses. Most systems include data from the student information system (SIS), the customer relationship management (CRM) software, and the learning management system (LMS). Some also integrate data from other early alert or case management systems; location data from swipe cards or wireless internet connections; student membership in clubs and other organizations; tutoring/support systems; attendance records; financial information; and adaptive learning platforms.²⁰

Colleges should also ask whether additional types of data increase the predictive power of the algorithms the vendor is using. It is particularly important to acknowledge the tradeoff between how closely the data reflect a student’s current situation and how intrusive it might seem. While historical data like high school GPA and demographic information might seem less sensitive then say location data, that information also might fail to predict a student’s current circumstances and/or lead to profiling while also failing to provide enough information to guide effective intervention.

It is also important for colleges to look at how often the data are updated in the system. The more the data reflect the current actions of students, the better the chance to intervene and put students on the right track. In addition, some data elements may need to be updated more often than others. Colleges should check to see if the vendor they are considering has the capacity to update data on different schedules.

Return to top

Colleges will most likely find multiple vendors that have products with features that align with what the college currently needs. But they should also be looking to the future. While colleges hire vendors based on an initial set of needs, those needs will change over time. Colleges should create a plan for how they expect to use predictive analytics over the next three to five years. Then, they should ask prospective vendors about their future upgrade plans and compare the vendors’ plans to their own to help determine which company might be the best fit. At the same time, colleges should ask about vendors’ flexibility in building out new functionality. Is there an opportunity to build more functionality if there is aggregated demand among colleges? How much demand is enough to trigger building new functionality? A successful partnership will have flexibility built in to adapt to those changing requirements.²¹

Colleges should also check how the vendor system affects their current work processes. The faculty members and administrators on campus who will be the frontline users of the predictive analytics product should be on the purchasing committee to flag where the processes of the college may not line up with the requirements of the software. For instance, if a college does transcript review and degree planning with students before they apply and the predictive analytics tool requires a student ID to work, that tool will not be useful.

Return to top

The software interface can be one of the most difficult things to get right. If the tool is clunky or does not answer frontline staff questions effectively, no one will use it. As a result, it is critical for colleges to ask about the vendors’ processes for testing the interface. Did they test it with real-world practitioners? Vendors should also allow colleges to test the tools with their own faculty and staff. As more and more of us do our jobs on the run, colleges should also ask about access via mobile device. If the system contains sensitive information and it is available on mobile devices, how is it protected in the mobile environment? Lastly, as more of these systems contain learning artifacts, does the system interface with open education resources?

Return to top

Colleges have an ethical and legal responsibility to make sure these systems are accessible for people with disabilities. This is particularly true for testing compatibility with screen readers.²² The vendor should be able to articulate exactly how the system supports accessibility requirements for staff who need accommodations under the Americans with Disabilities Act. Ideally, accessibility support should be a focus rather than an afterthought.²³ And colleges should not take the vendor’s word for it. Instead, they should test the systems with their staff to ensure it is accessible to all. Colleges should also include accessibility in their contracts with vendors.²⁴

Return to top

Implementation tends to take longer than expected because of data cleaning and integration. It also takes a while to get frontline staff members comfortable with using the tool in their everyday work. Colleges should ask vendors about their typical implementation timeline but know that the process will most likely take longer. If the vendor runs a community of practice, reach out to other clients to get a realistic sense of the effort and time it took to get the tool up and running. Colleges can get a sense of some of the typical implementation timelines in the vendor profiles in the publication Crossing the Finish Line: Vetting Tools that Support Student Success.²⁵

Return to top

One of the biggest frustrations for colleges in choosing a predictive analytics vendor is trying to peer into the companies’ algorithms, which are all too often a black box. Administrators want to know how the system comes up with its predictions but do not know which answers are reasonable to expect. Confusion over what vendors should be able to explain about the algorithm is particularly true for administrators who do not come from a technical background when they talk to salespeople who also do not entirely understand the system. The information asymmetry between the technical staff at the vendor and college can lead to misunderstandings and bad decisions. Colleges have an ethical obligation to know a certain amount about how the algorithms they are applying to their students work to ensure some accountability. Here are the things colleges should insist on knowing about the vendors’ algorithms:

• Ask about data used in the prediction
• Ask about training data
• Ask how effective the model is
• Ask vendors to test their algorithms
• Ask for a disparate impact analysis
• Ask for the factors that contribute to the prediction
• Ask how often the algorithm is refined

Ask the vendor to provide a list of all the data elements it uses in the predictions. With this information, the college can decide the sensitivity of the data elements and document where that data is stored. And the documentation of data elements also provides an important baseline for what the predictions are based on. Vendors should commit to providing the school with an updated list of variables they use and alert the institution if they change.

Return to Top

Colleges should ask vendors if they use data from other colleges to inform their models. Some vendors use data from across multiple schools to make their models more robust. But it is important to note that one way bias creeps into algorithms is through non-representative training data. Ask vendors to document the diversity of the types of data, the colleges, and the students that helped to train their models. Colleges should also ask how vendors plan to customize the algorithm to their specific school. Many vendors create the model with only the college’s data. In that case, administrators should ask how far back the training data goes. It is important to balance having the data be as recent as possible to reflect the current college population with ensuring there is a valid sample size to train the algorithm. College administrators should also ensure that the makeup of the college population has not changed significantly over that time.

Return to Top

Colleges should know that overall accuracy rates can be misleading and easily manipulated. Instead, colleges should ask vendors for an AUC (area under the curve)/C statistic. A C statistic indicates how accurate the model is. For most models, colleges should be looking for a C statistic of .75 or greater.²⁶ But watch out. If the C statistic is .99 or higher, there is most likely a problem with the model like leakage (where the data used to train the algorithm includes the data you are trying to predict) or overfitting (where the model is so tuned to the sample data that it fails to generalize to the larger population). Colleges can also ask for a Brier Score.²⁷ A Brier Score is a way to measure the accuracy of a probability forecast, like a weather forecaster saying there is an 80 percent chance of rain. On this scale a lower score is better. Anything under .22 is a solid model. Administrators should also bring colleagues with technical experience into these conversations. Vendors should also commit to presenting on these measures and the overall results of the model at regular intervals, such as every time they rebuild the model.

Return to Top

Some vendors will agree to conduct a pilot to test their algorithm with the college’s data. This can help indicate if the system is discriminatory by overidentifying as “at risk” certain groups of students on campus or if it identifies all of a college’s students as “at risk.” A pilot project can also help highlight where the integration of the institution’s data may need to be improved. Before signing a long-term contract with vendors, college administrators should ask if they are willing to conduct a pilot. Vendors may not be willing to conduct a pilot because they may need to do all the work with the data up front to show what they are capable of. In that case, administrators should ask for a de-identified example of analysis they did for a comparable school.

Return to Top

Vendors need to agree that they will conduct a disparate impact analysis of their tool output after getting the contract. If a vendor does over identify a certain group as “at risk”, that may be because we live in an imperfect world where certain groups are actually more at risk of failing. With the knowledge that a group of students is being overidentified, the college should come up with a plan to focus support on that population. The vendor also needs to ensure that its algorithm is useful in the college’s particular context. For example, is the algorithm identifying every student as “at risk” and if so, how can the vendor ensure this will be fixed?

Return to Top

It can be a tough sell to convince staff members who have worked with students for years to trust the predictions of an algorithm. This is particularly true if there is no transparency into why it is making its prediction. Having vendors list the most weighted factors that went into the prediction for a particular student on the dashboard or interface can bring transparency that helps users act on the prediction. However, it is also important to note that these factors are not causal. For instance, if one reason for the prediction is that the student did not enroll in very many credits that is not necessarily the reason the student is at risk of dropping out. It may just be an indication of something else going on in the student’s life.

Return to Top

As we noted in Predictive Analytics in Higher Education: Five Guiding Practices for Ethical Use, what colleges learn from their use of predictive analytics should be used to refine the algorithm. There should be a cycle of improvement. This can be difficult if the vendor does not support this type of systematic refinement. Models should generally be rebuilt at least once a year and as often as once a term. More frequently than once a term should not be necessary if it is a solid model and more than a year risks degradation of the model.

Return to Top

As with any vendor, colleges need to be sure that a contract with a predictive analytics company protects their students’ privacy and maintains the college’s ownership of that data. This is particularly critical when working with vendors that use cloud-based computing. All vendors must, at a minimum, meet the legal requirements of FERPA but there are other things to consider above and beyond that legal standard.²⁸ Although colleges’ legal and IT departments probably know what contracts with vendors should look like, here are a few things administrators should keep in mind. This list is not exhaustive but it highlights important components to consider.

• Ownership of the cleaned source data, the repurposed data, and the byproduct data
• Switching vendors
• Vendor bankruptcy, closure, or acquisition
• Vendor outsourcing
• Disaster mitigation, recovery, and breach plan
• Service level agreements
• Security

In some cases, the vendor will not agree to return the cleaned source data to the college, instead destroying it at the end of the contract. The contract should clearly lay out who owns what data and under what terms.²⁹ Of course, it should also prohibit the vendor or any partner of the vendor from selling student data.

Return to Top

To prevent vendor lock in, colleges need to have a plan for how they will switch vendors, if they need to, with the minimum amount of pain. To facilitate any transition, the contract should spell out the college’s right to access the data and lay out the process by which the vendor will return the data to the college at the end of the contract. The agreement should include the time frame for the data’s return as well as the format in which the data should be returned. The contract should also obligate the vendor to destroy and verify the destruction of any personally identifiable information when the contract ends.³⁰ Colleges should consider including transition assistance in the contract to ensure a smooth transition to a new solution.³¹ The contract also needs to address the end of the relationship, laying out timelines and each parties’ obligations beyond the ones listed above.³²

Return to Top

In the relatively new space of predictive analytics, there are bound to be acquisitions and closures. The contract needs to address that possibility. Colleges can find model language in the mergers and acquisitions section in this article: If It's in the Cloud, Get It on Paper.³³

Return to Top

Many vendors work with subcontractors or other partners. The contract should require them to disclose those partners and their roles in providing services. It should also clarify that the vendor remains responsible for executing all aspects of the contract with the college.³⁴

Return to Top

The contract should also outline how the contractor would respond to a disaster and outline a recovery plan, including how the vendor would respond if data are lost or corrupted. The section should detail how the vendor would notify the college, how it would correct the problem, and how it would continue to provide service. The contract should also outline the vendor’s plan of action if there is a data breach. And it should lay out who is responsible for what action and where the liability sits for the breach.

Return to Top

Service level agreements (SLAs) lay out the acceptable levels of service for each element of the service provided.³⁵ For example, if a product has an unacceptable outage, the penalty would be addressed in this section. The SLAs should also outline what the vendor will do if the expected level of service is not met.³⁶ Examples of SLAs for the University of California system can be found in this University of California Cloud Computing Service Agreement Template (starting on page 11).³⁷

Return to Top

The contract should clarify the level of security expected by the college. See the next section for more detail.

Return to Top

Ensuring that privacy and information security protections are in place is fundamental to the ethical use of data and thus to choosing a vendor. Most CIOs and IT departments have a good handle on what standards vendors should meet to merit access to student data. In fact, in the Higher Education Cloud Vendor Assessment Tool, there are over 300 information security and privacy questions for vendors to answer about their products.³⁸ However, there are a few things for non-technical decision makers to keep in mind. Vendors should be able to easily answer questions about the administrative, technological, and physical security and privacy safeguards they employ. If they struggle to answer these questions, that could be a signal to dig deeper into their security and privacy practices.

• Administrative safeguards
• Technological safeguards
• Physical safeguards

Administrative safeguards are the management statements that an organization puts into place to protect information security and privacy. Colleges should ask vendors about who can view the institution’s data in the vendor’s product and stored in its IT systems and how the companies control that access. As human error is one of the biggest sources of security breaches, it is important to check that vendors are training all of their staff, including technical staff, on security procedures and ethics. Colleges should ask if the vendor keeps a log of who has access to records so that an audit can be conducted to ensure that employees are not inappropriately viewing records to satisfy their curiosity about a friend or family member. Keeping these logs shows how the vendor views verifying how its staff is handling data and allows for forensic analysis if something goes wrong. Vendors should also meet all appropriate data standards like those from the National Institute of Standards and Technology. The Higher Education Cloud Vendor Assessment Tool shows how all of these standards apply to security practices.

Return to Top

Technological safeguards are the controls implemented in a college’s or vendor’s IT systems that protect those systems and the data located in them. There are a number of good technological practices for ensuring student data security, including industry-standard encryption and secure data connections. Ask if the data are encrypted using an industry-standard encryption both when stored at the vendor and when transferred between the institution and the vendor. Ask whether the vendor has passed an industry-standard security scan and if so, which one. Also ask if the vendor’s employees have remote access to the data and check if the tool can mask certain sensitive data from certain types of users, which allows colleges to present data only to staff members with a compelling need to know.

Return to Top

Physical safeguards are the controls put into place to keep unauthorized individuals out of controlled areas (e.g., data centers) or from accessing IT systems or data. These include controls designed to protect physical media like laptops, servers, storage media, and mobile devices. For example, Washington State University found that a hard drive containing personal information on over 1 million students had been stolen from a locked storage container in spring 2017.³⁹ Incidents like this show how securing the location of your data is an important consideration. Ask vendors if data will be stored in a physically secured location. It is also worth asking if employees are allowed to take data home or on travel; both increase chances of theft or carelessness that could lead to a data breach.

Return to Top

To use predictive analytics tools well, colleges will need to continuously improve their intervention efforts. At the same time, the tools themselves need to be externally validated through research.

• Evaluate the product’s research base
• Assist with intervention design and evaluation

Many vendors say that their products are evidence-based. But it is important for administrators to look critically at that research and see if it is independent and high quality. For instance, has the research been published in a peer reviewed journal? Organizations like Teachers College at Columbia University partner with vendors to increase and strengthen their evidence base.{{41}} There are also a series of tools, mostly designed for the K–12 environment, which could help colleges evaluate the evidence (see Box TK).

Return to Top

Once colleges have access to a predictive analytics tool, what should they do to change the trajectory for students? It can be helpful for vendors to support the design and evaluation of a college’s efforts intervene with students. Many vendors have experience with messages and interventions that worked in other college contexts. And they can also help evaluate how interventions perform by helping create well-matched control groups and processing the data. Since evaluation is an important part of using predictive analytics well, colleges should ask vendors how they can support these efforts.

Return to Top

Changing the way your faculty and staff do their jobs to incorporate a new tool can be a heavy lift. It is important to assess how the vendor will help overcome that hurdle by supporting implementation and providing technical support.

• Implementation consulting
• Technical support
• Communities of practice
• User training

Some vendors offer implementation consulting as part of the solution package. Colleges should consider whether they need this type of assistance given their experience level with data-led-decision-making because including it will increase the price of the product. Some vendors even assign the college a dedicated consultant to help them implement the tool. If that is the case, colleges should ask about the consultant’s experience with schools in their sector. A consultant that has worked with only small private liberal arts colleges may not be helpful to a large public flagship, for example.

Return to Top

It may seem like a small detail but establishing who is responsible for which aspects of technical support is crucial for a successful rollout. Colleges need to know what technical support the vendor provides and who will be responsible for day-to-day troubleshooting. For example, when a high level administrator loses her password, who is responsible for quickly getting her access? If users do not have access to support for their questions, they will not use the system. It is also important to establish how the vendor will help after the launch with adjustments to the system.

Return to Top

Many vendors bring their active users together, forming a community of practice. These communities are cultivated online and at annual conferences or regional meetings. If a college hires a vendor that has such a community, the school should explore it and see if it will be useful for the frontline users and others at the institution. These communities can provide helpful insight into what it is actually like to implement the tool and use it daily. While participants in a community of practice are self-selecting, they can still give colleges a sense of what using the vendor’s products is actually like.

Return to Top

Well-designed training is key to helping faculty and staff adopt a new tool. Most vendors will train a group of people on the campus and have those people train others. This model scales more quickly and can get everyone on board in an efficient way. If the vendor has agreed to participate in a pilot project, a core group of people with knowledge and buy in has a head start on training. That initial training should also include staff members from the school’s institutional research office, so they can get a better sense of how the tool works and how it will fit into the college’s information ecosystem. Training and assistance should also be embedded in the system so users can get help as they face challenges with using the new software. Ideally, the training should go beyond just the technical and should touch on how these tools can change the way administrators do their jobs and also show how to interpret data appropriately, with implicit bias in mind.

Return to Top

Colleges are continuing to take advantage of the increases in student success resulting from both home-grown and purchased predictive analytics systems. Our framework should help guide them through the complex choices they face as they try to decide which type of system to implement. Colleges need to consider the cost, their people, the ability to act on data, and the level of analysis needed when considering building a system in-house. If partnering with a vendor, colleges need to consider the flexibility of the tool, transparency, contract terms, privacy and security, research and evaluation, and professional development. Each of these considerations is important to creating a functional vendor/college relationship.

Laurie Heacock, vice president of data, technology and analytics, Achieving the Dream

September 27, 2017

Fiona Hollands, associate director and senior researcher, Center for Benefit-Cost Studies of Education, Columbia Teachers College

October 2, 2017

James Wiley, principal analyst, Eduventures

October 3, 2017

Ellen Wagner, former vice president of research, Hobsons

October 18, 2017

Scott James, data scientist, Hobsons

October 18, 2017

Lige Hensley, chief technology officer, Ivy Tech

October 18, 2017

Karen Vignare, executive director, Personalized Learning Consortium, APLU

October 20, 2017

James Cousins, senior statistical analyst, Rapid Insight

October 20, 2017

Scot Henley, vice president of Sales and Marketing, Rapid Insight

October 20, 2017

Amelia Parnell, vice president for Research and Policy, NASPA

October 23, 2017

Alexis Wesaw, director of data analytics, NASPA

October 23, 2017

Christine Keller, executive director, Association of Institutional Research

October 30, 2017

Darlena Jones, director of assessment & research, Association of Institutional Research

October 30, 2017

Dror Ben Naim, founder and CEO, Smart Sparrow

October 30, 2017

Laurie Fladd, associate dean of science and mathematics, Trident Technical College

October 31, 2017

Celeste Schwartz, vice president for information technology and college services, Montgomery County Community College

November 6, 2017

Sarah Zauner, founder, The Ada Center

November 8, 2017

Rahim Rajan, senior program officer, Gates Foundation

November 15, 2017

Vicki T. Tambellini, president and CEO, The Tambellini Group

November 20, 2017

Ed Venit, managing director, EAB

November 27, 2017

Jason Elwood, president, Elwoodway Consulting

December 7, 2017

Ellen P. Goodman, professor, Rutgers University Law School

December 8, 2017

Karen Ferguson, vice-provost, CSU Global

January 4, 2018

Dale Allen, president and co-founder, DXtera

January 9, 2018

Amanda Toohey, Director of Communications, Civitas Learning

January 9, 2018

Brendan Desetti, senior manager, Government & Stakeholder Relations, D2L

January 17, 2018

Shady Shehata, head of AI Platform & Data Science, D2L

January 17, 2018

Martin Kurzweil, director of Educational Transformation, Ithaka S+R

January 19, 2018

John Whitmer, learning analytics and research director, Blackboard

January 23, 2018

Derrick Tillman-Kelly, UIA fellow, Ohio State University

January 29, 2018

Timothy Harfield, senior product marketing manager for Analytics, Blackboard

February 8, 2018

Tristan Denley, executive vice chancellor for academic affairs and chief academic officer, University System of Georgia

March 14, 2018

Michelle Callaway, lead research programmer analyst, San Jacinto College

March 29, 2018

George Gonzalez, director of Institutional Research & Effectiveness, San Jacinto College

March 29, 2018

Amanda Vance, director of the Education Privacy Project and a Policy Counsel, Future of Privacy Forum

April 18, 2018

Ross Schulman, senior counsel and senior policy technologist, Open Technology Institute, New America

April 18, 2018

Nat Meysenburg, technologist, Open Technology Institute, New America

April 20, 2018

Glenda Morgan, research director, Gartner Research

May 1, 2018

Joanna Grama, director of the Cybersecurity Initiative, EDUCAUSE

May 9, 2018