Ensure that Data and Tools are Flexible and Fit the Need

Depending on their goals and profile, colleges have different needs when partnering with a predictive analytics vendor. Those needs will also change over time. Schools need to think through the following set of technical and ethical considerations to ensure that the vendors’ data and tools are flexible enough to meet those needs.

• Determine level of tool integration
• Make sure there is real interoperability
• Determine data needs
• Plan for the future
• Test the interface
• Make sure the tools are accessible
• Set an implementation timeline

It is important to look at the vendor landscape and decide how much integration the college should be looking for in the product. It is tempting to partner with a vendor that has all of the capabilities the college may ever need so that it would not have to go through another procurement process if it needs additional functionality. As a result, the trend in the marketplace is to build more horizontally integrated solutions. But a solution tailored to your current problem often works better. That is what colleges indicated in a recent Tyton Partners’ survey, which found that the majority of schools believe less integrated solutions perform better than solutions with more integration.¹ Colleges need to assess the pros and cons of partnering with a vendor of integrated solutions.

Return to top

Vendors all say they are interoperable with major university data systems, but what that means can vary widely. Is there out-of-the-box data integration or does the vendor need to create a program interface? Creating and maintaining an interface can mean a lot of time and money.² A good product must be able to receive data from many different sources, possibly through a robust set of application programming interfaces (APIs).³ One way to ensure that a vendor will truly integrate with a college’s data systems is to check if that company is compliant with the Learning Tools Interoperability (LTI) and the Caliper Analytics standards set out by IMS Global, a non-profit member collaborative that develops open interoperability standards.⁴ Another possibility is to partner with a vendor that specializes in supporting data integration. These products, like Dxtera, Jitterbit, and AcademyOne, facilitate secure information exchange between various technology tools and the institution’s data and can help support long-term integration.⁵

Return to top

Chances are colleges, particularly institutional research offices, will have to play a large role in cleaning college data so they are ready to put a predictive analytics model in place. Each college is unique in the way it codes and stores its data and college staff will most likely have a hands-on role in helping vendors accurately interpret institutional data. When choosing vendors, colleges should ask for a realistic picture of how much staff time this will take and ask to talk with similar colleges to check these claims.

When a college lays out a plan for using analytics, it should consider the types of data the school community is comfortable using to predict student outcomes. For instance, will the college be comfortable using location or student financial aid data or is that information too sensitive to include in these systems? When searching for a vendor, the college should make sure that it is comfortable with the types of data the system uses. Most systems include data from the student information system (SIS), the customer relationship management (CRM) software, and the learning management system (LMS). Some also integrate data from other early alert or case management systems; location data from swipe cards or wireless internet connections; student membership in clubs and other organizations; tutoring/support systems; attendance records; financial information; and adaptive learning platforms.⁶

Colleges should also ask whether additional types of data increase the predictive power of the algorithms the vendor is using. It is particularly important to acknowledge the tradeoff between how closely the data reflect a student’s current situation and how intrusive it might seem. While historical data like high school GPA and demographic information might seem less sensitive then say location data, that information also might fail to predict a student’s current circumstances and/or lead to profiling while also failing to provide enough information to guide effective intervention.

It is also important for colleges to look at how often the data are updated in the system. The more the data reflect the current actions of students, the better the chance to intervene and put students on the right track. In addition, some data elements may need to be updated more often than others. Colleges should check to see if the vendor they are considering has the capacity to update data on different schedules.

Return to top

Colleges will most likely find multiple vendors that have products with features that align with what the college currently needs. But they should also be looking to the future. While colleges hire vendors based on an initial set of needs, those needs will change over time. Colleges should create a plan for how they expect to use predictive analytics over the next three to five years. Then, they should ask prospective vendors about their future upgrade plans and compare the vendors’ plans to their own to help determine which company might be the best fit. At the same time, colleges should ask about vendors’ flexibility in building out new functionality. Is there an opportunity to build more functionality if there is aggregated demand among colleges? How much demand is enough to trigger building new functionality? A successful partnership will have flexibility built in to adapt to those changing requirements.⁷

Colleges should also check how the vendor system affects their current work processes. The faculty members and administrators on campus who will be the frontline users of the predictive analytics product should be on the purchasing committee to flag where the processes of the college may not line up with the requirements of the software. For instance, if a college does transcript review and degree planning with students before they apply and the predictive analytics tool requires a student ID to work, that tool will not be useful.

Return to top

The software interface can be one of the most difficult things to get right. If the tool is clunky or does not answer frontline staff questions effectively, no one will use it. As a result, it is critical for colleges to ask about the vendors’ processes for testing the interface. Did they test it with real-world practitioners? Vendors should also allow colleges to test the tools with their own faculty and staff. As more and more of us do our jobs on the run, colleges should also ask about access via mobile device. If the system contains sensitive information and it is available on mobile devices, how is it protected in the mobile environment? Lastly, as more of these systems contain learning artifacts, does the system interface with open education resources?

Return to top

Colleges have an ethical and legal responsibility to make sure these systems are accessible for people with disabilities. This is particularly true for testing compatibility with screen readers.⁸ The vendor should be able to articulate exactly how the system supports accessibility requirements for staff who need accommodations under the Americans with Disabilities Act. Ideally, accessibility support should be a focus rather than an afterthought.⁹ And colleges should not take the vendor’s word for it. Instead, they should test the systems with their staff to ensure it is accessible to all. Colleges should also include accessibility in their contracts with vendors.¹⁰

Return to top

Implementation tends to take longer than expected because of data cleaning and integration. It also takes a while to get frontline staff members comfortable with using the tool in their everyday work. Colleges should ask vendors about their typical implementation timeline but know that the process will most likely take longer. If the vendor runs a community of practice, reach out to other clients to get a realistic sense of the effort and time it took to get the tool up and running. Colleges can get a sense of some of the typical implementation timelines in the vendor profiles in the publication Crossing the Finish Line: Vetting Tools that Support Student Success.¹¹

Return to top