The Industry Relevance of an IT Transition Programme

There is a shortage of qualified people in the IT industry in the world, including New Zealand.¹¹1https://skillshortages.immigration.govt.nz To address this shortage, transition programmes have been created that help people change to careers in IT. At the University of Auckland the PGCertInfoTech is one such. The software industry always strives to recruit quality software developers. To provide a useful transition programme, we must understand what these developers need in order to fulfil industry’s expectations. Moreover, the IT industry undergoes continuous change, meaning a programme that was acceptable when it was first created may no longer be fit-for-purpose, that is, it no longer provides the necessary training to allow a change in career to what the industry needs today. Consequently we need to regularly review what the industry needs are in order to evaluate our programme. In this paper we present the results of a survey we conducted whose goal was to help us evaluate and, if necessary, update PGCertInfoTech.

The IT industry is very broad, and so no one programme can possibly cover it all. Thus it was never the intent that the PGCertInfoTech could cover everything that someone working in this industry would need. Instead the goal is to provide a core set of topics that potential employers of our graduates would find useful. This means that our primary goal was to evaluate how useful the topics we teach actually are for our graduates. To determine how useful topics are, rather than ask general questions about perceived usefulness, we instead asked specific questions about actual use. We surveyed only graduates from the programme, to ensure that our survey participants were familiar with the topics we asked about.

The PGCertInfoTech programme provides opportunities for people coming from a non-ICT background to gain essential software development skills. The programme is for students who have a bachelor’s degree in a non-ICT area and who want to upskill or pursue a career in IT.

The PGCertInfoTech programme consists of two courses, COMPSCI 718 and COMPSCI 719. COMPSCI 718, Programming for industry, focuses on the fundamentals of object-oriented programming using Java. COMPSCI 719, Programming with web development, focuses on developing web applications using various front-end and back-end web technologies. These are intensive courses covering a wide range of topics. We are reviewing the curriculum and our goal is to determine whether the topics in these courses are the right topics we should be teaching now. There are other topics that we might also include, perhaps by reducing the amount of content we currently cover of existing topics. To help us make this decision, we surveyed graduates of the PGCertInfoTech programme to determine which, and to what degree, of the topics in the programme our graduates have encountered in their jobs.

In this paper we describe our survey and present our findings from it. Our general conclusions are that the PGCertInfoTech is still acceptable as is, however there are some changes that we could make to improve it, such as changing the amount of material for some topics (either increasing or decreasing), and adding some new topics. We aim to answer the following research questions:

1. (1)

   Does the existing programme provide value to its graduates?

2. (2)

   What changes do we propose to our programme based on the needs of industry?

The rest of the paper is organised as follows. In the next section, we present related work in Computer Science (CS) and Software Engineering (SE) education. In Section 3 we summarise the PGCertInfoTech. We then present our research methodology in Section 4 and results and analysis in Section 5. We discuss results in Section 6, and then conclude with our findings.

The existing research provides insight into the exploration of the relevance and fitness of ICT programmes, along with their underlying courses, to the industry. Most studies report experiences with designing courses that encompass industry-relevant activities to prepare the students for their various industrial roles (Lundqvist et al., 2018; Tu et al., 2018; Tuzun et al., 2018). These studies analyze students’ feedback to understand the importance of the coursework after delivering the course. However, the decision on the design of such courses and the selection of the course contents involved might be better served by receiving inputs from industry professionals. Hence, the other smaller set of more recent studies follow this approach.

The PGCertInfoTech, introduced in 2015, is an intensive and fast-paced programme with the goal of providing industry-focused education to non-ICT students. The programme also provides an opportunity for people to gain a formal qualification in IT. The programme is equivalent to one semester (12 weeks) of full-time study, where students study 40 hours each week. Each lecture is 1–2 hours long. Students must study COMPSCI 718 and COMPSCI 719 to complete the programme. Class starts at 9am with lectures to learn new concepts and finishes at 4pm with a supervised lab to work on practical exercises. The programme also offers part-time study option where students study one course each semester over two semesters.

COMPSCI 718 aims to develop problem-solving skills in the context of software development. The course focuses on object-oriented programming using Java as the primary programming language. The course covers introductory and advanced programming concepts that are typically taught in computer science or software engineering first-year and second-year programming courses. The topics in the course include programming fundamentals, inheritance and polymorphism, exception handling, collections, recursion, concurrency, design patterns, testing and refactoring.

COMPSCI 719 aims to expose students to modern web development technologies. The course covers front-end and back-end web development. The front-end web topics in the course include HTML, CSS, and JavaScript. The back-end web topics include Java Servlets, cookies and sessions, introduction to SQL, and introduction to web security. The course also covers source and version control using Git, web deployment, and remote database servers. The part-time variation of COMPSCI 719 uses NodeJS instead of Java Servlets for the back-end development.

Table 1 summarises the topics covered and the lecture hours spent in each course.

Our general goal is to understand the relevance of PGCertInfoTech to industry, that is, those who take our programme find it useful in their new careers. To do so we need to determine what was of value to our graduates. This requires eliciting relevant information from our graduates. The most efficient means to do this is a survey, and so that is what we chose to do.

When conducting surveys, two important decisions are: what do we ask in the survey, and who do we survey. Given our goals, it seemed most appropriate to survey our graduates, rather than the industry in general. As noted earlier the IT industry is very broad, and the PGCertInfoTech was not intended to cover it all. So including participants from industry who were not graduates ran the risk receiving responses not relevant to our programme. Had our goal been to expand the PGCertInfoTech this would have been the better option. There would also have been the issue of getting a representative sample as contacting relevant participants would not be easy. As we had contacts for all of our graduates this seemed the best population to survey. We could be sure that this group had been exposed to the topics we were interested in, and provide insights to topics that might be the most relevant to the industry.

Our goal was to determine whether or not the topics taught in the programme were useful. Accordingly it is appropriate that a significant number of the questions we ask relate to the courses as they have been taught so far.

To develop the survey, we initially considered the topics which are taught, to some degree, in PGCertInfoTech. However, as technology changes, we also considered some topics not currently taught that are currently popular in industry²²2https://insights.stackoverflow.com/survey/2020(Garousi et al., 2019a). For large topics, such as Object-Oriented Programming, appropriate subtopics were identified (e.g. use of inheritance, polymorphism). For Object-Oriented Programming, we also asked questions to get a sense of the degree to which they used relevant concepts (see Section 5.9). The list of topics covered in our survey can be seen in Table 2.

How we asked our questions was important. We could have asked participants’ perceptions of whether or not a topic is important, but this would have introduced a degree of subjectivity in participants’ responses. Instead our goal is to ascertain the extent to which our participants use that topic in their day-to-day employment. Doing so meant that participants’ responses were based on concrete experience rather than opinion.

To this end, the majority of survey questions are of the form “In your employment, have you used [topic / subtopic] within [timeframe]?” or “Have you used [topic / subtopic] in your employment?”. Where further categorization or clarification is required, follow-up questions were asked. For example, after determining that a participant used version control, a follow-up question would be asked to elicit the specific version control tool used (e.g. git, svn). Open-ended free-text-entry questions were also added for each topic of the form “Please provide any further comments on your use of [topic / subtopic]”.

We also gather demographic data to capture the experience our participants had both since completing the programme and any previous experience they had. This was necessary to interpret the responses. The survey contains 52 questions in total.

Prior to releasing the survey, a small pilot was conducted where two participants in the target demographic filled out the survey in addition to providing feedback on the survey’s length and content. The feedback indicated that no changes needed to be made.

We have received ethics approval from the University to conduct the survey in January 2020. We distributed the survey via an alumni mailing list maintained by the University of Auckland. There were 148 graduates at the time of the distribution of the survey. The distribution was performed by the institution administration. The authors were not involved in this process.

As discussed in Section 1, we are looking for data to help us make decisions about possible changes to the courses. In this section we present just the raw data together with points of interest. We will discuss how we might use this data in making decisions in Section 6.

The overall results are shown in Table 3. The QID labels indicate the topic and will be used in the text below. Due to space constraints we report just the number of participants who answered positively to the questions we asked. While 48 started our survey, only 27 completed it. Our results are presented for those 27.

The results suggest that by and large the topics we already have are fit for purpose (RQ1). The results suggest no changes to databases, frameworks, GUI, object-oriented programming, recursion, refactoring, security, and version control. Our results indicate that exposure to these topics are important in industry.

The results give us confidence to teach object-oriented programming in COMPSCI 718 as most of our participants have reported working with code written in object-oriented languages. We noticed only few reported using polymorphism which seemed to be inconsistent with the number of participants reported working with object-oriented languages. Given many participants have not much experience, we suspect that they probably have been using polymorphism at work without knowing, suggesting we may need to discuss this more in the course.

For RQ2, the topics we should consider adjusting are concurrency, design patterns, and project management. The results suggest the use of asynchronous concurrency is more common than use of explicit threads. Currently, COMPSCI 718 covers concurrency with the use of explicit threads and SwingWorker. Based on the results, we think it would be appropriate to reduce the number of hours on explicit threads in COMPSCI 718, and introduce asynchronous concurrency in COMPSCI 719 as the topic is more applicable to web development.

The other topic to adjust is the design patterns taught in COMPSCI 718. The course currently covers adapter, composite, observer and template method patterns. The results show that while these design patterns are relevant to the industry, they were not the most frequently mentioned. Two people specifically mentioned MVC as important:

“MVC is the most important that everybody should know. The rest don’t need to be explicitly taught.”

From our results we conclude that introducing patterns such as MVC and Singleton would seem more appropriate for COMPSCI 718.

Currently, we spend one hour on planning using Scrum in COMPSCI 718. Based on the results, we think more time should be spent on project management in COMPSCI 718 as well as in COMPSCI 719. As most participants were exposed to at least one agile methodology, other agile methodologies such as Kanban should be introduced.

We also think testing should be adjusted in COMPSCI 718. To our surprise, not more than a third wrote unit tests and only one third of participants used test-driven development in their organisations. The results do not indicate unit testing or test-driven development being used as much as we expected. One participant, who has 6 – 10 years of experience as a developer, explained why they did not use test-driven development:

“Really depends on the pace of the company - a SME with about 20 devs moves too fast for TDD to be useful, as we rely on failing quickly to find issues that would otherwise take a lot of time to find (in our opinion).”

Some of our participants mentioned that they usually have a separate team writing unit tests or their roles do not require writing unit tests. They write unit tests only if they have to do so. Several participants commented on testing at their workplace:

“Currently they employ separate QAs who write tests concurrently as the developers write the code. However I think they are considering moving towards test-driven development.”

“Usually the unit tests and automated tests are written and run by a QA. We are expected to have ‘dev-tested’ it ourselves before submitting to the QA. However I did spend one week writing unit tests.”

“I very rarely unit test myself because it’s not possible for the infrastructure I work on, but other devs do and we require it as part of our application build process…”

Half of our participants also indicated the use of automated testing at work, which is an important element of continuous integration. In addition, the results reveal a number of participants have seen or used DevOps at work. Based on the results, we need to rethink what we teach in testing in COMPSCI 718 and consider at least exposing the concept of continuous integration and DevOps in either COMPSCI 718 or COMPSCI 719.

Documentation is a topic that we feel needs more investigation. The results do not give us a strong indication of the types of documentation that developers use at work. All but one participant had used some form of documentation, including inline comments. One participant made a comment on documentation:

“There’s never enough documentation. Generally documentation is stored in sharepoint, wiki or on a shared harddrive. Inline comments are a must for most jobs. GIT commits also a must”

It is worth commenting on our decision to ask about recursion. On the one hand, this is a fundamental concept in computer science and so would seem reasonable to teach, on the other hand there are a number of fundamental computer science topics that are not taught in the PGCertInfoTech so we wondered whether there was justification for including recursion and not these others topics. There was speculation within our team that it was not actually used in industry sufficiently to justify its inclusion and so we included a specific question on it to test this. The results clearly show that this is a concept worth covering in the programme. In fact, one participant implicitly indicated the importance industry attaches to this topic:

“Also had an interview question on recursion”

The results of our survey are helpful for us to evaluate our existing programme. In some cases the results might support multiple decisions (e.g. either increasing or decreasing content), but we can now make those decisions informed to some degree by what we know is happening in industry.

As with any survey-based research there are caveats. The most obvious question, and one common to any survey, is whether our sample was representative of the population. Because we limited our population, we were then able to contact almost everyone (148 in total). Of those contacted 48 started and 27 completed it, so a response rate of 18%. Whether this is sufficient is a matter of debate. Singer et al. report a consistent response rate of 5% in software engineering studies (Singer et al., 2008) so ours compares well with that.

However what is most important is whether the conclusions drawn from the responses could be invalidated by the non-responses. A key point is that our data is a lower bound on the usage of the different topics. If one person reports using a concept then we know at least one person in our population uses the concept, and it could easily be more. What we conclude from this, in particular whether we think this one usage is sufficient evidence to justify its inclusion in the PGCertInfoTech. is then dependent on our specific context. Other programmes, on seeing this evidence, may draw a different conclusion because they have a different context.

We only surveyed graduates of our programme. These graduates are likely to have sought positions that match the skills they have acquired from our programme, and so perhaps the responses to the topics in our survey are unsurprising. However what our results show is that graduates of our programme have, by and large, used the content provided in our programme, which gives us confidence it does provide value.

Our programme, and most of our graduates, exists in the New Zealand context. Nevertheless the original design of our programme was informed by international experience, and other research (Section 2) suggests the differences between countries is not great. Therefore we believe our results will be useful for others considering developing or revising similar programmes.

Our survey was based primarily on the existing topics in the PGCertInfoTech with only a few other topics that we thought we should consider. There are certainly other topics we might have included. However any choices made for any curriculum is a trade-off of topics and available resources (including time). The last question of our survey asked whether there were topics used in their work that we had not asked about. Five answered this question, mentioning people skills, use of frameworks (2), performance issues, and use of the command line.

The comment about command line was particularly interesting. It was:

“I reckon it’s really important to know how to navigate commandline comfortably, as a huge majority of development tools are commandline only”

This opinion was supported by the responses to the question on version control (Qvc), where a number of participants accessed their version control through the use of the command line. We currently do not specifically teach use of the command line. Based on these results we will consider introducing command line concepts to the programme.

In this paper we present the results of a survey of graduates of an IT transition programme with the view to evaluating its currency and to gather data on which to base decisions on updating it. Our conclusion is that our programme, the PGCertInfoTech at University of Auckland is largely relevant to our graduates working in the industry, but could be improved with adjustments in COMPSCI 718 to the coverage of concurrency, design patterns and testing, while COMPSCI 719 could introduce new topics such as asynchronous concurrency, continuous integration and project management.

For other institutions considering introduction a transition programme such as ours, they would likely come up with a similar set of topics to what we have. However, we hope that the data we have given will help provide more confidence with their decisions. For institutions who already have such programmes, we hope our data will help with their evaluation.