The :fileExtension property specifies the extension part of the filename of an asset.
File Extension
The :hasImage property relates a publication to an image.
Has Image
Publish Event
In order to differentiate these events further, the :PublishState vocabulary is used.
The :PublishEvent class represents events related to the publication of a document. This is intended in a strict sense, i.e. the time when a publication is made 'public'.
The :publicationName property specifies a string name for an external publication.
Publication Name
The :ExternalContributor class represents a contributor from an article which is externally published.
External Contributor
The :ExternalContributor class subclasses the :Contributor class.
The :hasNextIssue property relates a publication to the next issue.
Has Next Issue
Type
The :Type class groups together all domain-specific categorisations relevant to the NPG publishing world.
All of the :Type categorisations are implemented as self-contained reusable SKOS vocabularies (although not all of them are taxonomies in the strict sense of hierarchical vocabularies).
Citation (Data)
This is an alternative representation of the :citation property.
The :citationData property specifies the full citation of a publication as a single string for machine use.
The :hasPublishEvent property relates a publication to a publish-event that describes some recognized change in the status of a publication's publish history.
Has Publish Event
Volume Start
The :volumeStart property specifies the start volume for a serial publication. Note that this :volumeStart property relates to the publisher holdings which may not be the same as the actual publication volume start.
The :isNamedResource property marks whether an object resource is named or is unnamed (i.e. a blank node).
Is Named Resource
The :Person class represents a single person entity.
Person
The :hasFigure property relates a publication to a figure.
Has Figure
Issue End
The :issueEnd property specifies the end issue for a serial publication. Note that this :issueEnd property relates to the publisher holdings which may not be the same as the actual publication issue end.
The :publicationEndYearMonth property specifies the year and month date of a publication's end as a string. Note that this date relates to the publisher holdings which may not be the same as the actual publication end date.
Publication End YearMonth
Item
The :Item class represents the state of a document after this has been added to a :Collection by someone, at a specific time, and within a specific sequence.
The :PublishState class provides a categorisation of the possible states a document may have after it has been published. For example, whether it’s an 'Advanced online publication' or an 'Issue publication'. This categorisation is implemented as a SKOS vocabulary.
Publish State
Sort Letter
The :sortLetter property specifies the initial letter of an entity name to facilitate sort operations.
This property is generally computed from a build-time rule.
The :hasExternalArticle property relates an article to an external article.
Has External Article
The :volume property specifies a volume label for a journal.
Volume
The :coverDate property specifies the year and month date of a publication as it is displayed on its cover.
Cover YearMonth
This is not necessarily the actual date of publication.
Serial
The :Serial class represents publications which appear in a new edition on a regular schedule. For example, a journal, a magazine or a blog. The :PublishEvent class is used to provide more details about the time, place and other contextual information relevant to when a new edition gets published.
This property is similar to void:exampleResource. But see also skos:example.
The :example property specifies an example of an entity.
Example
The :hasExternalArticle property relates an article to an external contributor.
Has External Contributor
The :hasPublishState property relates a publish-event to a publish-state that describes some recognized change in the status of a publication's publish history.
Has Publish State
The :hasSection property relates an item to a section component.
Has Section
The :volumeEnd property specifies the end volume for a serial publication. Note that this :volumeEnd property relates to the publisher holdings which may not be the same as the actual publication volume end.
Volume End
The :hasBroaderSubject property relates a publication to a broader subject term than it is actually classified with.
Has Broader Subject
Caption (XML)
The :captionXml property specifies the caption of a figure or illustration as an XML string.
Summary
The :Summary class represents a brief description of a scholarly work, usually editorially generated.
This work is distributed under a Creative Commons Attribution 4.0 (CC BY 4.0) License <http://creativecommons.org/licenses/by/4.0/>.
Michele Pasin <mailto:michele.pasin@macmillan.com>
The Core Ontology is a formal model providing definitions for the key concepts of interest to content publishing at Macmillan Science and Education.
Core Ontology
For further information about this ontology or for any queries please use the contact address <mailto:developers@nature.com>.
Tony Hammond <mailto:tony.hammond@macmillan.com>
The :PublicationState class groups together publish-state and review-state individuals.
Publication State
The :hasPublication property relates a publication-event to a publication.
Has Publication
Tree Depth
The :treeDepth property specifies the depth of a term within a taxonomy tree.
This property is generally computed by a build-time rule
The :hasGraph property relates an entity to the RDF named graph associated with it.
Has Graph
The :isEventWorkflow property marks whether a publication is in the new metadata-based (event) workflow.
Is Event Workflow
The :idHash property specifies a string hash value for an ID.
ID Hash
Summary Type
The :SummaryType class represents an instance from a categorisation of types of :Summary. For example, 'Long Summary', 'Short Summary', etc. This categorisation is implemented as a SKOS concept scheme.
Number
The :number property specifies a sequence number for a component such as a figure.
The :hasReviewEvent property relates a publication to a review-event that describes some recognized change in the status of a publication's review history.
Has Review Event
The :hasXmlAsset property relates a publication to an xml-asset.
Has XML Asset
The :hasItem property relates a publication to an item.
Has Item
Given Name
The :givenName property specifies the given name of a person.
The :hasPrimaryArticleType property relates an article to an article-type that can be used for grouping purposes.
Has Primary Article Type
The :count property specifies a map which records the number of instances of an entity.
Count
The :Illustration class represents a numbered and captioned image within a publication. This image typically illustrates a point about the work described in the publication.
Illustration
The :publicationDate property specifies the full date of a publication as a string.
Publication Date
Asset Folder Name
The :assetFolderName property specifies a folder name for file storage assets.
The :timestamp property specifies a generic timestamp for an object.
Timestamp
The :hasIllustration property relates a publication to an illustration.
Has Illustration
Is Curated
The :isCurated property marks whether an image is editiorially managed.
Date Start
The :dateStart property specifies the start date for a serial publication. Note that this date relates to the publisher holdings which may not be the same as the actual publication start date.
The :isLeaf property marks whether a SKOS concept is a leaf SKOS concept.
Is Leaf
The :replacedBy property relates a publication to another publication that replaces it.
Replaced By
Credit
The :credit property specifies the photographer, illustrator, or copyright holder for an image in a publication.
Blog
The :Blog class represents an informal serial publication.
Item List
The :itemList property specifies an ordered list of items.
The :hasPdfAsset property relates a publication to a pdf-asset.
Has PDF Asset
Is Asset Of
The :isAssetOf property relates an asset to the publication it represents.
The :itemType property specifies the type of an item.
Item Type
The :itemReferenceList property specifies an ordered list of item references.
Item Reference List
Has Website
The :hasWebsite property relates a publication to a website.
The :BinaryAsset class represents the physical rendition of a generic binary object.
Binary Asset
The :webpage property relates a publication to a page on the web that describes it.
Web Page
The :id property provides a unique ID for a thing.
The :id value forms the local part of the full URI for an entity name.
The :caption property specifies the caption of a figure or illustration as a string.
Caption
The :Publication class groups together all individuals (or part of) created as the result of the publishing process. Note that we are not referring here to the physical artefacts, but just their abstract counterparts.
Publication
The :hasAbstract property relates an article to the abstract for an article.
Has Abstract
Is Issue Based
The :isIssueBased property marks whether a journal publishes issues.
The :Thing class groups together all individuals used by Macmillan Science and Education within the NPG publishing world.
Thing
Aggregation Event
The :AggregationEvent class groups together all events related to the construction of a publication.
Has Current Issue
The :hasCurrentIssue property relates a publication to the current issue.
The :filename property specifies the filename of an asset.
Filename
The :otherCode property specifies a number or string used by external applications as an ID for an entity.
Other Code
The :ReviewEvent class models events related to the reviewing of a document (before its publication). In order to differentiate these events further, the :ReviewState vocabulary is used.
Review Event
The :hasSummary property relates an article to an article summary.
Has Summary
The :hash property specifies a hash value over an entity URI used as a data integrity check.
Hash
Concept
The :Concept class groups together all individuals which do not exist in the physical world (i.e. have no extension in spacetime).
Publication Start Year
The :publicationEndYear property specifies the year date of a publication's end as a string. Note that this date relates to the publisher holdings which may not be the same as the actual publication end date.
The :hasBlog property relates a serial to a blog associated with it.
Has Blog
The :Organization class represents an organised body of people normally having a legal status.
Organization
The :class property specifies the class a domain model refers to.
Class
Has Article Type
The :hasArticleType relates an article to an article-type.
The :citationXML property specifies the full citation of a publication as a single string containing XML for user presentation.
Citation (XML)
1. INTRODUCTION. Knowledge tracing (KT) [1] is widely used to update an intelligent tutor?s estimate of the probability that a student has a given skill, based on the student?s observable performance on steps that use the skill. KT does not in itself address the issue of how to update multiple subskills used in the same step. This paper compares various approaches to this ?multiple subskills problem.? Section 2 frames the space of prior (and new) methods. Section 3 describes a recent method named LR-DBN [2]. Section 4 compares LR-DBN against previous methods on two data sets. Section 5 concludes. 2. COMPARATIVE FRAMEWORK. As a framework to compare previous and proposed methods for tracing multiple subskills, we use four aspects to characterize them: how they represent the KT model, how they fit the model parameters to observations of multi-subskill steps, how they use the model to predict performance on such steps, and how they update estimates of the subskills based on observed performance. 2.1 Represent model. Previous solutions represent the student?s knowledge at step n as a hidden state in a Hidden Markov Model (HMM), shown in Figure 1. It has knowledge parameters already know for the probability that the student knew the skill to start with, learn for the probability of the transition from not knowing the skill to knowing it, and forget (usually assumed to be 0) for the probability of the transition from knowing the skill to not knowing it. It also has performance parameters guess for the probability of performing the step correctly despite lacking the skill, and slip for the probability of performing the step incorrectly despite knowing the skill. Figure 1: Single-skill knowledge tracing architecture. 2.2 Fit parameters. One previous solution [3] tries to sidestep the problem by modeling each set of subskills as a distinct individual skill, e.g., computing the area of a circle embedded in a figure vs. by itself. However, modeling different sets of subskills as independent skills ignores transfer of learning between them. Other previous solutions [4-6] simply treat each subskill used in a step as if it were entirely responsible for that step. They train a separate KT model for each subskill on observations of all the steps that use it. Thus the same observed step appears in the training data for every subskill that it uses. They simply estimate the model parameters for each subskill using the same training procedure as for conventional KT. 2.3 Predict performance. In standard ?single-skill? KT, predicting performance at step n involving skill j is simple: Equation 1: Standard KT prediction. Previous solutions to the multiple-subskills problem predict performance on a step by combining in different ways the probabilities of correctly performing all the skills it requires. One way, based on an assumption that they are probabilistically independent, multiplies them [4, 6]: Equation 2: Independent subskills performance prediction. The weakest-subskill alternative [5] takes their minimum:. Equation 3: Weakest-subskill performance prediction. 2.4 Update estimate. To update its estimate of a skill j based on the observed success of a step n that uses it, standard KT applies Bayes? rule: Equation 4: Standard KT skill update for successful step. Conversely, the standard update rule if the step fails is:. Equation 5: Standard KT skill update for failed step. Either way, it estimates the probability of knowing the skill at the next step as either knowing without forgetting, or learning: Equation 6: Standard KT next-step update. When a step involves multiple subskills, previous methods use different ways to allocate responsibility among them for the observed success or failure of the step. The ?full responsibility? approach applies these equations to all the subskills. The ?update weakest subskill? approach simply applies the standard update equations above to whichever subskill in a step has the lowest probability, and leaves the others unchanged. Its ?blame weakest, credit rest? variant credits the other subskills as correct even if the step failed. Conjunctive knowledge tracing (CKT) [6] also predicts the probability of a step succeeding as a product of its subskill probabilities using Equation 2, and gives all of them full credit for success using Equation 4. However, rather than place full blame on each subskill for failure, CKT apportions blame among them differently. Instead of using Equation 5 to update each subskill based just on its own guess and slip probabilities, CKT takes into account those of the other subskills as well, as follows. Bayes? rule says how to update a skill based on performance: Equation 7: Bayes' rule for skill update. Conditioning on having skill j at step n reduces simplifying the numerator of Equation 9 to: Equation 8: CKT subskills update for failed step. CKT computes the denominator by assuming independence:. Equation 9: CKT prediction based on multiple subskills. Next, we introduce a different strategy of using logistic regression in KT to trace multiple subskills. 3. USING LOGISTIC REGRESSION TO TRACE MULTIPLE SUBSKILLS. We now describe two newer methods that trace multiple subskills using logistic regression. Previous KT methods fit their parameters independently using the same algorithm as for singleskill KT, thereby implicitly assigning full and equal responsibility to all the subskills in an observed step, and predict performance based on the weakest subskill or by multipying subskill estimates. Section 3.1 describes LR-DBN, which changes both these aspects by using Expectation Maximization (EM) [7] to fit parameters for multiple subskills simultaneously, and by using logistic regression to predict performance. As an ablation experiment to shed light on the relative impact of these two innovations, Section 3.2 introduces LR-DBN Minus, a hybrid method that fits the standard KT model just as previous methods do, but uses logistic regression to do prediction. 3.1 LR-DBN. LR-DBN is a recent but published method [2, 8] to trace multiple subskills, so we summarize it here only briefly in terms of the four aspects discussed in Section 2. Figure 2: Knowledge tracing with logistic regression. Represent model: Like standard KT, LR-DBN represents the knowledge for step n as a hidden knowledge state in a dynamic Bayes net. However, as Figure 2 illustrates, LR-DBN adds a layer of observable states as indicator variables to represent whether step n involves subskill j: 1 if so, 0 if not. LR-DBN uses logistic regression to model the initial hidden knowledge state at step 0 and the transition probabilities from step n-1 to n as follows: Equation 10: Logistic regression to combine subskills. These three conditional probabilities for each subskill replace the KT knowledge parameters already know, learn, and forget, but LR-DBN retains KT?s guess and slip parameters at each step. Fit parameters: LR-DBN uses Expectation Maximization (EM) [7] to fit parameters for all subskills together. Predict performance: LR-DBN uses logistic regression in a Dynamic Bayes Net to combine multiple subskills more flexibly than using Equation 2 to multiply their probabilities or Equation 3 to take their minimum, but it uses Equation 1 to predict expected performance based on estimated knowledge, guess, and slip. Update estimates: LR-DBN uses the same Bayes rule as singleskill KT to update its estimate of the hidden knowledge state in Equation 4, Equation 5, and Equation 6. 3.2 LR-DBN Minus. LR-DBN Minus is a hybrid of LR-DBN and standard KT. It combines KT?s single-skill fitting process with LR-DBN?s update and prediction based on logistic regression. The key is to convert the probability of knowing a subskill into a coefficient in logistic regression. LR-DBN uses logistic regression to model the transition probabilities between knowledge states, as well as the relation of the knowledge state at each step to the subskills it involves [2]. Thus, given the set of subskills used at step n, a set of coefficients exists such that: Equation 11: Logistic regression for the knowledge state. If we assume step n requires only a single subskill i, then for all the j?s such that j , and transformations between the and the coefficients are: Equation 12: Transformation between probabilities and logistic regression coefficients To update the estimates, we need to distribute the update at each step that is calculated either from Equation 4 or Equation 5 to the subskill coefficients. We assume that the coefficient for each subskill changes by the same amount when updated: Equation 13: Update coefficients in LR-DBN Minus. Then the update of each subskill becomes:. Equation 14: Update subskills in LR-DBN Minus. Next we still use the standard KT Equation 6 to update subskills at step n+1. Now we have successfully transformed LRDBN to upate on standard KT parameters. Note that we replace the separate guessj and slipj parameters for each subskill j with uniform guess and slip for all the steps. The reason is that LRDBN combines subskills to estimate the probability of the student knowing a step and then uses guess and slip to predict performance. In contrast, previous methods apply guessj and slipj to each subskill j before combining them to predict performance on the step. 4. EXPERIMENTAL EVALUATION. To compare LR-DBN and LR-DBN Minus to previous methods for tracing multiple subskills, we fit seven models to real data, summarized in Table 1: LR-DBN, LR-DBN Minus, CKT, and three variants of standard KT distinguished by how they update estimated skills: ?full responsibility,? ?blame weakest, credit rest,? and ?update weakest subskill,? with majority class as an additional baseline. Sections 4.1 and 4.2 describe our data and results. Table 1: Summary of models compared. 4.1 Data sets. We train and test the models on real data from two tutors used at schools. One data set is from children using Project LISTEN?s Reading Tutor [9] at primary schools during the 2005-2006 school year. To model their oral reading fluency, we define performance as whether the Reading Tutor scored a text word as read fluently at step n, i.e., read without help or hesitation and recognized by the automated speech recognizer. We assume that whether a student read a word fluently depended on whether the student knew the requisite subskills, namely the grapheme-tophoneme mappings in the word. Due to the large amount of data (1,792,103 read words from 275 students), we randomly selected 20 children who read a total of 80,268 words (3,972 distinct word types) with 320 unique grapheme-phoneme mappings. To counteract the prevalence of high-frequency words like the, we include at most the first 20 of each student?s encounters of a word in the training data, leaving 24,145 read words. We do not limit the test data, so it includes 40,867 words. The other data set [10] came from 123 high school students working on a geometry area unit of the Bridge to Algebra Cognitive Tutor®. The model for each student includes the same 50 subskills, and predicts whether the student will perform a step correctly. Again we include at most the first 20 of each student?s encounters of a step in the training data, leaving 11,730 algebra practice steps, but 22,737 steps of test data. The data sets from both tutors are unbalanced. The Reading Tutor scored 68.84% of the words in the training set as fluent, and 74.31% of the words in the test set. The Algebra Tutor rated 74.22% of the steps in the training data as correct, and 84.63% of the steps in the test data. We fit each model separately for each student, as opposed to training a single model on the data for all the students. One reason is computational expedience: unlike methods that fit a separate model for each subskill, LR-DBN fits a single model for all the subskills, which involves processing much more data at a time. Training this model on all the students? data at once would be computationally unwieldy. The other reason is to compare methods fairly. Except for LR-DBN, it is feasible to train a single model of a subskill on the data for all the students, and in fact we tried it, but the resulting model does not perform as well as training a separate model for each student. For all the methods, we fit the model for each student to the first half of the student?s steps, and test it on the second half. We report average per-student accuracy on the unseen test data, weighting its mean and variance by per-student sample size to derive 95% confidence intervals. We use paired T-tests, paired by student, to rate LR-DBN?s accuracy against each other method. 4.2 Results. Table 2 and Table 3 list all seven methods in decreasing order of their binary predictive accuracy on the test data. LR-DBN dramatically outpredicts all the other methods. LR-DBN?s overall accuracy on the Reading Tutor data is 13% higher than majority class, vs. only 1% for the next method. For the Algebra Tutor data, LR-DBN is the only method that beats the majority class, by 7%. That is, on both data sets, LR-DBN has only half the error rate of the next best method. For unbalanced data, accuracy on the minority class can be especially important. The minority class in our tutor data represents negative student outcomes to remediate by means of practice and instruction. LR-DBN beats every other method on the minority class by over 20% absolute in both data sets. What does comparison to LR-DBN Minus reveal about the relative contributions of the fitting and update procedures? LRDBN Minus uses the same fitting procedure as conventional knowledge tracing, but uses logistic regression to update estimates. It performs substantially worse than LR-DBN, and comparably to the other methods. We conclude that LR-DBN?s accuracy benefits more from its fitting procedure than from using logistic regression to combine estimates of hidden subskills. Table 2: Mean per-student accuracy on Reading Tutor data (95% confidence interval in parentheses) compared to LR-DBN is significantly (p<.01) worse if underlined, or better if italicized. Table 3: Mean per-student accuracy on Algebra Tutor data is significantly (p<.001) worse than LR-DBN?s where underlined; italicized values are significantly better. Why does LR-DBN outpredict the other methods? Possible reasons include the strong assumptions that it avoids, but which they make implicitly by fitting and updating subskill estimates separately, multiplying them to predict performance on a step, and assigning each subskill full responsibility for the step?s outcome. Inspection of Table 1 reveals that this last assumption is the only one they all have in common, implicating it as the likeliest culprit. Predictive accuracy is just one way to evaluate student models. A more sensitive metric is model fit as measured by data likelihood, penalized by model complexity. Table 4 and Table 5 list the complexity-penalized model fits of the methods on the two data sets in increasing order, as scored by the Akaike information criterion (AIC) [11] and Bayesian information criterion (BIC) [12], defined respectively as: Equation 15: Formulas for calculating AIC and BIC. Table 4: Complexity-adjusted Reading Tutor training data fit. Table 5: Complexity-adjusted Algebra Tutor training data fit. Both AIC and BIC measure model fit as log-likelihood of the training data, ln(L), penalized by model complexity (number of parameters, k). BIC also penalizes the number of observations, n. We calculate the number of parameters per student as follows: LR-DBN fits the 3 groups of coefficients for each of the subskills and one intercept in Equation 10, plus two shared parameters, guess and slip. For the Reading Tutor data set, this number totals 3 × (320 + 1) + 2 = 965, multiplied by the 20 children in the data sample. For the Algebra Tutor data set, it totals 3 × (50 + 1) + 2 = 155, multiplied by 123 students. LR-DBN Minus fits 2 parameters (already know and learn) per subskill, plus 2 shared parameters (guess and slip). This number of parameters per student totals (2 × 320) + 2 = 642 for the Reading Tutor and (2 × 50) + 2 = 102 for the Algebra Tutor. The other methods fit 4 parameters (already know, learn, guess and slip) per subskill for each student, totaling 4 × 320 = 1280 for the Reading Tutor, and 4 × 50 = 200 for the Algebra Tutor. Thus compared to previous methods, LR-DBN has about 1 less parameter per subskill, and LR-DBN Minus about 2 less. What about the number n of observations? LR-DBN uses one observation per step to fit all the subskill parameters. In contrast, the other methods fit each subskill separately, assigning it full responsibility for every step that uses it, as if observing it separately for each subskill. Counting such duplicate observations as separate, they use three times as many Reading Tutor observations as LR-DBN, and twice as many Algebra Tutor observations. Table 6 and Table 7 show the average log-likelihood of steps in the training and test data. All the methods except LR-DBN share the same likelihood on the training data because they fit parameters in the same way (as shown in Table 1). Table 6: Average log-likelihood for the Reading Tutor data. Table 7: Average log-likelihood for the Algebra Tutor data. Normally one might expect log-likelihood to be lower for test data than training data, by an amount reflecting the degree of overfitting. However, the models assign higher likelihood to correct steps because, as Section 4.1 mentioned, they are more common than incorrect steps in the training data, and this difference is more pronounced in the test data. Its likelihood is therefore higher, and hence is not a direct gauge of overfitting. Table 6 and Table 7 reveal that LR-DBN?s log-likelihood is by far the highest on unseen test data from both tutors, consistent with how dramatically it outpredicts the other methods, even though they have higher log-likelihood on the training data from the Algebra Tutor. This reversal from training to test data suggests that the other methods might overfit that training data. In summary, LR-DBN has a smaller number k of parameters than the other methods (except for LR-DBN Minus), a smaller number n of observations (counting duplicate observations as distinct), and higher likelihood on Reading Tutor training data, where it achieves the lowest AIC and BIC scores. Most important, LR-DBN far surpasses all the other methods in accuracy and log-likelihood on unseen test data from both tutors. 5. IMPLEMENTATION. To make LR-DBN publicly available , we added it to the Bayes Net Toolkit for Student Modeling (BNT-SM) [13]. BNT-SM inputs a data set and a DBN student model (not only the simple one used in standard knowledge tracing), specified in XML. It generates and executes BNT code to train and test the model, and outputs Excel files containing the parameter estimates and inference results. BNT is an open-source Matlab package that supports many learning and inference algorithms for both static and dynamic Bayes models. BNT-SM hides most of the BNT coding details, freeing users to focus on constructing the student models rather than on programming them. Using BNT-SM consists of four phases [13]: 1. Specify the data source in an XML specification. 2. Specify the DBN structure in XML. 3. Specify and initialize parameters in XML. 4. Call RunBnet.m in Matlab. To fit LR-DBN on the Reading Tutor data with 320 subskills, we specify the structure shown in Figure 2 to BNT-SM in XML, as shown in the APPENDIX. 6. CONCLUSIONS. This paper makes multiple contributions to knowledge tracing: First, we present a framework to characterize previous and new methods for tracing multiple subskills by how they (1) model knowledge tracing, (2) fit its parameters, (3) predict performance, and (4) update subskill estimates. Second, we use data sets from reading and algebra tutors to compare LR-DBN against previous methods in terms of AIC, BIC, and predictive accuracy on unseen data, and show that LR-DBN performs significantly better on both data sets on all three metrics, cutting the best previous prediction error rate in half. Third, we introduce the hybrid LR-DBN Minus method, which fits the same standard KT model as previous methods, but uses logistic regression to predict student performance. Fourth, by comparing LR-DBN Minus to LR-DBN, we show that using logistic regression to predict performance suffices to beat previous methods, but that using logistic regression EM to jointly estimate subskills accounts for most of LR-DBN?s superior performance. Finally, in order to amplify the impact of this work, we have made LR-DBN publicly available and easy to extend to other student modeling with dynamic Bayes nets, by incorporating it into the latest version of the BNT-SM student modeling toolkit [13] used in previous studies of knowledge tracing [e.g., 14]. This work has several limitations for future work to address. First, LR-DBN has so far been applied just to simple knowledge tracing of multiple subskills, but it can apply to any DBN. Future work could use LR-DBN to improve other DBN student models, for example to measure more accurately the scaffolding and learning effects of tutor help [14]. Second, LR-DBN needs 5.5 hours on average per student to fit and update; the other methods take less than 1 hour to fit a single set of parameters for all the students and subskills, and 2-5 minutes to update. Future work may train LR-DBN faster or develop other methods that are faster to train. Such work might adapt two previous types of cognitive diagnosis models that operate on static data and have statistical learning algorithms, both EM and MCMC [15]. NIDA (Noisy Inputs, Deterministic ?And? gate) models [16] resemble CKT because it applies guess and slip to individual subskills before combining them conjunctively. DINA (Deterministic Inputs, Noisy ?And? gate) models [17] resemble LR-DBN because it combines subskills (with logistic regression) before applying guess and slip to the resulting knowledge state. Extending either type of model to apply to knowledge tracing may improve LR-DBN itself. Finally, although LR-DBN traces multiple subskills better than previous methods, it (like them) must be told which steps use which subskills. Future work may infer this information automatically [18]. ACKNOWLEDGMENTS. This work was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A080628 to Carnegie Mellon University. The opinions expressed are those of the authors and do not necessarily represent the views of the Institute or U.S. Department of Education. We thank Ken Koedinger for his CKT implementation and algebra tutor data, and children, schools, and LISTENers for our Reading Tutor data.
Comparison of methods to trace multiple subskills: Is LR-DBN best?
A long-standing challenge for knowledge tracing is how to update estimates of multiple subskills that underlie a single observable step. We characterize approaches to this problem by how they model knowledge tracing, fit its parameters, predict performance, and update subskill estimates. Previous methods allocated blame or credit among subskills in various ways based on strong assumptions about their relation to observed performance. LRDBN relaxes these assumptions by using logistic regression in a Dynamic Bayes Net. LR-DBN significantly outperforms previous methods on data sets from reading and algebra tutors in terms of predictive accuracy on unseen data, cutting the error rate by half. An ablation experiment shows that using logistic regression to predict performance helps, but that using it to jointly estimate subskills explains most of this dramatic improvement. An implementation of LR-DBN is now publicly available in the BNT-SM student modeling toolkit.
The :Image class represents a graphical element within a publication.
Image
The :body property specifies the main body content of a publication as a string.
Body
The :xmlId property specifies an ID assigned to an XML element.
XML ID
The :hasPublisher property relates a publication to the organization that publishes it.
Has Publisher
The :publicationStartYear property specifies the year date of a publication's start as a string. Note that this date relates to the publisher holdings which may not be the same as the actual publication start date.
The :Subject class represents a topic, field of study or research area which can be used to categorise the contents of a publication. The instances of :Subject are organised hierarchically using the SKOS vocabulary.
Subject
The :sectionLevel property specifies the level number of a section.
Section Level
This property is used as a surrogate for the class name. There is a one-to-one relation between the class name and its associated graph, e.g. instances of the class :Subject are assigned to the subjects: graph.
The :type property specifies the local name of the graph namespace used for instances of a class.
The :pages property specifies a page range numbering for a document.
Pages
Is Group
The :isGroup property marks whether a contributor name refers to a collective agent, i.e. a group of people or an institution.
The :contributorList property specifies an ordered list of contributors.
Contributor List
The :status property specifies a status string for an ontology term.
Status
We are currently using this class for RDF datasets.
Dataset
The :Dataset class represents a collection of related sets of information modelled in a known way and stored in a computer system.
The :hasReviewState property relates a review-event to a review-state that describes some recognized change in the status of a publication's review history.
Has Review State
The :coverDate property specifies the full date of a publication as it is displayed on its cover.
Cover Date
Collection
The :Collection class represents a group of documents which are published (or republished) as part of a new composite publication. Often these documents have been published elsewhere, but since they share a common theme or belong to an identifiable narrative it is of interest to publish them again within a different container. (See also :Issue.)
The :graph property specifies the full URI of any graph.
Graph
The :isPublishReady property marks whether a type is ready for publishing.
Is Publish Ready
The :isRepeatable property marks whether a property can be repeated.
Is Repeatable
Blog Post
The :BlogPost class represents an informal work published as an item of a :Blog.
The :Document class groups together all publications which were created to be distributed and consumed as individual, self-contained entities.
Document
The :Event class groups together events of varying granularity. Events differ from :Concept entities because they have an extension in time; they differ from :Asset entities because they do not have an extension in space.
Event
Page End
The :pageEnd property specifies an ending page number for a document.
The :isTestInstance property marks whether a resource is to be used for test purposes only.
Is Test Instance
The :dateEnd property specifies the end date for a serial publication. Note that this date relates to the publisher holdings which may not be the same as the actual publication end date.
Date End
The :hasPreviousIssue property relates a publication to the previous issue.
Has Previous Issue
Has Issue
The :hasIssue property relates a publication-event to an issue.
Xml Asset
The :XmlAsset class represents the physical rendition of a document asset in XML format.
The :hasSerial property relates a publication to a serial.
Has Serial
Each Graph instance is associated with instances of a particular class. For example, the 'npgg:articles' :Graph instance is a named graph which contains all the instances of the :Article class. The :Graph instances themselves are then used to describe the RDF graph, providing sometimes known as an RDF annotation. This may furnish various kinds of information about the RDF graph: provenance, rights, metrics, validation reports, etc.
The :Graph class represents a collection of related sets of information modelled in RDF and organized as a named graph.
The :hasAsset property relates a publication to an asset.
Has Asset
The :datetime property specifies a system-generated datetime for an entity.
Datetime
The :bodyXml property specifies the main body content of a publication as an XML string.
Body (XML)
The :Journal class represents a scholarly serial publication, considered from both the scholarly and the commercial perspectives. Thus a journal is more than the sum of its publications (i.e. the set of all the :Issue instances it has published); it is an entity characterised by many properties, e.g. its founders, its impact factor, etc.
Journal
This property is used by the doamin ontologies for managing terms.
Code
The :code property specifies an integer number representing an internal ID for an object.
The :Figure class represents a numbered and captioned image within a publication. This image typically results as a direct outcome of the work described in the publication.
Figure
The :sortLetter property specifies the sequence number of an entity to facilitate sort operations.
Sort Number
The :Article class represents a scholarly work normally published as an item of a serial publication. In particular, this concept refers to the abstract notion of an article, rather than any of its realisations after it becomes embodied via a physical medium.
Article
With reference to the FRBR model [http://www.ifla.org/publications/functional-requirements-for-bibliographic-records], an :Article can be mapped directly to the frbr:Expression concept. It is worth noting though that within the NPG core ontology we do not have any notion of a frbr:Work. Hence multiple versions of the same article (e.g. language translations, or revisions) are characterised as such simply by means of some relationship.
The :KnowledgeBase class represents a collection of related sets of information modelled in RDF and consisting of various objects, e.g. one or more graphs, other RDF datasets etc.
KnowledgeBase
Differently from datasets, knowledge-bases refer to a composite dataset intended to be used as a consistent whole.
The :isCorresponding property marks whether a contributor is a corresponding author.
Is Corresponding
The :ArticleType class represents the various kinds of publication which can be used to index and group content published by Macmillan Science and Education. The instances of :ArticleType are organised hierarchically using the SKOS vocabulary.
Article Type
The :relation property relates a publication to a publication.
Relation
ORCID
As we currently have no persons as such registered in our data model, we use the :orcid property with the :Contributor publication component that proxies for a person.
The :orcid property relates a publication to an ORCID record.
Image Asset
The :ImageAsset class represents the physical rendition of an image asset in an encoded format.
Is PrimaryArticleType
The :isPrimaryArticleType property marks whether an article-type can be used for grouping purposes.
The :hasCorrespondingContributor property relates an article to the name of its corresponding author.
Has Corresponding Contributor
The :issn property specifies an ISSN number for a serial.
ISSN
Has Image Asset
The :hasImageAsset property relates an image to an image-asset.
The :subjectPath property specifies the position of a term within the subjects taxonomy by providing a path with the IDs of its successors.
Subject Path
The :date property specifies the year and month date of an event as a string.
YearMonth
The :figureList property specifies an ordered list of figures for a publication.
Figure List
The :titleXml property specifies a title to the main body content of a publication as an XML string.
Title (XML)
The :ReviewState class provides a categorisation of the possible states a document may have as a result of a reviewing process. For example, whether it’s been 'Received' or 'Accepted'. This categorisation is implemented as a SKOS vocabulary.
Review State
The :hasContributor property relates a publication to a string containing the name of one of the authors of that publication.
Has Contributor
Normally, the contributor name is a non-disambiguated person name, meaning that several contributor instances may refer to the same person.
The :test property specifies a general string test property.
Test
Year
The :date property specifies the year date of an event as a string.
The :hasCollection property relates a publication-event to a collection.
Has Collection
Name
The :name property specifies the full name for a thing.
The :mesh property relates a concept to a Medical Subject Heading (MeSH) term as a resource.
MeSH
Is Root
The :isRoot property marks whether a SKOS concept is a SKOS top concept.
The :namespace property provides the RDF namespace used in creating instances of a class.
Namespace
The :doiLink property relates a publication to the URI minted from the DOI.
DOI Link
Page Start
The :pageStart property specifies a starting page number for a document.
The :assetVersion property specifies the version of an :Asset associated with a :PublishEvent.
Asset Version
The :Website class represents a site that maintains one or more pages on the Web.
Website
Publication Event
The :PublicationEvent class groups together events that happen within the publishing process, broadly conceived. For example, the submission of an article, or the acceptance of a paper.
External Article
The :ExternalArticle class represents an article which is externally published.
The :ExternalArticle class subclasses the :Article class.
Replaces
The :replaces property relates a publication to another publication that it replaces.
The :hasRoot property relates a :Type instance to a toplevel :Type instance within the same domain model.
Has Root
The :citation property specifies the full citation of a publication as a single string for reference purposes.
Citation
The :hasJournal property relates a publication to a journal.
Has Journal
The :itemNumber property specifies the number of an item within a collection.
Item Number
The :Agent class groups together all entities that can be the subject of an action. These can be single persons (:Person), or groups of people organised as a legal entity (:Organization).
Agent
The :eissn property specifies an E-ISSN number for a serial.
E-ISSN
The :issueStart property specifies the start issue for a serial publication. Note that this :issueStart property relates to the publisher holdings which may not be the same as the actual publication issue start.
Issue Start
The :hasArticle property relates a publication-event to an article.
Has Article
Section
The :Section class represents a part of a document.
The :doi property specifies a DOI number for a publication.
DOI
Severity Level
The :SeverityLevel class represents a system status. The syslog standard (RFC 5424) for computer message logging is used to provide a reference set of states. This categorisation is implemented as a SKOS concept scheme.
The :shortTitle property specifies an abbreviated title for a publication.
Short Title
The :publicationStartYearMonth property specifies the year and month date of a publication's start as a string. Note that this date relates to the publisher holdings which may not be the same as the actual publication start date.
Publication Start YearMonth
The :Contributor class represents a document component detailing a contributor (personal or corporate) to the work being published. This class is disjoint from the :Agent class. Hence a :Contributor instance can be described as the name used to refer to a person or organisation in the context of a publication.
Contributor
Issue
The :issue property specifies an issue label for a journal.
The :dateTime property specifies the full date of an event as a string, including a time component.
DateTime
Pdf Asset
The :PdfAsset class represents the physical rendition of a document asset in PDF format.
Publication YearMonth
The :publicationYearMonth property specifies the year and month date of a publication as a string.
The :baseUriTemplate property specifies a string template for an asset base URI.
Base URI Template
The :repositoryId property specifies a (local) repository ID for an asset.
Repository ID
The :hasSubject property relates a publication to a subject term which describes one of the main topics the publication is about.
Has Subject
The :filesize property specifies an integer value for the size (in bytes) of an asset.
Filesize
The :imageList property specifies an ordered list of images for a publication.
Image List
Family Name
The :familyName property specifies the family name of a person.
The :Publisher class represents organizations involved in the publishing business.
Publisher
The :publicationDateTime property specifies the full date of a publication as a string, including the time of publication.
Publication DateTime
The :Issue class represents an aggregate object which is published periodically as part of a :Serial and is normally composed by one or more articles. (See also :Collection.)
Component
The :Component class groups together all individuals that normally come into existence as part of a :Document.
Abstract
The :Abstract class represents a brief description of a scholarly work, usually author generated. Instances of :Abstract normally do not exist in isolation, but are linked to a :Document instance (of which they provide a summary).
The :publicationYear property specifies the year date of a publication as a string.
Publication Year
Has Summary Type
The :hasSummaryType property relates a summary to a summary-type classification.
Cover Year
The :coverDate property specifies the year date of a publication as it is displayed on its cover.
Date
The :date property specifies the full date of an event as a string.
The :Asset class groups together all physical entities which can be stored in digital systems.
Asset
The :isVocabulary property marks whether a class is used for a controlled vocabulary.
Is Vocabulary
The :title property specifies a title to the main body content of a publication as a string.
Title
The :repository property relates an asset to a storage repository.
Repository
The :BlobAsset class groups together all assets used to store blobs - i.e. documents, images, or other media objects - within digital systems.
Blob Asset