<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:sy="http://purl.org/rss/1.0/modules/syndication/">

	<title>Xavablog</title>
	<id>tag:xavier.robin.info,2010-05-28:/fr/feed</id>
	<link rel="self" href="https://xavier.robin.info/fr/feed" />
	<link rel="alternate" href="https://xavier.robin.info/fr/blog"/><!--http://xavier.robin.name/comments for all comments http://xavier.robin.name/comments/path for comments on a specific post, http://xavier.robin.name/tag/(tag) for posts with 'tag'-->
	<updated>2026-11-13T21:16:15.637817000+01:00</updated>
	<sy:updatePeriod>daily</sy:updatePeriod>
	<sy:updateFrequency>2</sy:updateFrequency>
	<link rel="license" type="application/rdf+xml" href="http://creativecommons.org/licenses/by-sa/3.0/rdf" />

	<icon>https://xavier.robin.info/fr/img/favicon.ico</icon>

	<author>
		<name>Xavier Robin</name>
		<uri>https://xavier.robin.info/fr/contact</uri>
	</author>
	

	<entry xml:lang="en" xml:base="https://xavier.robin.info/fr/">
		<title type="html">131 bugs fixed in pROC 1.20</title>
		
			<category term="pROC" label="pROC" scheme="https://xavier.robin.info/fr/tag/pROC" />
		
		<link href="https://xavier.robin.info/fr/blog/2026/11/02/130-bugs-fixed-in-proc-1.20"/>
		<id>tag:xavier.robin.info,2026-11-02:/blog/2026/11/02/130-bugs-fixed-in-proc-1.20</id>
		<published>2026-11-02T10:36:54+01:00</published>
		<updated>2026-11-13T20:08:44+01:00</updated>
		<content type="html">&lt;p&gt;I let AI agents loose on pROC for several days in October. 
They found 131 bugs in 1.19.1, plus 77 problems in its documentation and tests.
I reviewed and revised everything. 
Below: how the hunt worked, how I checked it, what might break your code with the next update (1.20), and every fix with its commit.

&lt;div class=&quot;note&quot;&gt;
&lt;p&gt;&lt;b&gt;AI disclaimer.&lt;/b&gt; The bugs were found, fixed and
documented by AI agents (Claude Code, by Anthropic), working from my
instructions with a large degree of autonomy. The review was also done
with Claude, in a separate session: it audited the agents&#39; work,
reproduced the findings and prepared the decisions. Every decision about
what to fix, and how, was mine, and I reviewed the results. This post
was drafted by Claude from the agents&#39; reports and the review records,
and edited by me.&lt;/p&gt;
&lt;/div&gt;

&lt;h2 id=&quot;breaking-changes&quot;&gt;Breaking changes&lt;/h2&gt;
&lt;p&gt;These changes can break code that worked with 1.19.1:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;geom_polygon_auc()&lt;/code&gt; has no
&lt;code&gt;legacy.axes&lt;/code&gt; argument anymore: it follows the axes of the
&lt;code&gt;ggroc()&lt;/code&gt; plot.&lt;/li&gt;
&lt;li&gt;The formula methods of &lt;code&gt;plot.roc()&lt;/code&gt; and
&lt;code&gt;lines.roc()&lt;/code&gt; now take &lt;code&gt;subset&lt;/code&gt; and
&lt;code&gt;na.action&lt;/code&gt; after &lt;code&gt;...&lt;/code&gt;, so these must be named:
&lt;code&gt;plot.roc(y ~ x, data, subset = g == &quot;a&quot;)&lt;/code&gt;. A positional
&lt;code&gt;subset&lt;/code&gt; now fails.&lt;/li&gt;
&lt;li&gt;The unpaired DeLong test of &lt;code&gt;roc.test()&lt;/code&gt;
no longer returns a &lt;code&gt;parameter&lt;/code&gt; (df): it now compares its
statistic to the normal distribution (Hanley &amp;amp; McNeil, 1983), like
the paired DeLong and the bootstrap tests, instead of a Student t
distribution with ad hoc degrees of freedom. p-values change slightly
with small samples.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;results-that-change&quot;&gt;Results that change&lt;/h2&gt;
&lt;p&gt;Most fixes change results. Of the 131 bugs fixed, 55 change a
returned value (a wrong number, curve, confidence interval, p-value or
flag becomes right), 42 return a result where 1.19.1 failed with an
error, 11 fail with an error where 1.19.1 returned a wrong or
meaningless value, and 21 change only printed or plotted output. The
other two change where pROC writes its bug-report file, and add a
&lt;code&gt;paired&lt;/code&gt; argument to &lt;code&gt;cov()&lt;/code&gt; and
&lt;code&gt;power.roc.test()&lt;/code&gt;. Every fix is listed below with its
commit.&lt;/p&gt;
&lt;h2 id=&quot;what-was-done-the-bug-hunt&quot;&gt;What was done: the bug hunt&lt;/h2&gt;
&lt;p&gt;The hunt ran several days, mostly unattended.&lt;/p&gt;
&lt;h3&gt;The brief&lt;/h3&gt;
&lt;p&gt;I gave the agent a written brief:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The deliverable: one draft pull request with one
commit per bug.&lt;/li&gt;
&lt;li&gt;What counts as a bug: wrong results,
inconsistencies between interfaces, ignored arguments, input handling,
side effects, C++ problems, platform problems, wrong documentation and
wrong tests.&lt;/li&gt;
&lt;li&gt;The methods to use: review against the literature,
parameter sweeps, property tests, differential tests, edge cases, static
analysis and sanitizers, coverage, and the history.&lt;/li&gt;
&lt;li&gt;The evidence: every bug needed a standalone
reproducible example, with its output before and after the fix.&lt;/li&gt;
&lt;li&gt;Autonomy: the agent was not to stop and ask about
individual bugs. Anything that changed behaviour or the API went to an
approval queue instead. Nothing destructive was allowed: no force-push,
no push to the main branches, no merge.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;The organisation.&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;The main session kept a ledger on disk with one
entry per bug, and ran the static checks itself: compiler warnings,
cppcheck, &lt;code&gt;codetools&lt;/code&gt;, lintr, coverage, and a sanitizer
(ASAN/UBSAN) run of the whole test suite.&lt;/li&gt;
&lt;li&gt;Hunter agents each searched one area: the
&lt;code&gt;roc&lt;/code&gt; core; AUC, smoothing and multiclass; the CIs, bootstrap
and &lt;code&gt;coords&lt;/code&gt;; DeLong, var/cov and the tests; power; plotting
and printing; and the documentation. They couldn&#39;t touch git, and they
ran R against a fixed copy of the package.&lt;/li&gt;
&lt;li&gt;Fix agents worked in separate git worktrees. Each
fix came with a regression test that had to fail before the fix and pass
after it, and the full test suite ran on every commit.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;Six passes&lt;/h3&gt;
&lt;p&gt; Each pass added new methods:&lt;/p&gt;
&lt;table class=&quot;smarttable&quot;&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Pass&lt;/th&gt;
&lt;th&gt;Methods&lt;/th&gt;
&lt;th&gt;Bugs found&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;The eight methods of the brief, on every area&lt;/td&gt;
&lt;td&gt;98&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;Differential tests against brute force; features new in the
development version&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;Other ROC packages as references; mutation testing (2574 mutants);
real usage (all GitHub issues, Stack Overflow, the pROC calls of 216
reverse dependencies); papers; 8 older CRAN releases; the test suite on
R 4.0–4.3&lt;/td&gt;
&lt;td&gt;35&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;Re-review of recent commits; deep dives where bugs clustered; the
tests of 162 reverse dependencies; plot output; every use of
&lt;code&gt;...&lt;/code&gt;; unusual input types&lt;/td&gt;
&lt;td&gt;64&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;Mutation testing of every line the pull request added (2219 mutants:
no bug, the first method to find nothing); a crash fuzzer (about 20,000
generated programs)&lt;/td&gt;
&lt;td&gt;21&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;Review of the latest commits; an extended fuzzer (40,000 programs);
environment perturbation and a statistical calibration sweep, both cut
short when we stopped the hunt&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The first method to find nothing was the mutation testing of pass 5.
Pass 6 still found bugs, but minor ones, such as a
&lt;code&gt;plot.roc()&lt;/code&gt; option that ignored a partial AUC unless another
option was also set, a pairing rule that was not documented, and an
unclear error message. The hunt did not stop by itself.
During pass 6, the review session and I stopped it: read together, the
commits showed that in several places each point fix had widened the
previous one, and that needed rules before any more fixes (&quot;Policies&quot;,
in the next section).&lt;/p&gt;
&lt;p&gt;By the end, the hunt had produced 233 bug reports, from BUG-001 to
BUG-233, in one pull request: &lt;a
href=&quot;https://github.com/xrobin/pROC/pull/148&quot;&gt;#148&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id=&quot;how-it-was-reviewed&quot;&gt;How it was reviewed&lt;/h2&gt;
&lt;h3&gt;First reviews, by the agent itself.&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;Every candidate was re-run before it got a bug number.&lt;/li&gt;
&lt;li&gt;An adversarial review agent re-read all the commits before the first
push. It found two regressions and four incomplete fixes, which were
corrected before anything was published.&lt;/li&gt;
&lt;li&gt;GitHub&#39;s continuous integration ran on every push.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;These reviews looked at one commit at a time, though. Several later
fixes turned out to repair regressions introduced by earlier fixes of
the same pull request.&lt;/p&gt;
&lt;h3&gt;Policies&lt;/h3&gt;
&lt;p&gt;With the hunt stopped, I set
policies for the fixes:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;P1: degenerate input gives NA, as base R does;&lt;/li&gt;
&lt;li&gt;P2: no matching on warning messages, and no blanket
suppression of warnings;&lt;/li&gt;
&lt;li&gt;P3: no floating-point tolerances as fixes;&lt;/li&gt;
&lt;li&gt;P4: fix in one place;&lt;/li&gt;
&lt;li&gt;P5: test hygiene;&lt;/li&gt;
&lt;li&gt;P6: classes from packages pROC doesn&#39;t depend on
are out of scope;&lt;/li&gt;
&lt;li&gt;P7: every statistical change needs a source that
was actually read.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;Root causes&lt;/h3&gt;
&lt;p&gt;For each cluster of related fixes, the
agent wrote a design note, and I decided how to fix the cause once:
argument routing, smoothing settings, rounding, input normalisation,
degenerate input, and the bootstrap. The agent then rewrote the history
so that each commit holds one complete, reviewed fix.&lt;/p&gt;
&lt;h3&gt;An audit of the whole pull request&lt;/h3&gt;
&lt;p&gt;A second agent:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;checked every commit against the policies;&lt;/li&gt;
&lt;li&gt;compared the behaviour of the whole package with 1.19.1;&lt;/li&gt;
&lt;li&gt;loaded objects saved by 1.19.1 into the new version;&lt;/li&gt;
&lt;li&gt;compared performance with 1.19.1;&lt;/li&gt;
&lt;li&gt;measured every test that depends on a random seed over hundreds of
seeds.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I took each finding as a separate decision.&lt;/p&gt;
&lt;h3&gt;A check that every bug was reviewed&lt;/h3&gt;
&lt;p&gt;Some fixes had
only been accepted as a number in a list. So every bug was traced back
to the evidence I had actually seen:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;the 15 fixes that change numbers, or turn a value into an error,
were reviewed again in full: all 15 were real bugs, and a few needed
changes;&lt;/li&gt;
&lt;li&gt;the other 117 got a lighter pass, with each fix re-run against
1.19.1.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;Final check&lt;/h3&gt;
&lt;p&gt;After the last revision, 1520 test
cases were compared between 1.19.1, the previous version of the pull
request and the final one. Every difference traced back to a documented
decision.&lt;/p&gt;
&lt;h2 id=&quot;other-fixes&quot;&gt;Other fixes&lt;/h2&gt;
&lt;p&gt;A few bugs were also fixed in the development version, separately
from the hunt: seven bugs and two warning or error messages (DEV-1 to
DEV-10). They are included in the counts and in the lists below.&lt;/p&gt;
&lt;h2 id=&quot;not-fixed&quot;&gt;Not fixed&lt;/h2&gt;
&lt;p&gt;5 bugs of 1.19.1 are not fixed, by decision:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;three floating-point rounding cases (BUG-112, the
weighted ties of BUG-113, and the fraction case of BUG-226): no
floating-point tolerance as a fix;&lt;/li&gt;
&lt;li&gt;integer64 predictors (BUG-174): a class from a
package pROC doesn&#39;t depend on;&lt;/li&gt;
&lt;li&gt;a single density value (BUG-205): degenerate input,
which returns a value as in 1.19.1.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;lessons-learned&quot;&gt;Lessons learned&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;Settle the policies before the first fix, not after 230.
I only wrote P1 to P7 once the commits were already contradicting each other. 
By then the agent had made statistical claims from memory instead of checking them (P7), and was introducing tolerances in numeric comparisons everywhere (P3).
But how do you know what policies are needed beforehand?&lt;/li&gt;
&lt;li&gt;When a third fix lands in the same function, it&#39;s time to take a step back. Reviewing commit by commit didn&#39;t catch the clusters. Reviewing by root cause did.&lt;/li&gt;
&lt;li&gt;Record who decided what. Once, the review agent put into the agent&#39;s brief a decision I had never made. Now every decision file quotes my answer, and anything else is labelled a proposal.&lt;/li&gt;
&lt;li&gt;Freeze history before writing about it. The branch was rewritten three times during the review, and every commit hash in the decision files and the pull request had to be updated each time.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;all-the-changes&quot;&gt;All the changes&lt;/h2&gt;
&lt;p&gt;Each entry links to its commit, with a reproducible example and the
output before and after the fix.&lt;/p&gt;
&lt;h3 id=&quot;bugs-fixed-129&quot;&gt;Bugs fixed (131)&lt;/h3&gt;
&lt;table class=&quot;smarttable&quot;&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Bug&lt;/th&gt;
&lt;th&gt;Area&lt;/th&gt;
&lt;th&gt;Fix&lt;/th&gt;
&lt;th&gt;Commit&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;BUG-001&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;roc.test: the paired DeLong conf.int was on the 0-1 scale for
percent=TRUE curves&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/4664e42618bf3f8a2bda5e7ca0c2287916c8e69b&quot;&gt;4664e42&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-002&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;roc.test: conf.level unchecked with the automatic method (NaN or
inverted CI)&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/7563cd64f4f39e04af0951332177a4ec125703c8&quot;&gt;7563cd6&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-003&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;roc.test/cov: the partial AUC of auc objects of smoothed curves was
ignored&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/b511a3d25e5a90c70f12df7cf1a1a758b44c4c37&quot;&gt;b511a3d&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-004&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;roc.test/cov/var: numeric-density smoothed curves gave D=Inf or 0
instead of the documented error&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/5b8a819bf081ff91654b1deed9f7ae6a8d70cfab&quot;&gt;5b8a819&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-005&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;roc.test: unpaired Venkatraman mixing proportion used n1+n1 instead
of n1+n2&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/ba1d47140af082320c096f34221d5cebc0bc0b77&quot;&gt;ba1d471&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-006&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;cov: Obuchowski method returned NaN for percent curves with a
partial AUC&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/dd976e2db8fd05aab1aa19e4a08b12a3fccab4af&quot;&gt;dd976e2&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-008&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;roc.test: paired Venkatraman p-value depended on the row order with
ties&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/8822bcc3b4c7e40ad95fbd80ce2629892e3ba954&quot;&gt;8822bcc&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-011&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;cov: paired curves with NAs at different positions failed or gave a
wrong bootstrap value&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/b2718f53be71ea9d7cbbb1d20c4757a38f8487d7&quot;&gt;b2718f5&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-013&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;roc.test/cov: a partial AUC on roc2 only picked DeLong and crashed
(wrote pROC_bug.RData)&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/6adb5a31d3e85b27f0bcd0c673495b1b5b3c3530&quot;&gt;6adb5a3&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-014&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;roc.test: Venkatraman ran on whole curves when only roc2 had a
partial AUC&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/96194ddceec211af6ae90ed495bc388793fc0ff7&quot;&gt;96194dd&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-016&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;roc.test/cov: formula-built curves were not seen as paired with
default-built ones&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/b5f4619abea4d0367a122cfbf1adbc0412577d03&quot;&gt;b5f4619&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-017&lt;/td&gt;
&lt;td&gt;roc&lt;/td&gt;
&lt;td&gt;roc/auc/ci* (formula): subset failed (&quot;object not found&quot;) with
plot=TRUE or smooth=TRUE&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/aac4e69a4c73bc20e659c8ebd0861d4b692e99fd&quot;&gt;aac4e69&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-018&lt;/td&gt;
&lt;td&gt;roc&lt;/td&gt;
&lt;td&gt;roc (formula, data.frame), roc_: malformed list instead of NA/NaN
with NA data&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/3a34186f35e75b2dd294563ed92f453c2658e71d&quot;&gt;3a34186&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-019&lt;/td&gt;
&lt;td&gt;roc&lt;/td&gt;
&lt;td&gt;roc (densities): direction=&quot;auto&quot; was not resolved with auc=FALSE
(inverted curve)&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/ef34a6c874dfa606b14f285c3f4a1a1e5f937cc7&quot;&gt;ef34a6c&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-020&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;are.paired: user objects named oroc1/oroc2 broke roc.test and
cov&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/e784d77808577110590a45740d4573284cd6e71a&quot;&gt;e784d77&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-021&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;are.paired(return.paired.rocs=TRUE): unsmoothed curves returned for
smoothed input&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/8f1ccd08e303fc5219470027d944cdbb4f44b15b&quot;&gt;8f1ccd0&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-022&lt;/td&gt;
&lt;td&gt;auc&lt;/td&gt;
&lt;td&gt;has.partial.auc: TRUE for a malformed partial.auc of length 1&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/87507b2ef0d250f7e775b191d4f99cecce2369dc&quot;&gt;87507b2&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-025&lt;/td&gt;
&lt;td&gt;coords&lt;/td&gt;
&lt;td&gt;coords: an NA threshold (input=&quot;threshold&quot;) failed instead of giving
an NA row&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/ae37c686ffc98c68da1cf257c3d7f9716591f425&quot;&gt;ae37c68&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-026&lt;/td&gt;
&lt;td&gt;roc&lt;/td&gt;
&lt;td&gt;roc: na.action attribute lost when some responses were outside
levels&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/fb7b51a5c090ff5f704e2a7299accbab4f8e4abf&quot;&gt;fb7b51a&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-029&lt;/td&gt;
&lt;td&gt;roc&lt;/td&gt;
&lt;td&gt;roc(smooth=TRUE) via formula/data.frame: data names printed as
&quot;(unknown)&quot;&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/5f7115e52af8773134f64310274bb724d2c8eedb&quot;&gt;5f7115e&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-030&lt;/td&gt;
&lt;td&gt;roc&lt;/td&gt;
&lt;td&gt;roc_: failed when called from a function through Map or mapply&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/bf40bd1401872685ed0c98858a7e3d612f70a10f&quot;&gt;bf40bd1&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-031&lt;/td&gt;
&lt;td&gt;multiclass&lt;/td&gt;
&lt;td&gt;multiclass.roc: error when exactly half the classes had no predictor
column&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/4517c6015a1ed51d1aa9cc32264f3f4401b5280d&quot;&gt;4517c60&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-032&lt;/td&gt;
&lt;td&gt;smooth&lt;/td&gt;
&lt;td&gt;smooth: logcondens smoothing ignored direction=&quot;&amp;gt;&quot;&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/ae036360412ca20ca396ca9e3f4b933acd6e19ca&quot;&gt;ae03636&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-033&lt;/td&gt;
&lt;td&gt;smooth&lt;/td&gt;
&lt;td&gt;smooth(reuse.ci=TRUE): failed on every curve with a stored CI&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/97ae00f23ec8a6be78828c17b9b2e22c91bf1926&quot;&gt;97ae00f&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-035&lt;/td&gt;
&lt;td&gt;auc&lt;/td&gt;
&lt;td&gt;auc: partial.auc bounds outside [0, 1] returned NA or empty AUC
instead of an error&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/c0e00069b2c769dd7cbbd48348987467723f4921&quot;&gt;c0e0006&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-036&lt;/td&gt;
&lt;td&gt;smooth&lt;/td&gt;
&lt;td&gt;smooth(method=&quot;fitdistr&quot;): curve truncated to the data range (AUC
off by up to 0.27)&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/f3787d8aea63a359cebebcc90835e594184e4e4a&quot;&gt;f3787d8&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-037&lt;/td&gt;
&lt;td&gt;multiclass&lt;/td&gt;
&lt;td&gt;multiclass.roc: requested levels without observations failed for
non-factor responses&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/e9c1d73c1b49d7af8a252ed47257cc993bf3a8f7&quot;&gt;e9c1d73&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-040&lt;/td&gt;
&lt;td&gt;smooth&lt;/td&gt;
&lt;td&gt;smooth (binormal): points sharing one specificity ran backwards or
failed&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/bd0aa83b6bdf27faf9657f562a7ef33a5efb2c4d&quot;&gt;bd0aa83&lt;/a&gt;, &lt;a href=&quot;https://github.com/xrobin/pROC/commit/36092b8b52807c3241e040ed9970c1bc8b21d9ff&quot;&gt;36092b8&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-041&lt;/td&gt;
&lt;td&gt;multiclass&lt;/td&gt;
&lt;td&gt;multiclass.roc: a tibble predictor matrix failed (&quot;Predictor must be
numeric or ordered&quot;)&lt;/td&gt;
&lt;td&gt;&lt;a 
href=&quot;https://github.com/xrobin/pROC/commit/f64f87ef4d8be3891acd01aa1c43753836e7f943&quot;&gt;f64f87e&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-042&lt;/td&gt;
&lt;td&gt;smooth&lt;/td&gt;
&lt;td&gt;roc(smooth=TRUE, fitdistr): &quot;unused argument&quot; with the partial AUC
arguments&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/a2a71f531575147601ed2a63dd10b881a7faf7cd&quot;&gt;a2a71f5&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-043&lt;/td&gt;
&lt;td&gt;multiclass&lt;/td&gt;
&lt;td&gt;multiclass.roc: fewer than two usable levels gave AUC=Inf instead of
an error&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/eb399cfcbea42bd83dac83fecf7ef2843a6042e8&quot;&gt;eb399cf&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-044&lt;/td&gt;
&lt;td&gt;coords&lt;/td&gt;
&lt;td&gt;coords(x=&quot;best&quot;): wrong youden/closest.topleft with a specificity
partial AUC&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/c5b94e28907700db9a2395960d4dc047e3a0d4d5&quot;&gt;c5b94e2&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-045&lt;/td&gt;
&lt;td&gt;coords&lt;/td&gt;
&lt;td&gt;coords: fpr, fnr and other complementary inputs picked the wrong end
of flat segments&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/28487a4c40ddfbc72f3c38ed74925b9fd04112e8&quot;&gt;28487a4&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-046&lt;/td&gt;
&lt;td&gt;coords&lt;/td&gt;
&lt;td&gt;coords: NA threshold for exact points with numeric inputs other than
se/sp&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/4aed903af6f92f70e6ff1ec3ad3bb9437c1b4291&quot;&gt;4aed903&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-047&lt;/td&gt;
&lt;td&gt;coords&lt;/td&gt;
&lt;td&gt;coords on an auc object: ignored all arguments and returned the
whole curve&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/03366802c10064695dd4207afe61ec572beb5b52&quot;&gt;0336680&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-048&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;ci.auc: NA CI for smoothed percent curves with a corrected partial
AUC&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/fb4b263aa8dca9ee7a0e21d449b5fe0f2eb461a4&quot;&gt;fb4b263&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-049&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;ci.coords (smoothed): replicates shifted when some resamples failed
to smooth&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/d87f992ee57ad2bdea92eeba909a7fd7637a7141&quot;&gt;d87f992&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-050&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;ci.coords(x=&quot;best&quot;): failed when a resample had no point in the
partial AUC range&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/77ec97cf518f30699e67735b90ba6247e1db8a34&quot;&gt;77ec97c&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-051&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;ci/ci.* (formula, default): smooth=TRUE failed&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/10f5350c98fca63866c5be5740bd69aa50b2d7b0&quot;&gt;10f5350&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-052&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;ci.se/sp/coords/thresholds: non-stratified resamples losing a class
aborted&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/6c3051ae4d926a6b5a0a6dec0e75a612af39400c&quot;&gt;6c3051a&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-057&lt;/td&gt;
&lt;td&gt;coords&lt;/td&gt;
&lt;td&gt;coords(x=&quot;best&quot;): closest.topleft 100 times too large on percent
curves&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/c5b94e28907700db9a2395960d4dc047e3a0d4d5&quot;&gt;c5b94e2&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-058&lt;/td&gt;
&lt;td&gt;power.roc.test&lt;/td&gt;
&lt;td&gt;power.roc.test: paired curves with NAs at different positions failed
or misaligned&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/e94449d0a0f6e1fc516db9702ea66596b71b62b2&quot;&gt;e94449d&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-059&lt;/td&gt;
&lt;td&gt;power.roc.test&lt;/td&gt;
&lt;td&gt;power.roc.test: NaN for the auc of a percent curve&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/3b1311506bbd8a86b378bf3f48d2b60d999513aa&quot;&gt;3b13115&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-060&lt;/td&gt;
&lt;td&gt;power.roc.test&lt;/td&gt;
&lt;td&gt;var/cov/power.roc.test (Obuchowski): binormal A/B misused, pAUC
variances far too small&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/fefccc6c75228e9992697d5983c79553133b6e49&quot;&gt;fefccc6&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-061&lt;/td&gt;
&lt;td&gt;power.roc.test&lt;/td&gt;
&lt;td&gt;power.roc.test (Obuchowski): null variance used the alternative&#39;s
parameters&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/fba063edbf99b420addc742ae6e970914bc22b2c&quot;&gt;fba063e&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-061b&lt;/td&gt;
&lt;td&gt;power.roc.test&lt;/td&gt;
&lt;td&gt;power.roc.test (DeLong, bootstrap): result depended on curve order,
NaN if better first&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/31de1a44dbea6ec0453cdbd49f089b64ccda6f30&quot;&gt;31de1a4&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-062&lt;/td&gt;
&lt;td&gt;power.roc.test&lt;/td&gt;
&lt;td&gt;power.roc.test (two curves): kappa ignored in the variances&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/25f4144840290b70c4042523838bd4be291dc221&quot;&gt;25f4144&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-063&lt;/td&gt;
&lt;td&gt;power.roc.test&lt;/td&gt;
&lt;td&gt;power.roc.test (one curve): a partial AUC was silently replaced by
the full AUC&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/baf62093ea0ecb4d280e1db1811bd2a471bf24cc&quot;&gt;baf6209&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-064&lt;/td&gt;
&lt;td&gt;power.roc.test&lt;/td&gt;
&lt;td&gt;power.roc.test (parameter list): FPR bound order flipped the
covariance (107 vs 205)&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/ffe2ac2d1c2111f431f5e7e12e723d7d07d82ac5&quot;&gt;ffe2ac2&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-065&lt;/td&gt;
&lt;td&gt;power.roc.test&lt;/td&gt;
&lt;td&gt;power.roc.test (bootstrap): variances from the wrong margin, random
sample sizes&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/0abf5cc75ca88d21ea601fb5b0d309273a30414d&quot;&gt;0abf5cc&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-066&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;var: failed on smoothed curves built with percent=TRUE&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/7eedc3347759d3638f41d6fb238dafb02cc9666e&quot;&gt;7eedc33&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-067&lt;/td&gt;
&lt;td&gt;smooth&lt;/td&gt;
&lt;td&gt;print (fitdistr-smoothed curve): fitted distributions never
shown&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/2cc6e4d3ec2450bb779e5d8dfa9a8a2e86f135df&quot;&gt;2cc6e4d&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-068&lt;/td&gt;
&lt;td&gt;auc&lt;/td&gt;
&lt;td&gt;Ops on auc/ci: unary minus failed; auc + ci returned a ci with a
warning&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/74f2b2b35530de6a8527f82757269003d80a8239&quot;&gt;74f2b2b&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-069&lt;/td&gt;
&lt;td&gt;multiclass&lt;/td&gt;
&lt;td&gt;print (multivariate multiclass.roc, formula): predictor names pasted
together&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/5ba7ee0922e53897a9b5611be50701926f87d9c6&quot;&gt;5ba7ee0&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-070&lt;/td&gt;
&lt;td&gt;roc&lt;/td&gt;
&lt;td&gt;print (roc from densities): garbage &quot;Data: (unknown) in 0 controls&quot;
line&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/3a8cb557be9dd84e8218d4b98f1c9069b0c27dbc&quot;&gt;3a8cb55&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-071&lt;/td&gt;
&lt;td&gt;ggroc&lt;/td&gt;
&lt;td&gt;ggroc: partially named lists gave blank labels or &quot;Names must be
unique&quot;&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/66ad4caf4606669375c615bff7e46ae16b9a2cf7&quot;&gt;66ad4ca&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-072&lt;/td&gt;
&lt;td&gt;plot&lt;/td&gt;
&lt;td&gt;plot.roc/lines.roc (formula): returned curve printed &quot;(unknown)&quot;
names&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/87d85ad9ca877022cf7b9d7908a0d1ae54542587&quot;&gt;87d85ad&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-074&lt;/td&gt;
&lt;td&gt;ggroc&lt;/td&gt;
&lt;td&gt;geom_polygon_auc: default fill (dark grey) unlike plot.roc&#39;s
gainsboro polygon&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/a5ad49b4de3f38daffa0fbceb14267fcdc666d5c&quot;&gt;a5ad49b&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-075&lt;/td&gt;
&lt;td&gt;plot&lt;/td&gt;
&lt;td&gt;plot.ci.coords: zero-width ticks for a single x; cryptic error for
x=&quot;best&quot;&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/7d16372526fc5eb97436506cfa694522f927338a&quot;&gt;7d16372&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-076&lt;/td&gt;
&lt;td&gt;roc&lt;/td&gt;
&lt;td&gt;roc(ci=TRUE, of=&quot;coords&quot;, x=..., plot=TRUE): x reached plot.roc and
failed&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/552a364d2bacfe3f7f0cae63963f11b45ae63045&quot;&gt;552a364&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-090&lt;/td&gt;
&lt;td&gt;roc&lt;/td&gt;
&lt;td&gt;roc (densities): direction=&quot;auto&quot; with a corrected pAUC failed below
the diagonal&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/3509a281ff0c22b9b3820a618a08dd5a3bb887e0&quot;&gt;3509a28&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-091&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;roc.test (sensitivity/specificity methods): reported the AUCs as
estimates&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/c622f4ad7e9749742e07d3367b2a88e446908909&quot;&gt;c622f4a&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-095&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;roc.test: the documented data.name element was missing&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/9100cb8f88e5e002888ab4795a982d2c91f77299&quot;&gt;9100cb8&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-097&lt;/td&gt;
&lt;td&gt;coords&lt;/td&gt;
&lt;td&gt;coords (smoothed): transpose, drop and as.list gave the wrong
shape&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/d03dce35be1fd8c648c4ebe3a0b94fe291ffefd9&quot;&gt;d03dce3&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-098&lt;/td&gt;
&lt;td&gt;smooth&lt;/td&gt;
&lt;td&gt;smooth(method=&quot;fitdistr&quot;, density=&quot;t&quot;): always failed&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/ccfe199465e8f12c3670f7c40dad59737db61c21&quot;&gt;ccfe199&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-099&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;ci.auc/var: non-stratified bootstrap aborted when a resample lost
the controls&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/27d391436414ae24a5630b2398a20e59f494100d&quot;&gt;27d3914&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-106&lt;/td&gt;
&lt;td&gt;coords&lt;/td&gt;
&lt;td&gt;coords/ci.*: percent rates not exact (typed 57.5 missed points);
&quot;best&quot; lost exact ties&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/72fc41b5dae0df9b08981d21a99cf9db30983cd9&quot;&gt;72fc41b&lt;/a&gt;,
&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/42b3d2fe618cdd1ab61d5fc4515f4e0edb324bbc&quot;&gt;42b3d2f&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-107&lt;/td&gt;
&lt;td&gt;coords&lt;/td&gt;
&lt;td&gt;coords(&quot;local maximas&quot;): flat edges, straight segments and dents
reported&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/3c4920f6a33e9e672e0a5cfd215fb34f6d9c1a79&quot;&gt;3c4920f&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-108&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;roc.test: data.name garbled for long calls&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/bfa969cc62ca01e16de84aeb471bf50429678d3f&quot;&gt;bfa969c&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-114&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;roc.test: the unpaired Venkatraman statistic was not integrated over
[0, 1]&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/5472f257f6e07e3902e164f616d7146d5c7a4fb5&quot;&gt;5472f25&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-115&lt;/td&gt;
&lt;td&gt;smooth&lt;/td&gt;
&lt;td&gt;smooth(fitdistr, density function): an extra ... argument failed
(&quot;unused argument&quot;)&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/c091aa16abf86c869c850eda34dc834d3288238e&quot;&gt;c091aa1&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-116&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;roc.test (se/sp methods): D=NaN when the tested values never
differ&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/a2f9d330c0365d86259448ad3aab7b7ae4d664f1&quot;&gt;a2f9d33&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-117&lt;/td&gt;
&lt;td&gt;smooth&lt;/td&gt;
&lt;td&gt;smooth(fitdistr, discrete densities): pmf evaluated between
integers, wrong curve&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/b9e15e86e26a6b6d2ad33d48b69bd48e476e6c77&quot;&gt;b9e15e8&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-117b&lt;/td&gt;
&lt;td&gt;smooth&lt;/td&gt;
&lt;td&gt;smooth(fitdistr): density names in another case (&quot;Normal&quot;) rejected,
unlike MASS&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/e65b358805199d34ee102ae10b6cf12b48ee2af9&quot;&gt;e65b358&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-119&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;roc.test (one-sided paired DeLong): returned the two-sided
conf.int&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/13e735af10bd44de8bd6d9ebea7564913524af06&quot;&gt;13e735a&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-120&lt;/td&gt;
&lt;td&gt;other&lt;/td&gt;
&lt;td&gt;internal errors: pROC_bug.RData written to the working directory
without asking&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/9c3b658309245144ef6c194faf322fedcea5eb35&quot;&gt;9c3b658&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-121&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;roc.test (paired DeLong): single case/control crashed or gave p=1
instead of NA&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/dc5282935cf7900374982ea6cbe76029360aff1b&quot;&gt;dc52829&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-122&lt;/td&gt;
&lt;td&gt;auc&lt;/td&gt;
&lt;td&gt;auc/ci* (response, predictor): failed instead of returning roc()&#39;s
NA/NaN&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/6813bdc04ab8a5d89b66b0f9aa5c4a927ed4eb54&quot;&gt;6813bdc&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-133&lt;/td&gt;
&lt;td&gt;smooth&lt;/td&gt;
&lt;td&gt;ci.coords/roc(smooth.method=&quot;density&quot;): x of ci.coords passed to
density()&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/a2a71f531575147601ed2a63dd10b881a7faf7cd&quot;&gt;a2a71f5&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-140&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;roc.test: unpaired Venkatraman permuted raw ranks, not the mixture
CDF scale&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/55238cb95c1b2554f2e745dc0e8430acb2c584d3&quot;&gt;55238cb&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-141&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;ci.se/ci.sp (smoothed): &quot;dim(X)&quot; error when no resample could be
smoothed&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/b621cfa278afb1f16f1b610ff5934ca2495daaeb&quot;&gt;b621cfa&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-144&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;roc.test/multiclass.roc (formula): subset failed with plot=TRUE or
smooth=TRUE&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/aac4e69a4c73bc20e659c8ebd0861d4b692e99fd&quot;&gt;aac4e69&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-146&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;roc.test (unpaired DeLong): D=NaN for two curves with zero
variance&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/e6e02b2753dabff6d1e57e5540b8243c7783bf47&quot;&gt;e6e02b2&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-151&lt;/td&gt;
&lt;td&gt;multiclass&lt;/td&gt;
&lt;td&gt;print (multiclass.roc from a formula in a variable): failed&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/43c569def30b85faa702fb1596d60df9aeb83f1e&quot;&gt;43c569d&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-156&lt;/td&gt;
&lt;td&gt;plot&lt;/td&gt;
&lt;td&gt;plot.roc(print.thres): percent curves showed thresholds with 1
decimal&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/9646da2ccb6e3bd33dc87a4c20e90a1f947a9b54&quot;&gt;9646da2&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-157&lt;/td&gt;
&lt;td&gt;plot&lt;/td&gt;
&lt;td&gt;plot.ci/plot.roc(ci.type=&quot;shape&quot;): shapes crossed for unordered CI
points&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/bbf2933fc95d2d751684f9bb27a5e4c7e5ed6f59&quot;&gt;bbf2933&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-158&lt;/td&gt;
&lt;td&gt;plot&lt;/td&gt;
&lt;td&gt;plot.roc(print.auc, new partial.auc): printed the stored full AUC&#39;s
CI&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/34d40a97e1cbf5742e44257507a23b0fc55b9a72&quot;&gt;34d40a9&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-159&lt;/td&gt;
&lt;td&gt;plot&lt;/td&gt;
&lt;td&gt;plot.roc(legacy.axes=TRUE, percent): axis labels padded and
off-centre&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/050b5d0dbb692653c1a0802d51db701cd5086e8a&quot;&gt;050b5d0&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-160&lt;/td&gt;
&lt;td&gt;plot&lt;/td&gt;
&lt;td&gt;plot.roc: ci.coords CIs drawn on wrong axes, or failing for
x=&quot;best&quot;&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/f049d8fb05fea16fda54a9942056b2be7ce24eb9&quot;&gt;f049d8f&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-161&lt;/td&gt;
&lt;td&gt;plot&lt;/td&gt;
&lt;td&gt;plot.roc/roc(smooth=TRUE): panel.first/panel.last evaluated before
the plot&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/a2a71f531575147601ed2a63dd10b881a7faf7cd&quot;&gt;a2a71f5&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-162&lt;/td&gt;
&lt;td&gt;plot&lt;/td&gt;
&lt;td&gt;plot.roc/lines.roc (default, smooth=TRUE): curve printed internal
names&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/a4faf0a62886f0acd10ee0562d7dc1cf08cc3012&quot;&gt;a4faf0a&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-163&lt;/td&gt;
&lt;td&gt;plot&lt;/td&gt;
&lt;td&gt;plot.roc(print.thres=&quot;all&quot;, pAUC): window limits marked and labelled
&quot;NA&quot;&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/74aba204eb2e9500dd12be11cf5b2ebc0dc7b6c9&quot;&gt;74aba20&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-164&lt;/td&gt;
&lt;td&gt;plot&lt;/td&gt;
&lt;td&gt;plot.roc/lines.roc (formula): sub= taken as subset; subset must now
be named&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/aac4e69a4c73bc20e659c8ebd0861d4b692e99fd&quot;&gt;aac4e69&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-165&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;ci.auc/ci (smooth=TRUE): failed with reuse.auc=FALSE or
auc=FALSE&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/552a364d2bacfe3f7f0cae63963f11b45ae63045&quot;&gt;552a364&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-166&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;ci/ci.auc (response, predictor, smooth=TRUE): &#39;method&#39; collided with
smooth.roc&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/a2a71f531575147601ed2a63dd10b881a7faf7cd&quot;&gt;a2a71f5&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-167&lt;/td&gt;
&lt;td&gt;multiclass&lt;/td&gt;
&lt;td&gt;multiclass.roc: ci/auc arguments failed (&quot;matched by multiple
arguments&quot;)&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/552a364d2bacfe3f7f0cae63963f11b45ae63045&quot;&gt;552a364&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-168&lt;/td&gt;
&lt;td&gt;smooth&lt;/td&gt;
&lt;td&gt;roc(smooth=TRUE, plot=TRUE, main=quote(...)): CI all NA, roc.test
failed&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/24da24cc9deae82b752d054112235e18638c20c2&quot;&gt;24da24c&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-169&lt;/td&gt;
&lt;td&gt;smooth&lt;/td&gt;
&lt;td&gt;roc(smooth=TRUE, density, plot=TRUE): graphical adj changed the
curve or failed&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/a2a71f531575147601ed2a63dd10b881a7faf7cd&quot;&gt;a2a71f5&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-170&lt;/td&gt;
&lt;td&gt;plot&lt;/td&gt;
&lt;td&gt;plot.roc: graphical parameter lab failed&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/552a364d2bacfe3f7f0cae63963f11b45ae63045&quot;&gt;552a364&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-171&lt;/td&gt;
&lt;td&gt;plot&lt;/td&gt;
&lt;td&gt;plot.roc(print.auc/print.thres/polygons): cex, pch, adj...
failed&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/552a364d2bacfe3f7f0cae63963f11b45ae63045&quot;&gt;552a364&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-172&lt;/td&gt;
&lt;td&gt;smooth&lt;/td&gt;
&lt;td&gt;roc(smooth=TRUE, fitdistr, density=&amp;lt;function&amp;gt;): density name
recorded wrong&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/b9e095f4b3ffee8cdfe4780d7c90c0b6c65fb710&quot;&gt;b9e095f&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-173&lt;/td&gt;
&lt;td&gt;roc&lt;/td&gt;
&lt;td&gt;roc (ordered, numeric labels): observations of an NA level
dropped&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/0f43659a432ef5f77bf9dc1f6c5a1259367b8fff&quot;&gt;0f43659&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-175&lt;/td&gt;
&lt;td&gt;roc&lt;/td&gt;
&lt;td&gt;roc/multiclass.roc: duplicated levels compared a class with
itself&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/803c146313b5d13b4e8c816b9e275819bd587347&quot;&gt;803c146&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-176&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;are.paired/roc.test: explicit levels=c(0, 1) made paired curves
unpaired&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/f7f04de2ed345ef4fc9bfa1fbd7bb88bed347b9d&quot;&gt;f7f04de&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-177&lt;/td&gt;
&lt;td&gt;roc&lt;/td&gt;
&lt;td&gt;roc/smooth (densities): negative density values gave a wrong
AUC&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/cab602e4b3e3114b2513204dc17c496ed860be47&quot;&gt;cab602e&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-191&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;ci.auc/var/cov/power.roc.test: a percent curve with a stored
ci.coords failed&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/dd976e2db8fd05aab1aa19e4a08b12a3fccab4af&quot;&gt;dd976e2&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-192&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;ci.coords (smoothed): all-NA threshold column; ret=&quot;threshold&quot; alone
now errors&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/f4ec873034b3fa875af99019405e832c0b4eb243&quot;&gt;f4ec873&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-193&lt;/td&gt;
&lt;td&gt;coords&lt;/td&gt;
&lt;td&gt;coords (smoothed, numeric x): best.weights ignored for
youden/closest.topleft&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/3be234fde3d4cbb09b5c04cdbe01a18b4511ba1c&quot;&gt;3be234f&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-195&lt;/td&gt;
&lt;td&gt;coords&lt;/td&gt;
&lt;td&gt;coords/ci.*: NA in x, specificities, sensitivities or thresholds
failed&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/ae37c686ffc98c68da1cf257c3d7f9716591f425&quot;&gt;ae37c68&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-196&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;print (ci.coords with repeated x): &quot;duplicate row.names&quot; error&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/d1fd14d4a3c524aeeeececcc7ed11c2bfdc97270&quot;&gt;d1fd14d&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-198&lt;/td&gt;
&lt;td&gt;coords&lt;/td&gt;
&lt;td&gt;coords/ci.coords/plot.roc: negative cost in best.weights gave the
worst threshold&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/abd55366e236fb3821a819347c4847edd0493dfe&quot;&gt;abd5536&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-201&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;roc.test/are.paired: re-paired curves printed pROC internals&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/7822db24ba4a4b2a830e7fe192df5dff9f350622&quot;&gt;7822db2&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-202&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;boot.n: fractions/NA unchecked; Venkatraman p divided by a
fractional boot.n&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/41a8df66738fddf3d6092c5ba13e2cd5ad7c021f&quot;&gt;41a8df6&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-203&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;ci*/roc.test/power.roc.test: vector or NA conf.level, kappa... gave
garbled results&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/debd4d06e6576ef01fc196bdeb005cb7f81a0ed7&quot;&gt;debd4d0&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-204&lt;/td&gt;
&lt;td&gt;auc&lt;/td&gt;
&lt;td&gt;auc: partial.auc.correct=1 mislabelled; integer bounds warned in
roc.test/cov&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/55edcaf7f40e172463645d56217ed745dcd7c510&quot;&gt;55edcaf&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-213&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;ci.coords(ret=&quot;all&quot;, non-threshold input): failed instead of
skipping threshold&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/57cde0ef1f1783a9105e7ed6febd067bebfd6b06&quot;&gt;57cde0e&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-218&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;are.paired/smooth(reuse.ci=TRUE): stored &quot;best&quot; ci.coords recomputed
with Youden&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/e67489e3bbd100d2eb3b15a8cd14c189a1db1f63&quot;&gt;e67489e&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-223&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;ci.thresholds(&quot;all&quot;, pAUC): quantile() error from window-limit
points&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/24724b539ab2770094bcfd31cbcb36e2f839277f&quot;&gt;24724b5&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-224&lt;/td&gt;
&lt;td&gt;coords&lt;/td&gt;
&lt;td&gt;coords(&quot;best&quot;)/plot.roc(print.thres): pAUC window without points
failed&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/6558c2e23c2bc22587e2240082139178f17e2fcf&quot;&gt;6558c2e&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-230&lt;/td&gt;
&lt;td&gt;plot&lt;/td&gt;
&lt;td&gt;plot.roc(print.thres, partial.auc, reuse.auc=FALSE): best ignored
the pAUC&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/d0ae66ef5fcee72e4261edfb6baf61a856e93333&quot;&gt;d0ae66e&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-231&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;ci.coords (count input, non-stratified): failed when a resample had
too few&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/a0c465b1e0f51f7870d15ac243443b9a0506c753&quot;&gt;a0c465b&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-232&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;cov/power.roc.test: no &#39;paired&#39; override; roc.test(paired=FALSE)
warned&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/e9fe809215a215b1fef189a39f5ba4e5b11efeba&quot;&gt;e9fe809&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-234&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;roc.test/cov (bootstrap): extra smoothing argument failed, wrote
pROC_bug.RData&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/24da24cc9deae82b752d054112235e18638c20c2&quot;&gt;24da24c&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DEV-1 (develop)&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;ci.coords: fractions instead of percentages on percent=TRUE
curves&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/bbf227037cce77dfde07bb978a88aa0b1560b0bc&quot;&gt;bbf2270&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DEV-2 (develop)&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;cov/roc.test (bootstrap): an NA replicate dropped a whole curve&#39;s
values&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/bbf227037cce77dfde07bb978a88aa0b1560b0bc&quot;&gt;bbf2270&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DEV-3 (develop)&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;ci.coords (smoothed): failed for every x unless input was given,
always for &quot;best&quot;&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/be0d834103e796ccf56f75fbe22f67ec2d469422&quot;&gt;be0d834&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DEV-4 (develop)&lt;/td&gt;
&lt;td&gt;power.roc.test&lt;/td&gt;
&lt;td&gt;power.roc.test: two partial AUC curves failed instead of using the
bootstrap&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/152c499cb0ca32f50ecb45a2ca164e9607c73fc5&quot;&gt;152c499&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DEV-5 (develop)&lt;/td&gt;
&lt;td&gt;ggroc&lt;/td&gt;
&lt;td&gt;ggroc: a mixed list of roc and smooth.roc curves failed&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/6072b2b69b23d7fa547a7b0a330d0a358d7ba69a&quot;&gt;6072b2b&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DEV-6 (develop)&lt;/td&gt;
&lt;td&gt;plot&lt;/td&gt;
&lt;td&gt;plot.roc(print.auc=TRUE) on a curve with a CI: AUC label drawn
twice&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/b2f09c425a102c6cca31cc7e83abba150beaf1ef&quot;&gt;b2f09c4&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DEV-8 (develop)&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;ci.coords(best.policy=&quot;omit&quot;): resamples with several best points
put thresholds in the rate columns&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/9afd18f6ffcdd90d24eff25a462f6c0792fb2246&quot;&gt;9afd18f&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h3 id=&quot;documentation-and-tests-77&quot;&gt;Documentation and tests (77)&lt;/h3&gt;
&lt;table class=&quot;smarttable&quot;&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Bug&lt;/th&gt;
&lt;th&gt;Area&lt;/th&gt;
&lt;th&gt;Fix&lt;/th&gt;
&lt;th&gt;Commit&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;BUG-007&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;?roc.test: wrong method list, one-sided Venkatraman behaviour and
return values&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/15107f5e5fb4b763838706e4107010b15f64b5ba&quot;&gt;15107f5&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-009&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;?var/?cov: documented a bootstrap fallback where the code errors;
obuchowski unlisted&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/1d650b808b4bddb080c4c04f1eb9a555b052abed&quot;&gt;1d650b8&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-010&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;var: Obuchowski errors said &quot;Using bootstrap instead&quot; though nothing
ran&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/1d650b808b4bddb080c4c04f1eb9a555b052abed&quot;&gt;1d650b8&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-012&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;roc.test: an explicit predictor2 = NULL gave an unrelated error&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/f4039ed72075129b5bd3ac198c0f95c342d24cf6&quot;&gt;f4039ed&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-015&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;?roc.test: conf.level range described as excluding 0 and 1&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/703175de8bcb43280ae536abd1803b54697725fb&quot;&gt;703175d&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-023&lt;/td&gt;
&lt;td&gt;roc&lt;/td&gt;
&lt;td&gt;roc: the deprecated &#39;algorithm&#39; warning showed a literal %s&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/aee1c0633572cac22af02cbcd7559f23c79c7226&quot;&gt;aee1c06&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-024&lt;/td&gt;
&lt;td&gt;roc&lt;/td&gt;
&lt;td&gt;roc (formula): cryptic stopifnot error for formulas without response
or with odd terms&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/9b5f3e30a52c8845f904331d396829065cf89a6a&quot;&gt;9b5f3e3&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-027&lt;/td&gt;
&lt;td&gt;roc&lt;/td&gt;
&lt;td&gt;?roc: said combined input sets were ignored (they give an
error)&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/767e7a66e0c500e3b9376fec7cff060e72a49caa&quot;&gt;767e7a6&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-028&lt;/td&gt;
&lt;td&gt;roc&lt;/td&gt;
&lt;td&gt;roc: the density error named response/predictor instead of
cases/controls&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/3f2f0d9e6672950e026d77254aab885162e6c6ef&quot;&gt;3f2f0d9&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-034&lt;/td&gt;
&lt;td&gt;other&lt;/td&gt;
&lt;td&gt;pROC: partial matching of list elements warned with
warnPartialMatchDollar&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/70c22fa97f89f62d4f2b78c7252085a2e508647e&quot;&gt;70c22fa&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-038&lt;/td&gt;
&lt;td&gt;multiclass&lt;/td&gt;
&lt;td&gt;multiclass.roc: false &quot;classes not found&quot; warning with a subset of
levels&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/5e95a52e9de7acad4c26cdf0b979456e0c6fdf7c&quot;&gt;5e95a52&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-039&lt;/td&gt;
&lt;td&gt;multiclass&lt;/td&gt;
&lt;td&gt;?multiclass.roc/?auc: wrong value description and smoothing
docs&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/e98d1db3475acdd9b4865b90bc8a51de43926ec4&quot;&gt;e98d1db&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-053&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;?ci.auc: documented a warning and fallback for DeLong partial AUC
(it errors)&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/6eb2944e03cbb5626b7606b73352db5016f7c91e&quot;&gt;6eb2944&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-054&lt;/td&gt;
&lt;td&gt;coords&lt;/td&gt;
&lt;td&gt;?coords: ignore.partial.auc description reversed&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/1fdcb96c859b8b7455bf408f4260183f359a6afe&quot;&gt;1fdcb96&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-055&lt;/td&gt;
&lt;td&gt;other&lt;/td&gt;
&lt;td&gt;?ci and others: claimed char/style were passed to
txtProgressBar&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/fc65cae73600b03ab3689ff684e75dcb4c9d4b6a&quot;&gt;fc65cae&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-073&lt;/td&gt;
&lt;td&gt;ggroc&lt;/td&gt;
&lt;td&gt;geom_polygon_auc: &quot;no applicable method&quot; on a curve without AUC&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/4f5d7f367adaca7796be7b9427a0eec894297f46&quot;&gt;4f5d7f3&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-077&lt;/td&gt;
&lt;td&gt;other&lt;/td&gt;
&lt;td&gt;Help pages: wrong volume (7, not 12) of the pROC paper&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/607b7c72296494eb0fe15f1b48df37a02a2fd402&quot;&gt;607b7c7&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-078&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;?ci.auc: the partial AUC example computed the full-AUC CI&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/ddbaf52eeed019ef1cc8b0dbbcfea5b83fb5817e&quot;&gt;ddbaf52&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-079&lt;/td&gt;
&lt;td&gt;roc&lt;/td&gt;
&lt;td&gt;?roc: example claimed smooth(roc()) has no AUC&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/e81dfbd741c5ea3cabde0a149a6a1bf53e90f1fb&quot;&gt;e81dfbd&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-080&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;?ci.coords: wrong class and a broken link in the value section&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/051a08fee2b29343e54be08199a372141f2250bc&quot;&gt;051a08f&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-081&lt;/td&gt;
&lt;td&gt;coords&lt;/td&gt;
&lt;td&gt;?coords: wrong sign in the closest.topleft formula&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/46219a28bba587588ae42ab535ebda8339d08e21&quot;&gt;46219a2&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-082&lt;/td&gt;
&lt;td&gt;coords&lt;/td&gt;
&lt;td&gt;?coords_transpose: wrong 1.16 default and deprecation
conditions&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/bb74e6bfb07b2154131c3d0369fdc9822b271027&quot;&gt;bb74e6b&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-083&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;?cov: wrong bootstrap covariance formula, replicate shape and
example&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/4610f1a405f02c26272d8df763fb3eab22f7d67e&quot;&gt;4610f1a&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-084&lt;/td&gt;
&lt;td&gt;plot&lt;/td&gt;
&lt;td&gt;?plot.roc: the example built roc.ndka from wfns&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/d9b8fbb4ab066b11be612525b504d0457a81bfa6&quot;&gt;d9b8fbb&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-086&lt;/td&gt;
&lt;td&gt;other&lt;/td&gt;
&lt;td&gt;?pROC: wrong number of aSAH patients (141, not 113)&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/382237cec110f2c58801a11be3be15e722136fa4&quot;&gt;382237c&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-087&lt;/td&gt;
&lt;td&gt;other&lt;/td&gt;
&lt;td&gt;FAQ vignette: stale and incorrect statements&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/de17f5839aa7e4eb54cf65e9c55ff692851df979&quot;&gt;de17f58&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-088a&lt;/td&gt;
&lt;td&gt;other&lt;/td&gt;
&lt;td&gt;?groupGeneric: aliases of nonexistent Ops/Math methods&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/2d7286e8e2c00d7b8d04026d11db5dd7e8ddb945&quot;&gt;2d7286e&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-088b&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;?ci.auc: the middle value was called the median for every
method&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/9ff1d45f5c4f1401314c076cd02b076fb6b699ae&quot;&gt;9ff1d45&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-088c&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;?ci.auc/?cov/?var: copy-pasted roc.test references&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/91516042bd98a1155550882916e0b814a6fd6d27&quot;&gt;9151604&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-088d&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;?ci.auc: duplicated usage line&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/f3ed0ee92c8b257fb07e4190177cf1efc67f9faa&quot;&gt;f3ed0ee&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-088e&lt;/td&gt;
&lt;td&gt;other&lt;/td&gt;
&lt;td&gt;Help pages: cited the plyr paper, no longer used&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/39ac7aec4c0145092d91cd3fca6b88f1d793e057&quot;&gt;39ac7ae&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-088f&lt;/td&gt;
&lt;td&gt;plot&lt;/td&gt;
&lt;td&gt;?lines.roc: wrong x argument names and return value&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/6cf69a94861a7c8b3b754630339c1a5c5ba8cbb4&quot;&gt;6cf69a9&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-088g&lt;/td&gt;
&lt;td&gt;plot&lt;/td&gt;
&lt;td&gt;?plot.ci: type=&quot;shape&quot; did not mention ci.coords&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/c2d01839ca792d6c7c25ca7507e54694e0821c93&quot;&gt;c2d0183&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-088h&lt;/td&gt;
&lt;td&gt;smooth&lt;/td&gt;
&lt;td&gt;?smooth: quoted a missing-package error pROC never gives&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/9226d5d012b047b33a5807bf030deffe4efdf065&quot;&gt;9226d5d&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-088i&lt;/td&gt;
&lt;td&gt;smooth&lt;/td&gt;
&lt;td&gt;?smooth: incomplete description of &#39;start&#39;&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/81c0de3caaf4aa5a62049705d3893eb5caf3dbd9&quot;&gt;81c0de3&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-088j&lt;/td&gt;
&lt;td&gt;smooth&lt;/td&gt;
&lt;td&gt;?smooth: percent and direction said to come from arguments, not the
curve&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/bda5ddf6d76d233acfcf850c119a8aaf2d693a77&quot;&gt;bda5ddf&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-088l&lt;/td&gt;
&lt;td&gt;other&lt;/td&gt;
&lt;td&gt;?aSAH: one laboratory variable instead of two&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/fcd27fa99b612e761b2327183f40d396ef8c0d69&quot;&gt;fcd27fa&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-088m&lt;/td&gt;
&lt;td&gt;other&lt;/td&gt;
&lt;td&gt;?aSAH: example comment named a nonexistent &#39;set&#39; variable&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/75c269e406caadf5b531c0f594e7cfa624d427db&quot;&gt;75c269e&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-088o&lt;/td&gt;
&lt;td&gt;auc&lt;/td&gt;
&lt;td&gt;?auc: wrong destination of auc.smooth.roc&#39;s ...&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/65959d0a0c457973453d1b2af4461d818d708452&quot;&gt;65959d0&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-088p&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;?roc.test: wrong unpaired DeLong statistic formula&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/a5f77785bcd26225d5dc85ae0be296092fe95f62&quot;&gt;a5f7778&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-088r&lt;/td&gt;
&lt;td&gt;other&lt;/td&gt;
&lt;td&gt;Help pages: spelling mistakes, including in cited references&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/30c8670ce836b5c69eae41bfe17624068df959c7&quot;&gt;30c8670&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-089&lt;/td&gt;
&lt;td&gt;multiclass&lt;/td&gt;
&lt;td&gt;ci.se: multiclass error message named ci.sp&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/ba4000a0baed9a671055e3343195a9a254b39e5f&quot;&gt;ba4000a&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-092&lt;/td&gt;
&lt;td&gt;coords&lt;/td&gt;
&lt;td&gt;coords (smoothed): each deprecation warning given two or three
times&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/d03dce35be1fd8c648c4ebe3a0b94fe291ffefd9&quot;&gt;d03dce3&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-093&lt;/td&gt;
&lt;td&gt;roc&lt;/td&gt;
&lt;td&gt;?roc: said na.rm is ignored by roc.formula&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/bcf6e97794d2a739c74299084923bcbf6e59a8e7&quot;&gt;bcf6e97&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-094&lt;/td&gt;
&lt;td&gt;coords&lt;/td&gt;
&lt;td&gt;tests: the coords se/sp range test never used its smoothed percent
curve&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/5c9e145e775b5060dbc5363c3d744c3aeb1b52d2&quot;&gt;5c9e145&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-104&lt;/td&gt;
&lt;td&gt;plot&lt;/td&gt;
&lt;td&gt;plot.roc(ci=TRUE) on a curve without CI: cryptic &quot;need finite
&#39;xlim&#39;&quot; error&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/ae078fa3022b0b2f4cd15384b3b75483e342e734&quot;&gt;ae078fa&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-111&lt;/td&gt;
&lt;td&gt;coords&lt;/td&gt;
&lt;td&gt;?coords: did not say &quot;local maximas&quot; is unavailable on smoothed
curves&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/f1673eabf6fd76f0be0121d1ccce5d70c6468ae2&quot;&gt;f1673ea&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-125&lt;/td&gt;
&lt;td&gt;ggroc&lt;/td&gt;
&lt;td&gt;ggroc: partial argument match warning (&#39;lim&#39;) with
warnPartialMatchArgs&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/503079cf2a1803a4798b838debb7c86f33a2188e&quot;&gt;503079c&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-127&lt;/td&gt;
&lt;td&gt;roc&lt;/td&gt;
&lt;td&gt;?roc: how direction=&quot;auto&quot; is chosen was never described&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/ef34a6c874dfa606b14f285c3f4a1a1e5f937cc7&quot;&gt;ef34a6c&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-128&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;?roc.test/?cov: re-pairing of curves on common observations
undocumented&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/ea0a6e33a7912faa38829586820b49eeec87ff1c&quot;&gt;ea0a6e3&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-130&lt;/td&gt;
&lt;td&gt;roc&lt;/td&gt;
&lt;td&gt;roc/auc/ci* (response, predictor): cryptic error for a data.frame
response&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/162d3928af506750660ccdf0746dc8fe87a7d3ca&quot;&gt;162d392&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-131&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;?roc.test: misleading description of na.rm = FALSE&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/e20cabffd813cf902ce323ec1b8abb6834893e4a&quot;&gt;e20cabf&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-134&lt;/td&gt;
&lt;td&gt;roc&lt;/td&gt;
&lt;td&gt;?roc: wrong names of the arguments roc() forwards&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/4b3b20163f7bdd7dfe628c9d25747d2dd402a653&quot;&gt;4b3b201&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-136&lt;/td&gt;
&lt;td&gt;multiclass&lt;/td&gt;
&lt;td&gt;ci* (multivariate multiclass): misleading &quot;No valid data provided&quot;
error&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/a2c1682a77c7b827ff750d220a0f9758f507015b&quot;&gt;a2c1682&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-137&lt;/td&gt;
&lt;td&gt;roc&lt;/td&gt;
&lt;td&gt;?roc: did not say ci.coords&#39; ret cannot go through roc_&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/d2a918e8e221f94844b0f8159129125edde90b8e&quot;&gt;d2a918e&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-138&lt;/td&gt;
&lt;td&gt;other&lt;/td&gt;
&lt;td&gt;?smooth/?roc.test: wrong Zou, Rufibach and Pepe references&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/14460c6b24147cdcc4c0c17bbb2f66c5d3b1cbb0&quot;&gt;14460c6&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-139&lt;/td&gt;
&lt;td&gt;multiclass&lt;/td&gt;
&lt;td&gt;?multiclass.roc/?auc: univariate AUC wrongly attributed to Hand and
Till&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/3aeb7f26d16fdf804eb34fbc0ae2cfcc8c27b306&quot;&gt;3aeb7f2&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-147&lt;/td&gt;
&lt;td&gt;multiclass&lt;/td&gt;
&lt;td&gt;multiclass.roc: cryptic error for a data.frame response&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/162d3928af506750660ccdf0746dc8fe87a7d3ca&quot;&gt;162d392&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-148&lt;/td&gt;
&lt;td&gt;multiclass&lt;/td&gt;
&lt;td&gt;multiclass.roc(smooth=TRUE): cryptic &quot;$ operator&quot; error&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/552a364d2bacfe3f7f0cae63963f11b45ae63045&quot;&gt;552a364&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-149&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;var/ci.se/ci.sp/ci.coords (smoothed, corrected pAUC): a warning per
replicate&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/0de1e677aef75a40fe739340ebb352941e5d7492&quot;&gt;0de1e67&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-152&lt;/td&gt;
&lt;td&gt;multiclass&lt;/td&gt;
&lt;td&gt;?multiclass.roc: levels field wrong, rocs field undocumented (code
part reverted)&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/bd24db628b4f8ce9efeafcbd1203862328bc03b2&quot;&gt;bd24db6&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-154&lt;/td&gt;
&lt;td&gt;smooth&lt;/td&gt;
&lt;td&gt;smooth(fitdistr, logcondens): cryptic optim/logcondens errors for a
single-value class&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/ec0b962ad0296af8fa8befb894a99b7fb79d1633&quot;&gt;ec0b962&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-155&lt;/td&gt;
&lt;td&gt;power.roc.test&lt;/td&gt;
&lt;td&gt;power.roc.test: errors asked for an &#39;auc&#39; that was given or not
accepted&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/85b253c391578fc12a0c21fad9ae79a551a330f7&quot;&gt;85b253c&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-166b&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;ci.auc (smoothed curve): method=&quot;delong&quot; silently ignored (now
warns)&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/ae4327a2bb8454f2177e59e05cef725866907f3d&quot;&gt;ae4327a&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-181&lt;/td&gt;
&lt;td&gt;roc&lt;/td&gt;
&lt;td&gt;?roc: said na.action is not supported by the formula method&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/aac4e69a4c73bc20e659c8ebd0861d4b692e99fd&quot;&gt;aac4e69&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-184&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;?roc.test: wrong per-method details (boot.n, estimates,
warnings)&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/e568f7e639ffb6820100ce749ea987923c08dbfd&quot;&gt;e568f7e&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-187&lt;/td&gt;
&lt;td&gt;plot&lt;/td&gt;
&lt;td&gt;?plot.roc: x and value said &quot;roc&quot; only, not smooth.roc&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/ae078fa3022b0b2f4cd15384b3b75483e342e734&quot;&gt;ae078fa&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-188&lt;/td&gt;
&lt;td&gt;ci&lt;/td&gt;
&lt;td&gt;?ci.se/?ci.sp: quoted a density error that doesn&#39;t exist&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/8e96fc6a9eabeec71d348f85a3ecd067c9da4972&quot;&gt;8e96fc6&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-197&lt;/td&gt;
&lt;td&gt;coords&lt;/td&gt;
&lt;td&gt;coords/ci.coords: misspelt ret elements silently dropped (now
warns)&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/090b467fbcf87d26ca917ee3158469d31ef68b31&quot;&gt;090b467&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-207&lt;/td&gt;
&lt;td&gt;multiclass&lt;/td&gt;
&lt;td&gt;auc (multiclass): each pairwise warning emitted twice&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/276533146a4aaac4232d44a84d23c93d719e1e92&quot;&gt;2765331&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-221&lt;/td&gt;
&lt;td&gt;smooth&lt;/td&gt;
&lt;td&gt;?smooth: &#39;start&#39; documented as optional where fitdistr requires
it&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/cc977b6102d75064d6a2556cd0a3a56968aa1738&quot;&gt;cc977b6&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-222&lt;/td&gt;
&lt;td&gt;smooth&lt;/td&gt;
&lt;td&gt;smooth(fitdistr): a fixed parameter of a closed-form density gave a
cryptic error&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/ec0b962ad0296af8fa8befb894a99b7fb79d1633&quot;&gt;ec0b962&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-227&lt;/td&gt;
&lt;td&gt;other&lt;/td&gt;
&lt;td&gt;coords/roc warnings: typos &quot;cased by&quot; and &quot;values(s)&quot;&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/d7fd7b42996c790ec21a8646cf3228c805026862&quot;&gt;d7fd7b4&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-233&lt;/td&gt;
&lt;td&gt;multiclass&lt;/td&gt;
&lt;td&gt;multiclass.roc: misleading error when only one matrix column
matched&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/2396fd16f53f80003805b73cfde597b7774246aa&quot;&gt;2396fd1&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BUG-235&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;roc.test: curves it returned warned &quot;Different smoothing parameters&quot;
when reused&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/24da24cc9deae82b752d054112235e18638c20c2&quot;&gt;24da24c&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DEV-7 (develop)&lt;/td&gt;
&lt;td&gt;plot&lt;/td&gt;
&lt;td&gt;plot.ci.*: low-level &quot;plot.new has not been called&quot; error with no
plot open&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/4e084c364bb59384a70d75c7e4e527fd33c3338c&quot;&gt;4e084c3&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DEV-9 (develop)&lt;/td&gt;
&lt;td&gt;other&lt;/td&gt;
&lt;td&gt;parallel: the documented deprecation warning was never given&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/c332ddf4d512eacac2021555c388684d72f9d14d&quot;&gt;c332ddf&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DEV-10 (develop)&lt;/td&gt;
&lt;td&gt;roc.test/cov/var&lt;/td&gt;
&lt;td&gt;var and cov with method = &quot;obuchowski&quot; ignored partial.auc.correct&lt;/td&gt;
&lt;td&gt;&lt;a
href=&quot;https://github.com/xrobin/pROC/commit/f4d568e7876f346db56d2b43d617f6097151a1f5&quot;&gt;f4d568e&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
</content>
		</entry>

	<entry xml:lang="en" xml:base="https://xavier.robin.info/fr/">
		<title type="html">Breaking changes for ordered predictors in upcoming pROC 1.20.0</title>
		
			<category term="pROC" label="pROC" scheme="https://xavier.robin.info/fr/tag/pROC" />
		
		<link href="https://xavier.robin.info/fr/blog/2026/10/06/breaking-changes-for-ordered-predictors-in-upcoming-proc-1.20.0"/>
		<id>tag:xavier.robin.info,2026-10-06:/blog/2026/10/06/breaking-changes-for-ordered-predictors-in-upcoming-proc-1.20.0</id>
		<published>2026-10-06T14:26:48+02:00</published>
		<updated>2026-10-06T14:26:48+02:00</updated>
		<content type="html">&lt;p&gt;pROC has always accepted &lt;code&gt;ordered&lt;/code&gt; factors as predictors (for example &lt;code&gt;aSAH$wfns&lt;/code&gt;,
the WFNS neurological grade in the &lt;code&gt;aSAH&lt;/code&gt; dataset, with levels &lt;code&gt;1&lt;/code&gt; to &lt;code&gt;5&lt;/code&gt;).
Internally it simply called &lt;code&gt;as.numeric()&lt;/code&gt; on them and computed thresholds as the midpoints
between consecutive numeric codes (&lt;code&gt;1.5&lt;/code&gt;, &lt;code&gt;2.5&lt;/code&gt;, and so on).&lt;/p&gt;

&lt;p&gt;That&#39;s reasonable when the levels happen to be the numbers &lt;code&gt;1:n&lt;/code&gt;, but it is
not appropriate for an ordered factor whose levels are textual, for example &lt;code&gt;&quot;very low&quot;&lt;/code&gt;,
&lt;code&gt;&quot;low&quot;&lt;/code&gt;, &lt;code&gt;&quot;medium&quot;&lt;/code&gt;, &lt;code&gt;&quot;high&quot;&lt;/code&gt;, &lt;code&gt;&quot;very high&quot;&lt;/code&gt;. The thresholds
were still &lt;code&gt;-Inf, 1.5, 2.5, 3.5, 4.5, Inf&lt;/code&gt;, numbers that do not correspond to anything in
the data and are meaningless to show. This was reported as
&lt;a href=&quot;https://github.com/xrobin/pROC/issues/63&quot;&gt;issue #63&lt;/a&gt;, and a fix for it was just merged into
the &lt;code&gt;develop&lt;/code&gt; branch.&lt;/p&gt;

&lt;p&gt;In practice, this could potentially break some of your code. Before it reaches CRAN, I&#39;d like testers
who use ordered predictors to try the &lt;code&gt;develop&lt;/code&gt; version on their own data and tell me if
anything looks wrong.&lt;/p&gt;

&lt;h2&gt;What changes&lt;/h2&gt;

&lt;p&gt;On &lt;code&gt;develop&lt;/code&gt;, thresholds of an ordered ROC curve are now the predictor&#39;s own levels, plus
one infinite sentinel for the all-positive or all-negative cut-off. &lt;strong&gt;Sensitivities, specificities
and AUC are exactly the same as before&lt;/strong&gt;, only the threshold labels change, from numeric
midpoints to the levels you actually gave it:&lt;/p&gt;

&lt;pre&gt;library(pROC)
data(aSAH)

wfns.txt &amp;lt;- aSAH$wfns
levels(wfns.txt) &amp;lt;- c(&quot;very low&quot;, &quot;low&quot;, &quot;medium&quot;, &quot;high&quot;, &quot;very high&quot;)

r &amp;lt;- roc(aSAH$outcome, wfns.txt, quiet = TRUE)
r$thresholds
# very low, low, medium, high, very high, Inf
# Levels: very low &amp;lt; low &amp;lt; medium &amp;lt; high &amp;lt; very high &amp;lt; Inf

coords(r, &quot;medium&quot;)
#   threshold specificity sensitivity
# 1    medium   0.7916667   0.6585365&lt;/pre&gt;

&lt;p&gt;This is a &lt;strong&gt;breaking change&lt;/strong&gt; for any code that reads &lt;code&gt;roc$thresholds&lt;/code&gt; as numbers,
or passes a numeric cut-off into &lt;code&gt;coords()&lt;/code&gt;, &lt;code&gt;ci.thresholds()&lt;/code&gt;, &lt;code&gt;ci.coords()&lt;/code&gt;,
or &lt;code&gt;plot.roc(print.thres = ...)&lt;/code&gt; on an ordered ROC curve. Numeric predictors are entirely
unaffected by this change. Where you used to pass a midpoint, pass the level label instead:&lt;/p&gt;

&lt;pre&gt;# Incorrect (pROC &amp;lt;= 1.19.1 converted wfns to 1:5, thresholds were midpoints):
coords(roc(aSAH$outcome, aSAH$wfns), 2.5)

# Correct: pass the level label
coords(roc(aSAH$outcome, aSAH$wfns, quiet = TRUE), &quot;3&quot;)&lt;/pre&gt;

&lt;p&gt;The mapping from old midpoints to new labels is straightforward (for &lt;code&gt;direction = &quot;&amp;lt;&quot;&lt;/code&gt;,
the default): the old &lt;code&gt;-Inf&lt;/code&gt; becomes the lowest level, the old midpoint &lt;code&gt;k + 0.5&lt;/code&gt;
becomes the level in position &lt;code&gt;k + 1&lt;/code&gt;, and the old &lt;code&gt;Inf&lt;/code&gt; becomes the level literally
named &lt;code&gt;&quot;Inf&quot;&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;As a side effect of the same fix, &lt;code&gt;ci.coords(..., x = &quot;best&quot;, best.policy = &quot;omit&quot;)&lt;/code&gt; now
correctly &lt;em&gt;drops&lt;/em&gt; a bootstrap replicate that has several equally-good &quot;best&quot; points, instead of
silently keeping the first one.&lt;/p&gt;

&lt;h2&gt;How to test it&lt;/h2&gt;

&lt;p&gt;This is &lt;strong&gt;not released yet&lt;/strong&gt;: it is only on the &lt;code&gt;develop&lt;/code&gt; branch, ahead of the
next CRAN release, planned for early 2027. If you use ordered predictors with &lt;code&gt;roc()&lt;/code&gt;, please
install the development version and run your own code against it:&lt;/p&gt;

&lt;pre&gt;if (! requireNamespace(&quot;devtools&quot;)) install.packages(&quot;devtools&quot;)
devtools::install_github(&quot;xrobin/pROC@develop&quot;)&lt;/pre&gt;

&lt;h2&gt;Recommendations&lt;/h2&gt;

&lt;p&gt;If you have scripts or packages that build ROC curves from ordered predictors, please run them against
the &lt;code&gt;develop&lt;/code&gt; version and check the following:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;Any place you hard-code a numeric threshold for an ordered predictor (in &lt;code&gt;coords()&lt;/code&gt;,
		&lt;code&gt;ci.thresholds()&lt;/code&gt;, &lt;code&gt;ci.coords()&lt;/code&gt;, or &lt;code&gt;print.thres&lt;/code&gt;) will need to
		switch to the level label.&lt;/li&gt;
	&lt;li&gt;&lt;code&gt;roc()&lt;/code&gt; now requires &lt;code&gt;cases&lt;/code&gt; and &lt;code&gt;controls&lt;/code&gt; built from ordered
		vectors to have &lt;strong&gt;identical&lt;/strong&gt; levels; it errors instead of silently coercing when they
		differ (for example after an unbalanced &lt;code&gt;droplevels()&lt;/code&gt;).&lt;/li&gt;
	&lt;li&gt;Smoothing an ordered ROC curve now only supports &lt;code&gt;method = &quot;binormal&quot;&lt;/code&gt;; other smoothing
		methods error instead of silently falling back to a numeric conversion.&lt;/li&gt;
	&lt;li&gt;AUC, DeLong and bootstrap CIs, &lt;code&gt;roc.test()&lt;/code&gt;, &lt;code&gt;cov()&lt;/code&gt;/&lt;code&gt;var()&lt;/code&gt;, and
		&lt;code&gt;power.roc.test()&lt;/code&gt; all still match what you got before. Only the thresholds
		changed, not the statistics.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If anything looks off, please comment on &lt;a href=&quot;https://github.com/xrobin/pROC/issues/63&quot;&gt;issue #63&lt;/a&gt;
or open a new issue on &lt;a href=&quot;https://github.com/xrobin/pROC/issues&quot;&gt;GitHub&lt;/a&gt; before this reaches CRAN.
Thanks in advance for testing!&lt;/p&gt;
</content>
		</entry>

	<entry xml:lang="en" xml:base="https://xavier.robin.info/fr/">
		<title type="html">pROC 1.19.1</title>
		
			<category term="pROC" label="pROC" scheme="https://xavier.robin.info/fr/tag/pROC" />
		
		<link href="https://xavier.robin.info/fr/blog/2026/08/27/proc-1.19.1"/>
		<id>tag:xavier.robin.info,2026-08-27:/blog/2026/08/27/proc-1.19.1</id>
		<published>2026-08-27T13:10:29+02:00</published>
		<updated>2026-08-27T13:10:29+02:00</updated>
		<content type="html">&lt;p&gt;pROC 1.19.1 is a minor maintenance release. The only user-facing change is the introduction of a &lt;a href=&quot;https://cran.r-project.org/web/packages/pROC/vignettes/FAQ.html&quot;&gt;FAQ vignette&lt;/a&gt; which supersedes the old wiki.&lt;/p&gt;

&lt;p&gt;Here is the full changelog:&lt;/p&gt;

&lt;ul&gt;
    &lt;li&gt;Added a FAQ vignette&lt;/li&gt;
    &lt;li&gt;Removed unused &lt;code&gt;tcltk&lt;/code&gt; from Suggests&lt;/li&gt;
    &lt;li&gt;Tests now skip if Suggest&#39;ed packages are missing&lt;/li&gt;
    &lt;li&gt;Fix CRAN checks NOTE about &lt;code&gt;structure()&lt;/code&gt; special names (&lt;code&gt;.Names&lt;/code&gt;, &lt;code&gt;.Dim&lt;/code&gt;, &lt;code&gt;.Dimnames&lt;/code&gt;)&lt;/li&gt;
&lt;/ul&gt;
</content>
		</entry>

	<entry xml:lang="en" xml:base="https://xavier.robin.info/fr/">
		<title type="html">pROC 1.19.0</title>
		
			<category term="pROC" label="pROC" scheme="https://xavier.robin.info/fr/tag/pROC" />
		
		<link href="https://xavier.robin.info/fr/blog/2025/07/30/proc-1.19.0"/>
		<id>tag:xavier.robin.info,2025-07-30:/blog/2025/07/30/proc-1.19.0</id>
		<published>2025-07-30T18:56:51+02:00</published>
		<updated>2025-07-31T12:32:11+02:00</updated>
		<content type="html">&lt;p&gt;pROC version 1.19.0 was just released and will be available on CRAN very soon.&lt;/p&gt;

&lt;p&gt;Besides minor changes and fixes in the &lt;code&gt;coords&lt;/code&gt; and &lt;code&gt;ci.coords&lt;/code&gt; functions, 
  the main updates in this version focus on the core of the package, aiming to make it more modern, efficient, and easier
  to maintain. Several features that were difficult to maintain have been deprecated.&lt;/p&gt;
&lt;ul&gt;

  &lt;li&gt;The dependency on the retired &lt;a href=&quot;http://had.co.nz/plyr/&quot;&gt;plyr&lt;/a&gt; package has been removed (thanks to Michael Chirico for his contributions). 
    Unfortunately, as a side effect, progress bars and parallel processing have been removed. 
    Trying to set the &lt;code&gt;progress&lt;/code&gt; and &lt;code&gt;parallel&lt;/code&gt; arguments will now trigger a warning, and the arguments will be ignored.&lt;/li&gt;

  &lt;li&gt;As a followup from the changes to the output of the &lt;code&gt;coords&lt;/code&gt; function in version 1.16.0, the
    &lt;code&gt;transpose&lt;/code&gt;, &lt;code&gt;as.list&lt;/code&gt;, &lt;code&gt;as.matrix&lt;/code&gt; and &lt;code&gt;drop&lt;/code&gt; arguments have been
    deprecated.
    The &lt;code&gt;coords&lt;/code&gt; function currently has multiple exit points, depending on arguments and inputs, and can return
    retults in mutliple formats. This makes it difficult to use and to maintain.
    Going forward, &lt;code&gt;coords&lt;/code&gt; will only retrun data in a single, tidy &lt;code&gt;data.frame&lt;/code&gt; format, compatible following modern R
    coding practices.
    For compatibility reasons, deprecated arguments are still available, but setting them to non default
    values will trigger warnings. They will be removed in a future release.
    If your code still uses these arguments, please update it accordingly.&lt;/li&gt;

  &lt;li&gt;Finally, following years of performance improvements, an in an effort to simplify the codebase, all ROC computation
    algorithms other than 2 have been removed as they no longer provided meaningful performance advantages. The &lt;code&gt;algorithm&lt;/code&gt;
    argument to &lt;code&gt;roc&lt;/code&gt; has been deprecated. Setting it to a non-default value has no effect and triggers a warning.
    The &lt;code&gt;fun.sesp&lt;/code&gt; value of &lt;code&gt;roc&lt;/code&gt; objects is also deprecated. Calling it triggers a warning.
    Both will be removed in a future release of pROC.&lt;/li&gt;
&lt;/ul&gt;
	
&lt;p&gt;Here is the full changelog:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;&lt;code&gt;ci.coords&lt;/code&gt; can now take the same &lt;code&gt;input&lt;/code&gt; values as &lt;code&gt;coords&lt;/code&gt; (&lt;a href=&quot;https://github.com/xrobin/pROC/issues/90&quot;&gt;issue #90&lt;/a&gt;)&lt;/li&gt;
	&lt;li&gt;&lt;code&gt;ci.coords&lt;/code&gt; can be &lt;code&gt;plot&lt;/code&gt;ted&lt;/li&gt;
	&lt;li&gt;Added &quot;lr_pos&quot; and &quot;lr_neg&quot; to &lt;code&gt;coords&lt;/code&gt; (&lt;a href=&quot;https://github.com/xrobin/pROC/issues/102&quot;&gt;issue #102&lt;/a&gt;)&lt;/li&gt;
	&lt;li&gt;&lt;code&gt;coords&lt;/code&gt; with partial.auc now interpolates bounds when needed&lt;/li&gt;
	&lt;li&gt;Added &lt;code&gt;ignore.partial.auc&lt;/code&gt; argument to &lt;code&gt;coords&lt;/code&gt;&lt;/li&gt;
	&lt;li&gt;Deprecated &lt;code&gt;transpose&lt;/code&gt;, &lt;code&gt;as.list&lt;/code&gt;, &lt;code&gt;as.matrix&lt;/code&gt; and &lt;code&gt;drop&lt;/code&gt; in &lt;code&gt;coords&lt;/code&gt;&lt;/li&gt;
	&lt;li&gt;Deprecated the &lt;code&gt;algorithm&lt;/code&gt; argument to &lt;code&gt;roc&lt;/code&gt; and &lt;code&gt;fun.sesp&lt;/code&gt; value&lt;/li&gt;
	&lt;li&gt;Deprecated the &lt;code&gt;progress&lt;/code&gt; and &lt;code&gt;parallel&lt;/code&gt; argument for bootstrap operations.&lt;/li&gt;
	&lt;li&gt;Removed dependencies on &lt;a href=&quot;https://cran.r-project.org/web/packages/doParallel/index.html&quot;&gt;doParallel&lt;/a&gt; and retired package &lt;a href=&quot;http://had.co.nz/plyr/&quot;&gt;plyr&lt;/a&gt; (thanks to Michael Chirico, &lt;a href=&quot;https://github.com/xrobin/pROC/pull/134&quot;&gt;pr #134&lt;/a&gt;, &lt;a href=&quot;https://github.com/xrobin/pROC/pull/135&quot;&gt;#135&lt;/a&gt;, &lt;a href=&quot;https://github.com/xrobin/pROC/pull/136&quot;&gt;#136&lt;/a&gt;, &lt;a href=&quot;https://github.com/xrobin/pROC/pull/137&quot;&gt;#137&lt;/a&gt;, &lt;a href=&quot;https://github.com/xrobin/pROC/pull/138&quot;&gt;#138&lt;/a&gt;, &lt;a href=&quot;https://github.com/xrobin/pROC/pull/139&quot;&gt;#139&lt;/a&gt; and &lt;a href=&quot;https://github.com/xrobin/pROC/pull/140&quot;&gt;#140&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;You can update your installation by simply typing:&lt;/p&gt;

&lt;pre&gt;install.packages(&quot;pROC&quot;)&lt;/pre&gt;

&lt;hr&gt;

&lt;p&gt;Update: pROC 1.19.0 was rejected from CRAN. A patch revision 1.19.0.1 was created to workaround an issue with a reverse dependency but provides no meaningful change:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;Move &lt;code&gt;fun.sesp&lt;/code&gt; definition to work around &lt;a href=&quot;https://github.com/LudvigOlsen/cvms/pull/44&quot;&gt;LudvigOlsen/cvms#44&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;</content>
		</entry>

	<entry xml:lang="en" xml:base="https://xavier.robin.info/fr/">
		<title type="html">pROC 1.18.5</title>
		
			<category term="pROC" label="pROC" scheme="https://xavier.robin.info/fr/tag/pROC" />
		
		<link href="https://xavier.robin.info/fr/blog/2023/11/02/proc-1.18.5"/>
		<id>tag:xavier.robin.info,2023-11-02:/blog/2023/11/02/proc-1.18.5</id>
		<published>2023-11-02T17:01:20+01:00</published>
		<updated>2023-11-02T17:01:20+01:00</updated>
		<content type="html">&lt;p&gt;pROC 1.18.5 is now available on CRAN. It&#39;s a minor bugfix release:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;Fixed formula input when given as variable and combined with &lt;code&gt;with&lt;/code&gt; (&lt;a href=&quot;https://github.com/xrobin/pROC/issues/111&quot;&gt;issue #111&lt;/a&gt;)&lt;/li&gt;
	&lt;li&gt;Fixed formula containing variables with spaces (&lt;a href=&quot;https://github.com/xrobin/pROC/issues/120&quot;&gt;issue #120&lt;/a&gt;)&lt;/li&gt;
	&lt;li&gt;Fixed broken grouping when &lt;code&gt;colour&lt;/code&gt; argument was given in &lt;code&gt;ggroc&lt;/code&gt; (&lt;a href=&quot;https://github.com/xrobin/pROC/issues/121&quot;&gt;issue #121&lt;/a&gt;)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;You can update your installation by simply typing:&lt;/p&gt;

&lt;pre&gt;install.packages(&quot;pROC&quot;)&lt;/pre&gt;</content>
		</entry>

	<entry xml:lang="en" xml:base="https://xavier.robin.info/fr/">
		<title type="html">Deep Learning of MNIST handwritten digits</title>
		
			<category term="programming" label="Programmation" scheme="https://xavier.robin.info/fr/tag/programming" />
		
		<link href="https://xavier.robin.info/fr/blog/2022/06/11/deep-learning-of-mnist-handwritten-digits"/>
		<id>tag:xavier.robin.info,2022-06-11:/blog/2022/06/11/deep-learning-of-mnist-handwritten-digits</id>
		<published>2022-06-11T16:39:25+02:00</published>
		<updated>2022-06-11T16:39:25+02:00</updated>
		<content type="html">&lt;p&gt;In this document I am going create a video showing the training of the inner-most layer of Deep Belief Network (DBN) using the MNIST dataset of handwritten digits. I will use our &lt;code&gt;DeepLearning&lt;/code&gt; R package that implements flexible DBN architectures with an object-oriented interface.&lt;/p&gt;

&lt;h2&gt;MNIST&lt;/h2&gt;
&lt;p&gt;The MNIST dataset is a database of handwritten digits with 60,000 training images and 10,000 testing images. &lt;a href=&quot;https://en.wikipedia.org/wiki/MNIST_database&quot; title=&quot;Wikipedia: MNIST database&quot;&gt;You can learn everything about it on Wikipedia&lt;/a&gt;. In short, it is the go-to dataset to train and test handwritten digit recognition machine learning algorithms.&lt;/p&gt;


&lt;p&gt;I made an R package for easy access, named &lt;code&gt;mnist&lt;/code&gt;. The easiest way to install it is with &lt;code&gt;devtools&lt;/code&gt;. If you don&#39;t have it already, let&#39;s first install &lt;code&gt;devtools&lt;/code&gt;:&lt;/p&gt;
&lt;pre&gt;
if (!require(&quot;devtools&quot;)) {install.packages(&quot;devtools&quot;)}
&lt;/pre&gt;

&lt;p&gt;Now we can install &lt;code&gt;mnist&lt;/code&gt;:&lt;/p&gt;

&lt;pre&gt;
devtools::install_github(&quot;xrobin/mnist&quot;)
&lt;/pre&gt;

&lt;h2&gt;PCA&lt;/h2&gt;
&lt;p&gt;In order to see what the dataset looks like, let&#39;s use PCA to reduce it to two dimensions.&lt;/p&gt;

&lt;pre&gt;
pca &amp;lt;- prcomp(mnist$train$x)
plot.mnist(
	prediction = predict(pca, mnist$test$x),
	reconstruction = tcrossprod(
		predict(pca, mnist$test$x)[,1:2], pca$rotation[,1:2]),
	highlight.digits = c(72, 3, 83, 91, 6688, 7860, 92, 1, 180, 13))
&lt;/pre&gt;

&lt;p&gt;&lt;img src=&quot;/files/blog/2022/06/11/pca.png&quot; style=&quot;max-width: 100%&quot; alt=&quot;PCA Scatterplot&quot;&gt;&lt;/p&gt;
	
&lt;p&gt;Let&#39;s take a minute to describe this plot.
The central scatterplot shows first two components of the PCA of all digits in the test set.
On the left hand side, I picked 10 representative digits from the test set to highlight, which are shown as larger circles in the central scatterplot.
On the left are the &quot;reconstructed digits&quot;, which were reconstructed from the two first dimensions of the PCA. While we can see some digit-like structures, it is basically impossible to recognize them.
We can see some separation of the digits in the 2D space as well, but it is pretty weak and some pairs cannot be distinguished at all (like 4 and 9).
Of course the reconstructions would look much better had we kept all PCA dimensions, but so much for dimensionality reduction.&lt;/p&gt;


&lt;h2&gt;Deep Learning&lt;/h2&gt;
&lt;p&gt;Now let&#39;s see if we can do better with Deep Learning. We&#39;ll use a classical Deep Belief Network (DBN), based on Restricted Boltzmann Machines (RBM) similar to what Hinton described back in 2006 (Hinton &amp;amp; Salakhutdinov, 2006). The training happens in two steps: a pre-training step with contrastive divergence stochastic gradient descent brings the network to a reasonable starting point for a more conventional conjugate gradient optimization (hereafter referred to as fine-tuning).&lt;/p&gt;

&lt;p&gt;I implemented this algorithm with a few modifications in an R package which is available on GitHub. The core of the processing is done in C++ with &lt;a href=&quot;https://cran.r-project.org/web/packages/RcppEigen/index.html&quot;&gt;RcppEigen&lt;/a&gt; (Bates &amp;amp; Eddelbuettel, 2013) for higher speed. Using &lt;code&gt;devtools&lt;/code&gt; again:&lt;/p&gt;
&lt;pre&gt;
devtools::install_github(&quot;xrobin/DeepLearning&quot;)
&lt;/pre&gt;

&lt;p&gt;We will use this code to train a 5 layers deep network, that reduces the digits to an abstract, 2D representation. By looking at this last layer throughout the training process we can start to understand how the network learns to recognize digits. Let&#39;s start by loading the required packages and the MNIST dataset, and create the DBN.&lt;/p&gt;

&lt;pre&gt;library(DeepLearning)
library(mnist)
data(mnist)

dbn &amp;lt;- DeepBeliefNet(Layers(c(784, 1000, 500, 250, 2),
	input = &quot;continuous&quot;, output = &quot;gaussian&quot;),
	initialize = &quot;0&quot;)
&lt;/pre&gt;
&lt;p&gt;We just created the 5-layers DBN, with continuous, 784 nodes input (the digit image pixels), and a 2 nodes, gaussian output. It is initialized with 0, but we could have left out the &lt;code&gt;initialize&lt;/code&gt; to start from a random initilization (Bengio &lt;i&gt;et al.&lt;/i&gt;, 2007). Before we go, let&#39;s define a few useful variables:&lt;/p&gt;

&lt;pre&gt;
output.folder &amp;lt;- &quot;video&quot; # Where to save the output
maxiters.pretrain &amp;lt;- 1e6  # Number of pre-training iterations
maxiters.train &amp;lt;- 10000 # Number of fine-tuning iterations
run.training &amp;lt;- run.images &amp;lt;- TRUE # Turn any of these off 
# Which digits to highlight and reconstruct
highlight.digits = c(72, 3, 83, 91, 6688, 7860, 92, 1, 180, 13)
&lt;/pre&gt;

&lt;p&gt;We&#39;ll also need the following function to show the elapsed time:&lt;/p&gt;
&lt;pre&gt;
format.timediff &amp;lt;- function(start.time) {
    diff = as.numeric(difftime(Sys.time(), start.time, units=&quot;mins&quot;))
    hr &amp;lt;- diff%/%60
    min &amp;lt;- floor(diff - hr * 60)
    sec &amp;lt;- round(diff%%1 * 60,digits=2)
    return(paste(hr,min,sec,sep=&#39;:&#39;))
}
&lt;/pre&gt;

&lt;h2&gt;Pre-training&lt;/h2&gt;
&lt;p&gt;Initially, the network is a stack of RBMs that we need to &lt;em&gt;pre-train&lt;/em&gt; one by one. Hinton &amp;amp; Salakhutdinov (2006) showed that this step is critical to train deep networks. We will use 1000000 iterations (&lt;code&gt;maxiters.pretrain&lt;/code&gt;) of contrastive divergence, which takes a couple of days on a modern CPU. Let&#39;s start with the first three RBMs:&lt;/p&gt;

&lt;h3&gt;First three RBMs&lt;/h3&gt;
&lt;pre&gt;
if (run.training) {
	sprintf.fmt.iter &amp;lt;- sprintf(&quot;%%0%dd&quot;, nchar(sprintf(&quot;%d&quot;, maxiters.pretrain)))
	
	mnist.data.layer &amp;lt;- mnist
	for (i in 1:3) {
&lt;/pre&gt;
&lt;p&gt;We define a &lt;code&gt;diag&lt;/code&gt; function that will simply print where we are in the training. Because this function will be called a million times (&lt;code&gt;maxiters.pretrain&lt;/code&gt;), we can use &lt;code&gt;rate = &quot;accelerate&quot;&lt;/code&gt; to slow down the printing over time and save a few CPU cycles.&lt;/p&gt;
&lt;pre&gt;
		diag &amp;lt;- list(rate = &quot;accelerate&quot;, data = NULL, f = function(rbm, batch, data, iter, batchsize, maxiters, layer) {
			print(sprintf(&quot;%s[%s/%s] in %s&quot;, layer, iter, maxiters, format.timediff(start.time)))
		})
&lt;/pre&gt;
&lt;p&gt;We can get the current RBM, and we will work on it directly. Let&#39;s save it for good measure, as well as the current time for the progress function:&lt;/p&gt;
&lt;pre&gt;
		rbm &amp;lt;- dbn[[i]]
		save(rbm, file = file.path(output.folder, sprintf(&quot;rbm-%s-%s.RData&quot;, i, &quot;initial&quot;)))
		start.time &amp;lt;- Sys.time()
&lt;/pre&gt;
&lt;p&gt;Now we can start the actual pre-training:&lt;/p&gt;
&lt;pre&gt;
		rbm &amp;lt;- pretrain(rbm, mnist.data.layer$train$x,
			penalization = &quot;l2&quot;, lambda=0.0002, momentum = c(0.5, 0.9),
			epsilon=c(.1, .1, .1, .001)[i], batchsize = 100, maxiters=maxiters.pretrain,
			continue.function = continue.function.always, diag = diag)
&lt;/pre&gt;
&lt;p&gt;This can take some time, especially for the first layers which are larger. Once it is done, we predict the data through this RBM for the next layer and save the results:&lt;/p&gt;
&lt;pre&gt;
		mnist.data.layer$train$x &amp;lt;- predict(rbm, mnist.data.layer$train$x)
		mnist.data.layer$test$x &amp;lt;- predict(rbm, mnist.data.layer$test$x)
		save(rbm, file = file.path(output.folder, sprintf(&quot;rbm-%s-%s.RData&quot;, i, &quot;final&quot;)))
		dbn[[i]] &amp;lt;- rbm
	}
&lt;/pre&gt;

&lt;h3&gt;Last RBM&lt;/h3&gt;
&lt;p&gt;This is very similar to the previous three, but note that we save the RBM within the &lt;code&gt;diag&lt;/code&gt; function. We could generate the plot directly, but it is easier to do it later once we have some idea about the final axis we will need. Please note the &lt;code&gt;rate = &quot;accelerate&quot;&lt;/code&gt; here. You probably don&#39;t want to save a million RBM objects on your hard drive, both for speed and space reasons.&lt;/p&gt;

&lt;pre&gt;
	rbm &amp;lt;- dbn[[4]]
	print(head(rbm$b))
	diag &amp;lt;- list(rate = &quot;accelerate&quot;, data = NULL, f = function(rbm, batch, data, iter, batchsize, maxiters, layer) {
		save(rbm, file = file.path(output.folder, sprintf(&quot;rbm-4-%s.RData&quot;, sprintf(sprintf.fmt.iter, iter))))
		print(sprintf(&quot;%s[%s/%s] in %s&quot;, layer, iter, maxiters, format.timediff(start.time)))
	})
	save(rbm, file = file.path(output.folder, sprintf(&quot;rbm-%s-%s.RData&quot;, 4, &quot;initial&quot;)))
	start.time &amp;lt;- Sys.time()
	rbm &amp;lt;- pretrain(rbm, mnist.data.layer$train$x,  penalization = &quot;l2&quot;, lambda=0.0002,
		epsilon=.001, batchsize = 100, maxiters=maxiters.pretrain,
		continue.function = continue.function.always, diag = diag)
	save(rbm, file = file.path(output.folder, sprintf(&quot;rbm-4-%s.RData&quot;, &quot;final&quot;)))
	dbn[[4]] &amp;lt;- rbm
&lt;/pre&gt;

&lt;iframe src=&quot;https://www.youtube.com/embed/3EapaWpDqGQ&quot; title=&quot;YouTube video player&quot; allow=&quot;accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture&quot; allowfullscreen style=&quot;	width: 100%; aspect-ratio: 16/9; border: none;&quot;&gt;&lt;/iframe&gt;

&lt;p&gt;If we were not querying the last layer, we could have pre-trained the entire network at once with the following call:&lt;/p&gt;

&lt;pre&gt;
	dbn &amp;lt;- pretrain(dbn, mnist.data.layer$train$x, 
		penalization = &quot;l2&quot;, lambda=0.0002, momentum = c(0.5, 0.9),
		epsilon=c(.1, .1, .1, .001), batchsize = 100, 
		maxiters=maxiters.pretrain,
		continue.function = continue.function.always)
&lt;/pre&gt;	 

&lt;h3&gt;Pre-training parameters&lt;/h3&gt;
&lt;p&gt;Pre-training RBMs is quite sensitive to the use of proper parameters.
 With improper parameters, the network can quickly go crazy and start to generate infinite values. If that happens to you, you should try to tune one of the following parameters:
 
&lt;ul&gt;
 &lt;li&gt;&lt;code&gt;penalization&lt;/code&gt;: this is the penalty of introducing or increasing the value of a weight. We used L2 regularization, but &lt;code&gt;&quot;l1&quot;&lt;/code&gt; is available if a sparser weight matrix is needed.&lt;/li&gt;
 &lt;li&gt;&lt;code&gt;lambda&lt;/code&gt;: the regularization rate. In our experience 0.0002 works fine with the MNIST and other datasets of similar sizes such as cellular imaging data. Too small or large values will result in over- or under-fitted networks, respectively.&lt;/li&gt;
 &lt;li&gt;&lt;code&gt;momentum&lt;/code&gt;: helps avoiding oscillatory behaviors, where the network oscillate between iterations. Allowed values can range from 0 (no momentum) to 1 (full momentum = no training). Here we used an increasing gradient of momentum which starts at 0.5 and increases linearly to 0.9, in order to stabilize the final network without compromising early training steps.&lt;/li&gt;
 &lt;li&gt;&lt;code&gt;epsilon&lt;/code&gt;: the learning rate. Typically, 0.1 works well with binary and continuous output layers, and must be decreased to around 0.001 for gaussian outputs. Too large values will drive the network to generate infinities, while too small ones will slow down the training.&lt;/li&gt;
 &lt;li&gt;&lt;code&gt;batchsize&lt;/code&gt;: larger batch sizes will result in smoother but slower training. Small batch sizes will make the training &quot;jumpy&quot;, which can be compensated by lower learning rates (epsilon) or increased momentum.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;Fine-tuning&lt;/h2&gt;
&lt;p&gt;This is where the real training happens. We use conjugate gradients to find the optimal solution. Again, the &lt;code&gt;diag&lt;/code&gt; function saves the DBN. This time we use &lt;code&gt;rate = &quot;each&quot;&lt;/code&gt; to save every step of the training. First we have way fewer steps, but also the training itself happen at a much more stable speed than in the pre-training, where things slow down dramatically.
&lt;/p&gt;

&lt;pre&gt;
	sprintf.fmt.iter &amp;lt;- sprintf(&quot;%%0%dd&quot;, nchar(sprintf(&quot;%d&quot;, maxiters.train)))
	diag &amp;lt;- list(rate = &quot;each&quot;, data = NULL, f = function(dbn, batch, data, iter, batchsize, maxiters) {
		save(dbn, file = file.path(output.folder, sprintf(&quot;dbn-finetune-%s.RData&quot;, sprintf(sprintf.fmt.iter, iter))))
		print(sprintf(&quot;[%s/%s] in %s&quot;, iter, maxiters, format.timediff(start.time)))
	})
	save(dbn, file = file.path(output.folder, sprintf(&quot;dbn-finetune-%s.RData&quot;, &quot;initial&quot;)))
	start.time &amp;lt;- Sys.time()
	dbn &amp;lt;- train(unroll(dbn), mnist$train$x, batchsize = 100, maxiters=maxiters.train,
		continue.function = continue.function.always, diag = diag)
	save(dbn, file = file.path(output.folder, sprintf(&quot;dbn-finetune-%s.RData&quot;, &quot;final&quot;)))
}
&lt;/pre&gt;

&lt;iframe src=&quot;https://www.youtube.com/embed/wSfoZ_kMMTc&quot; title=&quot;YouTube video player&quot; allow=&quot;accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture&quot; allowfullscreen style=&quot;	width: 100%; aspect-ratio: 16/9; border: none;&quot;&gt;&lt;/iframe&gt;

&lt;p&gt;And that&#39;s it, our DBN is now fully trained!&lt;/p&gt;

&lt;h2&gt;Generating the images&lt;/h2&gt;
&lt;p&gt;Now we need to read in the saved network states again, pass the data through the network (&lt;code&gt;predict&lt;/code&gt;) and save this in HD-sized PNG file.&lt;/p&gt;

&lt;p&gt;The first three RBMs are only loaded into the DBN&lt;/p&gt;

&lt;pre&gt;
if (run.images) {
	for (i in 1:3) {
		load(file.path(output.folder, sprintf(&quot;rbm-%d-final.RData&quot;, i)))
		dbn[[i]] &amp;lt;- rbm
	}
&lt;/pre&gt;

&lt;p&gt;The last RBM is where interesting things happen.&lt;/p&gt;
&lt;pre&gt;
	for (file in list.files(output.folder, pattern = &quot;rbm-4-.+\\.RData&quot;, full.names = TRUE)) {
		print(file)
		load(file)
		dbn[[4]] &amp;lt;- rbm
		iter &amp;lt;- stringr::str_match(file, &quot;rbm-4-(.+)\\.RData&quot;)[,2]
&lt;/pre&gt;
&lt;p&gt;We now predict and reconstruct the data, and calculate the mean reconstruction error:&lt;/p&gt;
&lt;pre&gt;
		predictions &amp;lt;- predict(dbn, mnist$test$x)
		reconstructions &amp;lt;- reconstruct(dbn, mnist$test$x)
		iteration.error &amp;lt;- errorSum(dbn, mnist$test$x) / nrow(mnist$test$x)
&lt;/pre&gt;
&lt;p&gt;Now the actual plotting. Here I selected &lt;code&gt;xlim&lt;/code&gt; and &lt;code&gt;ylim&lt;/code&gt; values that worked well for my training run, but your mileage may vary.&lt;/p&gt;
&lt;pre&gt;
		png(sub(&quot;.RData&quot;, &quot;.png&quot;, file), width = 1280, height = 720) # hd output
		plot.mnist(model = dbn, x = mnist$test$x, label = mnist$test$y+1, predictions = predictions, reconstructions = reconstructions,
				   ncol = 16, highlight.digits = highlight.digits,
				   xlim = c(-12.625948, 8.329168), ylim = c(-10.50657, 13.12654))
		par(family=&quot;mono&quot;)
		legend(&quot;bottomleft&quot;, legend = sprintf(&quot;Mean error = %.3f&quot;, iteration.error), bty=&quot;n&quot;, cex=3)
		legend(&quot;bottomright&quot;, legend = sprintf(&quot;Iteration = %s&quot;, iter), bty=&quot;n&quot;, cex=3)
		dev.off()
	}
&lt;/pre&gt;
&lt;p&gt;&lt;img src=&quot;/files/blog/2022/06/11/rbm-4-final.png&quot; style=&quot;max-width: 100%&quot; alt=&quot;Scatterplot after pretraining&quot;&gt;&lt;/p&gt;

&lt;p&gt;We do the same with the fine-tuning:&lt;/p&gt;
&lt;pre&gt;
	for (file in list.files(output.folder, pattern = &quot;dbn-finetune-.+\\.RData&quot;, full.names = TRUE)) {
		print(file)
		load(file)
		iter &amp;lt;- stringr::str_match(file, &quot;dbn-finetune-(.+)\\.RData&quot;)[,2]
		predictions &amp;lt;- predict(dbn, mnist$test$x)
		reconstructions &amp;lt;- reconstruct(dbn, mnist$test$x)
		iteration.error &amp;lt;- errorSum(dbn, mnist$test$x) / nrow(mnist$test$x)
		png(sub(&quot;.RData&quot;, &quot;.png&quot;, file), width = 1280, height = 720) # hd output
		plot.mnist(model = dbn, x = mnist$test$x, label = mnist$test$y+1, predictions = predictions, reconstructions = reconstructions,
				   ncol = 16, highlight.digits = highlight.digits,
				   xlim = c(-22.81098,  27.94829), ylim = c(-17.49874,  33.34688))
		par(family=&quot;mono&quot;)
		legend(&quot;bottomleft&quot;, legend = sprintf(&quot;Mean error = %.3f&quot;, iteration.error), bty=&quot;n&quot;, cex=3)
		legend(&quot;bottomright&quot;, legend = sprintf(&quot;Iteration = %s&quot;, iter), bty=&quot;n&quot;, cex=3)
		dev.off()
	}
}
&lt;/pre&gt;
&lt;p&gt;&lt;img src=&quot;/files/blog/2022/06/11/dbn-finetune-final.png&quot; style=&quot;max-width: 100%&quot; alt=&quot;Scatterplot after fine-tuning&quot;&gt;&lt;/p&gt;

&lt;h2&gt;The video&lt;/h2&gt;

&lt;p&gt;I simply used &lt;a href=&quot;https://ffmpeg.org/&quot;&gt;ffmpeg&lt;/a&gt; to convert the PNG files to a video:&lt;/p&gt;
&lt;pre&gt;
cd video
ffmpeg -pattern_type glob -i &quot;rbm-4-*.png&quot; -b:v 10000000 -y ../rbm-4.mp4
ffmpeg -pattern_type glob -i &quot;dbn-finetune-*.png&quot; -b:v 10000000 -y ../dbn-finetune.mp4
&lt;/pre&gt;

&lt;p&gt;And that&#39;s it! Notice how the pre-training only brings the network to a state similar to that of a PCA, and the fine-tuning actually does the separation, and how it really makes the reconstructions accurate.&lt;/p&gt;

&lt;h2&gt;Application&lt;/h2&gt;
&lt;p&gt;We used this code to analyze changes in cell morphology upon drug resistance in cancer. With a 27-dimension space, we could describe all of the observed cell morphologies and predict whether a cell was resistant to ErbB-family drugs with an accuracy of 74%. The paper is available in Open Access in Cell Reports, DOI &lt;a href=&quot;https://doi.org/10.1016/j.celrep.2020.108657&quot; title=&quot;Deep neural networks identify signaling mechanisms of ErbB-family drug resistance from a continuous cell morphology space&quot;&gt;10.1016/j.celrep.2020.108657&lt;/a&gt;.

&lt;h2&gt;Concluding remarks&lt;/h2&gt;
&lt;p&gt;In this document I described how to build and train a DBN with the &lt;code&gt;DeepLearning&lt;/code&gt; package. I also showed how to query the internal layer, and use the generative properties to follow the training of the network on handwritten digits.&lt;/p&gt;
&lt;p&gt;DBNs have the advantage over Convolutional Networks (CN) that they are fully generative, at least during the pre-training. They are therefore easier to query and interpret as we have demonstrated here. 
 However keep in mind that CNs have demonstrated higher accuracies on computer vision tasks, such as the MNIST dataset.&lt;/p&gt;
&lt;p&gt;Additional algorithmic details are available in the &lt;code&gt;doc&lt;/code&gt; folder of the DeepLearning package.&lt;/p&gt;


&lt;h2&gt;References&lt;/h2&gt;
&lt;dl&gt;
	&lt;dt&gt;Our paper, 2021&lt;/dt&gt;
	&lt;dd&gt;Longden J., Robin X., Engel M., &lt;i&gt;et al.&lt;/i&gt;
 &lt;a href=&quot;https://doi.org/10.1016/j.celrep.2020.108657&quot;&gt;Deep neural networks identify signaling mechanisms of ErbB-family drug resistance from a continuous cell morphology space&lt;/a&gt;. &lt;i&gt;Cell Reports&lt;/i&gt;, 2021;34(3):108657.&lt;/dd&gt;

	&lt;dt&gt;Bates &amp;amp; Eddelbuettel, 2013&lt;/dt&gt;
	&lt;dd&gt;Bates D, Eddelbuettel D. &lt;a href=&quot;http://www.jstatsoft.org/v52/i05/&quot;&gt;Fast and Elegant Numerical Linear Algebra Using the RcppEigen Package&lt;/a&gt;. &lt;i&gt;Journal of Statistical Software&lt;/i&gt;, 2013;52(5):1&amp;ndash;24.&lt;/dd&gt;

	&lt;dt&gt;Bengio &lt;i&gt;et al.&lt;/i&gt;, 2007&lt;/dt&gt;
	&lt;dd&gt;Bengio Y, Lamblin P, Popovici D, Larochelle H. &lt;a href=&quot;https://papers.nips.cc/paper/3048-greedy-layer-wise-training-of-deep-networks.pdf&quot;&gt;Greedy layer-wise training of deep networks&lt;/a&gt;. &lt;i&gt;Advances in neural information processing systems&lt;/i&gt;. 2007;19:153&amp;ndash;60.&lt;/dd&gt;

	&lt;dt&gt;Hinton &amp;amp; Salakhutdinov, 2006&lt;/dt&gt;
	&lt;dd&gt;Hinton GE, Salakhutdinov RR. &lt;a href=&quot;http://dx.doi.org/10.1126/science.1127647&quot;&gt;Reducing the Dimensionality of Data with Neural Networks&lt;/a&gt;. &lt;i&gt;Science&lt;/i&gt;. 2006;313(5786):504&amp;ndash;7.&lt;/dd&gt;
&lt;/dl&gt;

&lt;h2&gt;Downloads&lt;/h2&gt;
&lt;ol&gt;
	&lt;li&gt;&lt;a href=&quot;/files/blog/2022/06/11/MNIST_video.tar.gz&quot;&gt;Code to generate the video&lt;/a&gt;&lt;/li&gt;
	&lt;li&gt;&lt;a href=&quot;https://github.com/xrobin/DeepLearning&quot;&gt;DeepLearning package source code&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;
</content>
		</entry>

	<entry xml:lang="en" xml:base="https://xavier.robin.info/fr/">
		<title type="html">pROC 1.18.0</title>
		
			<category term="pROC" label="pROC" scheme="https://xavier.robin.info/fr/tag/pROC" />
		
		<link href="https://xavier.robin.info/fr/blog/2021/09/06/proc-1.18.0"/>
		<id>tag:xavier.robin.info,2021-09-06:/blog/2021/09/06/proc-1.18.0</id>
		<published>2021-09-06T18:34:01+02:00</published>
		<updated>2021-09-06T18:34:01+02:00</updated>
		<content type="html">&lt;p&gt;pROC version 1.18.0 is now available on CRAN now. Only a few changes were implemented in this release:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;Add &lt;abbr title=&quot;Confidence Interval&quot;&gt;CI&lt;/abbr&gt; of the estimate for &lt;code&gt;roc.test&lt;/code&gt; (DeLong, paired only for now) (code contributed by &lt;a href=&quot;https://wz-billings.rbind.io/&quot;&gt;Zane Billings&lt;/a&gt;) (&lt;a href=&quot;https://github.com/xrobin/pROC/pull/95&quot;&gt;issue #95&lt;/a&gt;).&lt;/li&gt;
	&lt;li&gt;Fix documentation and alternative hypothesis for Venkatraman test (&lt;a href=&quot;https://github.com/xrobin/pROC/issues/92&quot;&gt;issue #92&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;You can update your installation by simply typing:&lt;/p&gt;

&lt;pre&gt;install.packages(&quot;pROC&quot;)&lt;/pre&gt;</content>
		</entry>

	<entry xml:lang="en" xml:base="https://xavier.robin.info/fr/">
		<title type="html">pROC 1.17.0.1</title>
		
			<category term="pROC" label="pROC" scheme="https://xavier.robin.info/fr/tag/pROC" />
		
		<link href="https://xavier.robin.info/fr/blog/2021/01/13/proc-1.17.0.1"/>
		<id>tag:xavier.robin.info,2021-01-13:/blog/2021/01/13/proc-1.17.0.1</id>
		<published>2021-01-13T16:19:16+01:00</published>
		<updated>2021-01-13T16:19:16+01:00</updated>
		<content type="html">&lt;p&gt;pROC version 1.17.0.1 is available on CRAN now. Besides several bug fixes and small changes, it introduces more values in &lt;code&gt;input&lt;/code&gt; of &lt;code&gt;coords&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Here is an example:&lt;/p&gt;

&lt;pre&gt;
library(pROC)
data(aSAH)
rocobj &amp;lt;- roc(aSAH$outcome, aSAH$s100b)
coords(rocobj, x = seq(0, 1, .1), input=&quot;recall&quot;, ret=&quot;precision&quot;)
#    precision
# 1        NaN
# 2  1.0000000
# 3  1.0000000
# 4  0.8601399
# 5  0.6721311
# 6  0.6307692
# 7  0.6373057
# 8  0.4803347
# 9  0.4517906
# 10 0.3997833
# 11 0.3628319
&lt;/pre&gt;

	
&lt;h2&gt;Getting the update&lt;/h2&gt;

&lt;p&gt;The update his available on CRAN now. You can update your installation by simply typing:&lt;/p&gt;

&lt;pre&gt;install.packages(&quot;pROC&quot;)&lt;/pre&gt;

&lt;p&gt;Here is the full changelog:&lt;/p&gt; 

&lt;p&gt;1.17.0.1 (2020-01-07):&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Fix CRAN incoming checks as requested by CRAN.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;1.17.0 (2020-12-29)&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;Accept more values in &lt;code&gt;input&lt;/code&gt; of &lt;code&gt;coords&lt;/code&gt; (&lt;a href=&quot;https://github.com/xrobin/pROC/issues/67&quot;&gt;issue #67&lt;/a&gt;).&lt;/li&gt;
	&lt;li&gt;Accept &lt;code&gt;kappa&lt;/code&gt; for the &lt;code&gt;power.roc.test&lt;/code&gt; of two ROC curves (&lt;a href=&quot;https://github.com/xrobin/pROC/issues/82&quot;&gt;issue #82&lt;/a&gt;).&lt;/li&gt;
	&lt;li&gt;The &lt;code&gt;input&lt;/code&gt; argument to &lt;code&gt;coords&lt;/code&gt; for &lt;code&gt;smooth.roc&lt;/code&gt; curves no longer has a default.&lt;/li&gt;
	&lt;li&gt;The &lt;code&gt;x&lt;/code&gt; argument to &lt;code&gt;coords&lt;/code&gt; for &lt;code&gt;smooth.roc&lt;/code&gt; can now be set to &lt;code&gt;all&lt;/code&gt; (also the default).&lt;/li&gt;
	&lt;li&gt;Fix bootstrap &lt;code&gt;roc.test&lt;/code&gt; and &lt;code&gt;cov&lt;/code&gt; with &lt;code&gt;smooth.roc&lt;/code&gt; curves.&lt;/li&gt;
	&lt;li&gt;The &lt;code&gt;ggroc&lt;/code&gt; function can now plot &lt;code&gt;smooth.roc&lt;/code&gt; curves (&lt;a href=&quot;https://github.com/xrobin/pROC/issues/86&quot;&gt;issue #86&lt;/a&gt;).&lt;/li&gt;
	&lt;li&gt;Remove warnings with &lt;code&gt;warnPartialMatchDollar&lt;/code&gt; option (&lt;a href=&quot;https://github.com/xrobin/pROC/issues/87&quot;&gt;issue #87&lt;/a&gt;).&lt;/li&gt;
	&lt;li&gt;Make tests depending on vdiffr conditional (&lt;a href=&quot;https://github.com/xrobin/pROC/issues/88&quot;&gt;issue #88&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;</content>
		</entry>

	<entry xml:lang="en" xml:base="https://xavier.robin.info/fr/">
		<title type="html">pROC 1.16.1</title>
		
			<category term="pROC" label="pROC" scheme="https://xavier.robin.info/fr/tag/pROC" />
		
		<link href="https://xavier.robin.info/fr/blog/2020/01/14/proc-1.16.1"/>
		<id>tag:xavier.robin.info,2020-01-14:/blog/2020/01/14/proc-1.16.1</id>
		<published>2020-01-14T08:52:57+01:00</published>
		<updated>2020-01-14T08:52:57+01:00</updated>
		<content type="html">&lt;p&gt;pROC version 1.16.1 is a minor release that disables a timing-dependent test based on the microbenchmark package that can sometimes cause random failures on CRAN. This version contains no user-visible changes. Users don&#39;t need to install this update.&lt;/p&gt;
</content>
		</entry>

	<entry xml:lang="en" xml:base="https://xavier.robin.info/fr/">
		<title type="html">pROC 1.16.0</title>
		
			<category term="pROC" label="pROC" scheme="https://xavier.robin.info/fr/tag/pROC" />
		
		<link href="https://xavier.robin.info/fr/blog/2020/01/12/proc-1.16.0"/>
		<id>tag:xavier.robin.info,2020-01-12:/blog/2020/01/12/proc-1.16.0</id>
		<published>2020-01-12T21:46:00+01:00</published>
		<updated>2020-01-12T21:46:00+01:00</updated>
		<content type="html">&lt;p&gt;pROC version 1.16.0 is available on CRAN now. Besides several bug fixes, the main change is the switch of the default value of the &lt;code&gt;transpose&lt;/code&gt; argument to the &lt;code&gt;coords&lt;/code&gt; function from &lt;code&gt;TRUE&lt;/code&gt; to &lt;code&gt;FALSE&lt;/code&gt;. As announced earlier, &lt;strong&gt;this is a backward incompatible change that will break any script that did not previously set the &lt;code&gt;transpose&lt;/code&gt; argument&lt;/strong&gt; and for now comes with a warning to make debugging easier. Scripts that set transpose explicitly are not unaffected.&lt;/p&gt;

&lt;h2&gt;New return values of &lt;code&gt;coords&lt;/code&gt; and &lt;code&gt;ci.coords&lt;/code&gt;&lt;/h2&gt;

&lt;p&gt;With &lt;code&gt;transpose = FALSE&lt;/code&gt;, the &lt;code&gt;coords&lt;/code&gt; returns a tidy &lt;code&gt;data.frame&lt;/code&gt; suitable for use in pipelines:&lt;/p&gt;

&lt;pre&gt;
data(aSAH)
rocobj &amp;lt;- roc(aSAH$outcome, aSAH$s100b)
coords(rocobj, c(0.05, 0.2, 0.5), transpose = FALSE)
#      threshold specificity sensitivity
# 0.05      0.05  0.06944444   0.9756098
# 0.2       0.20  0.80555556   0.6341463
# 0.5       0.50  0.97222222   0.2926829
&lt;/pre&gt;

&lt;p&gt;The function doesn&#39;t drop dimensions, so the result is always a &lt;code&gt;data.frame&lt;/code&gt;, even if it has only one row and/or one column.&lt;/p&gt;

&lt;p&gt;If speed is of utmost importance, you can get the results as a non-transposed matrix instead:
&lt;pre&gt;
coords(rocobj, c(0.05, 0.2, 0.5), transpose = FALSE, as.matrix = TRUE)
#      threshold specificity sensitivity
# [1,]      0.05  0.06944444   0.9756098
# [2,]      0.20  0.80555556   0.6341463
# [3,]      0.50  0.97222222   0.2926829
&lt;/pre&gt;

&lt;p&gt;In some scenarios this can be a tiny bit faster, and is used internally in &lt;code&gt;ci.coords&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Type &lt;code&gt;help(coords_transpose)&lt;/code&gt; for additional information.&lt;/p&gt;

&lt;h3&gt;&lt;code&gt;ci.coords&lt;/code&gt;&lt;/h3&gt;
&lt;p&gt;The &lt;code&gt;ci.coords&lt;/code&gt; function now returns a list-like object:&lt;/p&gt;
&lt;pre&gt;
ciobj &amp;lt;- ci.coords(rocobj, c(0.05, 0.2, 0.5))
ciobj$accuracy
#        2.5%       50%     97.5%
# 1 0.3628319 0.3982301 0.4424779
# 2 0.6637168 0.7433628 0.8141593
# 3 0.6725664 0.7256637 0.7787611
&lt;/pre&gt;

&lt;p&gt;The &lt;code&gt;print&lt;/code&gt; function prints a table with all the results, however this table is generated on the fly and not available directly.&lt;/p&gt;

&lt;pre&gt;ciobj
# 95% CI (2000 stratified bootstrap replicates):
#      threshold sensitivity.low sensitivity.median sensitivity.high
# 0.05      0.05          0.9268             0.9756           1.0000
# 0.2       0.20          0.4878             0.6341           0.7805
# 0.5       0.50          0.1707             0.2927           0.4390
#      specificity.low specificity.median specificity.high accuracy.low
# 0.05         0.01389            0.06944           0.1250       0.3628
# 0.2          0.70830            0.80560           0.8889       0.6637
# 0.5          0.93060            0.97220           1.0000       0.6726
#      accuracy.median accuracy.high
# 0.05          0.3982        0.4425
# 0.2           0.7434        0.8142
# 0.5           0.7257        0.7788
&lt;/pre&gt;

&lt;p&gt;The following code snippet can be used to obtain all the information calculated by the function:&lt;/p&gt;
&lt;pre&gt;
for (ret in attr(ciobj, &quot;ret&quot;)) {
	print(ciobj[[ret]])
}
#        2.5%       50%     97.5%
# 1 0.9268293 0.9756098 1.0000000
# 2 0.4878049 0.6341463 0.7804878
# 3 0.1707317 0.2926829 0.4390244
#         2.5%        50%     97.5%
# 1 0.01388889 0.06944444 0.1250000
# 2 0.70833333 0.80555556 0.8888889
# 3 0.93055556 0.97222222 1.0000000
#        2.5%       50%     97.5%
# 1 0.3628319 0.3982301 0.4424779
# 2 0.6637168 0.7433628 0.8141593
# 3 0.6725664 0.7256637 0.7787611
&lt;/pre&gt;
	
&lt;h2&gt;Getting the update&lt;/h2&gt;

&lt;p&gt;The update his available on CRAN now. You can update your installation by simply typing:&lt;/p&gt;

&lt;pre&gt;install.packages(&quot;pROC&quot;)&lt;/pre&gt;

&lt;p&gt;Here is the full changelog:&lt;/p&gt; 

&lt;ul&gt;
	&lt;li&gt;BACKWARD INCOMPATIBLE CHANGE: &lt;code&gt;transpose&lt;/code&gt; argument to &lt;code&gt;coords&lt;/code&gt; switched to &lt;code&gt;FALSE&lt;/code&gt; by default (&lt;a href=&quot;https://github.com/xrobin/pROC/issues/54&quot;&gt;issue #54&lt;/a&gt;).&lt;/li&gt;
	&lt;li&gt;BACKWARD INCOMPATIBLE CHANGE: &lt;code&gt;ci.coords&lt;/code&gt; return value is now of list type and easier to use.&lt;/li&gt;
	&lt;li&gt;Fix one-sided DeLong test for curves with &lt;code&gt;direction=&quot;&amp;gt;&quot;&lt;/code&gt; (&lt;a href=&quot;https://github.com/xrobin/pROC/issues/64&quot;&gt;issue #64&lt;/a&gt;).&lt;/li&gt;
	&lt;li&gt;Fix an error in &lt;code&gt;ci.coords&lt;/code&gt; due to expected &lt;code&gt;NA&lt;/code&gt; values in some coords (like &quot;precision&quot;) (&lt;a href=&quot;https://github.com/xrobin/pROC/issues/65&quot;&gt;issue #65&lt;/a&gt;).&lt;/li&gt;
	&lt;li&gt;Ordrered predictors are converted to numeric in a more robust way (&lt;a href=&quot;https://github.com/xrobin/pROC/issues/63&quot;&gt;issue #63&lt;/a&gt;).&lt;/li&gt;
	&lt;li&gt;Cleaned up &lt;code&gt;power.roc.test&lt;/code&gt; code (&lt;a href=&quot;https://github.com/xrobin/pROC/issues/50&quot;&gt;issue #50&lt;/a&gt;).&lt;/li&gt;
	&lt;li&gt;Fix pairing with &lt;code&gt;roc.formula&lt;/code&gt; and warn if &lt;code&gt;na.action&lt;/code&gt; is not set to &lt;code&gt;&quot;na.pass&quot;&lt;/code&gt; or &lt;code&gt;&quot;na.fail&quot;&lt;/code&gt; (&lt;a href=&quot;https://github.com/xrobin/pROC/issues/68&quot;&gt;issue #68&lt;/a&gt;).&lt;/li&gt;
	&lt;li&gt;Fix &lt;code&gt;ci.coords&lt;/code&gt; not working with &lt;code&gt;smooth.roc&lt;/code&gt; curves.&lt;/li&gt;
&lt;/ul&gt;</content>
		</entry>


</feed>
