Press "Enter" to skip to content

Coming Home

taidoka 0

This week has been so bad I feel like I'm under spiritual attack. It was supposed to be a vacation, but both my cats got sick, I got sick myself, and I had to work in the middle of it. I feel like the protagonist of a Neil Gaiman story I read in M is For Magic where a black cat is protecting a home from supernatural assault.

But now both of my cats are coming home. Gabby, the gold guy above, comes home tonight after a serious asthma attack, and Loki is already home after a serious urinary tract blockage.

Here's hoping two cats and prayers put things back on track.

-the Centaur

What is “Understanding”?

taidoka 0

When I was growing up - or at least when I was a young graduate student in a Schankian research lab - we were all focused on understanding: what did it mean, scientifically speaking, for a person to understand something, and could that be recreated on a computer? We all sort of knew it was what we'd call nowadays an ill-posed problem, but we had a good operational definition, or at least an operational counterexample: if a computer read a story and could not answer the questions that a typical human being could answer about that story, it didn't understand it at all.

But there are at least two ways to define a word. What I'll call a practical definition is what a semanticist might call the denotation of a word: a narrow definition, one which you might find in a dictionary, which clearly specifies the meaning of the concept, like a bachelor being an unmarried man. What I'll call a philosophical definition, the connotations of a word, are the vast web of meanings around the core concept, the source of the fine sense of unrightness that one gets from describing Pope Francis as a bachelor, the nuances of meaning embedded in words that Socrates spent his time pulling out of people, before they went and killed him for being annoying.

It's those connotations of "understanding" that made all us Schankians very leery of saying our computer programs fully "understood" anything, even as we were pursuing computer understanding as our primary research goal. I care a lot about understanding, deep understanding, because, frankly, I cannot effectively do my job of teaching robots to learn if I do not deeply understand robots, learning, computers, the machinery surrounding them, and the problem I want to solve; when I do not understand all of these things, I stumble in the dark, I make mistakes, and end up sad.

And it's pursuing a deeper understanding about deep learning where I got a deeper insight into deep understanding. I was "deep reading" the Deep Learning book (a practice in which I read, or re-read, a book I've read, working out all the equations in advance before reading the derivations), in particular section 5.8.1 on Principal Components Analysis, and the authors made the same comment I'd just seen in the Hands-On Machine Learning book: "the mean of the samples must be zero prior to applying PCA."

Wait, what? Why? I mean, thank you for telling me, I'll be sure to do that, but, like ... why? I didn't follow up on that question right away, because the authors also tossed off an offhand comment like, "XX is the unbiased sample covariance matrix associated with a sample x" and I'm like, what the hell, where did that come from? I had recently read the section on variance and covariance but had no idea why this would be associated with the transpose of the design matrix X multiplied by X itself. (In case you're new to machine learning, if x stands for an example input to a problem, say a list of the pixels of an image represented as a column of numbers, then the design matrix X is all the examples you have, but each example listed as a row. Perfectly not confusing? Great!)

So, since I didn't understand why Var[x] = XX, I set out to prove it myself. (Carpenters say, measure twice, cut once, but they'd better have a heck of a lot of measuring and cutting under their belts - moreso, they'd better know when to cut and measure before they start working on your back porch, or you and they will have a bad time. Same with trying to teach robots to learn: it's more than just practice; if you don't know why something works, it will come back to bite you, sooner or later, so, dig in until you get it). And I quickly found that the "covariance matrix of a variable x" was a thing, and quickly started to intuit that the matrix multiplication would produce it.

This is what I'd call surface level understanding: going forward from the definitions to obvious conclusions. I knew the definition of matrix multiplication, and I'd just re-read the definition of covariance matrices, so I could see these would fit together. But as I dug into the problem, it struck me: true understanding is more than just going forward from what you know: "The brain does much more than just recollect; it inter-compares, it synthesizes, it analyzes, it generates abstractions" - thank you, Carl Sagan. But this kind of understanding is a vast, ill-posed problem - meaning, a problem without a unique and unambiguous solution.

But as I was continuing to dig through the problem, reading through the sections I'd just read on "sample estimators," I had a revelation. (Another aside: "sample estimators" use the data you have to predict data you don't, like estimating the height of males in North America from a random sample of guys across the country; "unbiased estimators" may be wrong but their errors are grouped around the true value). The formula for the unbiased sample estimator for the variance actually doesn't look quite the matrix transpose - but it depends on the unbiased estimator of sample mean.

Suddenly, I felt that I understood why PCA data had to have a mean of 0. Not driving forward from known facts and connecting their inevitable conclusions, but driving backwards from known facts to hypothesize a connection which I could explore and see. I even briefly wrote a draft of the ideas behind this essay - then set out to prove what I thought I'd seen. Setting the mean of the samples to zero made the sample mean drop out of sample variance - and then the matrix multiplication formula dropped out. Then I knew I understood why PCA data had to have a mean of 0 - or how to rework PCA to deal with data which had a nonzero mean.

This I'd call deep understanding: reasoning backwards from what we know to provide reasons for why things are the way they are. A recent book on science I read said that some regularities, like the length of the day, may be predictive, but other regularities, like the tides, cry out for explanation. And once you understand Newton's laws of motion and gravitation, the mystery of the tides is readily solved - the answer falls out of inertia, angular momentum, and gravitational gradients. With apologies to Larry Niven, of course a species that understands gravity will be able to predict tides.

The brain does do more than just remember and predict to guide our next actions: it builds structures that help us understand the world on a deeper level, teasing out rules and regularities that help us not just plan, but strategize. Detective Benoit Blanc from the movie Knives Out claimed to "anticipate the terminus of gravity's rainbow" to help him solve crimes; realizing how gravity makes projectiles arc, using that to understand why the trajectory must be the observed parabola, and strolling to the target.

So I'd argue that true understanding is not just forward-deriving inferences from known rules, but also backward-deriving causes that can explain behavior. And this means computing the inverse of whatever forward prediction matrix you have, which is a more difficult and challenging problem, because that matrix may have a well-defined inverse. So true understanding is indeed a deep and interesting problem!

But, even if we teach our computers to understand this way ... I suspect that this won't exhaust what we need to understand about understanding. For example: the dictionary definitions I've looked up don't mention it, but the idea of seeking a root cause seems embedded in the word "under - standing" itself ... which makes me suspect that the other half of the word, standing, itself might hint at the stability, the reliability of the inferences we need to be able to make to truly understand anything.

I don't think we've reached that level of understanding of understanding yet.

-the Centaur

Pictured: Me working on a problem in a bookstore. Probably not this one.

Never Give Up

centaur 0

So, 2019. What a mess. More on that later; as for me, I've had neither the time nor even the capability to blog for a while. But one thing I've noticed is, at least for me, the point at which I want to give up is usually just prior to the point where I could have my big breakthrough.

For example: Scrivener.

I had just about given up on Scrivener, an otherwise great program for writers that helps with organizing notes, writing screenplays, and even for comic book scripts. But I'd become used to Google Docs and its keyboard shortcuts for hierarchical bulleted lists, not entirely different from my prior life using hierarchical notebook programs like GoldenSection Notes. But Scrivener's keyboard shortcuts were all different, and the menus didn't seem to support what I needed, so I had started trying alternatives. Then I gave on more shot at going through the manual, which had earlier got me nothing.At first this looked like a lost cause: Scrivener depended on Mac OS X's text widgets, which themselves implement a nonstandard text interface (fanboys, shut up, sit down: you're overruled. case in point: Home and End. I rest my case), and worse, depend on the OS even for the keyboard shortcuts, which require the exact menu item. But the menu item for list bullets actually was literally a bullet, which normally isn't a text character in most programs; you can't access it. But as it turns out, in Scrivener, you can. I was able to insert a bullet, find the bullet character, and even create a keyboard shortcut for it. And it did what it was supposed to!

Soon I found the other items I needed to fill out the interface that I'd come to know and love in Google Docs for increasing/decreasing the list bullet indention on the fly while organizing a list:

Eventually I was able to recreate the whole interface and was so happy I wrote a list describing it in the middle of the deep learning Scrivener notebook that I had been working on when I hit the snag that made me go down this rabbit hole (namely, wanting to create a bullet list):

Writing this paragraph itself required figuring out how to insert symbols for control characters in Mac OS X, but whatever: a solution was possible, even ready to be found, just when I was ready to give up.

I found the same thing with so many things recently: stuck photo uploads on Google Photos, configuration problems on various publishing programs, even solving an issue with the math for a paper submission at work.

I suspect this is everywhere. It's a known thing in mathematics that when you feel close to a solution you may be far from it; I often find myself that the solution is to be found just after the point you want to give up.

I've written about a related phenomenon called this "working a little bit harder than you want to" but this is slightly different: it's the idea that your judgment that you've exhausted your options is just that, a judgment.

It may be true.

Try looking just a bit harder for that answer.

-the Centaur

Pictured: a photo of the Greenville airport over Christmas, which finally uploaded today when I went back through the archives of Google Photos on my phone and manually stopped a stuck upload from December 19th.

Viiictory to the Twenty-Fifth Power!

centaur 0

SO! After yet another National Novel Writing Month, I have added yet another 50,000 words of rough draft to my writing output - making this the fourteenth time I have won Nano, and the twenty-fifth time I have won one of the Nanowrimo or CampNanowrimo challenges! Woohoo!

This month wasn't so bad, though there was a bit of a dip around the time I was writing report cards for our robot learning systems ("Little Johnny 5 tries very hard, but needs to work on his cornering!"). But, as usual, the week I took off for Thanksgiving "vacation" put me back on track:

Yes, one day I did indeed get 6000+ words written, which was a record for the 25th of the month, but nowhere near my record of 9074 words - written on the 30th(!) of November 2016, in what I recall was a delerious mad dash sitting on my sofa wracking my brain to produce enough words to make my goal for PHANTOM SILVER. Frankly speaking, that sucked, and since then I have redoubled my efforts to ensure that I'm never THAT far behind.

So this month looks typical. It's interesting to me how much Nano has become a part of my life. First tried in 2002, first made into a yearly habit in 2007, and first made into a thrice-yearly habit (Camp April, Camp July and November Nano) in 2014-2015 ... now I've done Nano 27 times, with 25 successes, for 1.36 million words of rough draft ... it's a heavy feeling.

Do I want to keep doing this? Absolutely. I wish I had more time to, like, edit my books, so I didn't have a backlog of 6 finished novels, 2 novellas, and 5 partially finished novels. (Gulp!) But I like having a roof more, and the time and money to pay for my laptop, my nice dinners, and my late nite teas and mochas, so, teaching robots to learn by day it is, for the time being.

One of the most interesting things for me is how Nano breaks through your creative barriers. When I started on MACHINERY OF THE APOCALYPSE, then titled TWO YEARS OF HELL, I had the idea of writing an action-adventure steampunk hard science fiction story around computer science concepts, and conceived it as a connected tale made of 16 short stories  --- two to the fourth power, a number beloved of many computer scientists.

But as I've written, the story has sprawled out from my original design, and there are at least two, perhaps three set pieces which may demand their own stories. Or perhaps existing stories will have to be cut or deleted. I don't know; I just create the worlds, but once they exist, they follow the laws of physics (plot and character physics). Here's an excerpt from one of those diversions, which may or may not make it into the final design:

The dark doorway loomed before her like a maw. Jeremiah steeled herself: she had been her at best a handful of times, but she felt like she knew every rivet of the damned hatch, felt like she was right back to waiting on the damn Keepers while they prepared themselves.

Oh, she did not, did not, did not want to be back here.

Yet she was, not a child, but a Major. She straightened, nodded.

“Major?” asked Thompson, looking back at her. “You look a bit green.”

Jeremiah smiled, to give herself a moment to speak. What would a cracker-jack young major say? Or … wasn’t that putting on airs? What would General Weiss have said? Perhaps she should just be … honest?

“Good eye, sir, but I don’t just look it: I feel it too,” Jeremiah said, forcing a grin—was that fake, or did she just want to take this in the best humor possible? “Every time I’m here, it takes me right back to my childhood.”

“Childhood?” Thompson asked. The white hairs in his saltpepper eyebrows sparkled as his brow beetled. “Why were you here as a child?”

“I, foolishly perhaps, asked to see the thing that killed my mother,” Jeremiah said. “And … foolishly perhaps, the powers that be let the granddaughter of Benjamin Willstone get what she asked for.”

Thompson stared at her strangely, then turned away.

“I would have let you,” he said at last. “Seems to have been the first step into forming a fine soldier who doesn’t flinch.”

“Oh, I assure you, I flinch,” Jeremiah said. “Just not from duty.”

“That’s the Major Willstone of my reports,” Thompson said. He leaned over and said a bit cheekily. “I hear you scream like a girl even when you’re firing both blasters at point-blank range—”

“Why, I never—” Jeremiah colored. “Well, that does speak to character—”

“Yes, yes, it does,” Thompson said, “and to good sense. Alright, in fairness: the report just said ‘cried out in shock before blasting the thing,’ but one could imagine the girlish scream—”

“Oi!” Jeremiah said. “Wait, what thing was this?”

“Er,” Thompson said, as the hatch opened. “I … don’t recall. Frankly, Major, with your record, the monsters start to blur—”

“Not all of them,” Jeremiah said, striding forward with a projected confidence she absolutely did not feel. “Have a look at that.”

Enjoy. Back to writing!

-the Centaur

My Novels and Nano

centaur 0

SO! I love to write, and four of my novels are published - FROST MOON, BLOOD ROCK, LIQUID FIRE, about magical tattoo artist Dakota Frost, and JEREMIAH WILLSTONE AND THE CLOCKWORK TIME MACHINE, about steampunk heroine Jeremiah Willstone.

You can read about the published ones at my Novels page, but even though life got a bit away from me this year, I haven't stopped writing - I have six more finished novels in the editing queue, not to mention half a dozen more in process.

And every single one of these novels, published or not, was largely written in National Novel Writing Month in November (or its sister challenge Camp Nanowrimo in April and July).

Nanowrimo is a 501(c)(3)that helps people find their creative voices - and certainly helped me transition from mostly not-writing to writing over a million words of fiction! (Way over, now).

Every year, I donate to the Nanowrimo foundation to help them not just keep the lights on but to support young writers everywhere with their Young Writers Program. This year, consider helping them bring literacy and creativity to more people all around the world!

-the Centaur

Work, Finish, Publish!

taidoka 0

So I think a lot about how to be a better scientist, and during my reading I found a sparkly little gem by one of the greatest experimentalists of all time, Michael Faraday. It's quoted in Analysis and Presentation of Experimental Results as above, but from Wikiquote we get the whole story:

"The secret is comprised in three words — Work, finish, publish."

His well-known advice to the young William Crookes, who had asked him the secret of his success as a scientific investigator, as quoted in Michael Faraday (1874) by John Hall Gladstone, p. 123

Well said. The middle part often seems the hardest for many people, in my experience: it's all too easy to work on something without finishing it, or to rush to publish something before it's really ready. The hard part is pushing through all three in the right order with the appropriate level of effort.

-the Centaur

Pictured: Michael Faraday, Photograph by Maull & Polyblank. Credit: Wellcome Collection. CC BY.

The Sole Test of Any Idea

taidoka 0

Inspirational physicist Richard Feynman once said "the sole test of any idea is experiment." I prefer the formulation "the sole test of any idea open to observation is experiment," because opening our ideas to observation - rather than relying on just belief, instrumentation, or arguments - is often the hardest challenge in making progress on otherwise seemingly unresolvable problems.

-the Centaur

Renovation in Process

centaur 0

So you may have noticed the blog theme and settings changing recently; that's because I'm trying to get some kind of slider or visual image above the fold. I love the look of the blog with the big banner image, but I'm concerned that people just won't scroll down to see what's in the blog if there's nothing on the first page which says what I do.

So I'll be experimenting. Stay tuned!

-the Centaur

Pictured: Yeah, this isn't the only renovation going on.

GROW, Every Day

centaur 0

So I read a lot and write a lot and occasionally edit what I write and even more rarely, something gets sent to an editor and turned into a publication. But this seems slow, for always there are more thoughts that I have in my head than I seem to have time to put onto a page and share.

Also I seem to get stuck in ruts. Actually I like ruts - I’m in one now, eating near my house and bank and pet food store at a restaurant with really good iced tea - but only ruts that are good for getting things done, like ruts in a well-trodden road.

When ruts leave you spinning your wheels in the mud, it’s time for a change. This can be as simple as engaging the locking hubs on your stuck all-wheel drive truck to get out of the mud, but you have to know that those locks are there to engage them. (True story).

So in order to grow, you need to learn. But if you learn, and you don’t tell anyone, then when you die, what you learn is gone. Fortunately, at the dawn of history humans learned how to speak to the dead, if only the dead are first willing to share, through their stories.

I don’t recommend waiting until you’re dead to tell your story. (Most people find that disturbing). Instead, it’s better to organize your thoughts - to reflect on what happened, what you’ve learned, and to package it the lesson with its context so it’s easy to share, like knowledge in a little case.

But I don’t do that all that well. I read for entertainment, and I occasionally write things down, but I rarely reflect, and I even more rarely share. But in my attempt to grow, I’ve read some things that made me think, and it made me want to find a way to make me share.

I like to LEARN, of course, usually some technical material related to writing or my job, and I’m now consciously reading books to GROW, like Art Matters by Neil Gaiman or It’s Not How Good You Are but How Good You Want To Be by Paul Arden. But I don’t take time to ORGANIZE those thoughts, nor do I seem to take time to SHARE them. That made me think.

I already take time to LEARN and GROW. I’ve already decided I need to take more time to ORGANIZE my thoughts, to be a better scientist. Can I also take some time to SHARE? Maybe I can put all those together into, like, an acronym! And that acronym will help me do it!

Okay, then, let’s go: LEARN-GROW-ORGANIZE-SHARE. LGOS!

Well.

That’s a terrible acronym.

Alright, alright, if first you don’t succeed, go home and rethink your life. Or something like that, like rethink your acronym. LEARN and GROW need to come before ORGANIZE and SHARE, but G is a better thing to start a word with, as GL or GR is a more common start than LG.

And I’m doing it to GROW. So perhaps it could be GROW LEARN, or GROW READ, naturally followed by ORGANIZE. That gets us GRO, which followed by S for SHARE is one letter short of GROSS; but what if instead we got to the point, and said what we have to do: WRITE.

So, here’s what I recommend to you (well, actually, to me): take some time every day to

  • GROW yourself by
  • READing to learn,
  • ORGANIZing your thoughts, and share them by
  • WRITING

GROW-READ-ORGANIZE-WRITE: GROW. Why, that’s nicely recursive: GROW to GROW! Since it is recursive, let me try this GROW thing out with this very GROW thought.

There.

How did it go?

-Anthony

Author Signing Now!

taidoka 0

Marriott, bottom floor, International Hall South. Follow the signs for the author signing, you can't miss it!