* Why Machines Learn: The Elegant Math Behind Modern AI, Anil Ananthaswamy, 2024. (Ananthaswamy is a science journalist, not an AI person, as I initially assumed. That said, he’s quite good.)
My Take on the Book
As discussed below, this is not the sort of book I’d usually read — my interest stops at the level or algorithms, and understanding the underlying math just doesn’t grab me. This really is a book for people interested in the math. But I learned some interesting, mostly-meta things from it.
Early learning systems. Early research focused on using simple networks (e.g. Perceptrons) to recognize fuzzy inputs like hand-written letters. This involved presenting a system with an input, having it try to map that input to a category, and then providing feedback to the system based on the mismatch between the input and the right (label for the input) result. Much of the math to do with measuring and tracking the degree of error. Even though they sound different things, this is also how adaptive filters: they try to learn the characteristics of noise so as to minimize it.
Matrices model networks! More generally, matrices and linear algebra are exceptionally useful for modeling networks.
A Lot of AI learning is just gradient descent. A lot of the math involves figuring out multi-dimensional spaces that characterize a domain, and using some sort of algorithm, usually involving gradient descent, to find the minima. The minima, of course, being where error is minimized, meaning that the system is performing as well as possible at its learning task.
People made jaw-droppingly crude simplifications, and yet the math still worked! Something I found very interesting was that, once you have an algorithm that appears to do what you want, you can make shockingly crude simplifications to it that (1) make it possible to run it in very high dimensional spaces, and (2) it will still work well. For example, if you are doing gradient descent in a multi-dimensional space, you can get away with finding the gradient for a randomly chosen single dimension (stochastic gradient descent), rather than in all dimensions. I still don’t get why that works, but it does. Not only does god play dice with the universe, but he’s OK with kludges tool.
Adding ‘noise’ can improve learning. Adding noise to data sets on which machines are trained can make the learning more robust. To me this seems sensible, in that, especially if you add different amounts of noise to the same features, you can multiply the training set and counter the tendency to overfit. Of course the noise ought to be ‘natural,’ which is to say that it ought to be native to the distribution from which you are sampling. I presume it is possible to figure that out for particular domains, but don’t actually know.
Monty Hall as Enabling Inside Trading. Hurrah. I finally understand the Monty Hall problem. In my defense, I will note that the ‘story problems’ I had in my primary education worked against my ability to solve it. In story problems, the stories didn’t actually matter, they were just frameworks for presenting a math problem. But in the Monty Hall problem, the key bit is Monty — he has inside information and will not choose to spoil the game by revealing where the prize is. Thus his action is not random, and provides additional information. If the Monty Hall problem were instead re-presented as an earthquake or windstorm accidentally revealing a goat behind one of the doors, the answer would be different.
WTF? — Why does deep learning work so well? We don’t, currently, really understand why deep learning works as well as it does. Spooky.,
Why do deep neural nets keep learning new things after they’ve overfit their training data? In particular, there is the paradox of benign overfitting / harmless interpolation: a deep neural network has so many parameters that it should perfectly overfit the training data, but — after enough training — it is able to generalize correctly to new data, something that it would not do if it had overfit. So what is going on? You’d think that once it was able to model the training data perfectly, it would stop learning. But perhaps the continued ‘training’ is introducing some kind of ‘noise’ or fluctuation which keeps perturbing the neural net?
Is something happening in the hidden layers??? I feel like there must be interesting stuff going on in the hidden layers of deep neural networks as their training continues beyond the point of benign overfitting/harmless interpolation. I don’t really understand what continued training of a deep neural net does — either directly or via back propagation. I have a vague idea idea that structure of the hidden layers undergoes some kind of phase change, in an analog to fractional crystallization. (In a magma chamber, where when the magma cools to a certain point a particular mineral crystalizes and precipitates, changing the nature of the magma and enabling further crystallization (and/or re-solution of prior crystalites) as the mix changes.)
In the epilogue the author raises the questions of whether LLMs have really learned, or are just doing statistics. I’m in the “stochastic parrot” camp.
Four Billion Years and Counting: Canada’s Geological Heritage. Produced by the Canadian Federation of Earth Sciences, by seven editors and dozens of authors. 2014.
November-December, 2024.
I am reading this with CJS. It is a nice overview of regional geology, and it is nice that all the examples come from Canada, and at least some of the discussion is relevant to Minnesota Geology. The book is notable for its beautifully done pictures and diagrams.
Although its a famous play, and does indeed contain some striking things — particularly Iago’s manipulation of Othello, and also the use of the hankerchief as symbol of fidelity and betrayal – I was not that keen on this play. Give me some comedy, or at least a little more magic!
Precis of the play
Othello, a famous general fighting for Venice, has just married Desdemona, to the dismay of her father. Othello is black, and an outsider, and knows little of the customs or society of Venice – but he is valued due to his military prowess, especially as the Turks seem about to attack. He has chosen the polished and bookish Cassio as his lieutenant, much to the distress and anger of Iago, who has spent his life in the field and believes he has earned the postion. Iago decides to get revenge, and aims to destroy Cassio and Desdemona and, through her, Othello.
After this, the play unfolds in a straightforward way. Iago subtly raises questions about Desdemona’s faithfulness – all the while pretending that he is reluctant to speak and is unsure of the truth of what he is saying – and in a famous scene transforms Othello’s trust of Desdemona into suspicion, suggesting that she is having an affair with Cassio. Iago is one of Shakespeare’s most famous villians – Coleridge referred to him as having “motiveless malignity.”
Othello wants visible proof, and here Desdemona’s hankerchief comes into play. It was her first gift from Othello, and it was woven by a fortune teller with magical properties. Iago secrets Desdemona’s hankerchief (which she had lost and Emilia found and given to Iago) in Cassio’s quarters. Cassio finds the hankerchief and gives it to the courtesian Bianca to copy – Othello watches this from a distance, and believes it proof of Desdemona’s infidelity. Othello orders Iago to kill Cassio, and Othello strangles Desdemona. When it is revealed that Desdemona was innocent, Othello kills himself.
This is an excellent book; interesting well-documented science, and some beautiful and erudite writing as well. The basic argument — that grammar determines what must be specified, rather than what can be specified, and in that manner instills certain habits of mind that effect how people see the world — seems correct, if not quite living up to the subtitle of the book: Why the World Looks Different in Other Languages.
Perhaps the most interesting and fun part of the book was to be introduced to languages that work very differently from English: The Mates language (in Peru) that requires speakers to specify whether the fact they report is based on personal observation, indirect evidence, or hearsay; and the Australian language that has no egocentric prepositions, but requires all positional information to be reported in terms of the cardinal directions, thus requiring their speakers to always be oriented.
This book was a pleasure to read. I plan to seek out other work by this writer.
Contents
Front Matter
On whether languages reflect the characteristics of their speakers, he writes:
Many a dinner table conversation is embellished by such vignettes, for few subjects lend themselves more readily to disquisition than the character of different languages and their speakers. And yet should these lofty observations be carried away from the conviviality of the dining room to the chill of the study, they would quickly collapse like a soufflé of airy anecdote-at best amusing and meaningless, at worst bigoted and absurd.
— p. 2
The basic argument of the book is this:
The effects that have emerged from recent research, however, are far more down to earth. They are to do with the habits of mind that language can instill on the ground level of thought: on memory, attention, perception, and associations. And while these effects may be less wild than those flaunted in the past, we shall see that some of them are no less striking for all that.
I think it is correct, but that the subtitle of the book – Why the World Looks Different in Other Languages – is a bit of an exaggeration.
C1-5: <Reprise of history and status of color terms>
C1: Naming the Rainbow
This chapter reprises now-unknown work by William Gladstone (now remembered as an English prime minister) on Homer and his writings, and focuses in on particular on one chapter in Gladstone’s monumental 3,000 page work: a chapter on Homer’s use of color terms.
Gladstone’s scrutiny of the Iliad and the Odyssey revealed that there is something awry about Homer’s descriptions of color, and the conclusions Gladstone draws from his discovery are so radical and so bewildering that his contemporaries are entirely unable to digest them and largely dismiss them out of hand. But before long, Gladstone’s conundrum will launch a thousand ships of learning, have a profound effect on the development of at least three academic disciplines, and trigger a war over the control of language between nature and culture that after 150 years shows no sign of abating.
Gladstone notes that Homer uses color terms in odd ways — the famous “wine dark sea” (really “wine-looking” sea) being just one example.
Mostly Homer, as well as other Greek authors of the period, use color very little in their descriptions: mostly they use black or white; terms for colors are used infrequently and inconsistently. For example, the only other use of “wine-looking” is to describe the color of oxen.
Gladstone’s fourth point is the vast predominance of the “most crude and elemental forms of color”-black and white-over every other. He counts that Homer uses the adjective melas (black) about 170 times in the poems, and this does not even include instances of the corresponding verb “to grow black,” as when the sea is described as “black-ening beneath the ripple of the West Wind that is newly risen.” Words meaning “white” appear around 100 times. In contrast to this abun-dance, the word eruthros (red) appears thirteen times, xanthos (yellow) is hardly found ten times, ioeis (violet) six times, and other colors even less often.
C6: Crying Whorf
This chapter describes the origin, rise and fall of linguistic relativity. Sapir is depicted as respectable but making over-stated claims; Whorf comes across as a charlatan, for example, making claims to have deeply studied Hopi, when he only had access to a single informant in New York – and making broad claims that are entirely wrong (e.g. that the Hopi language does not have a future tense).
Deutscher traces the origin of linguistic relativity to Wilhelm von Humboldt in 1799, whose “linguistic road to Damascus led through the Pyrennes.” Deutscher encountered the Basque language, and found that it was radically different from the languages linguists tended to study. He then sought out other ‘more exotic’ languages, which he found by going to the Vatican library and studying the notes of Jesuit missionaries to South and Central America: “…Humboldt was barely scratching the surface. But the dim ray of light that shown from his materials felt dazzling nonetheless because of the utter darkness in which he and his contemporaries had languished.” p. 135 Although Humboldt’s ideas led to linguistic relativity, it should be noted that he had a much more nuanced and correct view: In principle, any language may express any idea; the real differences among languages are not what they are able to express but in “what it encourages and stimulates its speakers to do from its own inner force.” But this view was not carried forward, and instead: “The Humboldtian ideas now underwent a process of rapid fermentation, and as the spirit of the new theory grew more powerful, the rhetoric became less sober. ”
All that said, Deutscher argues it is a mistake to dismiss the idea that language has no influence over thought. But rather than taking the strong case the language constrains thought, he instead argues the habits of language may lead to habits of mind. In the case of the influence of language, and refers to the idea that Boas introduces and that Jakobson crystalized into a maxim: “Languages differ in what they must convey, and not in what they may convey.”
Phrases I like
“…has still the power to disturb our hearts.” [Sapir, referring to Homer, Ovid, etc.] p. 129
“[His] linguistic road to Damascus led through the Pyrennes.” p. 134
“…Humboldt was barely scratching the surface. But the dim ray of light that shown from his materials felt dazzling nonetheless because of the utter darkness in which he and his contemporaries had languished.” p. 135
Background: Henry V is son of Henry IV, who obtained the throne by usurping it from Richard II – this means that there is some feeling that neither Henry is a legitimate ruler. Before becoming King, Henry V was a wild youth, dissipated and engaged in lascivious acts. But on his father’s death, Henry becomes a serious and mature ruler.
The play opens with a chorus praising Henry as an unmatched warrior King. But then, the next act depicts the Archbishop of Canterbery revealing his plan to avert a harsh tax on the Church by legitimizing and encouraging Henry’s plans to invade France and take its throne.
Act 2 begins with the chorus describing the desire of the young men of England to pursue honor by participating in this war. The first scene following this shows conversation – and almost a fight, between three old soldiers who are erstwhile companions of Henry – the depicts honor as the least of their concerns. The second scene of Act shows the unmasking of traitors among the Lords who support Henry.
Act III. The war has begun. The English army, led by Henry, lays siege to the French town of Harfleur. Before the gates, Henry delivers a rousing speech (“Once more unto the breach…”) to rally his soldiers; the siege takes a heavy toll, and the town eventually surrenders.
In Act IV, Henry has arrived at Agincourt; his army is weary and outnumbered. Henry, in disguise, walks among his soldiers at night, listening to their fears and doubts. In the morning, Henry delivers his famous St. Crispin’s Day speech, which lifts the English spirits.
In Act V, the battle of Agincourt is won by the English. Henry returns to England, where the victory is celebrated, and then to France, to negotiate the final terms of the peace. There he woos a reluctant Princess Katherine, which marriage will solidify his claim to the thrown. The play ends with a reminder that Henry will die, and things will unravel.
Structure of the Play
1. Invasion Groundwork
Prologue: Chorus wishes for a greater stage, and tells audience to use its imagination.
1.1 Theological Justification Bishops of Canterbury and Ely discuss bill that will seize money from the search; they plan to avoid it by providing a theological justification for Henry V’s claim to France, and thus his invasion. They also mention how much Henry V has changed since his father’s death: “And so the Prince obscured his contemplation / Under the veil of wildness / which, no doubt, grew like the summer grass, fastest by night / Unseen yet crescive in his facility”
1.2: Bishops assure H of invasion’s morality; tennis ball mock Henry V invites the Bishops to give an explication of the law regarding his claims to France, and they do so, even as Henry repeatedly asks them to be honest about it. Henry also raises the possibility of Scotland invading should he go to France, but the Bishops argue that that can be defended against. Finally, after deciding that he will take control of France, by invasion if necessary, he invites in the French ambassadors, who, in a message from the Dauphin, present him with a barrel of tennis balls. Henry says he will play play a set in France, and will “strike his father’s crown into the hazard.” Exter, uncle to the King, is present and speaks a line or two.
2. Preparations for War
Elimination of traitors; introduction of common solidiers; preparation by France
Chorus: The chorus describes the excitement in England about the coming war – They sell the pasture now to buy the horse – and provide notice that three nobles – Cambridge, Scroop, and Grey –have become traitors.
Bardolph, Henry’s former tavern companion, prevents two solidiers – Nym and Pistol – from fighting over Hostess Quickly, Pistol’s wife, and requires them to become friends. They are interrupted by news that Falstaff is dying.
Cambridge, Scroop, and Grey are brought into Henry V’s presence, not realizing that he knows they are traitors, and are asked about whether Henry should show mercy to someone who has spoken against it. They say no, and override Henry’s wishes to show clemency. He the reveals that he knows of their betrayals, and they are all condemned to death.
Falstaff has died. Bardolph, Nym, Pistol and Hostess Quickly morn his death. The three men prepare to depart for France, and Pistol bids Hostess Quickly goodbye.
The King of France and the Dauphin plan for the defense of France against Henry – the King is cautious, the Dauphin is not, being contemptuous of Henry, and ignoring warnings about Henry’s new ethos. Exter enters as ambassador, and asks the King of France to yield to Henry, and returns insults to the Dauphin. The King says he will answer in the morning: “A night is but small breath and little pause / To answer matters of this consequence.”
3. Invasion, part 1: Success as Harlefor surrenders
Initial success: Harlefor surrenders; commoners show cowardance; 5:1 odds
Chorus: Describes the departure of the English navy: … Play with your fancies and in them behold, Upon the hempen tackle, shipboys climbing. Hear the shrill whistle, which doth order give To sounds confused. Behold the threaden sails, Borne with th’ invisible and creeping wind, Draw the huge bottoms through the furrowed sea Breasting the lofty surge. O, do but think You stand upon the rivage and behold A city on th’ inconstant billows dancing… and notes that the French King offered the hand of his daughter and some small unprofitable dukedoms – this offer is disregarded (and is reported only after the navy is described as being launched).
The invasion begins: “Once more into the breach, dear friends, once more / Or close the wall up with our English dead.” Henry makes a speech as the prepare to advance.
The three soldiers show their cowardence in trying to withdraw from the assault – they are driven back to it by Captain Fluellen. Captain Fluellen then engages in discussions and disputations with three other Captains: Glower, Jamy, Macmorris. [Not quite sure of the point of this scene]
Henry gives a speech before the gates of Harlefor, saying it is their last chance, and that they will be to blame if they do not surrender and the city is ravaged:
I will not leave the half-achieved Harfleur Till in her ashes she lie buried. The gates of mercy shall be all shut up, And the fleshed soldier, rough and hard of heart In liberty of bloody hand, shall range With conscience wide as hell, mowing like grass Your fresh fair virgins and your flow’ring infant What is it then to me if impious war, Arrayed in flames like to the prince of fiends, Do with his smirched complexion all fell feats Unlinked to waste and desolation?
Katherine, Princess of France, has one of her maids teach her English. [The scene appears to be presented in French – would the audience have understood???]
The governor surrenders the town, and Henry spares its citizens. [Neither of these things happened in history.]
The French nobles are embarrassed by Henry’s successful invasion. But they convince themselves they will triumph, and send an ambassador to ask what ransom Henry will offer when he is captured.
Ancient Pistol has distinguished himself and pleads with Captain Fluellen for the life of Bardoph, who has been sentenced to death for stealing. His plea is rejected, and he departs with a curse. Captain Fluellen talks with Henry, and mentions Bardolph, whose execution Henry upholds. The French Ambassador, Mountjoy arrives to enquire about Henry’s ransom: Henry says ‘nothing but my body.’
The French nobles, confident of their victory on the eve of the battle, boast and banter among themselves.
4. Invasion, part 2: Triumph at Agincourt
Eve of battle; Henry & Williams & Fluellen; Pistol demands ransom; triumph at Agincourt
The Chorus draws a beautiful picture of the two armies the night before the battle, camped across from one another, awaiting the morning. The French confident, the English anxious… but with Henry moving among them to raise morale. Now entertain conjecture of a time When creeping murmur and the poring dark Fills the wide vessel of the universe From camp to camp, through the foul womb of night, The hum of either army stilly sounds, That the fixed sentinels almost receive The secret whispers of each other’s watch. Fire answers fire, and through their paly flames Each battle sees the other’s umbered face
Henry walks though his camp, in disguise. He encounters Pistol, overhears a conversation between Grover and Fluellen that leaves him impressed with the Welshman’s quality, and argues with a soldier – Williams – about the King’s responsibility for the spiritual fate of his solidiers – they exchange gloves with the intention of dueling later. Last, Henry laments his father’s usurpation of Richard II’s throne.
The French nobles, about to fight, lament that the English are so few and weak.
Henry gives a speech of encouragement again. Responding to someone wishing for more men, Henry says he does not wish for more, and furthermore that those who do not wish to figtht will be furnished with passage home. ‘I do not wish to share the honor more than I have to,’ is his sentiment.
The ambassador, Mountjoy, comes again to negotiate a ransom, which Henry refuses.
A French soldier surrenders to Pistol, who threatens to kill him unless he provides a ransom.
The French nobles recognize that they have been defeated, and, ashamed, vow that they will die in battle.
Henry hears of the deaths of York and Suffolk; unsure of whether he had victory, when he hears a French call to arms he orders all French prisoners killed.
Fluellen in conversation with Grover compare Henry to Alexander the Great. Montjoy arrives with the French surrender. Williams appears with the glove, which Henry does not acknowledge; but Henry give Fluellen the other glove and sends him after Williams, and then sends others after Fluellen to prevent a full fight.
William encounters Fluellen, and strikes him. The other men arrive and prevent an escalation. Henry arrives and explains what happens and ‘pardons’ Williams, and has his glove filled with crowns. [I’m not quite sure of what happens after this, especially between Williams and Fluellen—Fluellen seems to do an about face and now thinks well of Williams.] The scene ends with the numbers of the dead being announced, and Henry giving credit for the victory to god.
5. Treaty signed, and marriage
Treaty signed and Princess Kate agrees to marry Henry; Fluellen gets revenge
Chorus: Brings Henry back to England where he and his victory are celebrated, and then back to France where the treaty recognizing Henry as sovereign will be signed.
Fluellen, via use of a cudgel, forces Pistol to eat a leek to avenge his insults; Pistol decides to return to England where he will wear his cudgel wounds to pretend to be a wounded soldier.
Henry and the King of France meet, and Henry delegates negotiation to his nobles while he woos Princess Katherine – she consents to marrying him, but without, it seems to me, much understanding or enthusiasm. Henry rides roughshod over her preference not to kiss before the wedding: “O Kate, nice customs curtsy to great Kings.”
A few notes
Throughout the play we see that Henry has separated himself from his old base companions: Falstaff dies (and was previously exiled); Henry allows Barloph to be hanged for stealing; the Bishops remark on how Henry has changed.
Deception: Not much. Henry goes in disguise among his troops. Henry incident with William. Henry does not tell Fluellen what is up when he sends him after William. Henry uses lots of flowery words which it is unlikely Princess Kate will understand.
??? Is Henrys order to kill the prisoners proper?
??? Does Henry really think the war is just?
??? Henry says that if they do not surrender, governor will be responsible for soldiers’ depredations.
Quotes I like
Now entertain conjecture of a time When creeping murmur and the poring dark Fills the wide vessel of the universe. From camp to camp, through the foul womb of night, The hum of either army stilly sounds, That the fixed sentinels almost receive The secret whispers of each other’s watch. Fire answers fire, and through their paly flames Each battle sees the other’s umbered face;
I will not leave the half-achieved Harfleur Till in her ashes she lie buried. The gates of mercy shall be all shut up, And the fleshed soldier, rough and hard of heart In liberty of bloody hand, shall range With conscience wide as hell, mowing like grass: Your fresh fair virgins and your flow’ring infant What is it then to me if impious war, Arrayed in flames like to the prince of fiends, Do with his smirched complexion all fell feats Enlinked to waste and desolation?
Play with your fancies and in them behold, Upon the hempen tackle, shipboys climbing. Hear the shrill whistle, which doth order give To sounds confused. Behold the threaden sails, Borne with th’ invisible and creeping wind, Draw the huge bottoms through the furrowes Breasting the lofty surge. O, do but think You stand upon the rivage and behold A city on th’ inconstant billows dancing…
