Julie Beth Lovins (October 19, 1945, in Washington, D.C. – January 26, 2018, in Mountain View, California) was a computational linguist who published The Lovins Stemming Algorithm - a type of stemming algorithm for word matching - in 1968. The Lovins Stemmer is a single pass, context sensitive stemmer, which removes endings based on the longest-match principle. The stemmer was the first to be published and was extremely well developed considering the date of its release, having been the main influence on a large amount of the future work in the area. -Adam G., et al == Background == Born on October 19, 1945, in Washington, D.C., Lovins grew up in Amherst, Massachusetts. Her father Gerald H. Lovins was an engineer and her mother, Miriam Lovins, a social services administrator. Lovins' brother Amory Lovins is the co-founder and chief environmental scientist of Rocky Mountain Institute. For her undergraduate degree, Lovins attended Pembroke College, the women's college of Brown University, which later combined into Brown University in 1971. At Pembroke College, Lovins studied mathematics and linguistics, graduating with honors. Her thesis was named, A Study of Idioms. She received the inaugural Bloch Fellowship in 1970 from the Linguistic Society of America to attend graduate school. Lovins obtained her Master of Arts in 1970 and Doctor of Philosophy in 1973 from the University of Chicago, studying linguistics. At the University of Chicago, her dissertation was titled, Loan Phonology -- Subject Matter. A revision of her thesis on loanwords and the phonological structure of Japanese was published in 1975 by the Indiana University Linguistics Club. == Teaching career == Following Lovins' PhD, she spent a year working as a linguist-at-large at a University of Tokyo language research institute and as an English conversation teacher. She then joined the faculty at Tsuda College as a professor of English and linguistics, where she taught for seven years. During her time as a faculty member at Tsuda College, Lovins also served as a guest researcher in the University of Tokyo's Research Institute of Logopedics and Phoniatrics, a research center for speech science. == Industry career == After teaching Japanese phonology at Japanese universities abroad, Lovins moved back to the U.S. to work in the computing industry. She worked on early speech synthesis at Bell Labs in Murray Hill, New Jersey. At Bell Labs, Lovins worked with Osamu Fujimura, a Japanese linguist who is credited as a pioneer in speech sciences. Lovins also worked as a software engineer at various companies in Silicon Valley and served as a consultant for computational linguistics throughout the 1990s. As a consultant, she called her business, "The Language Doctor." == The Lovins Stemming Algorithm == Lovins published an article about her work on developing a stemming algorithm through the Research Laboratory of Electronics at MIT in 1968. Lovins' stemming algorithm is frequently referred to as the Lovins stemmer. A stemming algorithm is the process of taking a word with suffixes and reducing it to its root, or base word. Stemming algorithms are used to improve the accuracy in information retrieval and in domain analysis. These algorithms help find variants of the terms being queried. Stemming algorithms bring value in their reduction of a given query into its less complex form, allowing more similar documents to be retrieved for similar queries. Stemming algorithms are prevalent in search engines, such as Google Search, which did not implement word stemming until 2003. This means that up until 2003, a Google search for the word warm would not have explicitly returned results for related words like warmth or warming. As the first published stemming algorithm, Lovins' work set a precedent and influenced future work in stemming algorithms, such as the Porter Stemmer published by Martin Porter in 1980 which has been recognized widely as the most common stemming algorithm for stemming English. Additionally, the Dawson Stemmer developed by John Dawson is an extension of the Lovins stemmer. The Lovins stemmer follows a rule-based affix elimination approach. It first removes the longest identifiable suffix from the target word - producing a base stem word - then indexes a lookup table to convert the (potentially malformed) stem word to a valid word. This process can be split into two phases. In the first phase, a word is compared with a pre-determined list of endings, and when a word is found to contain one of these endings, the ending is removed, leaving only the stem of the word. The second phase standardizes spelling exceptions that come from the first phase, ensuring that words with only marginally varying stems are appropriately paired together. For example, with the word dried, phase one results in dri, which should match with the word dry. The second phase takes care of these exceptions. Compared to other stemmers, Lovins' algorithm is fast and equipped to handle irregular plural words like person and people. Disadvantages, however, include many suffixes not being available in the table of endings. Furthermore, it is sometimes highly unreliable and frequently fails to form valid words from the stems or to match the stems of like-meaning words. This is most often caused by the usage of specialist terminology and domain-specific vocabulary by the author. == Personal life == Lovins moved to Mountain View, California, in 1979, and later to Old Mountain View in 1981 with her partner and later husband Greg Fowler, a software engineer and advocate for environmental issues & the blind. In their free time, she and her husband enjoyed taking walks and volunteering for their local community. Lovins actively volunteered for organizations like the Old Mountain View Neighborhood Association, Mountain View Friends of the Library, League of Women Voters, Mountain View Cool Cities Team, and the Mountain View Sustainability Task Force. In 2016, Lovins' husband died unexpectedly, following a heart attack. Eighteen days after her husband died, Lovins was diagnosed with brain cancer. She died on January 26, 2018, at a hospice, surrounded by friends, family and caregivers.
Sensory, Inc.
Sensory, Inc. is an American company which develops software AI technologies for speech, sound and vision. It is based in Santa Clara, California. Sensory’s technologies have shipped in over three billion products from hundreds of leading consumer electronics manufacturers including AT&T, Hasbro, Huawei, Google, Amazon, Samsung, LG, Mattel, Motorola, Plantronics, GoPro, Sony, Tencent, Garmin, LG, Microsoft, Lenovo, and more. Sensory has over 60 issued patents covering speech recognition in consumer electronics, biometric authentication, sensor/speech combinations, wake word technology, and more. == History == Sensory, Inc. was founded in 1994, originally as Sensory Circuits, by Forrest Mozer, Mike Mozer and Todd Mozer. The three had also co-founded ESS Technology years earlier. In 1999 Sensory acquired Fluent Speech Technologies, which was formed and started by a group of professors out of the Oregon Graduate Institute (formerly OGI, now OHSU). Fluent Speech Technologies developed high performance embedded speech engines, the technology from this acquisition is now the core technology used throughout Sensory's chip and software line. === Company timeline === 1994 – Founded 1995 – Introduces the RSC 164 - first commercially successful speech recognition IC 1998 – Introduces first speaker verification IC 2000 – Acquires Oregon based Fluent-Speech Technologies 2002 – Acquires Texas Instruments line of speech output ICs (the SC series) 2007 – Introduces first Voice User Interface for Bluetooth silicon (CSR BC-5) - BlueGenie 2008 - Sensory and BlueAnt partner on the V1 - Revolutionary new Bluetooth headset with a voice user interface. First wearable to use a voice user interface for control and best-reviewed speech recognition product in history 2009 – Introduced world's smallest text to speech system (TTS) and Truly HandsfreeTM Triggers/ wake words. 2010 – Introduced the NLP-5x – First Natural Language Voice Processor and TrulyHandsfree wake words in SDKs for Android, iOS, Linux, and Windows. NLP5x used the first generation of TrulyHandsfree wake words with low power and enhanced accuracy. 2011 – Sensory partners with Google and Microsoft to enable TrulyHandsfree as a front end to Goog411 and Bing411 2012 – Partnered with Tensilica to offer ultra-low power TrulyHandsfree wake words; introduced Speaker Verification and Speaker Identification for mobile phones and other consumer electronics. 2012 - TrulyHandsfree released into Samsung's Galaxy S2 for "Hey Galaxy" wake word 2013 – TrulyHandsfree wake words migrated to many new platforms and began shipping as MotoVoice in the Google-owned MotoX. Sensory's TrulyHandsfree in mobile takes off with the Galaxy S3 and S4 and Galaxy Note and is licensed into wearables like Google Glass. 2014 – Announced new initiative in Vision; added LG and Motorola as customers; received the 2014 Global Mobile Award for Best Mobile Technology Breakthrough at the GSMA Mobile World Congress in Barcelona, Spain (judges commented, "A big advance for the wearables market, this offers many benefits for consumers, increasing uptake and usage of many mobile apps, driving revenue for operators and content providers.") 2015-2018 - Licensed Google, Amazon, MSFT, Baidu, Huawei, ZTE, and many others with TrulyHandsfree wake words. Sensory develops first wake words for OK Google, Hey Siri, and Hey Cortana. 2019 - Sensory launched two new solutions: SoundID, sound identification, and TrulyNatural, embedded large vocabulary speech recognition. Sensory also acquired Vocalize.ai, an independent testing lab. 2020 - Sensory introduced VoiceHub, which allows the automated generation of wake words. 2021 - Sensory expands VoiceHub with speech recognition and NLU capabilities. The company initiated a new cloud platform, SensoryCloud.ai. 2022-Sensory rolls out SensoryCloud.ai with speech to text, text to speech, face & voice biometrics 2024- Sensory Automotive & TrulyNatural Speech-to-text On-Device launched == Technology and products == Sensory originally developed both hardware (Integrated Circuit - IC or "chip") and software platforms but migrated to software only around 2005 and added cloud and hybrid computing capabilities in 2021. Sensory's RSC-164 IC (Integrated Circuit or "chip") was used on NASA's Mars Polar Lander in the Mars Microphone on the Lander. Speech Synthesis SC-6x chips – acquired some speech synthesis technology from Texas Instruments. Sensory’s embedded AI solutions include the following: TrulyHandsfree (THF) - wake word detection and phrase spotting. TrulyNatural (TNL) - large vocabulary continuous speech recognition with NLU. TrulySecure (TS) - face and voice biometrics. TrulySecureSpeakerVerification (TSSV) - speaker and sound identification. VoiceHub - Online portal for creating custom wake words and speech recognition models with NLU. Sensory Automotive- Sensory Automotive is a full voice and vision suite of AI technologies that operate efficiently in the car without connecting to a network. The cloud initiative, SensoryCloud.ai, is targeting Speech To Text (STT), Text To Speech (TTS), Wake Word verification, face and voice recognition, and sound identification.
WebCrow
The WebCrow is a research project carried out at the Information Engineering Department of the University of Siena with the purpose of automatically solving crosswords. == The Project == The scientific relevance of the project can be understood considering that cracking crosswords requires human-level knowledge. Unlike chess and related games and there is no closed world configuration space. A first nucleus of technology, such as search engines, information retrieval, and machine learning techniques enable computers to enfold with semantics real-life concepts. The project is based on a software system whose major assumption is to attack crosswords making use of the Web as its primary source of knowledge. WebCrow is very fast and often thrashes human challengers in competitions, especially on multi language crossword schemes. A distinct feature of the WebCrow software system is to combine properly natural language processing (NLP) techniques, the Google web search engine, and constraint satisfaction algorithms from artificial intelligence to acquire knowledge and to fill the schema. The most important component of WebCrow is the Web Search Module (WSM), which implements a domain specific web based question answering algorithm. The way WebCrow approaches crosswords solving is quite different with respect to humans: Whereas we tend to first answer clues we are sure of and then proceed filling the schema by exploiting the already answered clues as hints, WebCrow uses two clearly distinct stages. In the first one, it processes all the clues and tries to answer them all: For each clue it finds many possible candidates and sorts them according to complex ranking models mainly based on a probability criteria. In the second stage, WebCrow uses constraint satisfaction algorithms to fill the grid with the overall most likely combination of clue answers. In order to interact with Google, first of all, WebCrow needs to compose queries on the basis of the given clues. This is done by query expansion, whose purpose is to convert the clue into a query expressed by a simplified and more appropriate language for Google. The retrieved documents are parsed so as to extract a list of word candidates that are congruent with the crossword length constraints. Crosswords can hardly be faced by using encyclopedic knowledge only, since many clues are wordplays or are otherwise purposefully very ambiguous. This enigmatic component of crosswords is faced by a massive use of database of solved crosswords, and by automatic reasoning on a properly organized knowledge base of wired rules. Last but not the least, the final constraint satisfaction step is very effective to fill the correct candidate, even though, unlike humans, the system can not rely on very high confidence on the correctness of the answer. == Competitions == WebCrow speed and effectiveness has been tested many times in man-machine competitions on Italian, English and multi-language crosswords The outcome of the tests is that WebCrow can successfully compete with average human players on single language schemes and reaches expert level performance in multi-language crosswords. However, WebCrow has not reached expert level in single-language crosswords, yet. === ECAI-06 Competition === On August 30, 2006, at the European Conference on Artificial Intelligence (ECAI2006), 25 conference attendees and 53 internet connected crosswords lovers, competed with WebCrow in an official challenge organized within the conference program. The challenge consisted in 5 different crosswords (2 in Italian, 2 in English and one multi-language in Italian and English) and 15 minutes were assigned for each crossword. WebCrow ranked 21 out of 74 participants in the Italian competition, and won both the bilingual and English competitions. === Other Competitions === Several competitions have been held in Florence, Italy within the Creativity Festival in December 2006, and another official conference competition took place in Hyderabad, India in January 2007, within the International Conference of Artificial Intelligence, where it ranked second out of 25 participants.
Knights of Sidonia
Knights of Sidonia (Japanese: シドニアの騎士, Hepburn: Shidonia no Kishi) is a Japanese manga series written and illustrated by Tsutomu Nihei. It was serialized by Kodansha's seinen manga magazine Monthly Afternoon between April 2009 and September 2015, with its chapters collected in 15 tankōbon volumes. It tells the story of Nagate Tanikaze, an "under-dweller" destined to become a Garde pilot, whose mission is to defend the generation ship Sidonia from a hostile alien species called Gauna. The manga was licensed for English release in North America by Vertical. An anime television series adaptation was produced by Polygon Pictures. The first season aired from April to June 2014; the second between April and June 2015. An anime film sequel titled Knights of Sidonia: Love Woven in the Stars premiered in June 2021. In 2015, Knights of Sidonia received the 39th Kodansha Manga Award in the general category, as well as the 47th Seiun Award in the Best Comic category in 2016. == Plot == === Setting === The story is set in the year 3394, a thousand years after mankind flees from Earth after it was destroyed by a race of shapeshifting aliens called the Gauna (奇居子（ガウナ）), aboard hundreds of colossal spacecraft created from the remains of the planet. One such ship is the Sidonia, which has developed its own human culture closely based on that of Japan where human cloning, asexual reproduction, and human genetic engineering, such as granting humans photosynthesis, are commonplace. It is also revealed that the top echelons of this society have secretly been granted immortality. With a population of over 500,000 people, Sidonia is possibly the last human settlement remaining, as the fates of the other ships are unknown. Little is known about the true nature of the Gauna or their motivation for attacking humanity. At any given time, a Gauna consists of a nearly impenetrable core protected by a dense layer of malleable flesh known as "placenta" (胞衣, ena). Once the ena is shed away and the core is destroyed, the Gauna's body disintegrates. While Sidonia itself is heavily armed with an arsenal of high-output beam cannons and mass cannons including slow but powerful planet-destroying warheads, it is primarily defended by large mechanized weapons called Gardes (衛人, Morito) whose weaponry and mobility is powered by "Higgs particles" (ヘイグス粒子, Heigusu Ryūshi), armed with a high-output beam cannon for long range assaults and a special spear known as "Kabizashi" for close combat. The tip of the kabizashi is made of a rare and little-understood material which has the unique property of being able to destroy a Gauna's core. Later the Gardes are also equipped with firearms whose ammunition have the same material of the Kabizashi after a means to artificially mass-produce it is discovered. Most people in the surviving human population are screened and drafted as Garde pilots at a young age, if they are shown to be capable of piloting them. === Story === The story follows the adventures of Garde pilot Nagate Tanikaze, who lived in the underground layer of Sidonia since birth and was raised by his grandfather. Never having met anyone else, he trains himself in an old Guardian pilot simulator every day, eventually mastering it. After his grandfather's death, he emerges to the surface and is selected as a Garde pilot, just as Sidonia is once again threatened by the Gauna. == Media == === Manga === Written and illustrated by Tsutomu Nihei, Knights of Sidonia was serialized in Kodansha's seinen manga magazine Monthly Afternoon from April 25, 2009, to September 25, 2015. It was compiled in 15 tankōbon volumes. The manga has been licensed in North America by Vertical, who released all 15 volumes in English between February 5, 2013, and April 26, 2016. === Anime === An anime television series adaptation, produced by Polygon Pictures, aired its first season from April 10 to June 26, 2014, on MBS and later on TBS, CBC and BS-TBS. The series was directed by Kōbun Shizuno, assisted by Hiroyuki Seshita, with scripts by Sadayuki Murai and character designs by Yuki Moriyama. The opening theme song is "Sidonia" (シドニア, Shidonia), performed by Angela, while the ending theme song is "Show" (掌 -show-, Shō), performed by Eri Kitamura. A second season aired from April 11 to June 26, 2015. For the second season, the opening theme song is "Kishi Kōshinkyoku" (騎士行進曲, Knight March), performed by Angela, while the ending theme song is "Requiem" (鎮魂歌 -レクイエム-, Rekuiemu), performed by CustomiZ. The series was localized and streamed by Netflix in all of its territories since July 4, 2014, becoming the service's first original anime, as well as the first anime series on Netflix available in Dolby Vision/HDR. The first season has been licensed for home video release by Sentai Filmworks. The second season was released on Netflix on July 3, 2015, and has been licensed by Sentai Filmworks for home video distribution. In July 2021, Funimation announced they acquired the streaming rights from Netflix to both seasons. === Films === A compilation film of the first season with additional scenes and re-edited sound effects was released on March 6, 2015. A new anime film, titled Knights of Sidonia: Love Woven in the Stars, was announced on July 3, 2020. Hiroyuki Seshita served as chief director, while Tadahiro Yoshihira served as director for the new film, with Polygon Pictures returning for production. Sadayuki Murai and Tetsuya Yamada returned to write scripts, while Shūji Katayama composed the music. The rest of the staff and cast returned to reprise their roles. The first four minutes of the film were shown on YouTube on April 28, 2021. The film was set to premiere on May 14, 2021, but was delayed to June 4, 2021, due to the COVID-19 pandemic. Funimation screened the film in international theaters starting on September 13, 2021. == Reception == === Manga === Knights of Sidonia won the 39th Kodansha Manga Award in the general category in 2015. The manga won the 47th Seiun Award in the Best Comic category in 2016. It also won the Best Seinen category at the 26th Salón del Manga de Barcelona in 2020. It was one of the Jury Recommended works in the Manga Division at the 17th Japan Media Arts Festival in 2013. The Young Adult Library Services Association listed Knights of Sidonia in its 2014 list of Top 10 Graphic Novels for Teens. Carlo Santos from Anime News Network gave the first manga volume a B, stating, "It is got a young man piloting a giant robot against alien enemies, but Knight of Sidonia is no Neon Genesis Evangelion. Yet it is not as bleak or incomprehensible as Tsutomu Nihei works like Blame! or Biomega, either—rather, it is the best of both worlds, bringing Nihei's hard sci-fi mentality into a more conventional space-adventure environment". === Anime === The anime series received positive reviews, even from famous members of the Japanese anime/game industry, like Hideo Kojima, creator of the Metal Gear series, who claims that "It's a kind of anime that we haven't seen for a while that has that sci-fi spirit. Using digital technology cultivated through games, it creates animation that encapsulates Japan's cultural assets like manga, cel animation, kanji, giant robots, etc. What's born is a unique made-in-Japan work that could never be cooked up in Hollywood. Japanese culture has lost its 'cool', and Knights of Sidonia will be the white knight that saves it". Other industry pros left acknowledgements as well, including Akiko Higashimura, Digitarou and Yoshinao Dao.
Fuzzy concept
A fuzzy concept is an idea of which the boundaries of application can vary considerably according to context or conditions, instead of being fixed once and for all. That means the idea is somewhat vague or imprecise. Yet it is not unclear or meaningless. It has a definite meaning, which can often be made more exact with further elaboration and specification — including a closer definition of the context in which the concept is used. The inverse of a "fuzzy concept" is a "crisp concept" (i.e. a precise concept). Fuzzy concepts are often used to navigate imprecision in the real world, when precise information is not available and an approximate indication is sufficient to be helpful. Although the linguist George Philip Lakoff already defined the semantics of a fuzzy concept in 1973 (inspired by an unpublished 1971 paper by Eleanor Rosch,) the term "fuzzy concept" rarely received a standalone entry in dictionaries, handbooks and encyclopedias. Sometimes it was defined in encyclopedia articles on fuzzy logic, or it was simply equated with a mathematical “fuzzy set”. A fuzzy concept can be "fuzzy" for many different reasons in different contexts. This makes it harder to provide a precise definition that covers all cases. Paradoxically, the definition of fuzzy concepts may itself be somewhat "fuzzy". Lotfi A. Zadeh, known as "the father of fuzzy logic", claimed that "vagueness connotes insufficient specificity, whereas fuzziness connotes unsharpness of class boundaries". Not all scholars agree. With increasing academic literature on the subject, the term "fuzzy concept" is now more widely recognized as a philosophical, linguistic or scientific category, and the study of the characteristics of fuzzy concepts and fuzzy language is known as fuzzy semantics. “Fuzzy logic” has become a generic term for many different kinds of many-valued logics, and is applied in many different areas of research, computer programming and industrial design. For engineers, "Fuzziness is imprecision or vagueness of definition." For computer scientists, a fuzzy concept is an idea which is "to an extent applicable" in a situation. It means that the concept can have gradations of significance or unsharp (variable) boundaries of application — a "fuzzy statement" is a statement which is true "to some extent", and that extent can often be represented by a scaled value (a score). For mathematicians, a "fuzzy concept" is usually a fuzzy set or a combination of such sets (see fuzzy mathematics and fuzzy set theory). In cognitive linguistics, the things that belong to a "fuzzy category" exhibit gradations of family resemblance, and the borders of the category are not clearly defined. Through most of the 20th century, the idea of reasoning with fuzzy concepts faced considerable resistance from Western academic elites. They did not want to endorse the use of imprecise concepts in research or argumentation, and they often regarded fuzzy logic with suspicion, derision or even hostility. That may partly explain why the idea of a "fuzzy concept" did not get a separate entry in encyclopedias, handbooks and dictionaries. Yet although people might not be aware of it, the use of fuzzy concepts has risen gigantically in all walks of life from the 1970s onward. That is mainly due to advances in electronic engineering, fuzzy mathematics and digital computer programming. The new technology allows very complex inferences about "variations on a theme" to be anticipated and fixed in a program. The Perseverance Mars rover, a driverless NASA vehicle used to explore the Jezero crater on the planet Mars, features fuzzy logic programming that steers it through rough terrain. Similarly, to the North, the Chinese Mars rover Zhurong used fuzzy logic algorithms to calculate its travel route in Utopia Planitia from sensor data. New neuro-fuzzy computational methods make it possible for machines to identify, measure, adjust and respond to fine gradations of significance with great precision. It means that practically useful concepts can be coded, sharply defined, and applied to all kinds of tasks, even if ordinarily these concepts are never exactly defined. Nowadays engineers, statisticians and programmers often represent fuzzy concepts mathematically, using fuzzy logic, fuzzy values, fuzzy variables and fuzzy sets (see also fuzzy set theory). Fuzzy logic is not "woolly thinking", but a "precise logic of imprecision" which reasons with graded concepts and gradations of truth. Fuzzy concepts and fuzzy logic often play a significant role in artificial intelligence programming, for example because they can model human cognitive processes more easily than other methods. == Origins == Vagueness and fuzziness have probably always been a part of human experience. In the West, ancient texts show that philosophers and scientists were already thinking critically about this in classical antiquity. Most often, they regarded vagueness as a problem: as an obstacle to clear thinking, as a source of confusion, or as an evasive tactic. It got in the way of providing clear orientation, guidance, direction and leadership. Therefore, vagueness became associated with a hermeneutic of suspicion — it was considered as something to avoid, as something undesirable. By contrast, in the ancient Chinese tradition of Daoist thought of Laozi and Zhuang Zhou, "vagueness is not regarded with suspicion, but is simply an acknowledged characteristic of the world around us" — a subject for meditation and a source of insight. === Sorites paradox === The ancient Sorites paradox raised the logical problem, of how we could exactly define the threshold at which a change in quantitative gradation turns into a qualitative or categorical difference. With some physical processes, this threshold seems relatively easy to identify. For example, water turns into steam at 100 °C or 212 °F. Of course, the boiling point depends partly on atmospheric pressure, which decreases at higher altitudes; it is also affected by the level of humidity — in that sense, the boiling point is "somewhat fuzzy", because it can vary under different conditions. Nevertheless, for every altitude, level of air pressure and degree of humidity, we can predict accurately what the boiling point will be, if we know the relevant conditions. With many other processes and gradations, however, the point of change is much more difficult to locate, and remains somewhat vague. Thus, the boundaries between qualitatively different things may be unsharp: we know that there are boundaries, but we cannot define them exactly. For example, to identify "the oldest city in the world", we have to define what counts as a city, and at what point a growing human settlement becomes a city. === The continuum fallacy and Loki's wager === According to the modern idea of the continuum fallacy, the fact that a statement is to an extent vague, does not automatically mean that it has no validity. The question then arises, of how (by what method or approach) we could ascertain and define the validity that the fuzzy statement does have. The Nordic myth of Loki's wager suggested that concepts that lack precise meanings or lack precise boundaries of application cannot be operated with, because they evade any clear definition. However, the 20th-century idea of "fuzzy concepts" proposes that "somewhat vague terms" can be operated with, because we can explicate and define the variability of their application — by assigning numbers to gradations of applicability. This idea sounds simple enough, but it had large implications. === Precursors and pioneers === In Western civilization, the intellectual recognition of fuzzy concepts has been traced back to a diversity of famous and less well-known thinkers, including (among many others) Eubulides, Epicurus, Plato, Cicero, William Ockham and John Buridan, Georg Wilhelm Friedrich Hegel, Karl Marx and Friedrich Engels, Friedrich Nietzsche, William James, Hugh MacColl, Charles S. Peirce, Hans Reichenbach, Carl Gustav Hempel, Max Black, Arto Salomaa, Ludwig Wittgenstein, Jan Łukasiewicz, Emil Leon Post, Alfred Tarski, Georg Cantor, Nicolai A. Vasiliev, Kurt Gödel, Stanisław Jaśkowski, Willard Van Orman Quine, George J. Klir, Petr Hájek, Joseph Goguen, Ronald R. Yager, Enrique Héctor Ruspini, Jan Pavelka, Didier Dubois, Bernadette Bouchon-Meunier, and Donald Knuth. Across at least two and a half millennia, all of them had something to say about graded concepts with unsharp boundaries. This suggests at least that the awareness of the existence of concepts with "fuzzy" characteristics, in one form or another, has a very long history in human thought. Quite a few 20th century logicians, mathematicians and philosophers also tried to analyze the characteristics of fuzzy concepts as a recognized species, sometimes with the aid of some kind of many-valued logic or substructural logic. An early attempt in the post-WW2 era to create a mathematical theory of sets with gradations of
Weight initialization
In deep learning, weight initialization or parameter initialization describes the initial step in creating a neural network. A neural network contains trainable parameters that are modified during training: weight initialization is the pre-training step of assigning initial values to these parameters. The choice of weight initialization method affects the speed of convergence, the scale of neural activation within the network, the scale of gradient signals during backpropagation, and the quality of the final model. Proper initialization is necessary for avoiding issues such as vanishing and exploding gradients and activation function saturation. Note that even though this article is titled "weight initialization", both weights and biases are used in a neural network as trainable parameters, so this article describes how both of these are initialized. Similarly, trainable parameters in convolutional neural networks (CNNs) are called kernels and biases, and this article also describes these. == Constant initialization == We discuss the main methods of initialization in the context of a multilayer perceptron (MLP). Specific strategies for initializing other network architectures are discussed in later sections. For an MLP, there are only two kinds of trainable parameters, called weights and biases. Each layer l {\displaystyle l} contains a weight matrix W ( l ) ∈ R n l − 1 × n l {\displaystyle W^{(l)}\in \mathbb {R} ^{n_{l-1}\times n_{l}}} and a bias vector b ( l ) ∈ R n l {\displaystyle b^{(l)}\in \mathbb {R} ^{n_{l}}} , where n l {\displaystyle n_{l}} is the number of neurons in that layer. A weight initialization method is an algorithm for setting the initial values for W ( l ) , b ( l ) {\displaystyle W^{(l)},b^{(l)}} for each layer l {\displaystyle l} . The simplest form is zero initialization: W ( l ) = 0 , b ( l ) = 0 {\displaystyle W^{(l)}=0,b^{(l)}=0} Zero initialization is usually used for initializing biases, but it is not used for initializing weights, as it leads to symmetry in the network, causing all neurons to learn the same features. In this page, we assume b = 0 {\displaystyle b=0} unless otherwise stated. Recurrent neural networks typically use activation functions with bounded range, such as sigmoid and tanh, since unbounded activation may cause exploding values. (Le, Jaitly, Hinton, 2015) suggested initializing weights in the recurrent parts of the network to identity and zero bias, similar to the idea of residual connections and LSTM with no forget gate. In most cases, the biases are initialized to zero, though some situations can use a nonzero initialization. For example, in multiplicative units, such as the forget gate of LSTM, the bias can be initialized to 1 to allow good gradient signal through the gate. For neurons with ReLU activation, one can initialize the bias to a small positive value like 0.1, so that the gradient is likely nonzero at initialization, avoiding the dying ReLU problem. == Random initialization == Random initialization means sampling the weights from a normal distribution or a uniform distribution, usually independently. === LeCun initialization === LeCun initialization, popularized in (LeCun et al., 1998), is designed to preserve the variance of neural activations during the forward pass. It samples each entry in W ( l ) {\displaystyle W^{(l)}} independently from a distribution with mean 0 and variance 1 / n l − 1 {\displaystyle 1/n_{l-1}} . For example, if the distribution is a continuous uniform distribution, then the distribution is U ( ± 3 / n l − 1 ) {\displaystyle {\mathcal {U}}(\pm {\sqrt {3/n_{l-1}}})} . === Glorot initialization === Glorot initialization (or Xavier initialization) was proposed by Xavier Glorot and Yoshua Bengio. It was designed as a compromise between two goals: to preserve activation variance during the forward pass and to preserve gradient variance during the backward pass. For uniform initialization, it samples each entry in W ( l ) {\displaystyle W^{(l)}} independently and identically from U ( ± 6 / ( n l + 1 + n l − 1 ) ) {\displaystyle {\mathcal {U}}(\pm {\sqrt {6/(n_{l+1}+n_{l-1})}})} . In the context, n l − 1 {\displaystyle n_{l-1}} is also called the "fan-in", and n l + 1 {\displaystyle n_{l+1}} the "fan-out". When the fan-in and fan-out are equal, then Glorot initialization is the same as LeCun initialization. === He initialization === As Glorot initialization performs poorly for ReLU activation, He initialization (or Kaiming initialization) was proposed by Kaiming He et al. for networks with ReLU activation. It samples each entry in W ( l ) {\displaystyle W^{(l)}} from N ( 0 , 2 / n l − 1 ) {\displaystyle {\mathcal {N}}(0,2/n_{l-1})} . === Orthogonal initialization === (Saxe et al. 2013) proposed orthogonal initialization: initializing weight matrices as uniformly random (according to the Haar measure) semi-orthogonal matrices, multiplied by a factor that depends on the activation function of the layer. It was designed so that if one initializes a deep linear network this way, then its training time until convergence is independent of depth. Sampling a uniformly random semi-orthogonal matrix can be done by initializing X {\displaystyle X} by IID sampling its entries from a standard normal distribution, then calculate ( X X ⊤ ) − 1 / 2 X {\displaystyle \left(XX^{\top }\right)^{-1/2}X} or its transpose, depending on whether X {\displaystyle X} is tall or wide. For CNN kernels with odd widths and heights, orthogonal initialization is done this way: initialize the central point by a semi-orthogonal matrix, and fill the other entries with zero. As an illustration, a kernel K {\displaystyle K} of shape 3 × 3 × c × c ′ {\displaystyle 3\times 3\times c\times c'} is initialized by filling K [ 2 , 2 , : , : ] {\displaystyle K[2,2,:,:]} with the entries of a random semi-orthogonal matrix of shape c × c ′ {\displaystyle c\times c'} , and the other entries with zero. (Balduzzi et al., 2017) used it with stride 1 and zero-padding. This is sometimes called the Orthogonal Delta initialization. Related to this approach, unitary initialization proposes to parameterize the weight matrices to be unitary matrices, with the result that at initialization they are random unitary matrices (and throughout training, they remain unitary). This is found to improve long-sequence modelling in LSTM. Orthogonal initialization has been generalized to layer-sequential unit-variance (LSUV) initialization. It is a data-dependent initialization method, and can be used in convolutional neural networks. It first initializes weights of each convolution or fully connected layer with orthonormal matrices. Then, proceeding from the first to the last layer, it runs a forward pass on a random minibatch, and divides the layer's weights by the standard deviation of its output, so that its output has variance approximately 1. === Fixup initialization === In 2015, the introduction of residual connections allowed very deep neural networks to be trained, much deeper than the ~20 layers of the previous state of the art (such as the VGG-19). Residual connections gave rise to their own weight initialization problems and strategies. These are sometimes called "normalization-free" methods, since using residual connection could stabilize the training of a deep neural network so much that normalizations become unnecessary. Fixup initialization is designed specifically for networks with residual connections and without batch normalization, as follows: Initialize the classification layer and the last layer of each residual branch to 0. Initialize every other layer using a standard method (such as He initialization), and scale only the weight layers inside residual branches by L − 1 2 m − 2 {\displaystyle L^{-{\frac {1}{2m-2}}}} . Add a scalar multiplier (initialized at 1) in every branch and a scalar bias (initialized at 0) before each convolution, linear, and element-wise activation layer. Similarly, T-Fixup initialization is designed for Transformers without layer normalization. === Others === Instead of initializing all weights with random values on the order of O ( 1 / n ) {\displaystyle O(1/{\sqrt {n}})} , sparse initialization initialized only a small subset of the weights with larger random values, and the other weights zero, so that the total variance is still on the order of O ( 1 ) {\displaystyle O(1)} . Random walk initialization was designed for MLP so that during backpropagation, the L2 norm of gradient at each layer performs an unbiased random walk as one moves from the last layer to the first. Looks linear initialization was designed to allow the neural network to behave like a deep linear network at initialization, since W R e L U ( x ) − W R e L U ( − x ) = W x {\displaystyle W\;\mathrm {ReLU} (x)-W\;\mathrm {ReLU} (-x)=Wx} . It initializes a matrix W {\displaystyle W} of shape R n 2 × m {\displaystyle \mathbb {R} ^{{\frac {n}{2}}\times m}} by any method, such as orthogonal initialization, t
Vague set
In mathematics, vague sets are an extension of fuzzy sets. In a fuzzy set, each object is assigned a single value in the interval [0,1] reflecting its grade of membership. This single value does not allow a separation of evidence for membership and evidence against membership. Gau et al. proposed the notion of vague sets, where each object is characterized by two different membership functions: a true membership function and a false membership function. This kind of reasoning is also called interval membership, as opposed to point membership in the context of fuzzy sets. == Mathematical definition == A vague set V {\displaystyle V} is characterized by its true membership function t v ( x ) {\displaystyle t_{v}(x)} its false membership function f v ( x ) {\displaystyle f_{v}(x)} with 0 ≤ t v ( x ) + f v ( x ) ≤ 1 {\displaystyle 0\leq t_{v}(x)+f_{v}(x)\leq 1} The grade of membership for x is not a crisp value anymore, but can be located in [ t v ( x ) , 1 − f v ( x ) ] {\displaystyle [t_{v}(x),1-f_{v}(x)]} . This interval can be interpreted as an extension to the fuzzy membership function. The vague set degenerates to a fuzzy set, if 1 − f v ( x ) = t v ( x ) {\displaystyle 1-f_{v}(x)=t_{v}(x)} for all x. The uncertainty of x is the difference between the upper and lower bounds of the membership interval; it can be computed as ( 1 − f v ( x ) ) − t v ( x ) {\displaystyle (1-f_{v}(x))-t_{v}(x)} .