Spleak was an IM platform where users could publish and rate content. It existed in the form of six bots covering as many subject areas: CelebSpleak, SportSpleak, VoteSpleak, TVSpleak, GameSpleak, and StyleSpleak. == Overview == Users can add a "multi-Spleak" (which contains all of the different Spleak bots in one) or add the separate bots to their IM buddy lists on MSN and AIM. Users are also allowed access to Spleak online by using a CelebSpleak, SportSpleak, or VoteSpleak widget, or through the CelebSpleak and SportSpleak applications with Facebook. Spleak was an alternate reality game and is moving to its own company, Spleak Media Network. "Celebrate Spleak" was introduced throughout 2007, launched in 2008, and was forced to retire in 2009. == Key people == Spleak was co-founded by Morten Lund and Nicolaj Reffstrup. The company's chief executive officer is Morrie Eisenburg; Josh Scott is Vice President in Product and Tyler Wells is Vice President in Engineering.
Language model benchmark
A language model benchmark is a standardized test designed to evaluate the performance of language models on various natural language processing tasks. These tests are intended for comparing different models' capabilities in areas such as language understanding, generation, and reasoning. Benchmarks generally consist of a dataset and corresponding evaluation metrics. The dataset provides text samples and annotations, while the metrics measure a model's performance on tasks like answering questions, text classification, and machine translation. These benchmarks are developed and maintained by academic institutions, research organizations, and industry players to track progress in the field. In addition to accuracy, the metrics can include throughput, energy efficiency, bias, trust, and sustainability. == Overview == === Types === Benchmarks may be described by the following adjectives, not mutually exclusive: Classical: These tasks are studied in natural language processing, even before the advent of deep learning. Examples include the Penn Treebank for testing syntactic and semantic parsing, as well as bilingual translation benchmarked by BLEU scores. Question answering: These tasks have a text question and a text answer, often multiple-choice. They can be open-book or closed-book. Open-book QA resembles reading comprehension questions, with relevant passages included as annotation in the question, in which the answer appears. Closed-book QA includes no relevant passages. Closed-book QA is also called open-domain question-answering. Before the era of large language models, open-book QA was more common, and understood as testing information retrieval methods. Closed-book QA became common since GPT-2 as a method to measure knowledge stored within model parameters. Omnibus: An omnibus benchmark combines many benchmarks, often previously published. It is intended as an all-in-one benchmarking solution. Reasoning: These tasks are usually in the question-answering format, but are intended to be more difficult than standard question answering. Multimodal: These tasks require processing not only text, but also other modalities, such as images and sound. Examples include OCR and transcription. Agency: These tasks are for a language-model–based software agent that operates a computer for a user, such as editing images, browsing the web, etc. Adversarial: A benchmark is "adversarial" if the items in the benchmark are picked specifically so that certain models do badly on them. Adversarial benchmarks are often constructed after state of the art (SOTA) models have saturated (achieved 100% performance) a benchmark, to renew the benchmark. A benchmark is "adversarial" only at a certain moment in time, since what is adversarial may cease to be adversarial as newer SOTA models appear. Public/Private: A benchmark might be partly or entirely private, meaning that some or all of the questions are not publicly available. The idea is that if a question is publicly available, then it might be used for training, which would be "training on the test set" and invalidate the result of the benchmark. Usually, only the guardians of the benchmark have access to the private subsets, and to score a model on such a benchmark, one must send the model weights, or provide API access, to the guardians. The boundary between a benchmark and a dataset is not sharp. Generally, a dataset contains three "splits": training, test, and validation. Both the test and validation splits are essentially benchmarks. In general, a benchmark is distinguished from a test/validation dataset in that a benchmark is typically intended to be used to measure the performance of many different models that are not trained specifically for doing well on the benchmark, while a test/validation set is intended to be used to measure the performance of models trained specifically on the corresponding training set. In other words, a benchmark may be thought of as a test/validation set without a corresponding training set. Conversely, certain benchmarks may be used as a training set, such as the English Gigaword or the One Billion Word Benchmark, which in modern language is just the negative log-likelihood loss on a pretraining set with 1 billion words. Indeed, the distinction between benchmark and dataset in language models became sharper after the rise of the pretraining paradigm, whereby a model is first trained on massive, unlabeled datasets to learn general language patterns, syntax, and knowledge (pretraining), and the base model is then adapted to specific, downstream tasks using smaller, labeled datasets (fine-tuning). === Lifecycle === Generally, the life cycle of a benchmark consists of the following steps: Inception: A benchmark is published. It can be simply given as a demonstration of the power of a new model (implicitly) that others then picked up as a benchmark, or as a benchmark that others are encouraged to use (explicitly). Growth: More papers and models use the benchmark, and the performance on the benchmark grows. Maturity, degeneration or deprecation: A benchmark may be saturated, after which researchers move on to other benchmarks. Progress on the benchmark may also be neglected as the field moves to focus on other benchmarks. Renewal: A saturated benchmark can be upgraded to make it no longer saturated, allowing further progress. === Construction === Like datasets, benchmarks are typically constructed by several methods, individually or in combination: Web scraping: Ready-made question-answer pairs may be scraped online, such as from websites that teach mathematics and programming. Conversion: Items may be constructed programmatically from scraped web content, such as by blanking out named entities from sentences, and asking the model to fill in the blank. This was used for making the CNN/Daily Mail Reading Comprehension Task. Crowd sourcing: Items may be constructed by paying people to write them, such as on Amazon Mechanical Turk. This was used for making the MCTest. === Evaluation === Generally, benchmarks are fully automated. This limits the questions that can be asked. For example, with mathematical questions, "proving a claim" would be difficult to automatically check, while "calculate an answer with a unique integer answer" would be automatically checkable. With programming tasks, the answer can generally be checked by running unit tests, with an upper limit on runtime. The benchmark scores are of the following kinds: For multiple choice or cloze questions, common scores are accuracy (frequency of correct answer), precision, recall, F1 score, etc. pass@n: The model is given n {\displaystyle n} attempts to solve each problem. If any attempt is correct, the model earns a point. The pass@n score is the model's average score over all problems. k@n: The model makes n {\displaystyle n} attempts to solve each problem, but only k {\displaystyle k} attempts out of them are selected for submission. If any submission is correct, the model earns a point. The k@n score is the model's average score over all problems. cons@n: The model is given n {\displaystyle n} attempts to solve each problem. If the most common answer is correct, the model earns a point. The cons@n score is the model's average score over all problems. Here "cons" stands for "consensus" or "majority voting". The pass@n score can be estimated more accurately by making N > n {\displaystyle N>n} attempts, and use the unbiased estimator 1 − ( N − c n ) ( N n ) {\displaystyle 1-{\frac {\binom {N-c}{n}}{\binom {N}{n}}}} , where c {\displaystyle c} is the number of correct attempts. For less well-formed tasks, where the output can be any sentence, there are the following commonly used scores including BLEU ROUGE, METEOR, NIST, word error rate, LEPOR, CIDEr, and SPICE. === Issues === error: Some benchmark answers may be wrong. ambiguity: Some benchmark questions may be ambiguously worded. subjective: Some benchmark questions may not have an objective answer at all. This problem generally prevents creative writing benchmarks. Similarly, this prevents benchmarking writing proofs in natural language, though benchmarking proofs in a formal language is possible. open-ended: Some benchmark questions may not have a single answer of a fixed size. This problem generally prevents programming benchmarks from using more natural tasks such as "write a program for X", and instead uses tasks such as "write a function that implements specification X". inter-annotator agreement: Some benchmark questions may be not fully objective, such that even people would not agree with 100% on what the answer should be. This is common in natural language processing tasks, such as syntactic annotation. shortcut: Some benchmark questions may be easily solved by an "unintended" shortcut. For example, in the SNLI benchmark, having a negative word like "not" in the second sentence is a strong signal for the "Contradiction" category, regardless of what the se
Svetlana Lazebnik
Svetlana Lazebnik (born 1979) is a Ukrainian-American researcher in computer vision who works as a professor of computer science and Willett Faculty Scholar at the University of Illinois at Urbana–Champaign. Her research involves interactions between image understanding and natural language processing, including the automated captioning of images, and the development of a benchmark database of textually grounded images. == Education and career == Lazebnik was born in Kyiv in 1979 to a family of Ukrainian Jews, and emigrated with her family to the US as a teenager. She majored in computer science at DePaul University, minoring in mathematics and graduating with the highest honors in 2000. She completed her Ph.D. in 2006 at the University of Illinois at Urbana–Champaign, with the dissertation Local, Semi-Local and Global Models for Texture, Object and Scene Recognition supervised by Jean Ponce. After postdoctoral research at the University of Illinois, she became an assistant professor at the University of North Carolina at Chapel Hill in 2007. She returned to the University of Illinois as a faculty member in 2012. She is a co-editor-in-chief of the International Journal of Computer Vision. == Recognition == Lazebnik was named an IEEE Fellow in 2021, "for contributions to computer vision". With Cordelia Schmid and Jean Ponce, she won the Longuet-Higgins Prize in 2016 for the best work in computer vision from ten years earlier, for their work on spatial pyramid matching.
Bidyut Baran Chaudhuri
Bidyut Baran Chaudhuri (B. B. Chauduri) is a senior computer scientist and an emeritus professor of Techno India University in West Bengal, India. He is also adjuncted to Indian Statistical Institute, where he was a professor for about three decades. He was the founding Head of Computer Vision and Pattern Recognition Unit (which was established in 1994) of ISI. Moreover, he was a J.C. Bose Fellow and Indian National Academy of Engineering Distinguished Professor at ISI. He was the vice-president of the Society for Natural Language Technology Research (SNLTR). His primary research contributes to the fields of computer vision, image processing and pattern recognition. He is a pioneer of "Indian language script OCR". == Education == Chaudhuri received his BSc (Hons.), BTech and MTech degrees from University of Calcutta, India in 1969, 1972 and 1974, respectively and PhD Degree from Indian Institute of Technology Kanpur in 1980. He did his post-doc work during 1981-1982 from Queen's University, U.K, through Leverhulme Overseas Fellowship. He also worked as a visiting faculty at Tech University, Hannover during 1986-87 as well as at GSF Institute of Radiation Protection (now Leibnitz Institute), Munich in 1990 and 1992. == Awards and recognition == Chaudhuri has been elected as a Life Fellow of IEEE "for contributions to pattern recognition, especially Indian language script OCR, document processing and natural language processing". He has become a Fellow of International Association for Pattern Recognition (IAPR) "for contributions to character recognition and speech synthesis in Indian language". He is also Fellow of The World Academy of Sciences (TWAS), Indian National Science Academy (INSA), Indian National Academy of Engineering (INAE), National Academy of Sciences (NASI), and Institute of Electronics and Telecommunication Engineering (IETE). In 2011, Chaudhuri received the Om Prakash Bhasin Award for his contribution in the field of electronics and information technology. Chaudhuri's interview on some of his works has been reported in Indian newspaper as well. He is within world's top 2% scientists and top-10 Indian AI scientists according to a study conducted by Stanford University. He has also been featured as top-10 machine learning researcher from India.
Translation unit
In the field of translation, a translation unit is a segment of a text which the translator treats as a single cognitive unit for the purposes of establishing an equivalence. It may be a single word, a phrase, one or more sentences, or even a larger unit. When a translator segments a text into translation units, the larger these units are, the better chance there is of obtaining an idiomatic translation. This is true not only of human translation, but also where human translators use computer-assisted translation, such as translation memories, and when translations are performed by machine translation systems. == Perceptions on the concept of unit == Vinay and Darbelnet took to Saussure's original concepts of the linguistic sign when beginning to discuss the idea of a single word as a translation unit. According to Saussure, the sign is naturally arbitrary, so it can only derive meaning from contrast in other signs in that same system. However, Russian scholar Leonid Barkhudarov stated that, limiting it to poetry, for instance, a translation unit can take the form of a complete text. This seems to relate to his conception that a translation unit is the smallest unit in the source language with an equivalent in the target one, and when its parts are taken individually, they become untranslatable; these parts can be as small as phonemes or morphemes, or as large as entire texts. Susan Bassnett widened Barkhudarov's poetry perception to include prose, adding that in this type of translation text is the prime unit, including the idea that sentence-by-sentence translation could cause loss of important structural features. Swiss linguist Werner Koller connected Barkhudarov's idea of unit sizing to the difference between the two languages involved, by stating that the more different or unrelated these languages were, the larger the unit would be. One final perception on the idea of unit came from linguist Eugene Nida. To him, translation units have a tendency to be small groups of language building up into sentences, thus forming what he called meaningful mouthfuls of language. == Points of view towards translation units == === Process-oriented POV === According to this point of view, a translation unit is a stretch of text on which attention is focused to be represented as a whole in the target language. In this point of view we can consider the concept of the think-aloud protocol, supported by German linguist Wolfgang Lörscher: isolating units using self-reports by translating subjects. It also relates to how experienced the translator in question is: language learners take a word as a translation unit, whereas experienced translators isolate and translate units of meaning in the form of phrases, clauses or sentences. Since 1996 and 2005 keylogging and eyetracking technologies were introduced in Translation Process Research. These more advanced and non-invasive research methods made it possible to elaborate a finer-grained assessment of translation units as loops of (source or target text) reading and target text typing. Loops of translation units are thought to be the basic units by which translations are produced. Thus, Malmkjaer, for instance, defines process oriented translation units as a “stretch of the source text that the translator keeps in mind at any one time, in order to produce translation equivalents in the text he or she is creating” (p. 286). Records of keystrokes and eye movements allow to investigate these mental constructs through their physical (observable) behavioral traces in the translation process data. Empirical Translation Process Research has deployed numerous theories to explain and models the behavioral traces of these assumed mental units. === Product-oriented POV === Here, the target-text unit can be mapped into an equivalent source-text unit. A case study on this matter was reported by Gideon Toury, in which 27 English-Hebrew student-produced translations were mapped onto a source text. Those students that were less experienced had larger numbers of small units at word and morpheme level in their translations, while one student with translation experience had approximately half of those units, mostly at phrase or clause level.
TipTop Technologies
TipTop Technologies is a real-time web and social search engine with a platform for semantic analysis of natural language. Tip-Top Search provides results capturing individual and group sentiment, opinions, and experiences there from the content of various sorts such as real-time messages from Twitter or consumer product reviews on Amazon.com. TipTop Technologies and ITC Infotech collaborated to create a search interface suitable for both enterprise and consumer applications. Tip-Top's products are part of the "emerging Web 3.0 applications which use semantic technologies to augment the underlying Web system's functionalities." Their main product is 360, an AI tool that incorporates multiple AI applications under one wing. Jonathan AlBright professor at Elon University, found videos generated by TipTop Technologies software on YouTube in his research into artificial intelligence, described it as AI-generated "fake news". Through semantic analysis of large data sets, TipTop gleaned behavioral insights from Tweets around events like Halloween, Thanksgiving, Holiday Gifting, the Super Bowl, and the Oscar Nominees for the Academy Awards coverage. Sentiment analysis, concept trend tracking, and real-time market research are other applications included in the TipTop Search product. TipTop's insight engine solves the problem of real-time data noise, and its ability to "sort the 'good tweets' from the 'bad tweets' when it comes to a product, service, or a region..." In addition, products like TipTop Shopping with customizable search widgets bring together consumer reviews, social search, and sentiment analysis enabling product comparisons across attributes like the overall value and aiding purchasing decisions through user-driven product tips and pits. TipTop Finance adds another complexity to real-time search results by incorporating corporate sentiment, company stock tickers, and social media into TipTop's existing social search platform. Additional success applying semantic technologies has been with polling, "if you compare these Gallup results with TipTop, a sentiment engine based on Twitter, the results are not way off. It does surprise you but it tells me that sentiment analysis in case of public opinion about a burning social issue or a famous personality is relatively easier." With the increasing amount of unstructured, opinion-oriented, and user-generated content available on the Web, TipTop's technology aims to make sense of all this data, and deliver it in a useful way for consumer and enterprise users alike. TipTop Technologies is a privately held company with its headquarters in the San Francisco Bay Area, and team members are located globally.
Dilek Hakkani-Tür
Dilek Z. Hakkani-Tür is a Turkish-American computer scientist focusing on speech processing, speech recognition, and dialogue systems. She is a professor of computer science at the University of Illinois Urbana-Champaign. == Education and career == Hakkani-Tür is a 1994 graduate of Middle East Technical University in Ankara, Turkey. She continued her studies at Bilkent University, also in Ankara, where she earned a master's degree in 1996 and completed her Ph.D. in 2000. She worked as a researcher at AT&T Labs from 2001 to 2005, at the International Computer Science Institute from 2006 to 2010, at Microsoft Research from 2010 to 2016, at Google Research from 2016 to 2018, and at Amazon Alexa from 2018 to 2023. At Microsoft, she was in the team of scientists that built the first prototype of the Cortana virtual assistant. While working for Amazon Alexa, she also taught at the University of California, Santa Cruz as a distinguished visiting instructor. She joined the University of Illinois Urbana-Champaign faculty in 2023. She was editor-in-chief of IEEE/ACM Transactions on Audio, Speech and Language Processing from 2019 to 2021, and is president of the Special Interest Group on Discourse and Dialogue of the Association for Computational Linguistics for the 2023–2025 term. She has served as co-editor-in-chief of Transactions of the Association for Computational Linguistics since 2024. == Recognition == In 2014, Hakkani-Tür was elected as an IEEE Fellow "for contributions to spoken language processing", and as a Fellow of the International Speech Communication Association "for contributions to advancing the state-of-the-art in spoken language processing, especially for human/human and human/machine conversational understanding". In 2024, she was elected as a Fellow of the Association for Computational Linguistics for her contributions to spoken dialogue systems.