Topic modelling refers to the task of identifying topics that best describes a set of documents. These topics will only emerge during the topic modelling process (therefore called latent). And one…
Recent explosion in the popularity of large language models like ChatGPT has led to their increased usage in classical NLP tasks like language classification. This involves providing a context…
Perplexity is a useful metric to evaluate models in Natural Language Processing (NLP). This article will cover the two ways in which it is normally defined and the intuitions behind them. A language…
The ultimate guide to chatbot analytics. Find out what bot metrics and KPIs you should measure and discover easy ways to optimize your chatbot performance.
These measurements are indispensable for tracking the results of your chatbot, identifying any stumbling blocks and continuously improving its performance. But which metrics should you choose?
We’ve done a lot of looking over our shoulders at OpenAI. Who will cross the next milestone? What will the next move be?
But the uncomfortable truth is, we aren’t positioned to win this arms race and neither is OpenAI. While we’ve been squabbling, a third faction has been quietly eating our lunch.
I’m talking, of course, about open source. Plainly put, they are lapping us. Things we consider “major open problems” are solved and in people’s hands today.
Pandas AI is a Python library that integrates generative artificial intelligence capabilities into Pandas, making dataframes conversational - GitHub - gventuri/pandas-ai: Pandas AI is a Python library that integrates generative artificial intelligence capabilities into Pandas, making dataframes conversational
Build document-based question-answering systems using LangChain, Pinecone, LLMs like GPT-4, and semantic search for precise, context-aware AI solutions.
When a word appears in different contexts, its vector gets moved in different directions during updates. The final vector then represents some sort of weighted average over the various contexts. Averaging over vectors that point in different directions typically results in a vector that gets shorter with increasing number of different contexts in which the word appears. For words to be used in many different contexts, they must carry little meaning. Prime examples of such insignificant words are high-frequency stop words, which are indeed represented by short vectors despite their high term frequencies ...
When the downstream applications only care about the direction of the word vectors (e.g. they only pay attention to the cosine similarity of two words), then normalize, and forget about length.
However, if the downstream applications are able to (or need to) consider more sensible aspects, such as word significance, or consistency in word usage (see below), then normalization might not be such a good idea.
This page is a distribution site for movie-review data for use in sentiment-analysis experiments. Available are collections of movie-review documents labeled with respect to their overall sentiment polarity (positive or negative) or subjective rating (e.g., "two and a half stars") and sentences labeled with respect to their subjectivity status (subjective or objective) or polarity. These data sets were introduced in the following papers:
MACE (Multi-Annotator Competence Estimation) is an implementation of an item-response model that let's you evaluate redundant annotations of categorical data. It provides competence estimates of the individual annotators and the most likely answer to each item.
If we have 10 annotators answer a question, and five answer with 'yes' and five with 'no' (a surprisingly frequent event), we would normaly have to flip a coin to decide what the right answer is. If we knew, however, that one of the people who answered 'yes' is an expert on the question, while one of the others just alwas selects 'no', we would take this information into account to weight their answers. MACE does exactly that. It tries to find out which annotators are more trustworthy and upweighs their answers. All you need to provide is a CSV file with one item per line.
In tests, MACE's trust estimates correlated highly wth the annotators' true competence, and it achieved accuracies of over 0.9 on several test sets. MACE can take annotated items into account, if they are available. This helps to guide the training and improves accuracy.
Uniform Manifold Approximation and Projection (UMAP) is a dimension reduction technique that can be used for visualisation similarly to t-SNE, but also for general non-linear dimension reduction. The algorithm is founded on three assumptions about the data
The data is uniformly distributed on Riemannian manifold;
The Riemannian metric is locally constant (or can be approximated as such);
The manifold is locally connected.
From these assumptions it is possible to model the manifold with a fuzzy topological structure. The embedding is found by searching for a low dimensional projection of the data that has the closest possible equivalent fuzzy topological structure.
D-Tale is an interactive web-based library that consists of a Flask backend and a React front-end serving as an easy way to view & analyze Pandas data structures. It integrates seamlessly with ipython notebooks & python/ipython terminals. Currently, this tool supports such Pandas objects as DataFrame, Series, MultiIndex, DatetimeIndex & RangeIndex.
You want to discern how many clusters we have (or, if you prefer, how many gaussians components generated the data), and you don’t have information about the “ground truth”. A real case, where data do not have the nicety of behaving good as the simulated ones.
Definition of NLP coherence scores, in particular intrinsic UMass measure and PMI.
Human judgment not being correlated to perplexity (or likelihood of unseen documents) is the motivation for more work trying to model the human judgment. This is by itself a hard task as human judgment is not clearly defined; for example, two experts can disagree on the usefulness of a topic.
One can classify the methods addressing this problem into two categories. \textit{Intrinsic} methods that do not use any external source or task from the dataset, whereas \textit{extrinsic} methods use the discovered topics for external tasks, such as information retrieval [Wei06], or use external statistics to evaluate topics.
C. Tahri, X. Tannier, and P. Haouat. Proceedings of the first Workshop on Information Extraction from Scientific Publications, page 67--77. Online, Association for Computational Linguistics, (November 2022)
M. Windl, V. Winterhalter, A. Schmidt, and S. Mayer. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, New York, NY, USA, Association for Computing Machinery, (2023)