Feature engineering, statistical model, and learning from feedback — In NLP, segmenting a text document into its sentences is a useful basic operation. It is the first step in many NLP tasks that are more elaborate. Such as detecting and correcting errors in the text as it is being written [1], or detecting named entities [2]. In the former…

Domain-specific Feature Engineering From Labeled Data I watched many games in the recent soccer world cup. After a while, I began to realize that the team rosters provide an effective training set for the problem of predicting ethnicity from a person’s name. Lots of nationalities are covered. The data seems…

Detecting and correcting errors that involve modeling context — In the previous article, we discussed the problem of detecting and correcting common errors in text using methods from statistical NLP: Text Correction Using NLP Detecting and correcting common errors: problems and methodstowardsdatascience.com There we took an inventory of several issues, with accompanying real examples and discussion. Below are the ones that we had not fully resolved in that post. (The last…

Detecting and correcting common errors: problems and methods — Anyone who writes text will miss a comma here and there. Or use the wrong preposition in a certain context. Or make spelling errors. Or phrase awkwardly. Or use overly complicated or excessively long sentences. Or overly long paragraphs. Or ramble on excessively. On all but the shortest of writeups…

Extractive summarization using POS tagging, NER, and sentiment analysis — Consider a thread of text (or email) messages involving multiple people. Below is a sample involving two people planning a trip together. Individual messages are separated by empty lines. Here is some lodging near Pismo beach: Cliffs Hotel, SeaCrest, Madonna Inn, Shell beach inn Shell beach & Madonna - a bit…

Problems, models, and implementation issues — The driving use case for this is https://jagota-arun.medium.com/interactive-and-adaptive-menu-ordering-agent-bb447c58b3af (Not a prerequisite read.) Here we focus on the modeling. What is it? How does it relate to recommender systems? And then on the implementation. As in object-oriented design. Classes etc. The object-oriented exposition style extends the benefits of sequencing, modularity, composability…

A modeling case study in a restaurant setting — Imagine a smartphone app that does the following. You walk into a coffee house, restaurant, or bar. It brings up an interactive version of the menu. It offers top-level choices first. Such as drinks, appetizers, main dishes, and desserts in many restaurants. Or burgers, sandwiches, etc in others. …

Issues involving information retrieval, natural language processing, and machine learning — Imagine a software system that is able to answer questions posed in natural language effectively. Such a system is very useful. It is also very interesting from the perspective of what happens under the hood. Let’s start by enumerating some practical use cases of such a system. Online Shopping: Answering…

From generative to “plus adversarial” — Say we have a dataset of real images. Such as pictures of lions in various settings. From this data set, we want to machine-learn to generate new images that look like the real ones. Generative Adversarial Networks, GANs for short, are a compelling approach to this problem. A GAN comprises…