Glebashnik/feeding generate local llm performance test

tests/performance/feeding_generate_local_llm/app/schemas/passage.sd

@@ -0,0 +1,25 @@
+# Copyright Vespa.ai. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
+
+schema passage {
+    document passage {
+        field id type string {
+            indexing: summary | attribute
+        }
+
+        field text type string {
+            indexing: summary | index
+            index: enable-bm25
+        }
+    }
+
+    field local_llm_gen type string {
+        indexing: input text | generate local_llm_gen | index | summary
+        index: enable-bm25
+    }
+
+    rank-profile default {
+        first-phase {
+            expression: bm25(text)
+        }
+    }
+}

tests/performance/feeding_generate_local_llm/app/services.xml

@@ -0,0 +1,50 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<!-- Copyright Vespa.ai. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. -->
+<services version="1.0">
+
+    <container id="container" version="1.0">
+        <component id="local_llm" class="ai.vespa.llm.clients.LocalLLM">
+            <config name="ai.vespa.llm.clients.llm-local-client">
+                <!-- File is approx 130MB" -->
+                <!-- <model url="https://data.vespa-cloud.com/gguf_models/Llama-160M-Chat-v1.Q6_K.gguf"/>-->
+                <!-- File is approx 2.2GB" -->
+                <model url="https://data.vespa-cloud.com/gguf_models/Phi-3.5-mini-instruct-Q4_K_M.gguf"/>
+                <contextSize>5000</contextSize>
+                <parallelRequests>10</parallelRequests>
+                <maxQueueSize>5</maxQueueSize>
+                <maxQueueWait>100000</maxQueueWait>
+                <maxEnqueueWait>20000</maxEnqueueWait>
+                <maxTokens>200</maxTokens>
+                <maxPromptTokens>200</maxPromptTokens>
+            </config>
+        </component>
+
+        <component id="local_llm_gen" class="ai.vespa.llm.generation.LanguageModelTextGenerator">
+            <config name="ai.vespa.llm.generation.language-model-text-generator">
+                <providerId>local_llm</providerId>
+                <promptTemplate>Translate this text to Norwegian:\n{input}</promptTemplate>
+            </config>
+        </component>
+
+        <document-api/>
+
+        <search/>
+
+        <nodes>
+            <node hostalias="node1"/>
+        </nodes>
+    </container>
+
+    <content id="content" version="1.0">
+        <redundancy>1</redundancy>
+
+        <documents>
+            <document mode="index" type="passage"/>
+        </documents>
+
+        <nodes>
+            <node hostalias="node1" distribution-key="0"/>
+        </nodes>
+    </content>
+
+</services>

tests/performance/feeding_generate_local_llm/data/feed.jsonl

@@ -0,0 +1,40 @@
+{"put": "id:msmarco:passage::963", "fields": {"text": "Waterproof fabrics are fabrics that are inherently, or have been treated to become, resistant to penetration by water and wetting. The term waterproof refers to conformance to a governing specification and specific conditions of a laboratory test method.", "id": "963"}}
+{"put": "id:msmarco:passage::964", "fields": {"text": "Coated fabrics consist of woven and non-woven cloth with a coating or resin applied to the surface or saturated into the bulk of the material to provide some additional property. Specifications include: product type. material type. dimensions.", "id": "964"}}
+{"put": "id:msmarco:passage::968", "fields": {"text": "All cord piping is made by hand using premium marine vinyls and foam core (unless specified otherwise by customer). The average cost to completely remove, reupholster/rebuild, and reinstall all associated interior components in a 20 foot runabout is $2,500.00. Similar to the cost of a new couch, loveseat, and coffee table at a reasonably priced furniture store. Of course, prices will vary depending on the boat.", "id": "968"}}
+{"put": "id:msmarco:passage::982", "fields": {"text": "Good morning everyone, this is going to be a short post about the retention offers I received on my 5 Citi credit cards. Before I go on, I highly recommend reading Doctor of Credit’s post on Retention Bonus Rules & Tips For Each Card Issuer – Get More Than One Bonus Each Year. I called Citi yesterday to tell them I was going to make a large “purchase” at Target with my Citi American Airlines Executive Credit Card.", "id": "982"}}
+{"put": "id:msmarco:passage::983", "fields": {"text": "David Hecker, President, AFT Michigan. Keith Johnson, President, Detroit Federation of Teachers. John McDonald, President, Henry Ford Community College Fed of Teachers. Ruby Newbold, President, Detroit Assoc. of Educational Office Employees.", "id": "983"}}
+{"put": "id:msmarco:passage::984", "fields": {"text": "Lynda Carter was born on July 24, 1951 in Phoenix, Arizona, USA as Linda Jean Cordova Carter. She is an actress, known for Wonder Woman (1975), The Elder Scrolls IV: Oblivion (2006) and The Dukes of Hazzard (2005). She has been married to Robert Altman since January 29, 1984. They have two children.", "id": "984"}}
+{"put": "id:msmarco:passage::991", "fields": {"text": "There are three classes of diuretic drugs that are used to treat hypertension. Most commonly used are thiazide diuretics such as hydrochlorothiazide or chlorthalidone. There is not usually an increased urine flow after the first one or two days of taking these medications.", "id": "991"}}
+{"put": "id:msmarco:passage::992", "fields": {"text": "Monitor your pulse. You should know how to take your pulse – especially if you have an artificial pacemaker. 1  Put the second and third fingers of one hand on the inside of the wrist of the other hand, just below the thumb OR on the side of your neck, just below the corner of your jaw.  Feel for the pulse.", "id": "992"}}
+{"put": "id:msmarco:passage::993", "fields": {"text": "All medications have side effects, including drugs to treat arrhythmias. Most of the side effects aren't serious and disappear when the dose is changed or the medication is stopped. But some side effects are very serious. That's why some children are admitted to the hospital to begin the medication.", "id": "993"}}
+{"put": "id:msmarco:passage::994", "fields": {"text": "If you're being treated for arrhythmia and use any of these substances, be sure to discuss this with your doctor. Manage your risk factors. Just having an arrhythmia increases your risk of heart attack, cardiac arrest and stroke.", "id": "994"}}
+{"put": "id:msmarco:passage::43", "fields": {"text": "Common Treatments done by Medical Tourists in Costa Rica. Known initially for its excellent dental surgery services, medical tourism in Costa Rica has spread to a variety of other medical procedures, including: General and cosmetic dentistry; Cosmetic surgery; Aesthetic procedures (botox, skin resurfacing etc) Bariatric and Laparoscopic surgery", "id": "43"}}
+{"put": "id:msmarco:passage::44", "fields": {"text": "Medical Tours costa Rica office remains within the hospital and the Cook brothers 15 year relationship running the insurance office and seven years running the international patient department serves you the client very well.", "id": "44"}}
+{"put": "id:msmarco:passage::45", "fields": {"text": "About us. Medical Tours Costa Rica has helped thousands of patients and are the innovators in medical travel to Costa Rica. Brad and Bill Cook are visionaries that saw the writing on the wall while running the International insurance office for Costa busiest and most respected hospital The Clinica Biblica.", "id": "45"}}
+{"put": "id:msmarco:passage::46", "fields": {"text": "In an era of rising health care costs and decreased medical coverage, the concept of combining surgery with travel has taken off. The last decade has seen a boom in the health tourism sector in Costa Rica, especially in the area of plastic surgery.", "id": "46"}}
+{"put": "id:msmarco:passage::47", "fields": {"text": "The World Bank ranked Costa Rica as having the highest life expectancy, at 78.7 years. This figure is the highest amongst all countries in Latin America, and is equivalent to the level in Canada and higher than the United States by a year. Top Hospitals for Medical Tourism in Costa Rica", "id": "47"}}
+{"put": "id:msmarco:passage::48", "fields": {"text": "Over the last decade, Costa Rica has evolved from being a mere eco-tourism destination and emerged as a country of choice for foreigners, particularly from United States and Canada. These seek quality healthcare services and surgeries at a much lower price than their home countries.", "id": "48"}}
+{"put": "id:msmarco:passage::50", "fields": {"text": "I had 3 cups of coffee and a red bull today all in 4 hours. The first time I urinated, it was an amber color. Then I got worried and drank a lot of water and now my urine is back to normal (light yellow). This only happened once after drinking all that caffeine. Related Topics: Coffee, Urination, Drinking, Caffeine.", "id": "50"}}
+{"put": "id:msmarco:passage::51", "fields": {"text": "During the visual examination of the urine, the laboratorian observes the urine's color and clarity. These can be signs of what substances may be present in the urine. They are interpreted in conjunction with results obtained during the chemical and microscopic examinations to confirm what substances are present.", "id": "51"}}
+{"put": "id:msmarco:passage::52", "fields": {"text": "But the basic details of your urine -- color, smell, and how often you go -- can give you a hint about going on inside your body. Pee is your liquid waste, mainly made of water, salt, and chemicals called urea and uric acid. Your kidneys make it when they filter toxins and other bad stuff from your blood.", "id": "52"}}
+{"put": "id:msmarco:passage::53", "fields": {"text": "However, red-colored urine can also occur when blood is present in the urine and can be an indicator of disease or damage to some part of the urinary system. Another example is yellow-brown or greenish-brown urine that may be a sign of bilirubin in the urine (see The Chemical Examination section).", "id": "53"}}
+{"put": "id:msmarco:passage::54", "fields": {"text": "The shade, light or dark, also changes. If it has no color at all, that may be because been drinking a lot of water or taking a drug called a diuretic, which helps your body get rid of fluid. Very dark honey- or brown-colored urine could be a sign that you’re dehydrated and need to get more fluids right away.", "id": "54"}}
+{"put": "id:msmarco:passage::55", "fields": {"text": "A good rule of thumb is the darker your urine, the more water you need to drink. And if your urine is any other color besides a various shade of yellow (which get into down below) something may be wrong.", "id": "55"}}
+{"put": "id:msmarco:passage::56", "fields": {"text": "Color, density, and smell can reveal health problems. Human urine has been a useful tool of diagnosis since the earliest days of medicine. The color, density, and smell of urine can reveal much about the state of our health. Here, for starters, are some of the things you can tell from the hue of your liquid excreta. Advertising Policy.", "id": "56"}}
+{"put": "id:msmarco:passage::57", "fields": {"text": "More concentrated urine in the bladder can be darker. As long as your urine returned to a more-normal, light yellow color after you drank more water, there is no need to be concerned.", "id": "57"}}
+{"put": "id:msmarco:passage::58", "fields": {"text": "The color, density, and smell of urine can reveal much about the state of our health. Here, for starters, are some of the things you can tell from the hue of your liquid excreta. Cleveland Clinic is a non-profit academic medical center. Advertising on our site helps support our mission.", "id": "58"}}
+{"put": "id:msmarco:passage::59", "fields": {"text": "The most common cause for liver transplantation in adults is cirrhosis caused by various types of liver injuries such as infections (hepatitis B and C), alcohol, autoimmune liver diseases, liver cancer, metabolic and hereditary disorders, but also diseases of unknown aetiology.ombination therapy of ursodeoxycholic acid and corticosteroids for primary biliary cirrhosis with features of autoimmune hepatitis: a meta-analysis. A meta-analysis was performed of RCTs comparing therapies that combine UDCA and corticosteroids with UDCA monotherapy.", "id": "59"}}
+{"put": "id:msmarco:passage::60", "fields": {"text": "Inborn errors of bile acid synthesis can produce life-threatening cholestatic liver disease (which usually presents in infancy) and progressive neurological disease presenting later in childhood or in adult life.he neurological presentation often includes signs of upper motor neurone damage (spastic paraparesis). The most useful screening test for many of these disorders is analysis of urinary cholanoids (bile acids and bile alcohols); this is usually now achieved by electrospray ionisation tandem mass spectrometry.", "id": "60"}}
+{"put": "id:msmarco:passage::61", "fields": {"text": "Autoimmune liver disease and thyroid disease. Autoimmune disorders, including autoimmune thyroid disorders, occur in up to 34% of patients with autoimmune hepatitis. The presence of these disorders is associated with female sex, older age and certain human leukocyte antigens (HLAs).he liver might also be affected in patients with the genetic autoimmune disease, polyglandular autoimmune syndrome, which affects the thyroid gland. This interaction again demonstrates crosstalk between autoimmune disturbances in the thyroid system and the liver.", "id": "61"}}
+{"put": "id:msmarco:passage::62", "fields": {"text": "Primary biliary cirrhosis, or PBC, is a chronic, or long-term, disease of the liver that slowly destroys the medium-sized bile ducts within the liver. Bile is a digestive liquid that is made in the liver. It travels through the bile ducts to the small intestine, where it helps digest fats and fatty vitamins.In patients with PBC, the bile ducts are destroyed by inflammation. This causes bile to remain in the liver, where gradual injury damages liver cells and causes cirrhosis, or scarring of the liver.As cirrhosis progresses and the amount of scar tissue in the liver increases, the liver loses its ability to function.t travels through the bile ducts to the small intestine, where it helps digest fats and fatty vitamins. In patients with PBC, the bile ducts are destroyed by inflammation. This causes bile to remain in the liver, where gradual injury damages liver cells and causes cirrhosis, or scarring of the liver.", "id": "62"}}
+{"put": "id:msmarco:passage::63", "fields": {"text": "Hepatitis B and C, alcoholism, hemochromatosis, and primary biliary cirrhosis -- all causes of cirrhosis -- are some of the major risk factors for liver cancer. Cirrhosis due to hepatitis C is the leading cause of hepatocellular carcinoma in the United States.rimary Biliary Cirrhosis. Up to 95% of primary biliary cirrhosis (PBC) cases occur in women, usually around age 50. In people with PBC, the immune system attacks and destroys cells in the bile ducts. Like many autoimmune disorders, the causes of PBC are unknown.", "id": "63"}}
+{"put": "id:msmarco:passage::64", "fields": {"text": "The disorders of peroxisome biogenesis and peroxisomal oxidation that affect bile acid synthesis will be covered in the review by Ferdinandusse et al.he neurological presentation often includes signs of upper motor neurone damage (spastic paraparesis). The most useful screening test for many of these disorders is analysis of urinary cholanoids (bile acids and bile alcohols); this is usually now achieved by electrospray ionisation tandem mass spectrometry.", "id": "64"}}
+{"put": "id:msmarco:passage::65", "fields": {"text": "The neurological presentation often includes signs of upper motor neurone damage (spastic paraparesis). The most useful screening test for many of these disorders is analysis of urinary cholanoids (bile acids and bile alcohols); this is usually now achieved by electrospray ionisation tandem mass spectrometry.he neurological presentation often includes signs of upper motor neurone damage (spastic paraparesis). The most useful screening test for many of these disorders is analysis of urinary cholanoids (bile acids and bile alcohols); this is usually now achieved by electrospray ionisation tandem mass spectrometry.", "id": "65"}}
+{"put": "id:msmarco:passage::66", "fields": {"text": "Autoimmune Hepatitis. A liver disease in which the body's immune system damages liver cells for unknown reasons. PubMed Health Glossary. (Source: NIH-National Institute of Diabetes and Digestive and Kidney Diseases).ombination therapy of ursodeoxycholic acid and corticosteroids for primary biliary cirrhosis with features of autoimmune hepatitis: a meta-analysis. A meta-analysis was performed of RCTs comparing therapies that combine UDCA and corticosteroids with UDCA monotherapy.", "id": "66"}}
+{"put": "id:msmarco:passage::67", "fields": {"text": "1 itchiness (pruritus). 2  Pruritus is the primary symptom of cholestasis and is thought to be due to interactions of serum bile acids with opioidergic nerves. 3  In fact, the opioid antagonist naltrexone is used to treat pruritus due to cholestasis.ile is secreted by the liver to aid in the digestion of fats. Bile formation begins in bile canaliculi that form between two adjacent surfaces of liver cells (hepatocytes) similar to the terminal branches of a tree.", "id": "67"}}
+{"put": "id:msmarco:passage::68", "fields": {"text": "Primary Biliary Cirrhosis. Up to 95% of primary biliary cirrhosis (PBC) cases occur in women, usually around age 50. In people with PBC, the immune system attacks and destroys cells in the bile ducts. Like many autoimmune disorders, the causes of PBC are unknown.rimary Biliary Cirrhosis. Up to 95% of primary biliary cirrhosis (PBC) cases occur in women, usually around age 50. In people with PBC, the immune system attacks and destroys cells in the bile ducts. Like many autoimmune disorders, the causes of PBC are unknown.", "id": "68"}}
+{"put": "id:msmarco:passage::69", "fields": {"text": "However, a major motive with people today wanting to use the barley harvest to determine the start of the year is to justify starting the year AS EARLY AS POSSIBLE, frequently even before the end of winter. That is in fact the opposite of what the Talmud records the leaders of the Sanhedrin occasionally doing ...", "id": "69"}}
+{"put": "id:msmarco:passage::70", "fields": {"text": "Some people claim that the timing of the barley harvest in Israel should be the deciding factor as to when to start the new year for determining the observance of God's annual Feasts and Holy Days.", "id": "70"}}
+{"put": "id:msmarco:passage::71", "fields": {"text": "Barley (Hordeum vulgare L.), a member of the grass family, is a major cereal grain. It was one of the first cultivated grains and is now grown widely. Barley grain is a staple in Tibetan cuisine and was eaten widely by peasants in Medieval Europe. Barley has also been used as animal fodder, as a source of fermentable material for beer and certain distilled beverages, and as a component of various health foods.", "id": "71"}}
+{"put": "id:msmarco:passage::72", "fields": {"text": "The state of the barley harvest could PERHAPS cause the start of a year to be postponed to THE FOLLOWING NEW MOON (thereby giving the previous year a 13th month), but the state of the barley harvest could NEVER DETERMINE THAT AN EARLIER NEW MOON SHOULD BE USED TO START THE YEAR!", "id": "72"}}
+{"put": "id:msmarco:passage::73", "fields": {"text": "The grape harvest was usually completed before Tabernacles, but most of the olive harvest came after the autumn festivals. In ancient Israel the primary harvest season extended from April to November. This harvest period might be subdivided into three seasons and three major crops: the spring grain harvest, the summer grape harvest and the autumn olive harvest.", "id": "73"}}