GenAI Essentials – Full Course for Beginners - Ep104

that data andconvert it into its tokenized formatsright so if we wanted to implement thisourselves um it would be startup and aireally trickybecause what we would have to do is we'dhave to understand the architecture ofthe model and take off part of the modelthat does thetokenization um and and be able to takethe data and tokenize it and then alsoinject the data after that tokenizationstep step so you can see you'd have toreally mangle a model to do it um butanyway this thing is really great butit's only for very specific models umNot only would you gain um uh improvedresponse times for very large files orlarge documents but you could also savemoney so for example with Google Geminithe cash tokens might be build at areduced rate and what I say might theyare um but I don't know if that's thesame with anthropic Cloud Sonet and I'veonly again seen this in a few models soit's not like it's ubiquitous with allmodels um but it is something that'sworth checking out if you are doing oneof those two things working with verylarge files or um uh video files orlarge documents[Music]okay so structure Json is when you wantto force an element to produce structureJson as its output and there aremultiple techniques to force structureJson there's context free grammar uhfinite State machines and reg xes thatyou can utilize um and structured outputcan be a thirdparty library or it'sbuilt into the API of an llm and so thenthey're Implement implementing anotherstep um and so there's multiplestrategies but from experience I findthat trying to get structured Jsontelling an llm to just give you Jasonback is not easy and so you really doneed a secondary step outside of the llmto force it to do exactly what you wantand let's just talk about generally howthis process works well the idea is thatyou have an input you have an llm andthen uh what you're doing is that everytime it produces a letterokay you have some kind of schema thatis implemented either represented panticor or Jason schema and the idea is thatit's going to force the next letter tobe what would make sense so if you'reproducing Json and if it's the firstthing the first thing should either be asquare brace or curly so if the LM thefirst token comes out of it is not asquare brace or curly which it does notmatch the regular expression that itwill throw it away until it matches whatit expects it to match right and soyou're basically forc it token by tokento produce exactly what you want um andso hopefully that makes sense uh butyeah this stuff gets really complicatedsome LMS will tell you that the LM togenerate uh that you have to tell the LMto generate Json so for example if we'reusing coh here coh here specificallysays tell it to generate out uh Json butfor the most part if you're using athirdparty library that is not part ofthe um part of the uh API of the llmthen you don't have to do thatokay but let's take a look at um twoways that we can uh use structure Jsonspecifically where it's built into apisand a separate video we'll look at howwe can use a third party to do it sowith open AI they have an API forstructured output that requires the useof pantic pantic is um this validationtool for validating structures of datain um Python and the thing is is thatyou can use this to represent your Jsonso it does get converted to Json at somepoint but this is something you'llcommonly see with structured outputs isthey'll want pantic and so here we'redefining um I want like uh a bunch ofphrases and then those phrases arerepresented by action verbs and so thenI would pass that in as the responseformat and that would produce back JsonI have another example here with cohhere and it uses the Json schema basedon the Json schema.org website and thisis the exact same thing that we'reasking uh uh but now we have to use aJson scheme instead of pantic and thenwe're going to pass that alongum as the response format um so yeah butI do want to point out that gettingstructure Jason back is very challengingand sometimes you'll find that if youmake it too complicated it gives reallybad results sometimes the naming of theactual uh things within the uh the Jsonobject really help it um and so yeah youreally have to work hard to get um Jsonoutput back and sometimes it forces youto use specific models because sometimesthe only time you get reliable resultsis like with something with open AI uhand the only thing that's kind offrustrating is that we don't knowexactly how their um structure Jsonworks and a lot of times these are betafeatures and so I don't know if thesefeatures will vanish in the future uh infavor of third-party ones but we willlook at thirdparty ones that we canImplement as well[Music]okay let's take a look at instructorwhich is a thirdparty library that canproduce structure Json output I wassuper excited to find this as there wasa bunch of other ones that didn't reallywork but this one worked perfectly atleast for the code examples that Ipulled up for it uh which we can find atpython. usinordocomo um the only ones that I kind ofwish were there was something likeAmazon Bedrock but I understand thatit's not there at this time um butanyway uh here is a simple example ofGro and actually when we use gro we endup utilizing this implementation you cansee that this one is utilizing pythongantic um but the idea is that you canbasically uh use um this outputed formatwith any of these things so you likeit's not reliant on the built-in oneit'll just work with all these which Ithink is really cool uh but there you go[Music]okay hey this is angre brown and thisvideo I want to take a look at web UI soweb UI I believe it's called WebUI if it's not we'll find out what ismaybe it's called open web UI and so I'mtold that this thing allows us to have

(Editor: html)