If A and B are distinct, nonzero, relatively prime integers, write EA,B for the elliptic curve deﬁned by y2 = x(x + A)(x + B). Since EA,B and E−A,−B are isomorphic over the complex numbers (i.e., as Riemann surfaces), EA,B is modular if and only if E−A,−B is modular. If further AB(A B) is divisible by 16, then either EA,B or E−A,−B is semistable (this is easy to check directly; see for example I.1 of [24]). The Semistable Taniyama-Shimura Conjecture now implies that both § EA,B and E−A,−B are modular, and thus implies Theorem 1.1.\n\n−\n\nRemark . In 1.1 we saw that Theorem 1.1 and Ribet’s Theorem together imply Fermat’s Last Theorem. Therefore, the Semistable Taniyama-Shimura Conjecture implies Fermat’s Last Theorem.\n\n§\n\n1.6. Modular forms. In this paper we will work with a deﬁnition of modularity which uses modular forms.\n\nDeﬁnition. If N is a positive integer, a modular form f of weight k for Γ0(N ) is C which satisﬁes a holomorphic function f : H\n\n→\n\nf (γ(z)) = (cz + d)kf (z)\n\na b c d\n\nH,\n\n(3)\n\nΓ0(N ) and z\n\nfor every γ =\n\n∈\n\n∈\n\n(cid:1)\n\n(cid:0)\n\nand is holomorphic at the cusps (see Chapter 2 of [35]).\n\n1 1 0 1\n\nΓ0(N )), so ∞ n=0 ane2πinz, with complex numbers an and it has a Fourier expansion f (z) = (cid:1) . We say f is a cusp form if it with n vanishes at all the cusps; in particular for a cusp form the coeﬃcient a0 (the value at i\n\nA modular form f


the coeﬃcient a0 (the value at i\n\nA modular form f satisﬁes f (z) = f (z + 1) (apply (3) to\n\n∈\n\n(cid:0)\n\n0 because f is holomorphic at the cusp i\n\n≥\n\n∞\n\nP\n\n) is zero. Call a cusp form normalized if a1 = 1.\n\n∞ For ﬁxed N there are commuting linear operators (called Hecke operators) Tm, 1, on the (ﬁnite-dimensional) vector space of cusp forms of weight\n\nfor integers m two for Γ0(N ) (see Chapter 3 of [35]). If f (z) =\n\n≥\n\n∞ n=1 ane2πinz, then\n\nP danm/d2\n\n∞\n\ne2πinz\n\n(4)\n\nTmf (z) =\n\nn=1 X\n\n(d,N )=1 d|(n,m)\n\n(cid:0) X\n\n(cid:1)\n\nwhere (a, b) denotes the greatest common divisor of a and b and a b means that a divides b. The Hecke algebra T (N ) is the ring generated over Z by these operators.\n\n|\n\nDeﬁnition. In this paper an eigenform will mean a normalized cusp form of weight two for some Γ0(N ) which is an eigenfunction for all the Hecke operators.\n\n∞ n=1 ane2πinz is an eigenform, then Tmf = amf for all m.\n\nBy (4), if f (z) =\n\nP\n\n6\n\nK. RUBIN AND A. SILVERBERG\n\n1.7. Modularity, revisited. Suppose E is an elliptic curve over Q. If p is a prime, write Fp for the ﬁnite ﬁeld with p elements, and let E(Fp) denote the Fp- solutions of the equation for E (including the point at inﬁnity). We now give a second deﬁnition of modularity for an elliptic curve.\n\nDeﬁnition. An elliptic


of modularity for an elliptic curve.\n\nDeﬁnition. An elliptic curve E over Q is modular if there exists an eigenform\n\n∞ n=1 ane2πinz such that for all but ﬁnitely many primes q,\n\n#(E(Fq)).\n\n(5) P\n\naq = q + 1\n\n− 2. An overview\n\nThe ﬂow chart shows how Fermat’s Last Theorem would follow if one knew the Semistable Modular Lifting Conjecture (Conjecture 2.1) for the primes 3 and 5. 1 we discussed the upper arrow, i.e., the implication “Semistable Taniyama- In § Fermat’s Last Theorem”. In this section we will discuss the Shimura Conjecture other implications in the ﬂow chart. The implication given by the lowest arrow is straightforward (Proposition 2.3), while the middle one uses an ingenious idea of Wiles (Proposition 2.4).\n\n⇒\n\nFermat’s Last Theorem\n\n✻\n\nSemistable Taniyama-Shimura Conjecture\n\n✻\n\n(cid:0)\n\n❅ ❅\n\n(cid:0)\n\nSemistable Taniyama-Shimura for ¯ρE,3 irreducible\n\nSemistable Modular Lifting for p = 5\n\n✻\n\n(cid:0) (cid:0)\n\n❅\n\n❅\n\nSemistable Modular Lifting for p = 3\n\nLanglands-Tunnell Theorem\n\nSemistable Modular Lifting Conjecture\n\nFermat’s Last Theorem .\n\n⇒\n\nRemark . By the Modular Lifting Conjecture we will mean the Semistable Modular Lifting Conjecture with the hypothesis of semistability removed. The arguments of this section can also be used to show that the Modular Lifting Conjecture for p = 3 and 5, together with the Langlands-Tunnell Theorem, imply the full Taniyama- Shimura Conjecture.\n\nA REPORT ON WILES’ CAMBRIDGE


Shimura Conjecture.\n\nA REPORT ON WILES’ CAMBRIDGE LECTURES\n\n7\n\n2.1. Semistable Modular Lifting. Let ¯Q denote the algebraic closure of Q in C, and let GQ be the Galois group Gal( ¯Q/Q). If p is a prime, write\n\nF× p\n\n¯εp : GQ\n\n→\n\nfor the character giving the action of GQ on the p-th roots of unity. For the facts about elliptic curves stated below, see [39]. If E is an elliptic curve over Q and F is a subﬁeld of the complex numbers, there is a natural commutative group law on the set of F -solutions of E, with the point at inﬁnity as the identity element. Denote this group E(F ). If p is a prime, write E[p] for the subgroup of points in E( ¯Q) of order dividing p. Then E[p] ∼= F2 p. The action of GQ on E[p] gives a continuous representation\n\nGL2(Fp)\n\n¯ρE,p : GQ\n\n→\n\n(deﬁned up to isomorphism) such that\n\n(6)\n\ndet(¯ρE,p) = ¯εp\n\nand for all but ﬁnitely many primes q,\n\n#(E(Fq))\n\n(7)\n\ntrace(¯ρE,p(Frobq))\n\nq + 1\n\n(mod p).\n\n≡ (See Appendix A for the deﬁnition of the Frobenius elements Frobq ∈ to each prime number q.)\n\n−\n\nGQ attached\n\n∞ n=1 ane2πinz is an eigenform, let\n\nOf denote the ring of integers of the number ﬁeld Q(a2, a3, . . . ). (Recall that our eigenforms are normalized so that a1 = 1.)\n\nIf f (z) =\n\nP\n\nThe following conjecture is in the spirit of a conjecture of Mazur (see Conjectures\n\n3.2 and 4.2).\n\nConjecture 2.1


and 4.2).\n\nConjecture 2.1 (Semistable Modular Lifting Conjecture). Suppose p is an odd prime and E is a semistable elliptic curve over Q satisfying\n\n(a) ¯ρE,p is irreducible, (b) there are an eigenform f (z) =\n\n∞ n=1 ane2πinz and a prime ideal λ of\n\nOf\n\nsuch that p\n\nλ and for all but ﬁnitely many primes q,\n\n∈\n\nP\n\n#(E(Fq))\n\naq ≡\n\nq + 1\n\n(mod λ).\n\n−\n\nThen E is modular.\n\nThe Semistable Modular Lifting Conjecture is a priori weaker than the Semi- stable Taniyama-Shimura Conjecture because of the extra hypotheses (a) and (b). The more serious condition is (b); there is no known way to produce such a form in general. But when p = 3, the existence of such a form follows from the theorem below of Tunnell [41] and Langlands [20]. Wiles then gets around condition (a) by a clever argument (described below) which, when ¯ρE,3 is not irreducible, allows him to use p = 5 instead.\n\n8\n\nK. RUBIN AND A. SILVERBERG\n\n2.2. Langlands-Tunnell Theorem. In order to state the Langlands-Tunnell Theorem, we need weight-one modular forms for a subgroup of Γ0(N ). Let\n\na b c d\n\nSL2(Z) : c\n\n0 (mod N ), a\n\nd\n\n1 (mod N )\n\n.\n\nΓ1(N ) =\n\n∈\n\n≡\n\n≡\n\n≡\n\n(cid:1)\n\n(cid:9)\n\n(cid:8)(cid:0)\n\nReplacing Γ0(N ) by Γ1(N ) in 1.6, one can deﬁne the notion of cusp forms on § Γ1(N ). See Chapter 3 of [35] for the deﬁnitions of the Hecke operators on the


3 of [35] for the deﬁnitions of the Hecke operators on the space of weight-one cusp forms for Γ1(N ).\n\nTheorem 2.2 (Langlands-Tunnell). Suppose ρ : GQ GL2(C) is a continuous irreducible representation whose image in PGL2(C) is a subgroup of S4 (the sym- metric group on four elements ), τ is complex conjugation, and det(ρ(τ )) = 1. ∞ n=1 bne2πinz for some Γ1(N ), which is an Then there is a weight-one cusp form eigenfunction for all the corresponding Hecke operators, such that for all but ﬁnitely many primes q,\n\n→\n\n−\n\nP\n\n(8)\n\nbq = trace(ρ(Frobq)).\n\nThe theorem as stated by Langlands [20] and by Tunnell [41] produces an auto- morphic representation rather than a cusp form. Using the fact that det(ρ(τ )) = 1, standard techniques (see for example [12]) show that this automorphic repre-\n\n− sentation corresponds to a weight-one cusp form as in Theorem 2.2.\n\n2.3. Semistable Modular Lifting\n\nSemistable Taniyama-Shimura.\n\n⇒\n\nProposition 2.3. Suppose the Semistable Modular Lifting Conjecture is true for p = 3, E is a semistable elliptic curve, and ¯ρE,3 is irreducible. Then E is modular.\n\nProof. It suﬃces to show that hypothesis (b) of the Semistable Modular Lifting Conjecture is satisﬁed with the given curve E, for p = 3. There is a faithful representation\n\nGL2(Z[√\n\nGL2(C)\n\nψ : GL2(F3) ֒\n\n2])\n\n−\n\n⊂\n\n→\n\nGL2(F3),\n\nsuch that for every g\n\n∈ trace(ψ(g))\n\n(mod(1 +


for every g\n\n∈ trace(ψ(g))\n\n(mod(1 + √\n\n(9)\n\ntrace(g)\n\n2))\n\n≡\n\n−\n\nand\n\n(10)\n\ndet(ψ(g))\n\ndet(g)\n\n(mod 3).\n\n≡\n\nExplicitly, ψ can be deﬁned on generators of GL2(F3) by\n\n√\n\n1 1 1 0\n\n1 1 1 0\n\n1 1\n\n1 1\n\n2 1 1 0\n\n.\n\nψ\n\n=\n\nand ψ\n\n=\n\n− −\n\n− −\n\n−\n\n−\n\n(cid:19)\n\n(cid:18)(cid:18)\n\n(cid:19)(cid:19)\n\n(cid:18)\n\n(cid:18)(cid:18) ¯ρE,3. If τ is complex conjugation, then it follows from (6) and (10) that 1. The image of ψ in PGL2(C) is a subgroup of PGL2(F3) ∼= S4.\n\n(cid:19)\n\n(cid:19)(cid:19)\n\n(cid:18)\n\nLet ρ = ψ ◦ det(ρ(τ )) = Using that ¯ρE,3 is irreducible, one can show that ρ is irreducible.\n\n−\n\n∞ n=1 bne2πinz be a weight-one cusp form for some Γ1(N ) obtained by applying the Langlands-Tunnell\n\nLet p be a prime of ¯Q containing 1 + √\n\n2. Let g(z) =\n\n−\n\nP\n\nA REPORT ON WILES’ CAMBRIDGE LECTURES\n\n9\n\nTheorem (Theorem 2.2) to ρ. It follows from (6) and (10) that N is divisible by 3. The function\n\n0 if d 1 if d 1 if d\n\n0 (mod 3), 1 (mod 3), 2 (mod


d\n\n0 (mod 3), 1 (mod 3), 2 (mod 3)\n\n∞\n\n≡ ≡ ≡\n\nχ(d)e2πinz where χ(d) =\n\nE(z) = 1 + 6\n\n\uf8f1 \uf8f2\n\nn=1 X\n\nXd|n\n\n−\n\n∞ n=1 cne2πinz is a weight-one modular form for Γ1(3). The product g(z)E(z) = It is now is a weight-two cusp form for Γ0(N ) with cn ≡ bn possible to ﬁnd an eigenform f (z) = (mod p) for every n (see 6.10 and 6.11 of [4]). By (7), (8), and (9), f satisﬁes (b) of the Semistable Modular Lifting Conjecture with p = 3 and with λ = p\n\n\uf8f3\n\nbn (mod p) for all n. P n=1 ane2πinz on Γ0(N ) such that an ≡ ∩ Of .\n\n∞\n\nP\n\nProposition 2.4 (Wiles). Suppose the Semistable Modular Lifting Conjecture is true for p = 3 and 5, E is a semistable elliptic curve over Q, and ¯ρE,3 is reducible. Then E is modular.\n\nProof. The elliptic curves over Q for which both ¯ρE,3 and ¯ρE,5 are reducible are all known to be modular (see Appendix B.1). Thus we can suppose ¯ρE,5 is irreducible. It suﬃces to produce an eigenform as in (b) of the Semistable Modular Lifting Conjecture, but this time there is no analogue of the Langlands-Tunnell Theorem to help. Wiles uses the Hilbert Irreducibility Theorem, applied to a parameter space of elliptic curves, to produce another semistable elliptic curve E′ over Q satisfying\n\n(i) ¯ρE′,5 is isomorphic to ¯ρE,5, and (ii)


is isomorphic to ¯ρE,5, and (ii) ¯ρE′,3 is irreducible.\n\n(In fact there will be inﬁnitely many such E′; see Appendix B.2.) Now by Proposi- ∞ n=1 ane2πinz be a corresponding eigenform. tion 2.3, E′ is modular. Let f (z) = Then for all but ﬁnitely many primes q, P\n\n#(E′(Fq)) trace(¯ρE,5(Frobq))\n\naq = q + 1\n\ntrace(¯ρE′,5(Frobq)) #(E(Fq)) q + 1\n\n−\n\n≡ ≡\n\n(mod 5)\n\n≡\n\n−\n\nby (7). Thus the form f satisﬁes hypothesis (b) of the Semistable Modular Lifting Conjecture, and we conclude that E is modular.\n\nTaken together, Propositions 2.3 and 2.4 show that the Semistable Modular Lifting Conjecture for p = 3 and 5 implies the Semistable Taniyama-Shimura Con- jecture.\n\n3. Galois representations\n\nThe next step is to translate the Semistable Modular Lifting Conjecture into a conjecture (Conjecture 3.2) about the modularity of liftings of Galois repre- sentations. Throughout this paper, if A is a topological ring, a representation GL2(A) will mean a continuous homomorphism and [ρ] will denote the ρ : GQ isomorphism class of ρ. If p is a prime, let\n\n→\n\nZ× p\n\nεp : GQ\n\n→\n\nbe the character giving the action of GQ on p-power roots of unity.\n\n10\n\nK. RUBIN AND A. SILVERBERG\n\n3.1. The p-adic representation attached to an elliptic curve. Suppose E is an elliptic curve over Q and p is a prime number. For every positive integer n, write E[pn] for the subgroup in E( ¯Q) of points of order dividing pn and Tp(E) for the


in E( ¯Q) of points of order dividing pn and Tp(E) for the inverse limit of the E[pn] with respect to multiplication by p. For every n, E[pn] ∼= (Z/pnZ)2, and so Tp(E) ∼= Z2 p. The action of GQ induces a representation\n\nGL2(Zp)\n\nρE,p : GQ\n\n→\n\nsuch that det(ρE,p) = εp and for all but ﬁnitely many primes q,\n\n#(E(Fq)).\n\n(11)\n\ntrace(ρE,p(Frobq)) = q + 1\n\n−\n\nComposing ρE,p with the reduction map from Zp to Fp gives ¯ρE,p of\n\n2.1. §\n\n3.2. Modular representations. If f is an eigenform and λ is a prime ideal of Of at λ. Of , let\n\nOf,λ denote the completion of\n\nDeﬁnition. If A is a ring, a representation ρ : GQ if there are an eigenform f (z) = homomorphism ι :\n\nGL2(A) is called modular ∞ n=1 ane2πinz, a ring A′ containing A, and a\n\n→\n\nA′ such that for all but ﬁnitely many primes q,\n\nOf →\n\nP\n\ntrace(ρ(Frobq)) = ι(aq).\n\n∞ n=1 ane2πinz and a prime ideal λ of\n\nExamples. (i) Given an eigenform f (z) = Of , Eichler and Shimura (see\n\n7.6 of [35]) constructed a representation\n\n§\n\nP\n\nρf,λ : GQ\n\nGL2(\n\nOf,λ)\n\n→\n\nZ = pZ) and for all but ﬁnitely many primes q,\n\nsuch that det(ρf,λ) = εp (where λ\n\n∩\n\n(12)\n\ntrace(ρf,λ(Frobq)) = aq.\n\nThus ρf,λ is modular with ι taken to be


= aq.\n\nThus ρf,λ is modular with ι taken to be the inclusion of\n\nOf in\n\nOf,λ.\n\n(ii) Suppose p is a prime and E is an elliptic curve over Q. If E is modular, then ρE,p and ¯ρE,p are modular by (11), (7), and (5). Conversely, if ρE,p is modular, then it follows from (11) that E is modular. This proves the following.\n\nTheorem 3.1. Suppose E is an elliptic curve over Q. Then\n\nE is modular\n\nρE,p is modular for every p\n\nρE,p is modular for one p.\n\n⇔\n\n⇔\n\nRemark . In this language, the Semistable Modular Lifting Conjecture says that if p is an odd prime, E is a semistable elliptic curve over Q, and ¯ρE,p is modular and irreducible, then ρE,p is modular.\n\nA REPORT ON WILES’ CAMBRIDGE LECTURES\n\n11\n\n3.3. Liftings of Galois representations. Fix a prime p and a ﬁnite ﬁeld k of characteristic p. Recall that ¯k denotes an algebraic closure of k.\n\nGiven a map φ : A\n\nB, the induced map from GL2(A) to GL2(B) will also be\n\n→\n\ndenoted φ. If ρ : GQ A′ for the composition of ρ with the inclusion of GL2(A) in GL2(A′).\n\nGL2(A) is a representation and A′ is a ring containing A, we write\n\n→\n\nρ\n\n⊗\n\nDeﬁnition. If ¯ρ : GQ ρ : GQ Zp-algebra and there exists a homomorphism ι : A\n\nGL2(k) is a representation, we say that a representation GL2(A) is a lifting of ¯ρ (to A) if A is a complete noetherian local\n\n→\n\n→\n\n¯k such that the diagram\n\n→


local\n\n→\n\n→\n\n¯k such that the diagram\n\n→ GL2(A)\n\n✟✟✯\n\n[ρ]\n\n✟✟\n\nι ❄ GL2(¯k)\n\n✲\n\nGQ\n\n[ ¯ρ ⊗ ¯k]\n\n¯k].\n\ncommutes, in the sense that [ι\n\nρ] = [¯ρ\n\n\n\n⊗\n\nExamples. (i) If E is an elliptic curve then ρE,p is a lifting of ¯ρE,p.\n\n(ii) If E is an elliptic curve, p is a prime, and hypotheses (a) and (b) of Conjecture\n\n2.1 hold with an eigenform f and prime ideal λ, then ρf,λ is a lifting of ¯ρE,p.\n\n3.4. Deformation data. We will be interested not in all liftings of a given ¯ρ, but rather in those satisfying various restrictions. See Appendix A for the deﬁnition of GQ associated to primes q. We say that a representation ρ the inertia groups Iq ⊂ of GQ is unramiﬁed at a prime q if ρ(Iq) = 1. If Σ is a set of primes, we say ρ is unramiﬁed outside of Σ if ρ is unramiﬁed at every q / ∈\n\nΣ.\n\nDeﬁnition. By deformation data we mean a pair\n\n= (Σ, t)\n\nD where Σ is a ﬁnite set of primes and t is one of the words ordinary or ﬂat.\n\nZ×\n\nA× be the composition of the\n\nIf A is a Zp-algebra, let εA : GQ\n\np →\n\n→\n\ncyclotomic character εp with the structure map.\n\nDeﬁnition. Given deformation data type- outside of Σ, and ρ is t at p (where t\n\nGL2(A) is if A is a complete noetherian local Zp-algebra, det(ρ) = εA,


is a complete noetherian local Zp-algebra, det(ρ) = εA, ρ is unramiﬁed\n\n, a representation ρ : GQ\n\nD\n\n→\n\nD\n\nordinary, ﬂat }\n\n; see Appendix C).\n\n∈ {\n\nDeﬁnition. A representation ¯ρ : GQ eigenform f and a prime ideal λ of\n\nmodular if there are an\n\nGL2(k) is Of such that ρf,λ is a type-\n\n→\n\nD\n\nlifting of ¯ρ.\n\nD\n\nRemarks. (i) A representation with a type- fore if a representation is\n\nlifting must itself be type-\n\n. There-\n\nD\n\nD and modular.\n\nmodular, then it is both type-\n\nD\n\nD\n\n(ii) Conversely, if ¯ρ is type-\n\n, modular, and satisﬁes (ii) of Theorem 5.3 below, -modular, by work of Ribet and others (see [28]). This plays an important\n\nD\n\nthen ¯ρ is D role in Wiles’ work.\n\n12\n\nK. RUBIN AND A. SILVERBERG\n\n3.5. Mazur Conjecture.\n\nDeﬁnition. A representation ¯ρ : GQ ¯ρ\n\nGL2(k) is called absolutely irreducible if\n\n→\n\n¯k is irreducible.\n\n⊗\n\nThe following variant of a conjecture of Mazur (see Conjecture 18 of [23]; see\n\nalso Conjecture 4.2 below) implies the Semistable Modular Lifting Conjecture.\n\nConjecture 3.2 (Mazur). Suppose p is an odd prime, k is a ﬁnite ﬁeld of charac- GL2(k) is an absolutely irreducible teristic p, lifting of ¯ρ to the ring of integers of\n\nis deformation data, and ¯ρ : GQ -modular representation. Then every type-\n\nD\n\n→ D\n\nD a ﬁnite extension of Qp is


D\n\nD a ﬁnite extension of Qp is modular.\n\nRemark . Loosely speaking, Conjecture 3.2 says that if ¯ρ is modular, then every lifting which “looks modular” is modular.\n\nDeﬁnition. An elliptic curve E over Q has good (respectively, bad ) reduction at a prime q if E is nonsingular (respectively, singular) modulo q. An elliptic curve E over Q has ordinary (respectively, supersingular) reduction at q if E has good reduction at q and E[q] has (respectively, does not have) a subgroup of order q stable under the inertia group Iq.\n\nProposition 3.3. Conjecture 3.2 implies Conjecture 2.1.\n\nProof. Suppose p is an odd prime and E is a semistable elliptic curve over Q which satisﬁes (a) and (b) of Conjecture 2.1. We will apply Conjecture 3.2 with ¯ρ = ¯ρE,p. Write τ for complex conjugation. Then τ 2 = 1, and by (6), det(¯ρE,p(τ )) = 1. Since ¯ρE,p is irreducible and p is odd, a simple linear algebra argument now shows that ¯ρE,p is absolutely irreducible.\n\n−\n\nSince E satisﬁes (b) of Conjecture 2.1, ¯ρE,p is modular. Let\n\nΣ = t = ordinary if E has ordinary or bad reduction at p, t = ﬂat if E has supersingular reduction at p,\n\np\n\nprimes q : E has bad reduction at q\n\n,\n\n•\n\n{\n\n} ∪ {\n\n}\n\n= (Σ, t).\n\nD\n\nUsing the semistability of E, one can show that ρE,p is a type- (by combining results of several people; see [28]) that ¯ρE,p is 3.2 then says ρE,p is modular. By Theorem 3.1, E is modular.\n\nlifting of ¯ρE,p and -modular.


E is modular.\n\nlifting of ¯ρE,p and -modular. Conjecture\n\nD\n\nD\n\n4. Mazur’s deformation theory\n\nNext we reformulate Conjecture 3.2 as a conjecture (Conjecture 4.2) that the algebras which parametrize liftings and modular liftings of a given representation are isomorphic. It is this form of Mazur’s conjecture that Wiles attacks directly.\n\nA REPORT ON WILES’ CAMBRIDGE LECTURES\n\n13\n\n4.1. The universal deformation algebra R. Fix an odd prime p, a ﬁnite ﬁeld k of characteristic p, deformation data representation ¯ρ : GQ extension of Qp with residue ﬁeld k.\n\n, and an absolutely irreducible type-\n\nD\n\nD is the ring of integers of a ﬁnite\n\nGL2(k). Suppose\n\n→\n\nO\n\nDeﬁnition. We say ρ : GQ complete noetherian local commutes\n\n)-lifting of ¯ρ if ρ is type-\n\n, A is a → -algebra with residue ﬁeld k, and the following diagram\n\nGL2(A) is a (\n\n,\n\nD\n\nO\n\nD\n\nO\n\nGL2(A)\n\n✟✟✯\n\n[ρ]\n\n✟✟\n\n❄ GL2(k)\n\n✲\n\nGQ\n\n[ ¯ρ]\n\nwhere the vertical map is reduction modulo the maximal ideal of A.\n\nTheorem 4.1 (Mazur-Ramakrishna). With p, k, an that for every ( φρ : R\n\nas above, there are D GL2(R) of ¯ρ, with the property -algebra homomorphism\n\n, ¯ρ, and\n\nO\n\nalgebra R and a (\n\n)-lifting ρR : GQ )-lifting ρ of ¯ρ to A there is a unique\n\n,\n\nO\n\nD\n\nO\n\n→\n\n,\n\nD\n\nO\n\nO\n\nA such that the


such that the diagram\n\n→\n\n[ρR]\n\n✲\n\nGQ\n\nGL2(R)\n\n❍\n\n❍❍\n\nφρ ❄ GL2(A)\n\n[ρ]\n\n❍❍❥\n\ncommutes.\n\nThis theorem was proved by Mazur [21] in the case when\n\nis ordinary and is ﬂat. Theorem 4.1 determines R and ρR up to\n\nD\n\nby Ramakrishna [25] when isomorphism.\n\nD\n\n4.2. The universal modular deformation algebra T. Fix an odd prime p, a , and an absolutely irreducible ﬁnite ﬁeld k of characteristic p, deformation data -modular, and ﬁx an type- representation ¯ρ : GQ eigenform f and a prime ideal λ of lifting of ¯ρ. is the ring of integers of a ﬁnite extension of Qp with Suppose in addition that residue ﬁeld k, Of,λ ⊆ O\n\nD\n\nGL2(k). Assume ¯ρ is\n\nD\n\n→\n\nD\n\nOf such that ρf,λ is a type-\n\nD\n\nO , and the diagram\n\nGL2(\n\nOf,λ) ❄ GL2(k)\n\n✟✟✟✯ ✲\n\n[ρf,λ] ✟\n\nGQ\n\n[ ¯ρ]\n\ncommutes, where the vertical map is the reduction map.\n\n)-lifting of ¯ρ, and Wiles constructs a generalized Hecke algebra T which has the following properties (recall that Hecke algebras T (N ) were deﬁned in\n\nUnder these assumptions ρf,λ ⊗ O 1.6).\n\nis a (\n\n,\n\nD\n\nO\n\n§\n\n(T1) T is a complete noetherian local\n\nalgebra with residue ﬁeld k.\n\nO\n\n14\n\nK. RUBIN AND A. SILVERBERG\n\n(T2) There are an integer N divisible only by primes in Σ and a homomorphism by the Σ. By abuse


N divisible only by primes in Σ and a homomorphism by the Σ. By abuse of notation\n\nfrom the Hecke algebra T (N ) to T such that T is generated over images of the Hecke operators Tq for primes q / ∈ we write Tq also for its image in T.\n\nO\n\n(T3) There is a (\n\n,\n\n)-lifting\n\nD\n\nO\n\nGL2(T)\n\nρT : GQ\n\n→\n\nof ¯ρ with the property that trace(ρT(Frobq)) = Tq for every prime q / ∈\n\nΣ. )-lifting of ¯ρ to A, then there is a unique\n\n(T4) If ρ is modular and is a (\n\n,\n\nD\n\nO\n\nalgebra homomorphism ψρ : T\n\nA such that the diagram\n\nO\n\n→ [ρ T]\n\n✲\n\nGL2(T)\n\nGQ\n\n❍\n\n❍❍\n\nψρ ❄ GL2(A)\n\n[ρ]\n\n❍❍❥\n\ncommutes.\n\nSince ρT is a (\n\n,\n\n)-lifting of ¯ρ, by Theorem 4.1 there is a homomorphism\n\nD\n\nO\n\nT\n\nϕ : R\n\n→\n\nρR. By (T3), ϕ(trace(ρR(Frobq))) = Tq for every\n\nsuch that ρT is isomorphic to ϕ prime q / ∈\n\nΣ, so it follows from (T2) that ϕ is surjective.\n\n4.3. Mazur Conjecture, revisited. Conjecture 3.2 can be reformulated in the following way.\n\nConjecture 4.2 (Mazur). Suppose p, k, T is an isomorphism. above map ϕ : R\n\n, ¯ρ, and\n\nare as in\n\n4.2. Then the\n\nD\n\nO\n\n§\n\n→\n\nConjecture 4.2 was stated in [23] (Conjecture 18)


4.2 was stated in [23] (Conjecture 18) for\n\nordinary, and Wiles\n\nD\n\nmodiﬁed the conjecture to include the ﬂat case.\n\nProposition 4.3. Conjecture 4.2 implies Conjecture 3.2.\n\nProof. Suppose ¯ρ : GQ -modular, A is D the ring of integers of a ﬁnite extension of Qp, and ρ is a type- lifting of ¯ρ to A. to be the ring of integers of a suﬃciently large ﬁnite extension of Qp, and Taking and its residue ﬁeld, respectively, we may assume that ρ is extending ρ and ¯ρ to A, with φρ a ( as in Theorem 4.1. By (T3) and Theorem 4.1, ψ(Tq) = trace(ρ(Frobq)) for all but 3.5 of [35], given such a homomorphism ψ (and viewing A as ﬁnitely many q. By ∞ n=1 ane2πinz where aq = ψ(Tq) for all but a subring of C), there is an eigenform ﬁnitely many primes q. Thus ρ is modular.\n\nGL2(k) is absolutely irreducible and\n\n→\n\nD\n\nO )-lifting of ¯ρ. Assuming Conjecture 4.2, let ψ = φρ ◦\n\nO\n\nϕ−1 : T\n\n,\n\nD\n\nO\n\n→\n\n§\n\nP\n\nA REPORT ON WILES’ CAMBRIDGE LECTURES\n\n15\n\n5. Wiles’ approach to the Mazur Conjecture\n\nIn this section we sketch the major ideas of Wiles’ attack on Conjecture 4.2. The ﬁrst step (Theorem 5.2), and the key to Wiles’ proof, is to reduce Conjecture 4.2 to a bound on the order of the cotangent space at a prime of R. In 5.2 we § see that the corresponding tangent space is a Selmer group, and in 5.3 we outline a


that the corresponding tangent space is a Selmer group, and in 5.3 we outline a general procedure due to Kolyvagin for bounding sizes of Selmer groups. The input for Kolyvagin’s method is known as an Euler system. The most diﬃcult 5.4), and the part described as “not yet complete” in his part of Wiles’ work ( § December announcement, is his construction of a suitable Euler system. In 5.5 we state the results announced by Wiles (Theorems 5.3 and 5.4 and Corollary 5.5) and explain why Theorem 5.3 suﬃces for proving the Semistable Taniyama-Shimura Conjecture. As an application of Corollary 5.5 we write down an inﬁnite family of modular elliptic curves. , ¯ρ, 5 ﬁx p, k,\n\n§\n\n§\n\n∞ n=1 ane2πinz, and λ as in\n\n4.2.\n\nFor O By property (T4) there is a homomorphism\n\n, f (z) =\n\n§\n\n§\n\nD\n\nP\n\nπ : T\n\n→ O . By property (T2) and (12), π satisﬁes\n\nsuch that π π(Tq) = aq for all but ﬁnitely many q.\n\nρT is isomorphic to ρf,λ ⊗ O\n\n\n\n5.1. Key reduction. Wiles uses the following generalization of a theorem of Mazur, which says that T is Gorenstein.\n\nTheorem 5.1. There is a (noncanonical ) T-module isomorphism\n\n) ∼ →\n\nHomO(T,\n\nT.\n\nO\n\nLet η denote the ideal of\n\ngenerated by the image under the composition\n\nO HomO(T,\n\n) ∼ →\n\nT π\n\nO\n\n→ O\n\nHomO(T,\n\nof the element π ∈ choice of isomorphism in Theorem 5.1.\n\n). The ideal η is well deﬁned independent of


5.1.\n\n). The ideal η is well deﬁned independent of the\n\nO\n\nThe map π determines distinguished prime ideals of T and R,\n\nϕ) = ϕ−1(pT).\n\npT = ker(π),\n\npR = ker(π\n\n\n\nTheorem 5.2 (Wiles). If\n\n#(pR/p2\n\nR)\n\n#(\n\n/η) <\n\n, ∞\n\n≤\n\nO\n\nT is an isomorphism.\n\nthen ϕ : R\n\n→\n\nThe proof is entirely commutative algebra. The surjectivity of ϕ shows that /η). Thus if\n\n#(pR/p2 #(pR/p2\n\n#(pT/p2 #(\n\nT), and Wiles proves that #(pT/p2\n\nR) R)\n\nT)\n\n#(\n\n≥ ≤\n\n≥\n\nO\n\n/η), then\n\nO\n\n#(pR/p2\n\nR) = #(pT/p2\n\n(13)\n\nT) = #(\n\n/η).\n\nO\n\nThe ﬁrst equality in (13) shows that ϕ induces an isomorphism of tangent spaces. Wiles uses the second equality in (13) and Theorem 5.1 to deduce that T is a local\n\n16\n\nK. RUBIN AND A. SILVERBERG\n\ncomplete intersection over that\n\n(that is, there are f1, . . . , fr ∈ O\n\n[[x1, . . . , xr]] such\n\nO\n\nT ∼=\n\n[[x1, . . . , xr]]/(f1, . . . , fr)\n\nO\n\nas morphism.\n\nalgebras). Wiles then combines these two results to prove that ϕ is an iso-\n\nO\n\n5.2. Selmer groups. In general, if M is a torsion GQ-module, a Selmer group attached to M is a subgroup of the Galois cohomology group H 1(GQ, M ) deter- mined by certain


the Galois cohomology group H 1(GQ, M ) deter- mined by certain “local conditions” in the following way. If q is a prime with decomposition group Dq ⊂\n\nGQ, then there is a restriction map\n\nresq : H 1(GQ, M )\n\nH 1(Dq, M ).\n\n→ Jq ⊆\n\nH 1(Dq, M ) : q prime\n\n= For a ﬁxed collection of subgroups { the particular problem under consideration, the corresponding Selmer group is\n\ndepending on\n\nJ\n\n}\n\nres−1\n\nH 1(GQ, M ).\n\nS(M ) =\n\nq (Jq)\n\n⊆\n\nq \\ Write H i(Q, M ) for H i(GQ, M ), and H i(Qq, M ) for H i(Dq, M ).\n\nExample. The original examples of Selmer groups come from elliptic curves. Fix an elliptic curve E and a positive integer m, and take M = E[m], the subgroup of points in E( ¯Q) of order dividing m. There is a natural inclusion\n\nH 1(Q, E[m])\n\nE(Q)/mE(Q) ֒\n\n(14)\n\n→\n\nE( ¯Q) is any\n\nE(Q) to the cocycle σ\n\nobtained by sending x point satisfying my = x. Similarly, for every prime q there is a natural inclusion\n\nσ(y)\n\ny, where y\n\n∈\n\n7→\n\n−\n\n∈\n\nH 1(Qq, E[m]).\n\nE(Qq)/mE(Qq) ֒\n\n→ Deﬁne the Selmer group S(E[m]) in this case by taking the group Jq to be the image of E(Qq)/mE(Qq) in H 1(Qq, E[m]), for every q. This Selmer group is an important tool in studying the arithmetic of E because it contains (via (14)) E(Q)/mE(Q).\n\n5, let m denote the maximal ideal /mn) can be\n\nRetaining


let m denote the maximal ideal /mn) can be\n\nRetaining the notation from the beginning of\n\n§\n\nand ﬁx a positive integer n. The tangent space HomO(pR/p2 R,\n\nof identiﬁed with a Selmer group as follows. Let Vn be the matrix algebra M2(\n\nO\n\nO\n\n/mn), with GQ acting via the adjoint repre-\n\nO\n\nsentation σ(B) = ρf,λ(σ)Bρf,λ(σ)−1. There is a natural injection\n\ns : HomO(pR/p2 R,\n\n/mn) ֒\n\nH 1(Q, Vn)\n\nO\n\n→\n\nwhich is described in Appendix D (see also\n\n1.6 of [21]). Wiles deﬁnes a collection . Let SD(Vn) denote the associated Selmer\n\n§\n\nH 1(Qq, Vn) }\n\n=\n\nJq ⊆\n\ndepending on\n\nJ group. Wiles proves that s induces an isomorphism\n\n{\n\nD\n\n/mn) ∼ →\n\nHomO(pR/p2 R,\n\nSD(Vn).\n\nO\n\nA REPORT ON WILES’ CAMBRIDGE LECTURES\n\n17\n\n5.3. Euler systems. We have now reduced the proof of Mazur’s conjecture to bounding the size of the Selmer groups SD(Vn). About ﬁve years ago Kolyvagin [19], building on ideas of his own and of Thaine [40], introduced a revolutionary new method for bounding the size of a Selmer group. This new machinery, which is crucial for Wiles’ proof, is what we now describe.\n\nH 1(Qq,M ) is } 5.2. Let ˆM = a system of subgroups with associated Selmer group S(M ) as in Hom(M, µm), where µm is the group of m-th roots of unity. For every prime q, the cup product gives a nondegenerate Tate pairing\n\nSuppose M is a GQ-module of odd exponent m and\n\n=\n\nJq ⊆


M is a GQ-module of odd exponent m and\n\n=\n\nJq ⊆ §\n\nJ\n\n{\n\nH 2(Qq, µm) ∼ → H 1(Q, ˆM ), then\n\nH 1(Qq, ˆM )\n\niq : H 1(Qq, M )\n\nZ/mZ\n\n,\n\nh\n\n×\n\n→\n\nH 1(Q, M ) and d\n\n(see Chapters VI and VII of [3]). If c\n\n∈\n\n∈\n\n(15)\n\nresq(c), resq(d) h\n\niq = 0.\n\nq X\n\nH 1(Q, ˆM ) be the Selmer\n\nis a ﬁnite set of primes. Let S∗\n\nSuppose that\n\nL ⊆ H 1(Qq, ˆM ) }\n\nL group given by the local conditions\n\n∗ =\n\nJ ∗ q ⊆\n\n, where\n\nJ\n\n{\n\nthe orthogonal complement of Jq under H 1(Qq, ˆM )\n\n,\n\nif q / if q\n\n, ∈ L . ∈ L\n\niq\n\nJ ∗ q =\n\nh\n\n(\n\nH 1(Q, ˆM ), deﬁne\n\nIf d\n\n∈\n\nZ/mZ\n\nθd :\n\nJq →\n\nYq∈L\n\nby\n\nθd((cq)) =\n\ncq, resq(d) h\n\niq.\n\nXq∈L\n\nWrite resL : H 1(Q, M ) maps. By (15) and the deﬁnition of J ∗ in addition resL is injective on S(M ), then\n\nq∈L H 1(Qq, M ) for the product of the restriction ker(θd). If\n\n→\n\nS∗\n\nq , if d\n\nL, then resL(S(M


, if d\n\nL, then resL(S(M ))\n\n∈\n\n⊆\n\nQ\n\n#(S(M ))\n\n#\n\nker(θd)\n\n.\n\n≤\n\n(cid:0) \\d∈S∗\n\nL\n\n(cid:1)\n\nThe diﬃculty is to produce enough cohomology classes in S∗\n\nL to show that the right side of the above inequality is small. Following Kolyvagin, an Euler system is S∗ L for a large (inﬁnite) collection of sets of a compatible collection of classes κ( )) primes is related to resℓ(κ( )). Once an Euler system is given, Kolyvagin has an inductive procedure for choosing a set\n\n) L\n\n∈\n\n. Loosely speaking, compatible means that if ℓ /\n\n, then resℓ(κ(\n\nℓ\n\nL\n\n∈ L\n\nL ∪ {\n\n}\n\nL\n\nsuch that\n\nL\n\nresL is injective on S(M ),\n\n•\n\nP⊆L ker(θκ(P)) can be computed in terms of κ( ∅\n\n).\n\nT\n\n18\n\nK. RUBIN AND A. SILVERBERG\n\nS∗\n\nS∗\n\n, then S∗\n\nL.)\n\nL, so κ(\n\n)\n\n(Note that if\n\nP ⊆\n\nP For several important Selmer groups it is possible to construct Euler systems for\n\n∈\n\nP ⊆ L\n\nwhich Kolyvagin’s procedure produces a set\n\nactually giving an equality\n\nL ker(θκ(P))\n\n#(S(M )) = #\n\n.\n\n(cid:0) \\P⊆L This is what Wiles needs to do for the Selmer group SD(Vn). There are several examples in the literature where this kind of argument is worked out in some detail. For the simplest case, where the Selmer group in question is the ideal class group ) are constructed from cyclotomic


where the Selmer group in question is the ideal class group ) are constructed from cyclotomic units, of a real abelian number ﬁeld and the κ( L see [29]. For other cases involving ideal class groups and Selmer groups of elliptic curves, see [19], [31], [30], [13].\n\n(cid:1)\n\n5.4. Wiles’ geometric Euler system. The task now is to construct an Euler system of cohomology classes with which to bound #(SD(Vn)) using Kolyvagin’s method. This is the most technically diﬃcult part of Wiles’ proof and is the part of Wiles’ work he referred to as not yet complete in his December announcement. We give only general remarks about Wiles’ construction.\n\nThe ﬁrst step in the construction is due to Flach [10]. He constructed classes consisting of just one prime. This allows one to bound the ) L\n\nS∗\n\nκ( exponent of SD(Vn), but not its order.\n\nL for sets\n\n∈\n\nL\n\nEvery Euler system starts with some explicit, concrete objects. Earlier examples of Euler systems come from cyclotomic or elliptic units, Gauss sums, or Heegner points on elliptic curves. Wiles (following Flach) constructs his cohomology classes from modular units, i.e., meromorphic functions on modular curves which are holo- morphic and nonzero away from the cusps. More precisely, κ( ) comes from an explicit function on the modular curve X1(L, N ), the curve obtained by taking the quotient space of the upper half plane by the action of the group\n\nL\n\na b c d\n\nSL2(Z) : c\n\n1 (mod L) } ≡ ℓ∈L ℓ and where N is the N of (T2) of\n\n0\n\n(mod LN ),\n\na\n\nd\n\n,\n\nΓ1(L, N ) =\n\n∈\n\n≡\n\n≡\n\n{ (cid:1) (cid:0) and adjoining the cusps, where L = The construction and study of the classes κ( [8], [9] and others.\n\n4.2. ) rely heavily on results


[8], [9] and others.\n\n4.2. ) rely heavily on results of Faltings\n\n§\n\nL\n\nQ\n\n5.5. Wiles’ results. Wiles announced two main results (Theorems 5.3 and 5.4 below) in the direction of Mazur’s conjecture, under two diﬀerent sets of hypotheses on the representation ¯ρ. Theorem 5.3 implies the Semistable Taniyama-Shimura Conjecture and Fermat’s Last Theorem. Wiles’ proof of Theorem 5.3 depends on the not-yet-complete construction of an appropriate Euler system (as in 5.4), while his proof of Theorem 5.4 (though not yet fully checked) does not. For Theorem 5.4, Wiles bounds the Selmer group of 5.2 without constructing a new Euler system, by using results from the Iwasawa theory of imaginary quadratic ﬁelds. (These results in turn rely on Kolyvagin’s method and the Euler system of elliptic units; see [31].)\n\n§\n\n§\n\nSince for ease of exposition we deﬁned modularity of representations in terms of Γ0(N ) instead of Γ1(N ), the theorems stated below are weaker than those an- nounced by Wiles, but have the same applications to elliptic curves. (Note that by our deﬁnition of type-\n\n, if ¯ρ is type-\n\n, then det(¯ρ) = ¯εp.)\n\nD\n\nD\n\nA REPORT ON WILES’ CAMBRIDGE LECTURES\n\n19\n\nIf ¯ρ is a representation of GQ on a vector space V , Sym2(¯ρ) denotes the repre-\n\nsentation on the symmetric square of V induced by ¯ρ.\n\nTheorem 5.3 (Wiles). Suppose p, k, the following additional conditions :\n\n, ¯ρ, and\n\nare as in\n\n4.2 and ¯ρ satisﬁes\n\nD\n\nO\n\n§\n\n(i) Sym2(¯ρ) is absolutely irreducible, (ii) if ¯ρ is


Sym2(¯ρ) is absolutely irreducible, (ii) if ¯ρ is ramiﬁed at q and q (iii) if p is 3 or 5, then for some prime q, p divides #(¯ρ(Iq)).\n\n= p, then the restriction of ¯ρ to Dq is reducible,\n\n6\n\nT is an isomorphism.\n\nThen ϕ : R\n\n→\n\nSince Theorem 5.3 does not yield the full Mazur Conjecture (Conjecture 4.2) for 2 to see which elliptic curves §\n\np = 3 and 5, we need to reexamine the arguments of E can be proved modular using Theorem 5.3 applied to ¯ρE,3 and ¯ρE,5.\n\nHypothesis (i) of Theorem 5.3 will be satisﬁed if the image of ¯ρE,p is suﬃciently large in GL2(Fp) (for example, if ¯ρE,p is surjective). For p = 3 and p = 5, if ¯ρE,p satisﬁes hypothesis (iii) and is irreducible, then it satisﬁes hypothesis (i).\n\nIf E is semistable, p is an odd prime, and ¯ρE,p is irreducible and modular, then (see the proof of Proposition 3.3) and ¯ρE,p satisﬁes (ii) ¯ρE,p is D 14 of Appendix C of [39]). Therefore by Propositions and (iii) (use Tate curves; see 4.3 and 3.3, Theorem 5.3 implies that the Semistable Modular Lifting Conjecture (Conjecture 2.1) holds for p = 3 and for p = 5. As shown in 2, the Semistable Taniyama-Shimura Conjecture and Fermat’s Last Theorem follow.\n\nmodular for some\n\nD\n\n§\n\n§\n\nTheorem 5.4 (Wiles). Suppose p, k, contains no nontrivial p-th roots of unity. Suppose also that there are an imaginary quadratic


contains no nontrivial p-th roots of unity. Suppose also that there are an imaginary quadratic ﬁeld F of discriminant prime to p and a character χ : Gal( ¯Q/F ) × such that T is the induced representation Indχ of GQ is a ( an isomorphism.\n\n, ¯ρ, and\n\nare as in\n\n4.2 and\n\nD\n\nO\n\n§\n\nO\n\n→ O\n\n)-lifting of ¯ρ. Then ϕ : R\n\n,\n\nD\n\nO\n\n→\n\nCorollary 5.5 (Wiles). Suppose E is an elliptic curve over Q with complex mul- tiplication by an imaginary quadratic ﬁeld F and p is an odd prime at which E has good reduction. If E′ is an elliptic curve over Q satisfying\n\nE′ has good reduction at p and ¯ρE′,p is isomorphic to ¯ρE,p,\n\n•\n\nthen E′ is modular.\n\nProof of corollary. Let p be a prime of F containing p, and deﬁne = the ring of integers of the completion of F at p,\n\nO • • •\n\n/p primes at which E or E′ has bad reduction\n\nk = Σ = t = ordinary if E has ordinary reduction at p, t = ﬂat if E has supersingular reduction at p,\n\n,\n\nO {\n\nO\n\np\n\n,\n\n} ∪ {\n\n}\n\n= (Σ, t).\n\nD\n\nLet\n\nχ : Gal( ¯Q/F )\n\nAutO(E[p∞]) ∼=\n\n×\n\n→\n\nO\n\nbe the character giving the action of Gal( ¯Q/F ) on E[p∞] (where E[p∞] is the group of points of E killed by the endomorphisms of E which lie in some power of p). It is not hard to see that ρE,p ⊗ O\n\nis isomorphic to Indχ.\n\n20\n\nK. RUBIN AND A. SILVERBERG\n\nSince E has complex multiplication, it is well known that E and


SILVERBERG\n\nSince E has complex multiplication, it is well known that E and ¯ρE,p are mod- ular. Since E has good reduction at p, it can be shown that the discriminant of contains no nontrivial p-th roots of unity. One can show F is prime to p and that all of the hypotheses of Theorem 5.4 are satisﬁed with ¯ρ = ¯ρE,p ⊗ k. By our assumptions on E′, ρE′,p ⊗ O )-lifting of ¯ρ, and we conclude (using the D same reasoning as in the proofs of Propositions 3.3 and 4.3) that ρE′,p is modular and hence E′ is modular.\n\nO\n\nis a (\n\n,\n\nO\n\nRemarks. (i) The elliptic curves E′ of Corollary 5.5 are not semistable.\n\n(ii) Suppose E and p are as in Corollary 5.5 and p = 3 or 5. As in Appendix B.2 one can show that the elliptic curves E′ over Q with good reduction at p and with ¯ρE′,p isomorphic to ¯ρE,p give inﬁnitely many C-isomorphism classes.\n\nExample. Take E to be the elliptic curve deﬁned by\n\ny2 = x3\n\nx2\n\n3x\n\n1.\n\n−\n\n−\n\n−\n\nThen E has complex multiplication by Q(√ Deﬁne polynomials\n\n2), and E has good reduction at 3.\n\n−\n\n1512t3 3, a4(t) = a6(t) = 40824t6 + 31104t5 + 8370t4 + 504t3\n\n2430t4\n\n396t2\n\n56t\n\n−\n\n−\n\n−\n\n−\n\n−\n\n148t2\n\n24t\n\n1,\n\n−\n\n−\n\n−\n\nQ let Et be the elliptic curve\n\nand for each t\n\n∈\n\ny2 =


the elliptic curve\n\nand for each t\n\n∈\n\ny2 = x3\n\nx2 + a4(t)x + a6(t)\n\n−\n\nQ, ¯ρEt,3 is isomorphic to (note that E0 = E). It can be shown that for every t 0 or 1 (mod 3) (or more generally if t = 3a/b or t = 3a/b + 1 ¯ρE,3. If t with a and b integers and b not divisible by 3), then Et has good reduction at 3, for instance because the discriminant of Et is\n\n∈\n\nZ and t\n\n∈\n\n≡\n\n29(27t2 + 10t + 1)3(27t2 + 18t + 1)3.\n\nThus for these values of t, Corollary 5.5 shows that Et is modular and so is any elliptic curve over Q isomorphic over C to Et, i.e., any elliptic curve over Q with j-invariant equal to\n\n3\n\n4(27t2 + 6t + 1)(135t2 + 54t + 5) (27t2 + 10t + 1)(27t2 + 18t + 1)\n\n.\n\n(cid:18)\n\n(cid:19)\n\nThis explicitly gives inﬁnitely many modular elliptic curves over Q which are\n\nnonisomorphic over C.\n\n(For deﬁnitions of complex multiplication, discriminant, and j-invariant, see any\n\nstandard reference on elliptic curves, such as [39].)\n\nA REPORT ON WILES’ CAMBRIDGE LECTURES\n\n21\n\nAppendix A. Galois groups and Frobenius elements\n\nWrite GQ = Gal( ¯Q/Q). If q is a prime number and\n\nis a prime ideal dividing\n\nQ\n\nq in the ring of integers of ¯Q, there is a ﬁltration\n\nGQ\n\nDQ ⊃\n\nIQ\n\n⊃ where the decomposition group DQ and the inertia group IQ are deﬁned by\n\nDQ = IQ


group DQ and the inertia group IQ are deﬁned by\n\nDQ = IQ =\n\nσ\n\nGQ : σ ∈ Q ∈ DQ : σx\n\n=\n\n,\n\n{\n\nQ} x (mod\n\nσ\n\n) for all algebraic integers x }\n\n.\n\n≡ { There are natural identiﬁcations\n\nQ\n\nDQ/IQ ∼= Gal( ¯Fq/Fq),\n\nDQ ∼= Gal( ¯Qq/Qq),\n\nxq of GQ\n\nand FrobQ ∈ Gal( ¯Fq/Fq). If and\n\nDQ/IQ denotes the inverse image of the canonical generator x\n\n7→ for some σ\n\n′ is another prime ideal above q, then\n\n′ = σ\n\nQ DQ′ = σDQσ−1,\n\nQ\n\nQ\n\n∈\n\nFrobQ′ = σFrobQσ−1.\n\nIQ′ = σIQσ−1,\n\nSince we will care about these objects only up to conjugation, we will write Dq and GQ for any representative of a FrobQ. If ρ is a represen- Iq. We will write Frobq ∈ tation of GQ which is unramiﬁed at q, then trace(ρ(Frobq)) and det(ρ(Frobq)) are well deﬁned independent of any choices.\n\nAppendix B. Some details on the proof of Proposition 2.4\n\nB.1. The modular curve X0(15) can be viewed as a curve deﬁned over Q in such a way that the noncusp rational points correspond to isomorphism classes (over C) E( ¯Q) is a subgroup of pairs (E′, 42), of order 15 stable under GQ. An equation for X0(15) is y2 = x(x + 32)(x the elliptic curve discussed in 1. There are eight rational points on X0(15), four of § which are cusps. There are four modular elliptic curves, corresponding to a modular form for Γ0(50) (see p. 86 of [1]), which


a modular form for Γ0(50) (see p. 86 of [1]), which lie in the four distinct C-isomorphism classes that correspond to the noncusp rational points on X0(15).\n\n) where E′ is an elliptic curve over Q and\n\nC\n\nC ⊂\n\n−\n\nTherefore every elliptic curve over Q with a GQ-stable subgroup of order 15 is modular. Equivalently, if E is an elliptic curve over Q and both ¯ρE,3 and ¯ρE,5 are reducible, then E is modular.\n\nB.2. Fix a semistable elliptic curve E over Q. We will show that there are inﬁnitely many semistable elliptic curves E′ over Q such that\n\n(i) ¯ρE′,5 is isomorphic to ¯ρE,5, and (ii) ¯ρE′,3 is irreducible. Let\n\n1 0 0 1\n\na b c d\n\na b c d\n\nSL2(Z) :\n\n(mod 5) }\n\n.\n\nΓ(5) =\n\n≡\n\n∈\n\n{\n\nLet X be the twist of the classical modular curve X(5) (see [35]) by the cocycle (cid:0) induced by ¯ρE,5, and let S be the set of cusps of X. Then X is a curve deﬁned over Q which has the following properties. The rational points on X − (E′, φ) where E′ is an elliptic curve over Q and φ : E[5] module isomorphism.\n\n(cid:1)\n\n(cid:0)\n\n(cid:1)\n\n(cid:1)\n\n(cid:0)\n\nS correspond to isomorphism classes of pairs E′[5] is a GQ-\n\n\n\n→\n\n22\n\nK. RUBIN AND A. SILVERBERG\n\nS is four copies of H/Γ(5), so each component of\n\nAs a complex manifold X X has genus zero.\n\n\n\n−\n\nLet X 0 be the component of X containing the rational point


X 0 be the component of X containing the rational point corresponding to (E, identity). Then X 0 is a curve of genus zero deﬁned over Q with a rational point, so it has inﬁnitely many rational points. We want to show that inﬁnitely many of these points correspond to semistable elliptic curves E′ with ¯ρE′,3 irreducible.\n\nThere is another modular curve ˆX deﬁned over Q, with a ﬁnite set ˆS of cusps,\n\nwhich has the following properties. The rational points on ˆX (E′, φ, module isomorphism, and As a complex manifold ˆX The map that forgets the subgroup X deﬁned over Q and of degree [Γ(5) : Γ(5)\n\nˆS correspond to isomorphism classes of triples E′[5] is a GQ-\n\n\n\n−\n\n) where E′ is an elliptic curve over Q, φ : E[5]\n\nC\n\n→\n\nE′[3] is a GQ-stable subgroup of order 3.\n\nC ⊂ −\n\nˆS is four copies of H/(Γ(5)\n\nΓ0(3)).\n\n•\n\n∩ induces a surjective morphism θ : ˆX\n\nC\n\n→\n\nΓ0(3)] = 4.\n\n∩\n\nLet ˆX 0 be the component of ˆX which maps to X 0. The function ﬁeld of X 0 is Q(t), and the function ﬁeld of ˆX 0 is Q(t)[x]/f (t, x) where f (t, x) Q(t)[x] is irreducible and has degree 4 in x. If t′ Q is suﬃciently close 5-adically to the value of t which corresponds to E, then the corresponding elliptic curve is semistable at Q so that f (t1, x) is 5. By the Hilbert Irreducibility Theorem we can ﬁnd a t1 ∈ irreducible in Q[x]. It is possible to


we can ﬁnd a t1 ∈ irreducible in Q[x]. It is possible to ﬁx a prime ℓ = 5 such that f (t1, x) has no roots modulo ℓ. If t′ Q is suﬃciently close ℓ-adically to t1, then f (t′, x) has no rational roots, and thus t′ corresponds to a rational point of X 0 which is not the image of a rational point of ˆX 0. Therefore there are inﬁnitely many elliptic curves E′ over Q which are semistable at 5 and satisfy\n\n∈\n\n∈\n\n6\n\n∈\n\n(i) E′[5] ∼= E[5] as GQ-modules, and (ii) E′[3] has no subgroup of order 3 stable under GQ.\n\nIt follows from (i) and the semistability of E that E′ is semistable at all primes = 5, and thus E′ is semistable. We therefore have inﬁnitely many semistable q elliptic curves E′ which satisfy the desired conditions.\n\n6\n\nAppendix C. Representation types\n\nSuppose A is a complete noetherian local Zp-algebra and ρ : GQ\n\nGL2(A) is a |Dp for the restriction of ρ to the decomposition group Dp.\n\n→\n\nrepresentation. Write ρ We say ρ is\n\nordinary at p if ρ\n\n|Dp is (after a change of basis, if necessary) of the form ﬂat at p if ρ is not ordinary, and for every ideal a of ﬁnite index in A, the (cid:0) |Dp modulo a is the representation associated to the ¯Qp-points reduction of ρ of a ﬁnite ﬂat group scheme over Zp.\n\n\n\n∗ ∗ 0 χ\n\nwhere χ is unramiﬁed and the * are functions from Dp to A;\n\n(cid:1)\n\n\n\nAppendix D. Selmer groups\n\nWith notation as in\n\n5 (see especially §\n\n5.2),


notation as in\n\n5 (see especially §\n\n5.2), deﬁne\n\n§\n\n[ǫ]/(ǫ2, mn)\n\nOn =\n\nO\n\nA REPORT ON WILES’ CAMBRIDGE LECTURES\n\n23\n\nwhere ǫ is an indeterminate. Then v\n\n1 + ǫv deﬁnes an isomorphism\n\n7→ On) : δ GL2(\n\n∼ ∈ → { HomO(pR/p2 R,\n\n(16)\n\n1 (mod ǫ) } /mn) there is a unique -algebra homomorphism → On whose restriction to pR is ǫα. Composing with the representation ρR On. (In particular ρ0 )-lifting obtained when α = 0.) Deﬁne a one-cocycle cα on GQ\n\nδ\n\n.\n\nVn\n\n≡\n\nFor every α\n\nO\n\nO\n\n∈\n\nψα : R of Theorem 4.1 gives a ( denotes the ( by\n\n,\n\n)-lifting ρα = ψα ◦\n\nρR of ¯ρ to\n\nD\n\nO\n\n,\n\nD\n\nO\n\ncα(g) = ρα(g)ρ0(g)−1.\n\nH 1(Q, Vn). This deﬁnes a\n\nSince ρα ≡ homomorphism\n\nρ0 (mod ǫ), using (16) we can view cα ∈\n\ns : HomO(pR/p2 R,\n\n/mn)\n\nH 1(Q, Vn),\n\nO and it is not diﬃcult to see that s is injective. The fact that ρ0 and ρα are type- D gives information about the restrictions resq(cα) for various primes q, and using this H 1(Q, Vn) and veriﬁes that s information Wiles deﬁnes a Selmer group SD(Vn) is an isomorphism onto SD(Vn).\n\n→\n\n⊂\n\nReferences\n\n[1] B.


B. Birch and W. Kuyk, eds., Modular functions of one variable. IV, Lecture Notes in Math.,\n\nvol. 476, Springer-Verlag, New York, 1975, pp. 74–144.\n\n[2] J. Buhler, R. Crandall, R. Ernvall, and T. Mets¨ankyl¨a, Irregular primes and cyclotomic\n\ninvariants to four million, Math. Comp. 61 (1993), 151–153.\n\n[3] J. W. S. Cassels and A. Frohlich, Algebraic number theory, Academic Press, London, 1967. [4] P. Deligne and J.-P. Serre, Formes modulaires de poids 1, Ann. Sci. ´Ecole Norm. Sup. (4) 7\n\n(1974), 507–530.\n\n[5] L. E. Dickson, History of the theory of numbers (Vol. II), Chelsea Publ. Co., New York, 1971. [6] H. M. Edwards, Fermat’s Last Theorem. A genetic introduction to algebraic number theory,\n\nSpringer-Verlag, New York, 1977.\n\n[7] M. Eichler, Quatern¨are quadratische Formen und die Riemannsche Vermutung f¨ur die Kon-\n\ngruenzzetafunktion, Arch. Math. (Basel) 5 (1954), 355–366.\n\n[8] G. Faltings, p-adic Hodge theory, J. Amer. Math. Soc. 1 (1988), 255–299. [9]\n\n, Crystalline cohomology and p-adic Galois representations, Algebraic Analysis, Ge- ometry and Number Theory, Proceedings of the JAMI Inaugural Conference (J. I. Igusa, ed.), Johns Hopkins Univ. Press, Baltimore, MD, 1989, pp. 25–80.\n\n[10] M. Flach, A ﬁniteness theorem for the symmetric square of an elliptic curve, Invent. Math.\n\n109 (1992), 307–327.\n\n[11]


Math.\n\n109 (1992), 307–327.\n\n[11] G. Frey, Links between solutions of A − B = C and elliptic curves, Number Theory, Ulm 1987, Proceedings, Lecture Notes in Math., vol. 1380, Springer-Verlag, New York, 1989, pp. 31–62.\n\n[12] S. Gelbart, Automorphic forms on adele groups, Ann. of Math. Stud., vol. 83, Princeton\n\nUniv. Press, Princeton, NJ, 1975.\n\n[13] B. Gross, Kolyvagin’s work on modular elliptic curves, L-functions and Arithmetic, London Math. Soc. Lecture Note Ser., vol. 153, Cambridge Univ. Press, Cambridge, 1991, pp. 235–256. [14] G. H. Hardy and E. M. Wright, An introduction to the theory of numbers, Fourth ed., Oxford\n\nUniv. Press, London, 1971.\n\n[15] Y. Hellegouarch, ´Etude des points d’ordre ﬁni des vari´et´es de dimension un d´eﬁnies sur un\n\nanneau principal, J. Reine Angew. Math. 244 (1970), 20–36.\n\n, Points d’ordre ﬁni des vari´et´es ab´eliennes de dimension un, Colloque de Th´eorie des Nombres (Univ. Bordeaux, Bordeaux, 1969), Bull. Soc. Math. France, M´em. 25, Soc. Math. France, Paris, 1971, pp. 107–112.\n\n[16]\n\n, Points d’ordre ﬁni sur les courbes elliptiques, C. R. Acad. Sci. Paris S´er. A-B 273\n\n[17]\n\n(1971), A540–A543.\n\n24\n\nK. RUBIN AND A. SILVERBERG\n\n, Points d’ordre 2ph sur les courbes elliptiques, Acta. Arith. 26 (1974/75), 253–263. [18] [19] V. A.


253–263. [18] [19] V. A. Kolyvagin, Euler systems, The Grothendieck Festschrift (Vol. II) (P. Cartier et al.,\n\neds.), Birkh¨auser, Boston, 1990, pp. 435–483.\n\n[20] R. Langlands, Base change for GL(2), Ann. of Math. Stud., vol. 96, Princeton Univ. Press,\n\nPrinceton, NJ, 1980.\n\n[21] B. Mazur, Deforming Galois representations, Galois groups over Q (Y. Ihara, K. Ribet, and J.-P. Serre, eds.), Math. Sci. Res. Inst. Publ., vol. 16, Springer-Verlag, New York, 1989, pp. 385–437.\n\n, Number theory as gadﬂy, Amer. Math. Monthly 98 (1991), 593–610.\n\n[22] [23] B. Mazur and J. Tilouine, Repr´esentations galoisiennes, diﬀ´erentielles de K¨ahler et “conjec-\n\ntures principales”, Inst. Hautes ´Etudes Sci. Publ. Math. 71 (1990), 65–103.\n\n[24] J. Oesterl´e, Nouvelles approches du “th´eor`eme” de Fermat, S´eminaire Bourbaki no. 694\n\n(1987–1988), Ast´erisque 161/162 (1988) 165–186.\n\n, On a variation of Mazur ’s deformation functor, Compositio Math. 87 (1993), 269–\n\n[25]\n\n286.\n\n[26] P. Ribenboim, 13 lectures on Fermat ’s Last Theorem, Springer-Verlag, New York, 1979. [27] K. Ribet, On modular representations of Gal( ¯Q/Q) arising from modular forms, Invent.\n\nMath. 100 (1990), 431–476.\n\n, Report on mod ℓ representations of Gal(


431–476.\n\n, Report on mod ℓ representations of Gal( ¯Q/Q), Motives (U. Jannsen, S. Kleiman, and J-P. Serre, eds.), Proc. Sympos. Pure Math., vol. 55 (Part 2), Amer. Math. Soc., Providence, RI, 1994 (to appear).\n\n[28]\n\n[29] K. Rubin, The main conjecture. (Appendix to Cyclotomic ﬁelds I and II, S. Lang), Graduate\n\nTexts in Math., vol. 121, Springer-Verlag, New York, 1990, pp. 397–419.\n\n, Kolyvagin’s system of Gauss sums, Arithmetic Algebraic Geometry (G. van der Geer, F. Oort, and J. Steenbrink, eds.), Progr. Math., vol. 89, Birkh¨auser, Boston, 1991, pp. 309–324.\n\n[30]\n\n, The “main conjectures” of Iwasawa theory for imaginary quadratic ﬁelds, Invent.\n\n[31]\n\nMath. 103 (1991), 25–68.\n\n[32] J.-P. Serre, Sur les repr´esentations modulaires de degr´e 2 de Gal( ¯Q/Q), Duke Math. J. 54\n\n(1987), 179–230.\n\n[33] G. Shimura, Correspondances modulaires et les fonctions ζ de courbes alg´ebriques, J. Math.\n\nSoc. Japan 10 (1958), 1–28.\n\n, Construction of class ﬁelds and zeta functions of algebraic curves, Ann. of Math.\n\n[34]\n\n85 (1967), 58–159.\n\n, Introduction to the arithmetic theory of automorphic functions, Princeton Univ.\n\n[35]\n\nPress, Princeton, NJ, 1971.\n\n, On elliptic curves with complex multiplication as factors of the Jacobians of modular\n\n[36]\n\nfunction ﬁelds, Nagoya Math. J. 43 (1971), 199–208.\n\n,


Nagoya Math. J. 43 (1971), 199–208.\n\n, On the factors of the jacobian variety of a modular function ﬁeld, J. Math. Soc.\n\n[37]\n\nJapan 25 (1973), 523–544.\n\n, Yutaka Taniyama and his time. Very personal recollections, Bull. London Math.\n\n[38]\n\nSoc. 21 (1989), 186–196.\n\n[39] J. Silverman, The arithmetic of elliptic curves, Graduate Texts in Math., vol. 106, Springer-\n\nVerlag, New York, 1986.\n\n[40] F. Thaine, On the ideal class groups of real abelian number ﬁelds, Ann. of Math. (2) 128\n\n(1988), 1–18.\n\n[41] J. Tunnell, Artin’s conjecture for representations of octahedral type, Bull. Amer. Math. Soc.\n\n(N.S.) 5 (1981), 173–175.\n\n[42] A. Weil, ¨Uber die Bestimmung Dirichletscher Reihen durch Funktionalgleichungen, Math.\n\nAnn. 168 (1967), 149–156.\n\nDepartment of Mathematics, Ohio State University, Columbus, Ohio 43210 E-mail address: rubin@math.ohio-state.edu\n\nDepartment of Mathematics, Ohio State University, Columbus, Ohio 43210 E-mail address: silver@math.ohio-state.edu' metadata={'source': '/var/folders/l1/lphj87z16c3282pjwy91wtm80000gn/T/tmpdh5kk5yb/tmp.pdf'}


page_content='This is text file' metadata={'source': 'dropbox:///test.txt', 'title': 'test.txt'}


This notebook covers how to load documents from Docugami. It provides the advantages of using this system over alternative data loaders.
Appropriate chunking of your documents is critical for retrieval from documents. Many chunking techniques exist, including simple ones that rely on whitespace and recursive chunk splitting based on character length. Docugami offers a different approach:
If the DOCUGAMI_API_KEY environment variable is set, there is no need to pass it in to the loader explicitly otherwise you can pass it in as the access_token parameter.
[Document(page_content='MUTUAL NON-DISCLOSURE AGREEMENT This Mutual Non-Disclosure Agreement (this “ Agreement ”) is entered into and made effective as of April 4 , 2018 between Docugami Inc. , a Delaware corporation , whose address is 150 Lake Street South , Suite 221 , Kirkland , Washington 98033 , and Caleb Divine , an individual, whose address is 1201 Rt 300 , Newburgh NY 12550 .', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:ThisMutualNon-disclosureAgreement', 'id': '43rj0ds7s0ur', 'source': 'NDA simple layout.docx', 'structure': 'p', 'tag': 'ThisMutualNon-disclosureAgreement'}),
Document(page_content='The above named parties desire to engage in discussions regarding a potential agreement or other transaction between the parties (the “Purpose”). In connection with such discussions, it may be necessary for the parties to disclose to each other certain confidential information or materials to enable them to evaluate whether to enter into such agreement or transaction.', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Discussions', 'id': '43rj0ds7s0ur', 'source': 'NDA simple layout.docx', 'structure': 'p', 'tag': 'Discussions'}),


Document(page_content='In consideration of the foregoing, the parties agree as follows:', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Consideration', 'id': '43rj0ds7s0ur', 'source': 'NDA simple layout.docx', 'structure': 'p', 'tag': 'Consideration'}),
Document(page_content='1. Confidential Information . For purposes of this Agreement , “ Confidential Information ” means any information or materials disclosed by one party to the other party that: (i) if disclosed in writing or in the form of tangible materials, is marked “confidential” or “proprietary” at the time of such disclosure; (ii) if disclosed orally or by visual presentation, is identified as “confidential” or “proprietary” at the time of such disclosure, and is summarized in a writing sent by the disclosing party to the receiving party within thirty ( 30 ) days after any such disclosure; or (iii) due to its nature or the circumstances of its disclosure, a person exercising reasonable business judgment would understand to be confidential or proprietary.', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:Purposes/docset:ConfidentialInformation-section/docset:ConfidentialInformation[2]', 'id': '43rj0ds7s0ur', 'source': 'NDA simple layout.docx', 'structure': 'div', 'tag': 'ConfidentialInformation'}),


Document(page_content="2. Obligations and Restrictions . Each party agrees: (i) to maintain the other party's Confidential Information in strict confidence; (ii) not to disclose such Confidential Information to any third party; and (iii) not to use such Confidential Information for any purpose except for the Purpose. Each party may disclose the other party’s Confidential Information to its employees and consultants who have a bona fide need to know such Confidential Information for the Purpose, but solely to the extent necessary to pursue the Purpose and for no other purpose; provided, that each such employee and consultant first executes a written agreement (or is otherwise already bound by a written agreement) that contains use and nondisclosure restrictions at least as protective of the other party’s Confidential Information as those set forth in this Agreement .", metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:Obligations/docset:ObligationsAndRestrictions-section/docset:ObligationsAndRestrictions', 'id': '43rj0ds7s0ur', 'source': 'NDA simple layout.docx', 'structure': 'div', 'tag': 'ObligationsAndRestrictions'}),
Document(page_content='3. Exceptions. The obligations and restrictions in Section 2 will not apply to any information or materials that:', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:Exceptions/docset:Exceptions-section/docset:Exceptions[2]', 'id': '43rj0ds7s0ur', 'source': 'NDA simple layout.docx', 'structure': 'div', 'tag': 'Exceptions'}),


Document(page_content='(i) were, at the date of disclosure, or have subsequently become, generally known or available to the public through no act or failure to act by the receiving party;', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:TheDate/docset:TheDate/docset:TheDate', 'id': '43rj0ds7s0ur', 'source': 'NDA simple layout.docx', 'structure': 'p', 'tag': 'TheDate'}),
Document(page_content='(ii) were rightfully known by the receiving party prior to receiving such information or materials from the disclosing party;', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:TheDate/docset:SuchInformation/docset:TheReceivingParty', 'id': '43rj0ds7s0ur', 'source': 'NDA simple layout.docx', 'structure': 'p', 'tag': 'TheReceivingParty'}),
Document(page_content='(iii) are rightfully acquired by the receiving party from a third party who has the right to disclose such information or materials without breach of any confidentiality obligation to the disclosing party;', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:TheDate/docset:TheReceivingParty/docset:TheReceivingParty', 'id': '43rj0ds7s0ur', 'source': 'NDA simple layout.docx', 'structure': 'p', 'tag': 'TheReceivingParty'}),


Document(page_content='4. Compelled Disclosure . Nothing in this Agreement will be deemed to restrict a party from disclosing the other party’s Confidential Information to the extent required by any order, subpoena, law, statute or regulation; provided, that the party required to make such a disclosure uses reasonable efforts to give the other party reasonable advance notice of such required disclosure in order to enable the other party to prevent or limit such disclosure.', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:Disclosure/docset:CompelledDisclosure-section/docset:CompelledDisclosure', 'id': '43rj0ds7s0ur', 'source': 'NDA simple layout.docx', 'structure': 'div', 'tag': 'CompelledDisclosure'}),
Document(page_content='5. Return of Confidential Information . Upon the completion or abandonment of the Purpose, and in any event upon the disclosing party’s request, the receiving party will promptly return to the disclosing party all tangible items and embodiments containing or consisting of the disclosing party’s Confidential Information and all copies thereof (including electronic copies), and any notes, analyses, compilations, studies, interpretations, memoranda or other documents (regardless of the form thereof) prepared by or on behalf of the receiving party that contain or are based upon the disclosing party’s Confidential Information .', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:TheCompletion/docset:ReturnofConfidentialInformation-section/docset:ReturnofConfidentialInformation', 'id': '43rj0ds7s0ur', 'source': 'NDA simple layout.docx', 'structure': 'div', 'tag': 'ReturnofConfidentialInformation'}),


Document(page_content='6. No Obligations . Each party retains the right to determine whether to disclose any Confidential Information to the other party.', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:NoObligations/docset:NoObligations-section/docset:NoObligations[2]', 'id': '43rj0ds7s0ur', 'source': 'NDA simple layout.docx', 'structure': 'div', 'tag': 'NoObligations'}),
Document(page_content='7. No Warranty. ALL CONFIDENTIAL INFORMATION IS PROVIDED BY THE DISCLOSING PARTY “AS IS ”.', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:NoWarranty/docset:NoWarranty-section/docset:NoWarranty[2]', 'id': '43rj0ds7s0ur', 'source': 'NDA simple layout.docx', 'structure': 'div', 'tag': 'NoWarranty'}),
Document(page_content='8. Term. This Agreement will remain in effect for a period of seven ( 7 ) years from the date of last disclosure of Confidential Information by either party, at which time it will terminate.', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:ThisAgreement/docset:Term-section/docset:Term', 'id': '43rj0ds7s0ur', 'source': 'NDA simple layout.docx', 'structure': 'div', 'tag': 'Term'}),


Document(page_content='9. Equitable Relief . Each party acknowledges that the unauthorized use or disclosure of the disclosing party’s Confidential Information may cause the disclosing party to incur irreparable harm and significant damages, the degree of which may be difficult to ascertain. Accordingly, each party agrees that the disclosing party will have the right to seek immediate equitable relief to enjoin any unauthorized use or disclosure of its Confidential Information , in addition to any other rights and remedies that it may have at law or otherwise.', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:EquitableRelief/docset:EquitableRelief-section/docset:EquitableRelief[2]', 'id': '43rj0ds7s0ur', 'source': 'NDA simple layout.docx', 'structure': 'div', 'tag': 'EquitableRelief'}),
Document(page_content='10. Non-compete. To the maximum extent permitted by applicable law, during the Term of this Agreement and for a period of one ( 1 ) year thereafter, Caleb Divine may not market software products or do business that directly or indirectly competes with Docugami software products .', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:TheMaximumExtent/docset:Non-compete-section/docset:Non-compete', 'id': '43rj0ds7s0ur', 'source': 'NDA simple layout.docx', 'structure': 'div', 'tag': 'Non-compete'}),


Document(page_content='11. Miscellaneous. This Agreement will be governed and construed in accordance with the laws of the State of Washington , excluding its body of law controlling conflict of laws. This Agreement is the complete and exclusive understanding and agreement between the parties regarding the subject matter of this Agreement and supersedes all prior agreements, understandings and communications, oral or written, between the parties regarding the subject matter of this Agreement . If any provision of this Agreement is held invalid or unenforceable by a court of competent jurisdiction, that provision of this Agreement will be enforced to the maximum extent permissible and the other provisions of this Agreement will remain in full force and effect. Neither party may assign this Agreement , in whole or in part, by operation of law or otherwise, without the other party’s prior written consent, and any attempted assignment without such consent will be void. This Agreement may be executed in counterparts, each of which will be deemed an original, but all of which together will constitute one and the same instrument.', metadata={'xpath': '/docset:MutualNon-disclosure/docset:MutualNon-disclosure/docset:MUTUALNON-DISCLOSUREAGREEMENT-section/docset:MUTUALNON-DISCLOSUREAGREEMENT/docset:Consideration/docset:Purposes/docset:Accordance/docset:Miscellaneous-section/docset:Miscellaneous', 'id': '43rj0ds7s0ur', 'source': 'NDA simple layout.docx', 'structure': 'div', 'tag': 'Miscellaneous'}),
Document(page_content='[SIGNATURE PAGE FOLLOWS] IN WITNESS WHEREOF, the parties hereto have executed this Mutual Non-Disclosure Agreement by their duly authorized officers or representatives as of the date first set forth above.', metadata={'xpath': '/docset:MutualNon-disclosure/docset:Witness/docset:TheParties/docset:TheParties', 'id': '43rj0ds7s0ur', 'source': 'NDA simple layout.docx', 'structure': 'p', 'tag': 'TheParties'}),


Document(page_content='DOCUGAMI INC . : \n\n Caleb Divine : \n\n Signature: Signature: Name: \n\n Jean Paoli Name: Title: \n\n CEO Title:', metadata={'xpath': '/docset:MutualNon-disclosure/docset:Witness/docset:TheParties/docset:DocugamiInc/docset:DocugamiInc/xhtml:table', 'id': '43rj0ds7s0ur', 'source': 'NDA simple layout.docx', 'structure': '', 'tag': 'table'})]
The metadata for each Document (really, a chunk of an actual PDF, DOC or DOCX) contains some useful additional information:
You can use the Docugami Loader like a standard loader for Document QA over multiple docs, albeit with much better chunks that follow the natural contours of the document. There are many great tutorials on how to do this, e.g. this one. We can just use the same code, but use the DocugamiLoader for better chunking, instead of loading text or PDF files directly with basic splitting techniques.
The documents returned by the loader are already split, so we don't need to use a text splitter. Optionally, we can use the metadata on each document, for example the structure or tag attributes, to do any post-processing we want.
We will just use the output of the DocugamiLoader as-is to set up a retrieval QA chain the usual way.
{'query': 'What can tenants do with signage on their properties?',
'result': " Tenants can place or attach signs (digital or otherwise) to their premises with written permission from the landlord. The signs must conform to all applicable laws, ordinances, etc. governing the same. Tenants can also have their name listed in the building's directory at the landlord's cost.",


'source_documents': [Document(page_content='ARTICLE VI SIGNAGE 6.01 Signage . Tenant may place or attach to the Premises signs (digital or otherwise) or other such identification as needed after receiving written permission from the Landlord , which permission shall not be unreasonably withheld. Any damage caused to the Premises by the Tenant ’s erecting or removing such signs shall be repaired promptly by the Tenant at the Tenant ’s expense . Any signs or other form of identification allowed must conform to all applicable laws, ordinances, etc. governing the same. Tenant also agrees to have any window or glass identification completely removed and cleaned at its expense promptly upon vacating the Premises.', metadata={'Landlord': 'BUBBA CENTER PARTNERSHIP', 'Lease Date': 'April 24 \n\n ,', 'Lease Parties': 'This OFFICE LEASE AGREEMENT (this "Lease") is made and entered into by and between BUBBA CENTER PARTNERSHIP (" Landlord "), and Truetone Lane LLC , a Delaware limited liability company (" Tenant ").', 'Tenant': 'Truetone Lane LLC', 'id': 'v1bvgaozfkak', 'source': 'TruTone Lane 2.docx', 'structure': 'div', 'tag': '_601Signage', 'xpath': '/docset:OFFICELEASEAGREEMENT-section/docset:OFFICELEASEAGREEMENT/docset:Article/docset:ARTICLEVISIGNAGE-section/docset:_601Signage-section/docset:_601Signage'}),


Document(page_content='Signage. Tenant may place or attach to the Premises signs (digital or otherwise) or other such identification as needed after receiving written permission from the Landlord , which permission shall not be unreasonably withheld. Any damage caused to the Premises by the Tenant ’s erecting or removing such signs shall be repaired promptly by the Tenant at the Tenant ’s expense . Any signs or other form of identification allowed must conform to all applicable laws, ordinances, etc. governing the same. Tenant also agrees to have any window or glass identification completely removed and cleaned at its expense promptly upon vacating the Premises. \n\n ARTICLE VII UTILITIES 7.01', metadata={'Landlord': 'GLORY ROAD LLC', 'Lease Date': 'April 30 , 2020', 'Lease Parties': 'This OFFICE LEASE AGREEMENT (this "Lease") is made and entered into by and between GLORY ROAD LLC (" Landlord "), and Truetone Lane LLC , a Delaware limited liability company (" Tenant ").', 'Tenant': 'Truetone Lane LLC', 'id': 'g2fvhekmltza', 'source': 'TruTone Lane 6.pdf', 'structure': 'lim', 'tag': 'chunk', 'xpath': '/docset:OFFICELEASEAGREEMENT-section/docset:OFFICELEASEAGREEMENT/docset:Article/docset:ArticleIiiUse/docset:ARTICLEIIIUSEANDCAREOFPREMISES-section/docset:ARTICLEIIIUSEANDCAREOFPREMISES/docset:AnyTime/docset:Addition/dg:chunk'}),


Document(page_content='Landlord , its agents, servants, employees, licensees, invitees, and contractors during the last year of the term of this Lease at any and all times during regular business hours, after 24 hour notice to tenant, to pass and repass on and through the Premises, or such portion thereof as may be necessary, in order that they or any of them may gain access to the Premises for the purpose of showing the Premises to potential new tenants or real estate brokers. In addition, Landlord shall be entitled to place a "FOR RENT " or "FOR LEASE" sign (not exceeding 8.5 ” x 11 ”) in the front window of the Premises during the last six months of the term of this Lease .', metadata={'Landlord': 'BIRCH STREET , LLC', 'Lease Date': 'October 15 , 2021', 'Lease Parties': 'The provisions of this rider are hereby incorporated into and made a part of the Lease dated as of October 15 , 2021 between BIRCH STREET , LLC , having an address at c/o Birch Palace , 6 Grace Avenue Suite 200 , Great Neck , New York 11021 (" Landlord "), and Trutone Lane LLC , having an address at 4 Pearl Street , New York , New York 10012 (" Tenant ") of Premises known as the ground floor space and lower level space, as per floor plan annexed hereto and made a part hereof as Exhibit A (“Premises”) at 4 Pearl Street , New York , New York 10012 in the City of New York , Borough of Manhattan , to which this rider is annexed. If there is any conflict between the provisions of this rider and the remainder of this Lease , the provisions of this rider shall govern.', 'Tenant': 'Trutone Lane LLC', 'id': 'omvs4mysdk6b', 'source': 'TruTone Lane 1.docx', 'structure': 'p', 'tag': 'Landlord', 'xpath': '/docset:Rider/docset:RIDERTOLEASE-section/docset:RIDERTOLEASE/docset:FixedRent/docset:TermYearPeriod/docset:Lease/docset:_42FLandlordSAccess-section/docset:_42FLandlordSAccess/docset:LandlordsRights/docset:Landlord'}),


Document(page_content="24. SIGNS . No signage shall be placed by Tenant on any portion of the Project . However, Tenant shall be permitted to place a sign bearing its name in a location approved by Landlord near the entrance to the Premises (at Tenant's cost ) and will be furnished a single listing of its name in the Building's directory (at Landlord 's cost ), all in accordance with the criteria adopted from time to time by Landlord for the Project . Any changes or additional listings in the directory shall be furnished (subject to availability of space) for the then Building Standard charge .", metadata={'Landlord': 'Perry & Blair LLC', 'Lease Date': 'March 29th , 2019', 'Lease Parties': 'THIS OFFICE LEASE (the "Lease") is made and entered into as of March 29th , 2019 , by and between Landlord and Tenant . "Date of this Lease" shall mean the date on which the last one of the Landlord and Tenant has signed this Lease .', 'Tenant': 'Shorebucks LLC', 'id': 'dsyfhh4vpeyf', 'source': 'Shorebucks LLC_CO.pdf', 'structure': 'div', 'tag': 'SIGNS', 'xpath': '/docset:OFFICELEASE-section/docset:OFFICELEASE/docset:THISOFFICELEASE/docset:WITNESSETH-section/docset:WITNESSETH/docset:GrossRentCreditTheRentCredit-section/docset:GrossRentCreditTheRentCredit/docset:ThisLease-section/docset:ThisLease/docset:Guaranty-section/docset:Guaranty[2]/docset:TheTransfer/docset:TheTerms/docset:Indemnification/docset:INDEMNIFICATION-section/docset:INDEMNIFICATION/docset:Waiver/docset:Waiver/docset:Signs/docset:SIGNS-section/docset:SIGNS'})]}


One issue with large documents is that the correct answer to your question may depend on chunks that are far apart in the document. Typical chunking techniques, even with overlap, will struggle with providing the LLM sufficent context to answer such questions. With upcoming very large context LLMs, it may be possible to stuff a lot of tokens, perhaps even entire documents, inside the context but this will still hit limits at some point with very long documents, or a lot of documents.
For example, if we ask a more complex question that requires the LLM to draw on chunks from different parts of the document, even OpenAI's powerful LLM is unable to answer correctly.
At first glance the answer may seem reasonable, but if you review the source chunks carefully for this answer, you will see that the chunking of the document did not end up putting the Landlord name and the rentable area in the same context, since they are far apart in the document. The retriever therefore ends up finding unrelated chunks from other documents not even related to the DHA Group landlord. That landlord happens to be mentioned on the first page of the file Shorebucks LLC_NJ.pdf file, and while one of the source chunks used by the chain is indeed from that doc that contains the correct answer (13,500), other source chunks from different docs are included, and the answer is therefore incorrect.


[Document(page_content='1.1 Landlord . DHA Group , a Delaware limited liability company authorized to transact business in New Jersey .', metadata={'Landlord': 'DHA Group', 'Lease Date': 'March 29th , 2019', 'Lease Parties': 'THIS OFFICE LEASE (the "Lease") is made and entered into as of March 29th , 2019 , by and between Landlord and Tenant . "Date of this Lease" shall mean the date on which the last one of the Landlord and Tenant has signed this Lease .', 'Tenant': 'Shorebucks LLC', 'id': 'md8rieecquyv', 'source': 'Shorebucks LLC_NJ.pdf', 'structure': 'div', 'tag': 'DhaGroup', 'xpath': '/docset:OFFICELEASE-section/docset:OFFICELEASE/docset:THISOFFICELEASE/docset:WITNESSETH-section/docset:WITNESSETH/docset:TheTerms/dg:chunk/docset:BasicLeaseInformation/docset:BASICLEASEINFORMATIONANDDEFINEDTERMS-section/docset:BASICLEASEINFORMATIONANDDEFINEDTERMS/docset:DhaGroup/docset:DhaGroup/docset:DhaGroup/docset:Landlord-section/docset:DhaGroup'}),


Document(page_content='WITNESSES: LANDLORD: DHA Group , a Delaware limited liability company', metadata={'Landlord': 'DHA Group', 'Lease Date': 'March 29th , 2019', 'Lease Parties': 'THIS OFFICE LEASE (the "Lease") is made and entered into as of March 29th , 2019 , by and between Landlord and Tenant . "Date of this Lease" shall mean the date on which the last one of the Landlord and Tenant has signed this Lease .', 'Tenant': 'Shorebucks LLC', 'id': 'md8rieecquyv', 'source': 'Shorebucks LLC_NJ.pdf', 'structure': 'p', 'tag': 'DhaGroup', 'xpath': '/docset:OFFICELEASE-section/docset:OFFICELEASE/docset:THISOFFICELEASE/docset:WITNESSETH-section/docset:WITNESSETH/docset:GrossRentCreditTheRentCredit-section/docset:GrossRentCreditTheRentCredit/docset:Guaranty-section/docset:Guaranty[2]/docset:SIGNATURESONNEXTPAGE-section/docset:INWITNESSWHEREOF-section/docset:INWITNESSWHEREOF/docset:Behalf/docset:Witnesses/xhtml:table/xhtml:tbody/xhtml:tr[3]/xhtml:td[2]/docset:DhaGroup'}),


Document(page_content="1.16 Landlord 's Notice Address . DHA Group , Suite 1010 , 111 Bauer Dr , Oakland , New Jersey , 07436 , with a copy to the Building Management Office at the Project , Attention: On - Site Property Manager .", metadata={'Landlord': 'DHA Group', 'Lease Date': 'March 29th , 2019', 'Lease Parties': 'THIS OFFICE LEASE (the "Lease") is made and entered into as of March 29th , 2019 , by and between Landlord and Tenant . "Date of this Lease" shall mean the date on which the last one of the Landlord and Tenant has signed this Lease .', 'Tenant': 'Shorebucks LLC', 'id': 'md8rieecquyv', 'source': 'Shorebucks LLC_NJ.pdf', 'structure': 'div', 'tag': 'LandlordsNoticeAddress', 'xpath': '/docset:OFFICELEASE-section/docset:OFFICELEASE/docset:THISOFFICELEASE/docset:WITNESSETH-section/docset:WITNESSETH/docset:GrossRentCreditTheRentCredit-section/docset:GrossRentCreditTheRentCredit/docset:Period/docset:ApplicableSalesTax/docset:PercentageRent/docset:PercentageRent/docset:NoticeAddress[2]/docset:LandlordsNoticeAddress-section/docset:LandlordsNoticeAddress[2]'}),


Document(page_content='1.6 Rentable Area of the Premises. 9,753 square feet . This square footage figure includes an add-on factor for Common Areas in the Building and has been agreed upon by the parties as final and correct and is not subject to challenge or dispute by either party.', metadata={'Landlord': 'Perry & Blair LLC', 'Lease Date': 'March 29th , 2019', 'Lease Parties': 'THIS OFFICE LEASE (the "Lease") is made and entered into as of March 29th , 2019 , by and between Landlord and Tenant . "Date of this Lease" shall mean the date on which the last one of the Landlord and Tenant has signed this Lease .', 'Tenant': 'Shorebucks LLC', 'id': 'dsyfhh4vpeyf', 'source': 'Shorebucks LLC_CO.pdf', 'structure': 'div', 'tag': 'RentableAreaofthePremises', 'xpath': '/docset:OFFICELEASE-section/docset:OFFICELEASE/docset:THISOFFICELEASE/docset:WITNESSETH-section/docset:WITNESSETH/docset:TheTerms/dg:chunk/docset:BasicLeaseInformation/docset:BASICLEASEINFORMATIONANDDEFINEDTERMS-section/docset:BASICLEASEINFORMATIONANDDEFINEDTERMS/docset:PerryBlair/docset:PerryBlair/docset:Premises[2]/docset:RentableAreaofthePremises-section/docset:RentableAreaofthePremises'})]
Docugami can help here. Chunks are annotated with additional metadata created using different techniques if a user has been using Docugami. More technical approaches will be added later.
Specifically, let's look at the additional metadata that is returned on the documents returned by docugami, in the form of some simple key/value pairs on all the text chunks:
We can use a self-querying retriever to improve our query accuracy, using this additional metadata:
from langchain.chains.query_constructor.schema import AttributeInfo
from langchain.retrievers.self_query.base import SelfQueryRetriever


EXCLUDE_KEYS = ["id", "xpath", "structure"]
metadata_field_info = [
AttributeInfo(
name=key,
description=f"The {key} for this chunk",
type="string",
)
for key in documents[0].metadata
if key.lower() not in EXCLUDE_KEYS
]

document_content_description = "Contents of this chunk"
llm = OpenAI(temperature=0)
vectordb = Chroma.from_documents(documents=documents, embedding=embedding)
retriever = SelfQueryRetriever.from_llm(
llm, vectordb, document_content_description, metadata_field_info, verbose=True
)
qa_chain = RetrievalQA.from_chain_type(
llm=OpenAI(), chain_type="stuff", retriever=retriever, return_source_documents=True
)
Let's run the same question again. It returns the correct result since all the chunks have metadata key/value pairs on them carrying key information about the document even if this information is physically very far away from the source chunk used to generate the answer.
/root/Source/github/docugami.langchain/libs/langchain/langchain/chains/llm.py:275: UserWarning: The predict_and_parse method is deprecated, instead pass an output parser directly to LLMChain.
warnings.warn(

query='rentable area' filter=Comparison(comparator=<Comparator.EQ: 'eq'>, attribute='Landlord', value='DHA Group') limit=None


{'query': 'What is rentable area for the property owned by DHA Group?',
'result': ' The rentable area for the property owned by DHA Group is 13,500 square feet.',
'source_documents': [Document(page_content='1.6 Rentable Area of the Premises. 13,500 square feet . This square footage figure includes an add-on factor for Common Areas in the Building and has been agreed upon by the parties as final and correct and is not subject to challenge or dispute by either party.', metadata={'Landlord': 'DHA Group', 'Lease Date': 'March 29th , 2019', 'Lease Parties': 'THIS OFFICE LEASE (the "Lease") is made and entered into as of March 29th , 2019 , by and between Landlord and Tenant . "Date of this Lease" shall mean the date on which the last one of the Landlord and Tenant has signed this Lease .', 'Tenant': 'Shorebucks LLC', 'id': 'md8rieecquyv', 'source': 'Shorebucks LLC_NJ.pdf', 'structure': 'div', 'tag': 'RentableAreaofthePremises', 'xpath': '/docset:OFFICELEASE-section/docset:OFFICELEASE/docset:THISOFFICELEASE/docset:WITNESSETH-section/docset:WITNESSETH/docset:TheTerms/dg:chunk/docset:BasicLeaseInformation/docset:BASICLEASEINFORMATIONANDDEFINEDTERMS-section/docset:BASICLEASEINFORMATIONANDDEFINEDTERMS/docset:DhaGroup/docset:DhaGroup/docset:Premises[2]/docset:RentableAreaofthePremises-section/docset:RentableAreaofthePremises'}),


Document(page_content='1.6 Rentable Area of the Premises. 13,500 square feet . This square footage figure includes an add-on factor for Common Areas in the Building and has been agreed upon by the parties as final and correct and is not subject to challenge or dispute by either party.', metadata={'Landlord': 'DHA Group', 'Lease Date': 'March 29th , 2019', 'Lease Parties': 'THIS OFFICE LEASE (the "Lease") is made and entered into as of March 29th , 2019 , by and between Landlord and Tenant . "Date of this Lease" shall mean the date on which the last one of the Landlord and Tenant has signed this Lease .', 'Tenant': 'Shorebucks LLC', 'id': 'md8rieecquyv', 'source': 'Shorebucks LLC_NJ.pdf', 'structure': 'div', 'tag': 'RentableAreaofthePremises', 'xpath': '/docset:OFFICELEASE-section/docset:OFFICELEASE/docset:THISOFFICELEASE/docset:WITNESSETH-section/docset:WITNESSETH/docset:TheTerms/dg:chunk/docset:BasicLeaseInformation/docset:BASICLEASEINFORMATIONANDDEFINEDTERMS-section/docset:BASICLEASEINFORMATIONANDDEFINEDTERMS/docset:DhaGroup/docset:DhaGroup/docset:Premises[2]/docset:RentableAreaofthePremises-section/docset:RentableAreaofthePremises'}),


Document(page_content='1.11 Percentage Rent . (a) 55 % of Gross Revenue to Landlord until Landlord receives Percentage Rent in an amount equal to the Annual Market Rent Hurdle (as escalated); and', metadata={'Landlord': 'DHA Group', 'Lease Date': 'March 29th , 2019', 'Lease Parties': 'THIS OFFICE LEASE (the "Lease") is made and entered into as of March 29th , 2019 , by and between Landlord and Tenant . "Date of this Lease" shall mean the date on which the last one of the Landlord and Tenant has signed this Lease .', 'Tenant': 'Shorebucks LLC', 'id': 'md8rieecquyv', 'source': 'Shorebucks LLC_NJ.pdf', 'structure': 'p', 'tag': 'GrossRevenue', 'xpath': '/docset:OFFICELEASE-section/docset:OFFICELEASE/docset:THISOFFICELEASE/docset:WITNESSETH-section/docset:WITNESSETH/docset:GrossRentCreditTheRentCredit-section/docset:GrossRentCreditTheRentCredit/docset:Period/docset:ApplicableSalesTax/docset:PercentageRent/docset:PercentageRent/docset:PercentageRent/docset:PercentageRent-section/docset:PercentageRent[2]/docset:PercentageRent/docset:GrossRevenue[1]/docset:GrossRevenue'}),


Document(page_content='1.11 Percentage Rent . (a) 55 % of Gross Revenue to Landlord until Landlord receives Percentage Rent in an amount equal to the Annual Market Rent Hurdle (as escalated); and', metadata={'Landlord': 'DHA Group', 'Lease Date': 'March 29th , 2019', 'Lease Parties': 'THIS OFFICE LEASE (the "Lease") is made and entered into as of March 29th , 2019 , by and between Landlord and Tenant . "Date of this Lease" shall mean the date on which the last one of the Landlord and Tenant has signed this Lease .', 'Tenant': 'Shorebucks LLC', 'id': 'md8rieecquyv', 'source': 'Shorebucks LLC_NJ.pdf', 'structure': 'p', 'tag': 'GrossRevenue', 'xpath': '/docset:OFFICELEASE-section/docset:OFFICELEASE/docset:THISOFFICELEASE/docset:WITNESSETH-section/docset:WITNESSETH/docset:GrossRentCreditTheRentCredit-section/docset:GrossRentCreditTheRentCredit/docset:Period/docset:ApplicableSalesTax/docset:PercentageRent/docset:PercentageRent/docset:PercentageRent/docset:PercentageRent-section/docset:PercentageRent[2]/docset:PercentageRent/docset:GrossRevenue[1]/docset:GrossRevenue'})]}
This time the answer is correct, since the self-querying retriever created a filter on the landlord attribute of the metadata, correctly filtering to document that specifically is about the DHA Group landlord. The resulting source chunks are all relevant to this landlord, and this improves answer accuracy even though the landlord is not directly mentioned in the specific chunk that contains the correct answer.


DuckDB
DuckDB is an in-process SQL OLAP database management system.
Load a DuckDB query with one document per row.
from langchain.document_loaders import DuckDBLoader
Team,Payroll
Nationals,81.34
Reds,82.20
loader = DuckDBLoader("SELECT * FROM read_csv_auto('example.csv')")

data = loader.load()
[Document(page_content='Team: Nationals\nPayroll: 81.34', metadata={}), Document(page_content='Team: Reds\nPayroll: 82.2', metadata={})]
Specifying Which Columns are Content vs Metadata
loader = DuckDBLoader(
"SELECT * FROM read_csv_auto('example.csv')",
page_content_columns=["Team"],
metadata_columns=["Payroll"],
)

data = loader.load()
[Document(page_content='Team: Nationals', metadata={'Payroll': 81.34}), Document(page_content='Team: Reds', metadata={'Payroll': 82.2})]
loader = DuckDBLoader(
"SELECT Team, Payroll, Team As source FROM read_csv_auto('example.csv')",
metadata_columns=["source"],
)

data = loader.load()
[Document(page_content='Team: Nationals\nPayroll: 81.34\nsource: Nationals', metadata={'source': 'Nationals'}), Document(page_content='Team: Reds\nPayroll: 82.2\nsource: Reds', metadata={'source': 'Reds'})]


Email
This notebook shows how to load email (.eml) or Microsoft Outlook (.msg) files.
Using Unstructured
#!pip install unstructured
from langchain.document_loaders import UnstructuredEmailLoader
loader = UnstructuredEmailLoader("example_data/fake-email.eml")
[Document(page_content='This is a test email to use for unit tests.\n\nImportant points:\n\nRoses are red\n\nViolets are blue', metadata={'source': 'example_data/fake-email.eml'})]
Retain Elements
Under the hood, Unstructured creates different "elements" for different chunks of text. By default we combine those together, but you can easily keep that separation by specifying mode="elements".
loader = UnstructuredEmailLoader("example_data/fake-email.eml", mode="elements")
Document(page_content='This is a test email to use for unit tests.', metadata={'source': 'example_data/fake-email.eml', 'filename': 'fake-email.eml', 'file_directory': 'example_data', 'date': '2022-12-16T17:04:16-05:00', 'filetype': 'message/rfc822', 'sent_from': ['Matthew Robinson <mrobinson@unstructured.io>'], 'sent_to': ['Matthew Robinson <mrobinson@unstructured.io>'], 'subject': 'Test Email', 'category': 'NarrativeText'})
Processing Attachments
You can process attachments with UnstructuredEmailLoader by setting process_attachments=True in the constructor. By default, attachments will be partitioned using the partition function from unstructured. You can use a different partitioning function by passing the function to the attachment_partitioner kwarg.
loader = UnstructuredEmailLoader(
"example_data/fake-email.eml",
mode="elements",
process_attachments=True,
)


mode="elements",
process_attachments=True,
)
Document(page_content='This is a test email to use for unit tests.', metadata={'source': 'example_data/fake-email.eml', 'filename': 'fake-email.eml', 'file_directory': 'example_data', 'date': '2022-12-16T17:04:16-05:00', 'filetype': 'message/rfc822', 'sent_from': ['Matthew Robinson <mrobinson@unstructured.io>'], 'sent_to': ['Matthew Robinson <mrobinson@unstructured.io>'], 'subject': 'Test Email', 'category': 'NarrativeText'})
Using OutlookMessageLoader
#!pip install extract_msg
from langchain.document_loaders import OutlookMessageLoader
loader = OutlookMessageLoader("example_data/fake-email.msg")
Document(page_content='This is a test email to experiment with the MS Outlook MSG Extractor\r\n\r\n\r\n-- \r\n\r\n\r\nKind regards\r\n\r\n\r\n\r\n\r\nBrian Zhou\r\n\r\n', metadata={'subject': 'Test for TIF files', 'sender': 'Brian Zhou <brizhou@gmail.com>', 'date': 'Mon, 18 Nov 2013 16:26:24 +0800'})


Embaas
embaas is a fully managed NLP API service that offers features like embedding generation, document text extraction, document to embeddings and more. You can choose a variety of pre-trained models.
Prerequisites
Create a free embaas account at https://embaas.io/register and generate an API key
The document text extraction API allows you to extract the text from a given document. The API supports a variety of document formats, including PDF, mp3, mp4 and more. For a full list of supported formats, check out the API docs (link below).
# Set API key
embaas_api_key = "YOUR_API_KEY"
# or set environment variable
os.environ["EMBAAS_API_KEY"] = "YOUR_API_KEY"
Using a blob (bytes)
from langchain.document_loaders.embaas import EmbaasBlobLoader
from langchain.document_loaders.blob_loaders import Blob
blob_loader = EmbaasBlobLoader()
blob = Blob.from_path("example.pdf")
documents = blob_loader.load(blob)
# You can also directly create embeddings with your preferred embeddings model
blob_loader = EmbaasBlobLoader(params={"model": "e5-large-v2", "should_embed": True})
blob = Blob.from_path("example.pdf")
documents = blob_loader.load(blob)

print(documents[0]["metadata"]["embedding"])
Using a file
from langchain.document_loaders.embaas import EmbaasLoader
file_loader = EmbaasLoader(file_path="example.pdf")
documents = file_loader.load()
# Disable automatic text splitting
file_loader = EmbaasLoader(file_path="example.mp3", params={"should_chunk": False})
documents = file_loader.load()
For more detailed information about the embaas document text extraction API, please refer to the official embaas API documentation.


EPub
EPUB is an e-book file format that uses the ".epub" file extension. The term is short for electronic publication and is sometimes styled ePub. EPUB is supported by many e-readers, and compatible software is available for most smartphones, tablets, and computers.
This covers how to load .epub documents into the Document format that we can use downstream. You'll need to install the pandoc package for this loader to work.
from langchain.document_loaders import UnstructuredEPubLoader
loader = UnstructuredEPubLoader("winter-sports.epub")
Retain Elements
Under the hood, Unstructured creates different "elements" for different chunks of text. By default we combine those together, but you can easily keep that separation by specifying mode="elements".
loader = UnstructuredEPubLoader("winter-sports.epub", mode="elements")
Document(page_content='The Project Gutenberg eBook of Winter Sports in\nSwitzerland, by E. F. Benson', lookup_str='', metadata={'source': 'winter-sports.epub', 'page_number': 1, 'category': 'Title'}, lookup_index=0)


EverNote is intended for archiving and creating notes in which photos, audio and saved web content can be embedded. Notes are stored in virtual "notebooks" and can be tagged, annotated, edited, searched, and exported.
A document will be created for each note in the export.
[Document(page_content='testing this\n\nwhat happens?\n\nto the world?', metadata={'title': 'testing', 'created': time.struct_time(tm_year=2023, tm_mon=2, tm_mday=9, tm_hour=3, tm_min=47, tm_sec=46, tm_wday=3, tm_yday=40, tm_isdst=-1), 'updated': time.struct_time(tm_year=2023, tm_mon=2, tm_mday=9, tm_hour=3, tm_min=53, tm_sec=28, tm_wday=3, tm_yday=40, tm_isdst=-1), 'note-attributes.author': 'Harrison Chase', 'source': 'example_data/testing.enex'}),
Document(page_content='**Jan - March 2022**', metadata={'title': 'Summer Training Program', 'created': time.struct_time(tm_year=2022, tm_mon=12, tm_mday=27, tm_hour=1, tm_min=59, tm_sec=48, tm_wday=1, tm_yday=361, tm_isdst=-1), 'note-attributes.author': 'Mike McGarry', 'note-attributes.source': 'mobile.iphone', 'source': 'example_data/testing.enex'})]


This notebook covers how to load data from an .ipynb notebook into a format suitable by LangChain.
NotebookLoader.load() loads the .ipynb notebook file into a Document object.
loader.load(include_outputs=True, max_output_length=20, remove_newline=True)


The UnstructuredExcelLoader is used to load Microsoft Excel files. The loader works with both .xlsx and .xls files. The page content will be the raw text of the Excel file. If you use the loader in "elements" mode, an HTML representation of the Excel file will be available in the document metadata under the text_as_html key.
Document(page_content='\n \n \n Team\n Location\n Stanley Cups\n \n \n Blues\n STL\n 1\n \n \n Flyers\n PHI\n 2\n \n \n Maple Leafs\n TOR\n 13\n \n \n', metadata={'source': 'example_data/stanley-cups.xlsx', 'filename': 'stanley-cups.xlsx', 'file_directory': 'example_data', 'filetype': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', 'page_number': 1, 'page_name': 'Stanley Cups', 'text_as_html': '
\n
\n
\n Team	\n Location	\n Stanley Cups	\n
\n
\n Blues	\n STL	\n 1	\n
\n
\n Flyers	\n PHI	\n 2	\n
\n
\n Maple Leafs	\n TOR	\n 13	\n
\n
\n
', 'category': 'Table'})


This notebook covers how to load data from the Facebook Chats into a format that can be ingested into LangChain.
[Document(page_content='User 2 on 2023-02-05 03:46:11: Bye!\n\nUser 1 on 2023-02-05 03:43:55: Oh no worries! Bye\n\nUser 2 on 2023-02-05 03:24:37: No Im sorry it was my mistake, the blue one is not for sale\n\nUser 1 on 2023-02-05 03:05:40: I thought you were selling the blue one!\n\nUser 1 on 2023-02-05 03:05:09: Im not interested in this bag. Im interested in the blue one!\n\nUser 2 on 2023-02-05 03:04:28: Here is $129\n\nUser 2 on 2023-02-05 03:04:05: Online is at least $100\n\nUser 1 on 2023-02-05 02:59:59: How much do you want?\n\nUser 2 on 2023-02-04 22:17:56: Goodmorning! $50 is too low.\n\nUser 1 on 2023-02-04 14:17:02: Hi! Im interested in your bag. Im offering $50. Let me know if you are interested. Thanks!\n\n', metadata={'source': 'example_data/facebook_chat.json'})]


from langchain.document_loaders.fauna import FaunaLoader

secret = "<enter-valid-fauna-secret>"
query = "Item.all()" # Fauna query. Assumes that the collection is called "Item"
field = "text" # The field that contains the page content. Assumes that the field is called "text"

loader = FaunaLoader(query, field, secret)
docs = loader.lazy_load()

for value in docs:
print(value)
You get a after value if there are more data. You can get values after the curcor by passing in the after string in query.