Patents - the Appeal in Comptroller v Emotional Perceptions

 

Jane Lambert

Court of Appeal (Lady Justice Nicola Davies and Lords Justices Birss and Arnold) Comptroller General of Patents, Designs and Trade Marks v Emotional Perception AI Ltd [2024] EWCA Civ 825 (19 July 2024)

This was an appeal by the Comptroller-General of Patents, Designs and Trade Marks ("the Comptroller") from the decision of Sir Anthony Mann sitting as a judge of the High Court in  Emotional Perception AI Ltd v Comptroller-General of Patents, Designs and Trade Marks [2023] [2024] Bus LR 14, [2023] WLR(D) 500, [2023] EWHC 2948 (Ch)) (21 Nov 2023). Sir Anthony had allowed an appeal by Emotional Perceptions AI Ltd ("EPL") against the decision of the hearing officer Phil Thorpe in Re Emotional Perception AI Limited's Application BL/O/542/22 of 22 June 2022. In that decision, Mr Thorpe upheld the examiner's objection to UK Patent application GB1904713.3 for a Method of training a neural network to reflect emotional perception and related system and method for categorizing and finding associated content on the ground that that invention was excluded from patentability by s.1 (2) (c) the Patents Act 1977.

The Issue

A patent may be granted for an invention that complies with all the conditions of s.1 (1) of the Act.  One of those conditions is that the grant of a patent is not excluded by s.1 (2).  S.1 (2) (c) excludes from patentability a program for a computer as such.  Despite lengthy correspondence and several proposed amendments, the examiner objected to the application for the reasons set out in his examination report of 28 June 2021.  The issue in the appeal was summarized by Lord Justice Birss in para [1] of his judgment in Comptroller General of Patents, Designs and Trade Marks v Emotional Perception AI Ltd [2024] EWCA Civ 825 (19 July 2024):

"The first question in this appeal is whether the exclusion from patentability of a program for a computer 'as such' by s.1 (2) of the Patents Act 1977 has any application to artificial neural networks. These networks are the backbone of the machine learning systems on which modern artificial intelligence systems are based. If artificial neural networks do engage s.1 (2) then the second question arises. This concerns how that exclusion would apply to the particular patent application in this case."

Sir Anthony had held that no computer program was involved at all, at least for a hardware-implemented artificial neural network ("ANN") and therefore s.1 (2) (c) did not apply. He also held that even if that provision did apply, the invention was not excluded in the circumstances of this case.

The Invention

Lord Justice Birss described the invention as "a system for providing media file recommendations to a user." He gave the following typical example of the invention's use on a music website,  The system learns the sort of music that a user likes and suggests a track that meets his or her preferences.  Previous systems categorized music into genres such as classical, folk, rock and so on and users into fans of such genres.  Thus, they could recommend a folk song to a fan of folk music but not a track to that user from any of the other genres.    EPL's invention is said to be able to offer suggestions that meet a user's preferences in terms of human perception and emotion, irrespective of the genre of music and the apparently similar tastes of other people. The invention arrives at these suggestions by passing music through a trained ANN.

The Abstract

The abstract summarizes the invention as follows:

"Training an artificial intelligence to reflect human subjective responses to stimuli, e.g. audio, music, images, videos or text, employs an artificial neural network (ANN) to identify similarity between the content of two files, e.g. music tracks. For audio files, measurable signal qualities, e.g. rhythm, tonality, timbre and texture, are extracted from both files to identify musical properties. The ANN outputs a property vector for each musical property, and the property vectors are assembled into a multi-dimensional vector for each file. The weights or bias values of the ANN are adjusted by backpropagation based on the distance between the two multi-dimensional vectors and any discrepancy between this distance and a quantified semantic dissimilarity distance between the two files in semantic space, such as textual description of the track or other semantic associations with events, feelings, themes or environments. Thus, the ANN is trained to align the distance between the files in property space to correspond to the distance between the files in semantic space. The ANN can then identify music files which are subjectively semantically similar to a target file based only on the target file’s objective musical properties."

ANNs

The abstract, claim 4 and all the judgments refer to ANNs.  An ANN is a network of artificial neurons which are similar to neurons in the brain,  Each of those artificial neurons is connected to other artificial neurons and arranged in layers.  Each artificial neuron is capable of processing and outputting information.   Such outputs can be passed on to other artificial neurons in other layers.  The first layer receives inputs from outside the ANN and the last layer produces an output from the system.   

The features of an ANN appear in Figure 6 of ELP's patent application which is reproduced at the head of this article.  The right-hand side of the diagram shows a network of artificial neurons.  Each circle represents an artificial neuron and each line its connection to other artificial neurons.  In the diagram layers are called "levels". The input level appears on the left of the right-hand side of the diagram at 702 and the output level is on the right at 720. The levels in between the input and output levels are hidden. The diagram shows "n" input signals and "m" output signals.

The box on the left-hand side of the diagram shows a single artificial neuron.  It takes inputs 

x₁, x₂, x₃ to x_r 

and applies a specific weight 

w_i,1, w_i,2, w_i,3 to w_i,r 

to each input.  It then adds up all the weighted values as signified by the letter 𝝨 and applies a further weight called a "bias" signified by b_i and then a function f to the output.  The function converts the result of the biased sum into an overall output represented by y_i

The system can be made to perform a specific operation by a process called "training".  Weights and biases are adjusted iteratively as the system processes input data.  Once the training is complete the system can recognize data that it has not encountered before,  In the example of the music website such data could include music from an unfamiliar genre that meets a user's tastes.  After an ANN has been trained, its weights can be extracted and used to set up other ANNs of the same kind in order for them to perform the same operation.   Such an ANN can be set up by a computer program or it can be hardwired.

The Claims

The disputes claims were claims 1 and 4.   Both claims are appended to the judgments of Sir Anthony Mann and the Court of Appeal,   Most of the argument focussed on claim 4: which was for:

"A method of providing semantically relevant file recommendations in a system including an artificial neural network "ANN" having an output capable of generating a property vector in property space, the method comprising:

a) training the ANN by subjecting the ANN to a multiplicity of pairs of training data files sharing a content modality and where for each pair of training data files there are two independently derived separation distances, namely:

• a first independently derived separation distance that expresses a measure of relative distance between a first pair of training data files in semantic embedding space, where the first independently derived separation distance is obtained from natural language processing "NLP" of a semantic description of the nature of the data associated with each one of the first pair of training data files; and
• a second independently derived separation distance that expresses a measure of relative distance similarity between the first pair of training data files in property embedding space, where the second

independently derived separation distance is a property distance derived from measurable properties extracted from each one of the first pair of training data files,

and wherein shared content modality is: (i) video data files; or alternatively (ii) audio data files; or alternatively (iii) static image files; or alternatively (iv) text files;

b) in a backpropagation process in the ANN, using output vectors generated at the output of the ANN from processing of said multiplicity of pairs to adjust weighting factors in the ANN, thereby adapting the ANN during training to converge distances of generated output vectors, in property embedding space, towards corresponding pairwise semantic distances in semantic space, and

b) storing, in a database, a multiplicity of reference data files with content modality with target data and a stored association between each reference data file and a related individual property vector, wherein each related individual property vector is obtained from processing, within the trained ANN, of file properties extracted from its respective reference data file and each related individual property vector encodes the semantic description of its respective reference data file;

c) in response to the trained ANN receiving target data as an input and for which target data an assessment of relative semantic similarity of its content is to be made, and the ANN producing a file vector (Vnie) in property space for the target data based on processing within the trained ANN of file properties extracted from the target data;

d) comparing the file vector of the target data with individual property vectors of the multiplicity of reference data files in the database to produce an ordered list which identifies relevant reference files that are measurably similar to the property vector and thus identifying relevant reference data files that are semantically similar to the target data;

e) sending, over the communications network, relevant reference data files to the user device; and

f) at the user device, receiving the relevant reference files and outputting the content thereof."

The Aerotel Guidelines and HTC Signposts

Lord Justice Birss noted that the Court of Appeal had established the following guidelines for applying s.1 (2) in  Aerotel Ltd v Telco Holdings Ltd and Re Macrossan [2006] EWCA Civ 1371, [2006] Info TLR 215, [2007] BusLR 634, [2007] RPC 7, [2007] Bus LR 634, [2007] 1 All ER 225:

"(1) Properly construe the claim.
(2) Identify the actual contribution (although at the application stage this might have to be the alleged contribution).
(3) Ask whether it falls solely within the excluded matter.
(4) If the third step has not covered it, check whether the actual or alleged contribution is actually technical."

He also referred to HTC Europe Co Ltd v Apple Inc (Rev 1) [2013] RPC 30, [2013] EWCA Civ 451 which approved with one modification Mr Justice Lewison's judgment in AT &T Knowledge Ventures LP's Patent Application 2009] EWHC 343 (Pat), [2009] FSR 19, [2009] Bus LR D51 where his lordship had proposed 5 signposts to use when considering whether a computer program makes a technical contribution.

Hearing Officer's Decision

In the hearing before Mr Thorpe, EPL had argued that s.1 (2) (c) did not apply to the invention at all whether software implemented or hard-wired for the following reasons. 

First, a hardware-implemented ANN would fall outside the statutory exclusion altogether.  Secondly, s.1 (2) (c) is not supposed to operate to exclude inventions that would otherwise be patentable but for their implementation as software.  Thirdly,  software implemented ANN could not be excluded by that provision either.  EPL argued that a distinction could be made between programming which is the work of a programmer and training which results from processing data.

EPL's second argument was that the task that the trained ANN was required to perform was sending a file (namely a recommended music track) over a network. Such a task was external to the computer and not within any of the excluded areas.

The third reason advanced by EPL was that recommending a music file likely to appeal to a user was analogous to technically superior monitoring of electronic communications which Mr Justice Floyd had held to be patentable in Protecting Kids the World Over (PKTWO) Ltd's Application [2012] RPC 13, [2011] EWHC 2720 (Pat).

Mr Thorpe rejected all those contentions.  In response to the argument that a distinction could be drawn between programming and training, he said at para [61]:  "The programmer defines the problem and the training approach, and the ANN operates within those boundaries to build a suitable model. This is still no more than a computer program in my opinion."  As for the second argument, he said at [69] that while the recommendation of a music track was more than the standard transmission of a file in a network in that it was a song that the user was likely to enjoy "… such a beneficial effect is of a subjective and cognitive nature and does not suggest there is any technical effect over and above the running of a program on a computer."  Turning to the third reason the hearing officer said that any similarity between Protecting Kids the World Over and the case before him was superficial.   The technical superiority in monitoring communications had been a technical contribution.  The recommendation of music tracks had been based on their semantic similarity which was not a relevant technical effect.  Mr Thorpe also considered the HTC signposts but found no technical contribution in the invention.

Sir Anthony Mann's Judgment

EPL challenged the hearing officer's decision on the following grounds which Sir Anthony set out at para [31] of his judgment:

"(a) The computer program exclusion is not engaged at all; one does not get as far as finding a relevant computer program.
(b) The reasoning of the Hearing Officer fails to acknowledge a line of cases which [EPL's counsel] described as the 'patentable ignoring a computer program' line of cases.
(c) If there is a computer program and the exclusion is prima facie engaged, it does not apply because the claim reveals a technical contribution and the claim is not to a program for a computer 'as such.'"

At the end of the hearing of the appeal, the judge requested further written submissions from counsel on what is the computer, and where is the program which is said to engage the exclusion?  Counsel for the Comptroller responded as follows:

"(a) In the case of a hardware ANN, the 'computer' was the hardware itself.
(b) In the case of a hardware ANN, 'there is no relevant computer program to which the exclusion applies.' Accordingly, if the claim had involved only a hardware ANN 'it is unlikely' that the exclusion would be engaged. By this I took her to mean that it was her case that it would not have been engaged. This was not a position which she clearly adopted at the hearing, and is not consistent with some of her submissions. However, she did make that point clearly in her additional written submissions, so it is now her stance.
(c) Insofar as the invention was implemented in software, the computer is that which permits the implementation of the ANN, ie the 'computer comprising code that, when executed by processor intelligence, performs the method of the various aspects recited herein, and, particularly, in the claims' (Application at p11 lines 18-20). The 'code' is said by [the Comptroller's counsel] to be the code by which the input file is taken in, its vectors compared with the vectors of other known files, and the end file recommendation is sent out. In other words, code which is equivalent to the workings of a trained ANN.
(d) However, she went on to say that there was no material distinction between the computer program used to implement the trained ANN (ie the end product) and the computer program used to train the ANN. It was not clear from her final submissions, but it seems that she thereby sought to apply the computer program exclusion to the training software. That would be consistent with submissions that she made orally. She submitted that the 'training instructions' were within the exclusion and the 'key contribution' is the training, which she said was a computer program.
(e) The ANN itself, in a general and abstract sense (ie stripped of the software by which it was trained and/or implemented) was simply an algorithm and, as such, related wholly to a mathematical method and was excluded from patentability on that basis. She relied on the finding by the Hearing Officer in paragraph 63 of the Decision.
(f) Insofar as the invention is implemented in software form, the trained ANN is not a computer, but the trained ANN is supported by a computer.
(g) She submitted that the method of training the ANN and the operation of the trained ANN (including sending out the file recommendation) were each a computer program to which the exclusion applied. She submitted that the Hearing Officer found that the former was a computer program (Decision paragraph 61)."

EPL's counsel submitted that a hard-wired ANN was not a computer and that its internal workings were not a computer program. So far as an emulated computer was concerned, he relied on a distinction between the enabling software on the one hand and the emulated ANN which it enabled on the other. If the former involved a computer program, the operation of the ANN itself did not because that was not the nature of what was going on internally.

Sir Anthony understood that the Comptroller had submitted that a hard-wired ANN was a computer but it was a computer without a program. There was no relevant computer program to which the exclusion applied.  It followed that the Comptroller accepted that if the patent application had been confined to a hardware-implemented ANN, then it could not have been excluded from patentability.   As for a software-implemented ANN, Sir Anthony said at para [49] of his judgment that a computer program is involved at the training stage.   However, relying on the Comptroller's concession on the absence of programs in hardware-implemented ANNs he could not see why the same should not apply to a software-emulated ANN.  He held that it is appropriate to look at a software ANN as operating at a different level (metaphorically) from the underlying software on the computer. If a hardware ANN is not operating a program then neither is a software ANN working in the same way. Therefore he held at para [58] that an emulated ANN is not a program for a computer at all.

His lordship considered the technical contribution.  He agreed with the hearing officer that the transmission of a recommended music file was an effect external to the computer but held that he had been wrong to hold that a subjective appreciation of the output of the system was not a technical effect.   As Lord Justice Birss put it at para [51] of his judgment:

"The output file is a file identified as being semantically similar by the application of technical criteria which the system worked out for itself. It is 'not just any old file', the output is a technical effect outside the computer and when coupled with the purpose and method of selection it fulfils the requirement of a technical effect to escape the exclusion. The (music) file goes on to have an effect on the user if the thing works at all, but it would not matter if the user never listened to the file. The file with its similarity characteristics is still produced. Therefore the system is not excluded."

Grounds of Appeal

The Comptroller appealed to the Court of Appeal on the following grounds:

"i) Ground 1: the Judge erred in holding that the exclusion from patent protection for 'a program for a computer … as such' was not engaged;
ii) Ground 2: the Judge was wrong to rely on the Appellant's 'concession' that a hardware ANN was a computer but it was a computer with no program, or words to that effect;
iii) Ground 3: the Judge was wrong to exclude the consideration of the mathematical model exclusion; and
iv) Ground 4: the Judge was wrong to hold that the claimed invention involves a substantive technical contribution."

Lord Justice Birss decided to address Grounds 1 and 2 by asking what is a computer program and whether there is a computer program in an ANN?   If there is a computer program in an ANN then the question of whether Sir Anthony had been wrong to hold that the claimed invention involved a substantive technical contribution.  Ground 3 would arise only if the Comptroller were to lose Grounds 1 and 2.

The Appeal

The appeal was heard by Lady Justice Nicola Davies, Lord Justice Arnold and Lord Justice Birss on 14 and 15 May 2024.  The hearing was filmed and may be viewed on the Court of Appeal's YouTube channel.  The recording of 14 May is here and that of 15 May is here.  Judgment was handed down on 19 July 2024.  The lead judgment was delivered by Lord Justice Birss with concurring judgments from Lady Justice Nicola Davies and Lord Justice Arnold. 

Grounds 1 and 2: What is a Computer Program? and Whether there is a Progran in an Ann.

At para [61] of his judgment, Lord Justice Birss defined a computer as "a machine which processes information". Neither party came up with a better definition and Lord Justice believed that it was a useful one.

 He held that a computer program was "a set of instructions for a computer to do something."  He explained:

"These two definitions work together, so one can say that a computer is a machine which does something, and that thing it does is to process information in a particular way. The program is the set of instructions which cause the machine to process the information in that particular way, rather than in another way." 

He added that the focus on instructions was consistent with the approach of the Court of Appeal in Gale's Application [1991] RPC 305, on page 321 and in Aerotel at para [31].

He rejected EPL's submission that a program should refer to a human programmer as neither relevant nor helpful.  He did not see any relevance in the fact that the particular values for the weights are produced by a training process in which the machine learns for itself rather than by a human being,  Nor did he see any relevance in the fact that some computers are hard-wired such as the chips in washing machines or payment cards while others can be programmed,

Turning to an ANN it was clearly a machine for processing information and therefore a computer.   The weights and biases in an artificial neuron govern how such neurons process information.  They are therefore a program for such computers.   The learned Lord Justice noted that the Technical Board of Appeal of the European Patent Office had followed a similar approach in T 702/20 Mitsubishi/Sparsely connected neural network.  He concluded at para [70] of his judgment:
  
"Therefore the exclusion from patentability of a program for a computer as such in s.1 (2) of the 1977 Act is engaged in this case. Nor is there any difference for this purpose between a hardware ANN and a software ANN. However it is implemented, the weights (by which I mean weights and biases) of the ANN are a program for a computer and therefore within the purview of the exclusion."

Ground 4:  Did the Invention Involve a Substantial Technical Contribution?

In para [58] Lord Justice Birss said:

"The issue boils down to whether the Hearing Officer was right to find the exclusion applied because the beneficial effect was of a subjective and cognitive nature (HO[69]) or whether the judge was right to hold as he did in [76] that the exclusion did not apply because even though what made the file recommendation better was not technical criteria (because the semantic similarity is a subjective matter) the ANN had reached that result by 'going about its analysis and selection in a technical way'. As the judge put it in that paragraph: 'It is not just any old file; it is a file identified as being semantically similar by the application of technical criteria […]'".

In the Lord Justice's judgment, the hearing officer's conclusion was the right one.  He explained at [79]:

"What makes the recommended file worth recommending are its semantic qualities. This is a matter of aesthetics or, in the language used by the Hearing Officer, they are subjective and cognitive in nature. They are not technical and do not turn this into a system which produces a technical effect outside the excluded subject matter."

He added in the next paragraph:

"It is true that as the judge said, the system has gone about its analysis and selection in a technical way but that is because it is an ANN, i.e. a computer. The fact the computer is using properties it can measure to make this semantic recommendation makes no difference. I think the flaw is that this approach imports the undoubtedly technical nature of computer systems (including ANNs) into the analysis. If that was appropriate then the same could be said of the other cases of excluded matter such as the computer implemented financial trading system of Merrill Lynch."

He said at [82] that he had considered the HTC signposts but found none that could assist EPL.

He allowed the appeal on Ground 4 and upheld Mr Thorpe's decision that EPL's patent application was excluded from patentability under s.1 (2) (c) of the Patents Act 1977.

Further Information 

As this is the first case in the United Kingdom on the application of s.1 (2) of the Patents Act 1977 to the training of artificial neural networks I am willing to give an online talk before the end of October if there is sufficient interest among intellectual property practitioners.  By "sufficient"  I mean an audience of 20 or more.   I am also happy to give a similar talk in-house should a firm of solicitors or patent attorneys or some other organization make the arrangements.   Anyone wishing to discuss this article may contact me on +44 (0)20 7404 5252 during UK office hours or send me a message through my contact form.