点击上方蓝字关注我们
背景
AIDocumentLibraryChat 项目已扩展至生成测试代码（Java 代码已通过测试）。该项目可为公开的 Github 项目生成测试代码。只需提供要测试的类的网址，该类就会被加载、分析导入，项目中的依赖类也会被加载。这样，LLM 就有机会在为测试生成模拟时考虑导入的源类。可以提供 testUrl，为 LLM 提供一个示例，以便生成测试。我们已使用 Ollama 对 granite-code、deepseek-coder-v2 和 codestral 模型进行了测试。
目的是测试 LLM 对开发人员创建测试的帮助有多大。
配置
要选择 LLM 模型，需要更新 application-ollama.properties 文件
spring.ai.ollama.base-url=${OLLAMA-BASE-URL:http://localhost:11434}spring.ai.ollama.embedding.enabled=falsespring.ai.embedding.transformer.enabled=truedocument-token-limit=150embedding-token-limit=500spring.liquibase.change-log=classpath:/dbchangelog/db.changelog-master-ollama.xml...spring.ai.ollama.chat.options.num-thread=8spring.ai.ollama.chat.options.keep_alive=1sspring.ai.ollama.chat.model=codestral:22bspring.ai.ollama.chat.options.num-ctx=32768
spring.ai.olama.chat.model "选择要使用的 LLM 代码模型。
spring.ollama.chat.options.num-ctx "设置上下文窗口中的令牌数量。上下文窗口包含请求所需的令牌和响应所需的令牌。
如果 Ollama没有选择正确的内核数量，可以使用 “spring.olama.chat.options.num-thread”。spring.olama.chat.options.keep_alive "设置了保留上下文窗口的秒数。
Controller
Controller是获取信号源和生成测试的接口：
public class CodeGenerationController {private final CodeGenerationService codeGenerationService;public CodeGenerationController(CodeGenerationServicecodeGenerationService) {this.codeGenerationService = codeGenerationService;}public String getGenerateTests( String url,String testUrl) {return this.codeGenerationService.generateTest(URLDecoder.decode(url,StandardCharsets.UTF_8),Optional.ofNullable(testUrl).map(myValue -> URLDecoder.decode(myValue,StandardCharsets.UTF_8)));}public GithubSources getSources( String url,String testUrl) {var sources = this.codeGenerationService.createTestSources(URLDecoder.decode(url, StandardCharsets.UTF_8), true);var test = Optional.ofNullable(testUrl).map(myTestUrl ->this.codeGenerationService.createTestSources(URLDecoder.decode(myTestUrl, StandardCharsets.UTF_8), false)).orElse(new GithubSource("none", "none", List.of(), List.of()));return new GithubSources(sources, test);}}
代码生成控制器 “有一个 ”getSources(...) "方法。该方法获取要生成测试的类的 URL 和可选的 testUrl，以及可选的示例测试。它对请求参数进行解码，并调用 “createTestSources(...) ”方法。该方法会返回 “GithubSources”，其中包含要测试的类的源代码、其在项目中的依赖关系以及测试示例。
getGenerateTests(...) “方法获取测试类的 ”url “和可选的 ”testUrl “以进行 url 解码，并调用 ”CodeGenerationService “的 ”generateTests(...) "方法。
Services
CodeGenerationService "从 Github 收集类，并为被测类生成测试代码。
带有提示的服务看起来像这样：
public class CodeGenerationService {private static final Logger LOGGER = LoggerFactory.getLogger(CodeGenerationService.class);private final GithubClient githubClient;private final ChatClient chatClient;private final String ollamaPrompt = """You are an assistant to generate spring tests for the class under test.Analyse the classes provided and generate tests for all methods. Baseyour tests on the example.Generate and implement the test methods. Generate and implement completetests methods.Generate the complete source of the test class.Generate tests for this class:{classToTest}Use these classes as context for the tests:{contextClasses}{testExample}""";private final String ollamaPrompt1 = """You are an assistant to generate a spring test class for the sourceclass.1. Analyse the source class2. Analyse the context classes for the classes used by the source class3. Analyse the class in test example to base the code of the generatedtest class on it.4. Generate a test class for the source class and use the context classesas sources for creating the test class.5. Use the code of the test class as test example.6. Generate tests for each of the public methods of the source class.Generate the complete source code of the test class implementing thetests.{testExample}Use these context classes as extension for the source class:{contextClasses}Generate the complete source code of the test class implementing thetests.Generate tests for this source class:{classToTest}""";private Long contextWindowSize;public CodeGenerationService(GithubClient githubClient, ChatClientchatClient) {this.githubClient = githubClient;this.chatClient = chatClient;}
这是带有 “GithubClient ”和 “ChatClient ”的 “CodeGenerationService”。GithubClient 用于从公开可用的资源库加载源代码，而 ChatClient 是 Spring AI 接口，用于访问 AI/LLM。
ollamaPrompt "是 IBM Granite LLM 的提示符，上下文窗口有 8k 个 token。classToTest}（测试类）用被测类的源代码代替。{contextClasses}“可以替换为被测类的从属类，”{testExample}"是可选项，可以替换为测试类，作为代码生成的示例。
olamaPrompt2 "是 Deepseek Coder V2 和 Codestral LLM 的提示符。这些 LLM 可以 “理解 ”或使用思维链提示，其上下文窗口超过 32k 个词组。{...}“占位符的工作原理与 ”olamaPrompt "相同。由于上下文窗口较长，因此可以为代码生成添加上下文类。
Spring 注入了 “contextWindowSize ”属性，用于控制 LLM 的上下文窗口是否足够大，以便在提示符中添加“{contextClasses}”。
我们也可抽取提示词，经笔者完善后，单独测试下效果如下
https://lxblog.com/qianwen/share?shareId=6ebef3fc-1c25-4abf-aa11-7122e540e964e?shareId=6ebef3fc-1c25-4abf-aa11-7122e540e964
示例提示词
You are an assistant to generate a spring test class for the source class.1. Analyse the source class2. Analyse the context classes for the classes used by the source class3. Analyse the class in test example to base the code of the generated test class on it.4. Generate a test class for the source class and use the context classes as sources for creating the test class.5. Use the code of the test class as test example.6. Generate tests for each of the public methods of the source class.Your additional guidelines:1.Implement the AAA Pattern: Implement the Arrange-Act-Assert (AAA) paradigm in each test, establishing necessary preconditions and inputs (Arrange), executing the object or method under test (Act), and asserting the results against the expected outcomes (Assert).2.Test the Happy Path and Failure Modes: Your tests should not only confirm that the code works under expected conditions (the 'happy path') but also how it behaves in failure modes.3.Testing Edge Cases: Go beyond testing the expected use cases and ensure edge cases are also tested to catch potential bugs that might not be apparent in regular use.4.Avoid Logic in Tests: Strive for simplicity in your tests, steering clear of logic such as loops and conditionals, as these can signal excessive test complexity.5.Leverage TypeScript's Type System: Leverage static typing to catch potential bugs before they occur, potentially reducing the number of tests needed.6.Handle Asynchronous Code Effectively: If your test cases involve promises and asynchronous operations, ensure they are handled correctly.7.Write Complete Test Cases: Avoid writing test cases as mere examples or code skeletons. You have to write a complete set of tests. They should effectively validate the functionality under test.Generate the complete source code of the test class implementing the tests.{@ExtendWith(MockitoExtension.class)public class MockitoHelloTest {@MockRecordDao mockDao;@MockNotificationService mockNotification;@MockSequenceGenerator mockGenerator;@InjectMocksRecordService service;@Testpublic void testSaveRecord() {Record record = new Record();record.setName("Test Record");when(mockGenerator.getNext()).thenReturn(100L);when(mockDao.saveRecord(record)).thenReturn(record);Record savedRecord = service.saveRecord(record);verify(mockGenerator, times(1)).getNext();verify(mockDao, times(1)).saveRecord(any(Record.class));assertEquals("Test Record", savedRecord.getName());assertEquals(100L, savedRecord.getId());}}}Use these context classes as extension for the source class:{package com.app.login.domain;@Entity@Table(name = "app_user")@Getter@Setterpublic class User implements Serializable {private static final long serialVersionUID = 1L;@Id@GeneratedValue(strategy = GenerationType.IDENTITY)private Long id;@NotNull@Pattern(regexp = Constants.LOGIN_REGEX)@Size(min = 1, max = 50)@Column(length = 50, unique = true, nullable = false)private String login;@JsonIgnore@NotNull@Size(min = 60, max = 60)@Column(name = "password_hash", length = 60)private String password;@Size(max = 50)@Column(name = "first_name", length = 50)private String firstName;@Size(max = 50)@Column(name = "last_name", length = 50)private String lastName;@Size(min = 5, max = 100)@Column(length = 100, unique = true)private String email;@Column(name = "created_date")private Instant createdDate = Instant.now();@NotNull@Column(nullable = false)private boolean activated = false;@Size(min = 2, max = 5)@Column(name = "lang_key", length = 5)private String langKey;@Size(max = 256)@Column(name = "image_url", length = 256)private String imageUrl;@Size(max = 20)@Column(name = "activation_key", length = 20)@JsonIgnoreprivate String activationKey;@Size(max = 20)@Column(name = "reset_key", length = 20)@JsonIgnoreprivate String resetKey;@Column(name = "reset_date")private Instant resetDate = null;@Size(min = 7, max = 15)@Column(name = "ip_address", length = 15)String ipAddress;@JsonIgnore@ManyToMany@JoinTable(name = "app_user_authority", joinColumns = { @JoinColumn(name = "user_id", referencedColumnName = "id") }, inverseJoinColumns = { @JoinColumn(name = "authority_name", referencedColumnName = "name") })@BatchSize(size = 20)private Set<Authority> authorities = new HashSet<>();@Overridepublic boolean equals(Object o) {if (this == o) {return true;}if (o == null || getClass() != o.getClass()) {return false;}User user = (User) o;return login.equals(user.login);}@Overridepublic int hashCode() {return login.hashCode();}}@Entity@Table(name = "app_authority")public class Authority implements Serializable {private static final long serialVersionUID = 1L;@NotNull@Size(min = 0, max = 50)@Id@Column(length = 50)private String name;public String getName() {return name;}public void setName(String name) {this.name = name;}@Overridepublic boolean equals(Object o) {if (this == o) {return true;}if (o == null || getClass() != o.getClass()) {return false;}Authority authority = (Authority) o;return !(name != null ? !name.equals(authority.name) : authority.name != null);}@Overridepublic int hashCode() {return name != null ? name.hashCode() : 0;}}package com.app.login.repository;public interface UserRepository extends JpaRepository<User, Long> {Optional<User> findOneByActivationKey(String activationKey);Optional<User> findOneByResetKey(String resetKey);Optional<User> findOneByEmail(String email);Optional<User> findOneByLogin(String login);@EntityGraph(attributePaths = "authorities")User findOneWithAuthoritiesById(Long id);@EntityGraph(attributePaths = "authorities")Optional<User> findOneWithAuthoritiesByLogin(String login);Page<User> findAllByLoginNot(Pageable pageable, String login);List<User> findAllByIpAddressAndCreatedDateBetween(String ipAddress, Instant startDate, Instant currentDate);}package com.app.login.security;public class UserNotActivatedException extends AuthenticationException {private static final long serialVersionUID = 1L;public UserNotActivatedException(String message) {super(message);}public UserNotActivatedException(String message, Throwable t) {super(message, t);}}}Generate the complete source code of the test class implementing the tests.Generate tests for this source class:package com.app.login.security;import com.app.login.domain.User;import com.app.login.repository.UserRepository;import lombok.extern.slf4j.Slf4j;import org.springframework.security.core.GrantedAuthority;import org.springframework.security.core.authority.SimpleGrantedAuthority;import org.springframework.security.core.userdetails.UserDetails;import org.springframework.security.core.userdetails.UserDetailsService;import org.springframework.security.core.userdetails.UsernameNotFoundException;import org.springframework.stereotype.Component;import org.springframework.transaction.annotation.Transactional;import java.util.*;import java.util.stream.Collectors;@Component("userDetailsService")@Slf4jpublic class DomainUserDetailsServiceImpl implements UserDetailsService {private final UserRepository userRepository;public DomainUserDetailsServiceImpl(UserRepository userRepository) {this.userRepository = userRepository;}@Override@Transactional(rollbackFor = Exception.class)public UserDetails loadUserByUsername(final String login) {if (login==null){throw new UsernameNotFoundException("User login is null");}log.debug("Authenticating {}", login);String lowercaseLogin = login.toLowerCase(Locale.ENGLISH);Optional<User> userFromDatabase = userRepository.findOneWithAuthoritiesByLogin(lowercaseLogin);return userFromDatabase.map(user -> {if (!user.getActivated()) {throw new UserNotActivatedException("User " + lowercaseLogin + " was not activated");}List<GrantedAuthority> grantedAuthorities = user.getAuthorities().stream().map(authority -> new SimpleGrantedAuthority(authority.getName())).collect(Collectors.toList());return new org.springframework.security.core.userdetails.User(lowercaseLogin, user.getPassword(), grantedAuthorities);}).orElseThrow(() -> new UsernameNotFoundException("User " + lowercaseLogin + " was not found in the " + "database"));}}}
方法 “createTestSources(...) ”收集并返回 AI/LLM 提示的源代码：
public GithubSource createTestSources(String url, final booleanreferencedSources) {final var myUrl = url.replace("https://github.com",GithubClient.GITHUB_BASE_URL).replace("/blob", "");var result = this.githubClient.readSourceFile(myUrl);final var isComment = new AtomicBoolean(false);final var sourceLines = result.lines().stream().map(myLine ->myLine.replaceAll("[\t]", "").trim()).filter(myLine -> !myLine.isBlank()).filter(myLine ->filterComments(isComment, myLine)).toList();final var basePackage = List.of(result.sourcePackage().split("\\.")).stream().limit(2).collect(Collectors.joining("."));final var dependencies = this.createDependencies(referencedSources, myUrl,sourceLines, basePackage);return new GithubSource(result.sourceName(), result.sourcePackage(),sourceLines, dependencies);}private List<GithubSource> createDependencies(final booleanreferencedSources, final String myUrl, final List<String> sourceLines,final String basePackage) {return sourceLines.stream().filter(x -> referencedSources).filter(myLine -> myLine.contains("import")).filter(myLine -> myLine.contains(basePackage)).map(myLine -> String.format("%s%s%s",myUrl.split(basePackage.replace(".", "/"))[0].trim(),myLine.split("import")[1].split(";")[0].replaceAll("\\.","/").trim(), myUrl.substring(myUrl.lastIndexOf('.')))).map(myLine -> this.createTestSources(myLine, false)).toList();}private boolean filterComments(AtomicBoolean isComment, String myLine) {var result1 = true;if (myLine.contains("/*") || isComment.get()) {isComment.set(true);result1 = false;}if (myLine.contains("*/")) {isComment.set(false);result1 = false;}result1 = result1 && !myLine.trim().startsWith("//");return result1;}
方法 “createTestSources(...) ”使用 Github 源代码 “url”，并根据 “referencedSources ”的值提供项目中依赖类的源代码 “GithubSource ”记录。
为此，我们创建了 “myUrl ”来获取类的原始源代码。然后使用 “githubClient ”以字符串形式读取源文件。然后使用 “filterComments(...) ”方法将源代码字符串转换为不带格式和注释的源代码行。
要读取项目中的依赖类，需要使用基础包。例如，在 “ch.xxx.aidoclibchat.usecase.service ”包中，基础包是 “http://ch.xxx"。方法 “createDependencies(...) ”用于为基础软件包中的依赖类创建 “GithubSource ”记录。使用 “basePackage ”参数筛选出类，然后递归调用 “createTestSources(...) ”方法，并将参数 “referencedSources ”设为 false 以停止递归。这就是依赖类 “GithubSource ”记录的创建过程。
generateTest(...) "方法用于使用 AI/LLM 创建被测类的测试源：
public String generateTest(String url, Optional<String> testUrlOpt) {var start = Instant.now();var githubSource = this.createTestSources(url, true);var githubTestSource = testUrlOpt.map(testUrl ->this.createTestSources(testUrl, false)).orElse(new GithubSource(null, null, List.of(), List.of()));String contextClasses = githubSource.dependencies().stream().filter(x -> this.contextWindowSize >= 16 * 1024).map(myGithubSource -> myGithubSource.sourceName() + ":" +System.getProperty("line.separator")+ myGithubSource.lines().stream().collect(Collectors.joining(System.getProperty("line.separator"))).collect(Collectors.joining(System.getProperty("line.separator")));String testExample = Optional.ofNullable(githubTestSource.sourceName()).map(x -> "Use this as test example class:" +System.getProperty("line.separator") +githubTestSource.lines().stream().collect(Collectors.joining(System.getProperty("line.separator")))).orElse("");String classToTest = githubSource.lines().stream().collect(Collectors.joining(System.getProperty("line.separator")));LOGGER.debug(new PromptTemplate(this.contextWindowSize >= 16 * 1024 ?this.ollamaPrompt1 : this.ollamaPrompt, Map.of("classToTest",classToTest, "contextClasses", contextClasses, "testExample",testExample)).createMessage().getContent());LOGGER.info("Generation started with context window: {}",this.contextWindowSize);var response = chatClient.call(new PromptTemplate(this.contextWindowSize >= 16 * 1024 ? this.ollamaPrompt1 :this.ollamaPrompt, Map.of("classToTest", classToTest, "contextClasses",contextClasses, "testExample", testExample)).create());if((Instant.now().getEpochSecond() - start.getEpochSecond()) >= 300) {LOGGER.info(response.getResult().getOutput().getContent());}LOGGER.info("Prompt tokens: " +response.getMetadata().getUsage().getPromptTokens());LOGGER.info("Generation tokens: " +response.getMetadata().getUsage().getGenerationTokens());LOGGER.info("Total tokens: " +response.getMetadata().getUsage().getTotalTokens());LOGGER.info("Time in seconds: {}", (Instant.now().toEpochMilli() -start.toEpochMilli()) / 1000.0);return response.getResult().getOutput().getContent();}
为此，我们使用 “createTestSources(...) ”方法创建包含源代码行的记录。然后创建字符串 “contextClasses ”来替换提示符中的“{contextClasses}”占位符。如果上下文窗口小于 16k 字节，则字符串为空，以便为被测类和测试示例类提供足够的字节。然后创建可选的 “testExample ”字符串，以替换提示符中的“{testExample}”占位符。如果没有提供 “testUrl”，则字符串为空。然后创建 “classToTest ”字符串来替换提示符中的“{classToTest}”占位符。
调用 “chatClient ”将提示发送到人工智能/LLM。提示将根据 “contextWindowSize ”属性中上下文窗口的大小进行选择。PromptTemplate "会用准备好的字符串替换占位符。
响应 "用于记录提示标记、生成标记和总标记的数量，以便检查上下文窗口边界是否得到遵守。然后记录生成测试源的时间，并返回测试源。如果生成测试源的时间超过 5 分钟，则会记录测试源，以防止浏览器超时。
通过测试，两种模型都能生成 Spring 控制器测试和 Spring 服务测试。测试网址如下
http://localhost:8080/rest/code-generation/test?url=https://github.com/Angular2Guy/MovieManager/blob/master/backend/src/main/java/ch/xxx/moviemanager/adapter/controller/ActorController.java&testUrl=https://github.com/Angular2Guy/MovieManager/blob/master/backend/src/test/java/ch/xxx/moviemanager/adapter/controller/MovieControllerTest.javahttp://localhost:8080/rest/code-generation/test?url=https://github.com/Angular2Guy/MovieManager/blob/master/backend/src/main/java/ch/xxx/moviemanager/usecase/service/ActorService.java&testUrl=https://github.com/Angular2Guy/MovieManager/blob/master/backend/src/test/java/ch/xxx/moviemanager/usecase/service/MovieServiceTest.java
Ollama 上的 “granite-code:20b ”LLM 的上下文窗口只有 8k 个 token。这对于提供 “contextClasses ”并有足够的标记进行响应来说太小了。这意味着 LLM 只能使用被测类和测试示例。
Ollama 上的 “deepseek-coder-v2:16b ”和 “codestral:22b ”LLM 的上下文窗口超过 32k 个 token。这使得 “contextClasses”（上下文类）可以添加到提示中，从而使模型可以与思维链提示一起工作。
测试结果
测试结果
Granite-Code LLM 能够为 Spring 服务测试生成错误但有用的基础。没有一个测试成功，但缺失的部分可以用缺失的上下文类来解释。Spring Controller 测试则不太理想。它遗漏了太多代码，无法作为有用的基础。在中等配置的笔记本电脑上生成测试花费了 10 多分钟。
Deepseek-Coder-V2 LLM 能够创建一个 Spring 服务测试，其中大部分测试都能正常工作。这是一个很好的工作基础，缺失的部分很容易修复。Spring Controller 测试的错误较多，但也是一个有用的起点。在中等配置的笔记本电脑 CPU 上生成测试不到十分钟。
Codestral LLM 能够创建一个 Spring 服务测试，只有一个测试失败。这个更复杂的测试需要一些修正。Spring Controller 测试只有 1 个失败的测试用例，但这是因为缺少一个配置调用，导致测试成功而没有进行测试。两个生成的测试都是一个很好的起点。在中等配置的笔记本电脑 CPU 上生成测试花费了半个多小时。
结论
Deepseek-Coder-V2 和 Codestral LLM 可以帮助编写 Spring 应用程序的测试。Codestal 是更好的模型，但需要更多的处理能力和内存。这两个模型都需要 GPU 加速，才能用于生产。即使有可用的上下文类，LLM 也无法正确创建非琐碎代码。LLM 所能提供的帮助非常有限，因为 LLM 并不理解代码。代码对 LLM 来说只是字符，如果不了解语言语法，结果就不会令人印象深刻。开发人员必须能够修复测试中的所有错误，并添加断言等缺失部分。这意味着它只是节省了一些键入测试的时间。
Github Copilot 的使用体验与 Granite-Code LLM 类似。在 2024 年 9 月，上下文窗口太小，无法很好地生成代码，代码自动补全建议也经常被忽略。
LLM 是否有帮助 -> 是的
LLM 是否节省大量时间 -> 不是
