前提 笔者很久之前就有个想法:参考现有的主流ORM
框架的设计,造一个ORM
轮子,在基本不改变使用体验的前提下把框架依赖的大量的反射设计去掉,这些反射API
构筑的组件使用动态编译 加载的实例去替代,从而可以得到接近于直接使用原生JDBC
的性能。于是带着这样的想法,深入学习Java
的动态编译。编写本文的时候使用的是JDK11
。
基本原理 下面这个很眼熟的图来源于《深入理解Java虚拟机》前端编译与优化的章节,主要描述编译的过程:
上图看起来只有三步,其实每一步都有大量的步骤,下图尝试相对详细地描述具体的步骤(图比较大难以分割,直接放原图):
实际上,仅仅对于编译这个过程来说,开发者或者使用者不必要完全掌握其中的细节,JDK
提供了一个工具包javax.tools
让使用者可以用简易的API
进行编译(其实在大多数情况下,开发者是面向业务功能开发,像编译和打包这些细节一般直接由开发工具、Maven
、Gradle
等工具完成):
具体的使用过程包括:
获取一个javax.tools.JavaCompiler
实例。
基于Java
文件对象初始化一个编译任务javax.tools.JavaCompiler$CompilationTask
实例。
CompilationTask
实例执行结果代表着编译过程的成功与否。
我们熟知的javac
编译器其实就是JavaCompiler
接口的实现,在JDK1.6+
中,对应的实现类为com.sun.tools.javac.api.JavacTool
。
因为JVM
里面的Class
是基于ClassLoader
隔离的,所以编译成功之后可以通过自定义的类加载器加载对应的类实例,然后就可以应用反射API
进行实例化和后续的调用。
JDK动态编译 JDK
动态编译的步骤在上一节已经清楚地说明,这里造一个简单的场景。假设存在一个接口如下:
package club.throwable.compile;public interface HelloService { void sayHello (String name) ; } package club.throwable.compile;public class DefaultHelloService implements HelloService { @Override public void sayHello (String name) { System.out.println(String.format("%s say hello [by default]" , name)); } }
我们可以通过字符串SOURCE_CODE
定义一个类:
static String SOURCE_CODE = "package club.throwable.compile;\n" + "\n" + "public class JdkDynamicCompileHelloService implements HelloService{\n" + "\n" + " @Override\n" + " public void sayHello(String name) {\n" + " System.out.println(String.format(\"%s say hello [by jdk dynamic compile]\", name));\n" + " }\n" + "}" ; package club.throwable.compile;public class JdkDynamicCompileHelloService implements HelloService { @Override public void sayHello (String name) { System.out.println(String.format("%s say hello [by jdk dynamic compile]" , name)); } }
在组装编译任务实例之前,还有几项工作需要完成:
内置的JavaFileObject
标准实现SimpleJavaFileObject
是面向类源码文件,由于动态编译时候输入的是类源码文件的内容字符串,需要自行实现JavaFileObject
。
内置的JavaFileManager
是面向类路径下的Java
源码文件进行加载,这里也需要自行实现JavaFileManager
。
需要自定义一个ClassLoader
实例去加载编译出来的动态类。
实现JavaFileObject 自行实现一个JavaFileObject
,其实可以简单点直接继承SimpleJavaFileObject
,覆盖需要用到的方法即可:
public class CharSequenceJavaFileObject extends SimpleJavaFileObject { public static final String CLASS_EXTENSION = ".class" ; public static final String JAVA_EXTENSION = ".java" ; private static URI fromClassName (String className) { try { return new URI(className); } catch (URISyntaxException e) { throw new IllegalArgumentException(className, e); } } private ByteArrayOutputStream byteCode; private final CharSequence sourceCode; public CharSequenceJavaFileObject (String className, CharSequence sourceCode) { super (fromClassName(className + JAVA_EXTENSION), Kind.SOURCE); this .sourceCode = sourceCode; } public CharSequenceJavaFileObject (String fullClassName, Kind kind) { super (fromClassName(fullClassName), kind); this .sourceCode = null ; } public CharSequenceJavaFileObject (URI uri, Kind kind) { super (uri, kind); this .sourceCode = null ; } @Override public CharSequence getCharContent (boolean ignoreEncodingErrors) throws IOException { return sourceCode; } @Override public InputStream openInputStream () { return new ByteArrayInputStream(getByteCode()); } @Override public OutputStream openOutputStream () { return byteCode = new ByteArrayOutputStream(); } public byte [] getByteCode() { return byteCode.toByteArray(); } }
如果编译成功之后,直接通过自行添加的CharSequenceJavaFileObject#getByteCode()
方法即可获取目标类编译后的字节码对应的字节数组(二进制内容)。这里的CharSequenceJavaFileObject
预留了多个构造函数用于兼容原有的编译方式。
实现ClassLoader 只要简单继承ClassLoader
即可,关键是要覆盖原来的ClassLoader#findClass()
方法,用于搜索自定义的JavaFileObject
实例,从而提取对应的字节码字节数组进行装载,为了实现这一点可以添加一个哈希表作为缓存,键-值分别是全类名的别名(xx.yy.MyClass
形式,而非URI
模式)和目标类对应的JavaFileObject
实例。
public class JdkDynamicCompileClassLoader extends ClassLoader { public static final String CLASS_EXTENSION = ".class" ; private final Map<String, JavaFileObject> javaFileObjectMap = Maps.newConcurrentMap(); public JdkDynamicCompileClassLoader (ClassLoader parentClassLoader) { super (parentClassLoader); } @Override protected Class<?> findClass(String name) throws ClassNotFoundException { JavaFileObject javaFileObject = javaFileObjectMap.get(name); if (null != javaFileObject) { CharSequenceJavaFileObject charSequenceJavaFileObject = (CharSequenceJavaFileObject) javaFileObject; byte [] byteCode = charSequenceJavaFileObject.getByteCode(); return defineClass(name, byteCode, 0 , byteCode.length); } return super .findClass(name); } @Nullable @Override public InputStream getResourceAsStream (String name) { if (name.endsWith(CLASS_EXTENSION)) { String qualifiedClassName = name.substring(0 , name.length() - CLASS_EXTENSION.length()).replace('/' , '.' ); CharSequenceJavaFileObject javaFileObject = (CharSequenceJavaFileObject) javaFileObjectMap.get(qualifiedClassName); if (null != javaFileObject && null != javaFileObject.getByteCode()) { return new ByteArrayInputStream(javaFileObject.getByteCode()); } } return super .getResourceAsStream(name); } void addJavaFileObject (String qualifiedClassName, JavaFileObject javaFileObject) { javaFileObjectMap.put(qualifiedClassName, javaFileObject); } Collection<JavaFileObject> listJavaFileObject () { return Collections.unmodifiableCollection(javaFileObjectMap.values()); } }
实现JavaFileManager JavaFileManager
是Java
文件的抽象管理器,它用于管理常规的Java
文件,但是不局限于文件,也可以管理其他来源的Java
类文件数据。下面就通过实现一个自定义的JavaFileManager
用于管理字符串类型的源代码。为了简单起见,可以直接继承已经存在的ForwardingJavaFileManager
:
public class JdkDynamicCompileJavaFileManager extends ForwardingJavaFileManager <JavaFileManager > { private final JdkDynamicCompileClassLoader classLoader; private final Map<URI, JavaFileObject> javaFileObjectMap = Maps.newConcurrentMap(); public JdkDynamicCompileJavaFileManager (JavaFileManager fileManager, JdkDynamicCompileClassLoader classLoader) { super (fileManager); this .classLoader = classLoader; } private static URI fromLocation (Location location, String packageName, String relativeName) { try { return new URI(location.getName() + '/' + packageName + '/' + relativeName); } catch (URISyntaxException e) { throw new IllegalArgumentException(e); } } @Override public FileObject getFileForInput (Location location, String packageName, String relativeName) throws IOException { JavaFileObject javaFileObject = javaFileObjectMap.get(fromLocation(location, packageName, relativeName)); if (null != javaFileObject) { return javaFileObject; } return super .getFileForInput(location, packageName, relativeName); } @Override public JavaFileObject getJavaFileForOutput (Location location, String className, JavaFileObject.Kind kind, FileObject sibling) throws IOException { JavaFileObject javaFileObject = new CharSequenceJavaFileObject(className, kind); classLoader.addJavaFileObject(className, javaFileObject); return javaFileObject; } @Override public ClassLoader getClassLoader (Location location) { return classLoader; } @Override public String inferBinaryName (Location location, JavaFileObject file) { if (file instanceof CharSequenceJavaFileObject) { return file.getName(); } return super .inferBinaryName(location, file); } @Override public Iterable<JavaFileObject> list (Location location, String packageName, Set<JavaFileObject.Kind> kinds, boolean recurse) throws IOException { Iterable<JavaFileObject> superResult = super .list(location, packageName, kinds, recurse); List<JavaFileObject> result = Lists.newArrayList(); if (location == StandardLocation.CLASS_PATH && kinds.contains(JavaFileObject.Kind.CLASS)) { for (JavaFileObject file : javaFileObjectMap.values()) { if (file.getKind() == JavaFileObject.Kind.CLASS && file.getName().startsWith(packageName)) { result.add(file); } } result.addAll(classLoader.listJavaFileObject()); } else if (location == StandardLocation.SOURCE_PATH && kinds.contains(JavaFileObject.Kind.SOURCE)) { for (JavaFileObject file : javaFileObjectMap.values()) { if (file.getKind() == JavaFileObject.Kind.SOURCE && file.getName().startsWith(packageName)) { result.add(file); } } } for (JavaFileObject javaFileObject : superResult) { result.add(javaFileObject); } return result; } public void addJavaFileObject (Location location, String packageName, String relativeName, JavaFileObject javaFileObject) { javaFileObjectMap.put(fromLocation(location, packageName, relativeName), javaFileObject); } }
注意在这个类中引入了自定义类加载器JdkDynamicCompileClassLoader
,目的是为了实现JavaFileObject
实例的共享以及为文件管理器提供类加载器实例。
动态编译和运行 前置准备工作完成,我们可以通过JavaCompiler
去编译这个前面提到的字符串,为了字节码的兼容性更好,编译的时候可以指定稍低的JDK
版本例如1.6
:
public class Client { static String SOURCE_CODE = "package club.throwable.compile;\n" + "\n" + "public class JdkDynamicCompileHelloService implements HelloService{\n" + "\n" + " @Override\n" + " public void sayHello(String name) {\n" + " System.out.println(String.format(\"%s say hello [by jdk dynamic compile]\", name));\n" + " }\n" + "}" ; static DiagnosticCollector<JavaFileObject> DIAGNOSTIC_COLLECTOR = new DiagnosticCollector<>(); public static void main (String[] args) throws Exception { JavaCompiler compiler = ToolProvider.getSystemJavaCompiler(); List<String> options = new ArrayList<>(); options.add("-source" ); options.add("1.6" ); options.add("-target" ); options.add("1.6" ); StandardJavaFileManager manager = compiler.getStandardFileManager(DIAGNOSTIC_COLLECTOR, null , null ); JdkDynamicCompileClassLoader classLoader = new JdkDynamicCompileClassLoader(Thread.currentThread().getContextClassLoader()); JdkDynamicCompileJavaFileManager fileManager = new JdkDynamicCompileJavaFileManager(manager, classLoader); String packageName = "club.throwable.compile" ; String className = "JdkDynamicCompileHelloService" ; String qualifiedName = packageName + "." + className; CharSequenceJavaFileObject javaFileObject = new CharSequenceJavaFileObject(className, SOURCE_CODE); fileManager.addJavaFileObject( StandardLocation.SOURCE_PATH, packageName, className + CharSequenceJavaFileObject.JAVA_EXTENSION, javaFileObject ); JavaCompiler.CompilationTask compilationTask = compiler.getTask( null , fileManager, DIAGNOSTIC_COLLECTOR, options, null , Lists.newArrayList(javaFileObject) ); Boolean result = compilationTask.call(); System.out.println(String.format("编译[%s]结果:%s" , qualifiedName, result)); Class<?> klass = classLoader.loadClass(qualifiedName); HelloService instance = (HelloService) klass.getDeclaredConstructor().newInstance(); instance.sayHello("throwable" ); } }
输出结果如下:
编译[club.throwable.compile.JdkDynamicCompileHelloService]结果:true throwable say hello [by jdk dynamic compile]
可见通过了字符串的类源码,实现了动态编译、类加载、反射实例化以及最终的方法调用。另外,编译过程的诊断信息可以通过DiagnosticCollector
实例获取。为了复用,这里可以把JDK
动态编译的过程抽取到一个方法中:
public final class JdkCompiler { static DiagnosticCollector<JavaFileObject> DIAGNOSTIC_COLLECTOR = new DiagnosticCollector<>(); @SuppressWarnings("unchecked") public static <T> T compile (String packageName, String className, String sourceCode, Class<?>[] constructorParamTypes, Object[] constructorParams) throws Exception { JavaCompiler compiler = ToolProvider.getSystemJavaCompiler(); List<String> options = new ArrayList<>(); options.add("-source" ); options.add("1.6" ); options.add("-target" ); options.add("1.6" ); StandardJavaFileManager manager = compiler.getStandardFileManager(DIAGNOSTIC_COLLECTOR, null , null ); JdkDynamicCompileClassLoader classLoader = new JdkDynamicCompileClassLoader(Thread.currentThread().getContextClassLoader()); JdkDynamicCompileJavaFileManager fileManager = new JdkDynamicCompileJavaFileManager(manager, classLoader); String qualifiedName = packageName + "." + className; CharSequenceJavaFileObject javaFileObject = new CharSequenceJavaFileObject(className, sourceCode); fileManager.addJavaFileObject( StandardLocation.SOURCE_PATH, packageName, className + CharSequenceJavaFileObject.JAVA_EXTENSION, javaFileObject ); JavaCompiler.CompilationTask compilationTask = compiler.getTask( null , fileManager, DIAGNOSTIC_COLLECTOR, options, null , Lists.newArrayList(javaFileObject) ); Boolean result = compilationTask.call(); System.out.println(String.format("编译[%s]结果:%s" , qualifiedName, result)); Class<?> klass = classLoader.loadClass(qualifiedName); return (T) klass.getDeclaredConstructor(constructorParamTypes).newInstance(constructorParams); } }
Javassist动态编译 既然有JDK
的动态编译,为什么还存在Javassist
这样的字节码增强工具?撇开性能或者效率层面,JDK
动态编译存在比较大的局限性,比较明显的一点就是无法完成字节码插桩,换言之就是无法基于原有的类和方法进行修饰或者增强,但是Javassist
可以做到。再者,Javassist
提供的API
和JDK
反射的API
十分相近,如果反射平时用得比较熟练,Javassist
的上手也就变得比较简单。这里仅仅列举一个增强前面提到的DefaultHelloService
的例子,先引入依赖:
<dependency > <groupId > org.javassist</groupId > <artifactId > javassist</artifactId > <version > 3.27.0-GA</version > </dependency >
编码如下:
public class JavassistClient { public static void main (String[] args) throws Exception { ClassPool pool = ClassPool.getDefault(); CtClass cc = pool.get("club.throwable.compile.DefaultHelloService" ); CtMethod ctMethod = cc.getDeclaredMethod("sayHello" , new CtClass[]{pool.get("java.lang.String" )}); ctMethod.insertBefore("System.out.println(\"insert before by Javassist\");" ); ctMethod.insertAfter("System.out.println(\"insert after by Javassist\");" ); Class<?> klass = cc.toClass(); System.out.println(klass.getName()); HelloService helloService = (HelloService) klass.getDeclaredConstructor().newInstance(); helloService.sayHello("throwable" ); } }
输出结果如下:
club.throwable.compile.DefaultHelloService insert before by Javassist throwable say hello [by default] insert after by Javassist
Javaassist
这个单词其实是Java
和Assist
两个单词拼接在一起,意为Java
助手,是一个Java
字节码增强类库:
可以基于已经存在的类进行字节码增强,例如修改已经存在的方法、变量,甚至是直接在原有的类中添加新的方法等。
可以完全像积木拼接一样,动态拼出一个全新的类。
不像ASM
(ASM
的学习曲线比较陡峭,属于相对底层的字节码操作类库,当然从性能上来看ASM
对字节码增强的效率远高于其他高层次封装的框架)那样需要对字节码编程十分了解,Javaassist
降低了字节码增强功能的入门难度。
进阶例子 现在定义一个接口MysqlInfoMapper
,用于动态执行一条已知的SQL
,很简单,就是查询MySQL
的系统表mysql
里面的用户信息SELECT Host,User FROM mysql.user
:
@Data public class MysqlUser { private String host; private String user; } public interface MysqlInfoMapper { List<MysqlUser> selectAllMysqlUsers () ; }
假设现在只提供一个MySQL
的驱动包(mysql:mysql-connector-java:jar:8.0.20
),暂时不能依赖任何高层次的框架,要动态实现MysqlInfoMapper
接口,优先整理需要的组件:
需要一个连接管理器去管理MySQL
的连接。
需要一个SQL
执行器用于执行查询SQL
。
需要一个结果处理器去提取和转换查询结果。
为了简单起见,笔者在定义这三个组件接口的时候顺便在接口中通过单例进行实现(部分配置完全写死):
public interface ConnectionManager { String USER_NAME = "root" ; String PASS_WORD = "root" ; String URL = "jdbc:mysql://localhost:3306/mysql?useUnicode=true&characterEncoding=utf-8&serverTimezone=GMT%2B8&useSSL=false" ; Connection newConnection () throws SQLException ; void closeConnection (Connection connection) ; ConnectionManager X = new ConnectionManager() { @Override public Connection newConnection () throws SQLException { return DriverManager.getConnection(URL, USER_NAME, PASS_WORD); } @Override public void closeConnection (Connection connection) { try { connection.close(); } catch (Exception ignore) { } } }; } public interface SqlExecutor { ResultSet execute (Connection connection, String sql) throws SQLException ; SqlExecutor X = new SqlExecutor() { @Override public ResultSet execute (Connection connection, String sql) throws SQLException { Statement statement = connection.createStatement(); statement.execute(sql); return statement.getResultSet(); } }; } public interface ResultHandler <T > { T handleResultSet (ResultSet resultSet) throws SQLException ; ResultHandler<List<MysqlUser>> X = new ResultHandler<List<MysqlUser>>() { @Override public List<MysqlUser> handleResultSet (ResultSet resultSet) throws SQLException { try { List<MysqlUser> result = Lists.newArrayList(); while (resultSet.next()) { MysqlUser item = new MysqlUser(); item.setHost(resultSet.getString("Host" )); item.setUser(resultSet.getString("User" )); result.add(item); } return result; } finally { resultSet.close(); } } }; }
接着需要动态编译MysqlInfoMapper
的实现类,它的源文件的字符串内容如下(注意不要在类路径下新建这个DefaultMysqlInfoMapper
类):
package club.throwable.compile;import java.sql.Connection;import java.sql.ResultSet;import java.util.List;public class DefaultMysqlInfoMapper implements MysqlInfoMapper { private final ConnectionManager connectionManager; private final SqlExecutor sqlExecutor; private final ResultHandler resultHandler; private final String sql; public DefaultMysqlInfoMapper (ConnectionManager connectionManager, SqlExecutor sqlExecutor, ResultHandler resultHandler, String sql) { this .connectionManager = connectionManager; this .sqlExecutor = sqlExecutor; this .resultHandler = resultHandler; this .sql = sql; } @Override public List<MysqlUser> selectAllMysqlUsers () { try { Connection connection = connectionManager.newConnection(); try { ResultSet resultSet = sqlExecutor.execute(connection, sql); return (List<MysqlUser>) resultHandler.handleResultSet(resultSet); } finally { connectionManager.closeConnection(connection); } } catch (Exception e) { throw new IllegalStateException(e); } } }
然后编写一个客户端进行动态编译和执行:
public class MysqlInfoClient { static String SOURCE_CODE = "package club.throwable.compile;\n" + "import java.sql.Connection;\n" + "import java.sql.ResultSet;\n" + "import java.util.List;\n" + "\n" + "public class DefaultMysqlInfoMapper implements MysqlInfoMapper {\n" + "\n" + " private final ConnectionManager connectionManager;\n" + " private final SqlExecutor sqlExecutor;\n" + " private final ResultHandler resultHandler;\n" + " private final String sql;\n" + "\n" + " public DefaultMysqlInfoMapper(ConnectionManager connectionManager,\n" + " SqlExecutor sqlExecutor,\n" + " ResultHandler resultHandler,\n" + " String sql) {\n" + " this.connectionManager = connectionManager;\n" + " this.sqlExecutor = sqlExecutor;\n" + " this.resultHandler = resultHandler;\n" + " this.sql = sql;\n" + " }\n" + "\n" + " @Override\n" + " public List<MysqlUser> selectAllMysqlUsers() {\n" + " try {\n" + " Connection connection = connectionManager.newConnection();\n" + " try {\n" + " ResultSet resultSet = sqlExecutor.execute(connection, sql);\n" + " return (List<MysqlUser>) resultHandler.handleResultSet(resultSet);\n" + " } finally {\n" + " connectionManager.closeConnection(connection);\n" + " }\n" + " } catch (Exception e) {\n" + " // 暂时忽略异常处理,统一封装为IllegalStateException\n" + " throw new IllegalStateException(e);\n" + " }\n" + " }\n" + "}\n" ; static String SQL = "SELECT Host,User FROM mysql.user" ; public static void main (String[] args) throws Exception { MysqlInfoMapper mysqlInfoMapper = JdkCompiler.compile( "club.throwable.compile" , "DefaultMysqlInfoMapper" , SOURCE_CODE, new Class[]{ConnectionManager.class, SqlExecutor.class, ResultHandler.class, String.class}, new Object[]{ConnectionManager.X, SqlExecutor.X, ResultHandler.X, SQL}); System.out.println(JSON.toJSONString(mysqlInfoMapper.selectAllMysqlUsers())); } }
最终的输出结果是:
编译[club.throwable.compile.DefaultMysqlInfoMapper]结果:true [{ "host": "%", "user": "canal" }, { "host": "%", "user": "doge" }, { "host": "localhost", "user": "mysql.infoschema" }, { "host": "localhost", "user": "mysql.session" }, { "host": "localhost", "user": "mysql.sys" }, { "host": "localhost", "user": "root" }]
然后笔者查看本地安装的MySQL
中的结果,验证该查询结果是正确的。
这里笔者为了简化整个例子,没有在MysqlInfoMapper#selectAllMysqlUsers()
方法中添加查询参数,可以尝试一下查询的SQL
是SELECT Host,User FROM mysql.user WHERE User = 'xxx'
场景下的编码实现。
如果把动态实现的DefaultMysqlInfoMapper
注册到IOC
容器中,就可以实现MysqlInfoMapper
按照类型自动装配。 如果把SQL
和参数处理可以抽离到单独的文件中,并且实现一个对应的文件解析器,那么就可以把类文件和SQL
隔离,Mybatis
和Hibernate
都是这样做的。
小结 动态编译或者更底层的面向字节码层面的编程,其实是一个十分有挑战性但是可以创造无限可能的领域,本文只是简单分析了一下Java
源码编译的过程,并且通过一些简单的例子进行动态编译的模拟,离使用于实际应用中还有不少距离,后面需要花更多的时间去分析一下相关领域的知识。
参考资料:
(本文完 c-4-d e-a-20200606 0:23 r-a-20200718 封面来源于《龙与虎》)