使用ANTLR处理文本

引用

使用 Antlr 处理文本
https://www.ibm.com/developerworks/cn/java/j-lo-antlrtext/index.html
该文章写的非常好,无耐是2011年写的,与现有的antlr版本差别较大,编译不过去,编译过去,也测试不出来正确的结果,以下为用antlr4.2重写的

新项目使用maven和ant构建,需要以下几个文件

• pom.xml
• build.xml
• SqlExtrator.g4语法文件
• SqlExtrator.clj测文件
• Test.java 测试代码

测试方法,

1. 先用ant执行compile任务,生成和编译生成的一堆词法解析器和语法解析器代码,
然后执行ant的test任务,解析SqlExtrator.clj测文件里的文本

2. 使有Test.java,手动编程调用

使用ant任务的截图,






pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>com.xxx.lang</groupId>
	<artifactId>fieldTypeUpdate</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<packaging>jar</packaging>

	<name>fieldTypeUpdate</name>
	<url>http://maven.apache.org</url>


	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
	</properties>

	<dependencies>
		<dependency>
			<groupId>junit</groupId>
			<artifactId>junit</artifactId>
			<version>3.8.1</version>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>org.antlr</groupId>
			<artifactId>antlr4</artifactId>
			<version>4.2</version>
		</dependency>
	</dependencies>

	<build>
		<plugins>
			<plugin>
				<groupId>org.codehaus.mojo</groupId>
				<artifactId>build-helper-maven-plugin</artifactId>
				<version>1.8</version>
				<executions>
					<execution>
						<id>add-source</id>
						<phase>generate-sources</phase>
						<goals>
							<goal>add-source</goal>
						</goals>
						<configuration>
							<sources>
								<source>src/generated/java</source>
							</sources>
						</configuration>
					</execution>
				</executions>
			</plugin>
		</plugins>
	</build>

</project>

build.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <project basedir="." default="test" name="mylang">
    <property environment="env"/>
    <property name="debuglevel" value="source,lines,vars"/>
    <property name="target" value="1.8"/>
    <property name="source" value="1.8"/>
    <property name="language" value="sqlExtrator"/>
    <path id="mylang.classpath">
    	<pathelement location="lib/antlr-2.7.7.jar"/>
		<pathelement location="lib/antlr-runtime-3.5.jar"/>
		<pathelement location="lib/antlr4-4.2.jar"/>
		<pathelement location="lib/antlr4-annotations-4.2.jar"/>
		<pathelement location="lib/antlr4-runtime-4.2.jar"/>
		<pathelement location="lib/junit-3.8.1.jar"/>
		<pathelement location="lib/org.abego.treelayout.core-1.0.1.jar"/>
		<pathelement location="lib/ST4-4.0.7.jar"/>
		<pathelement location="lib/stringtemplate-3.2.1.jar"/>
    </path>
    
    <path id="antlr.classpath">
        <pathelement location="antlr-4.7.1-complete.jar"/>
    </path>
    
    <path id="compile.path">
    	 <pathelement location="target/classes"/>
    </path>
    

    
    <target name="clean">
        <delete dir="target"></delete>
    	<delete dir="src/main/java/com/xxx/lang/mylang/${language}"></delete>
    </target>
    
    <target depends="clean" name="gen">
        <echo message="generate java from g4 file"/>
       <java classname="org.antlr.v4.Tool" fork="yes" failonerror="true">
    				<classpath refid="mylang.classpath"/>
    				<arg value="src/main/resources/SqlExtrator.g4"/>
    				<arg line="-package "/>
    				<arg value="com.xxx.lang.mylang.${language}"/>
    				<arg line="-o "/>
    				<arg value="src/main/java/com/xxx/lang/mylang/${language}/"/>
       				<arg value="-visitor"/>
       				<arg value="-no-listener"/>
       				<arg value="-encoding"/>
       				<arg value="UTF-8"/>
    			</java>
    </target>
    
    <target depends="gen" name="compile">
        <echo message="compile generate java file"/>
         <mkdir dir="target/classes"/>
        <javac debug="true" debuglevel="${debuglevel}" destdir="target/classes" includeantruntime="false" source="${source}" target="${target}">
            <src path="src/main/java"/>
        	<compilerarg line="-encoding UTF-8 "/>
            <classpath refid="mylang.classpath"/>
        </javac>
    </target>

    	
    
    <target name="test"  description="Run the main class" >
    			<java classname="org.antlr.v4.gui.TestRig" fork="yes" failonerror="true">
    				<classpath refid="antlr.classpath"/>
    				<classpath refid="compile.path"/>
    				<sysproperty key="file.encoding" value="UTF-8"/>
    				<arg value="com.xxx.lang.mylang.${language}.SqlExtrator"></arg>
    				<arg value="sql"></arg>
    				<arg value="-gui"></arg>
    				<arg value="src/test/java/SqlExtrator.clj"></arg>
    			</java>
    </target>
    	

</project>

SqlExtrator.g4 语法文件 该语法文件,仅可以识别词法规定的字符,词法外的字符将会报错

	

grammar SqlExtrator; 


WS : (' ' |'\t' |'\r' |'\n' )+  ; 
 
INT: '0'..'9' + ;   
  
ID : ('a'..'z' |'A'..'Z' |'_' ) ('a'..'z' |'A'..'Z' |'_' |'0'..'9' )*;

EOL: ('\n' | '\r' | '\r\n')*;

SUCCESS:'DB20000I  The SQL command completed successfully.'EOL  ; 

SqlFrg :'INSERT INTO SYSA.' ID '(' ID ',' ID ')' WS 'VALUES' '(\'' ID '\',\'' INT '\')'EOL ;
 

txt:mysql=SqlFrg {System.out.println($mysql.text);} SUCCESS;

sql:(txt)+;

第二个版本的语法,添加了:
FILTER: .? -> skip;
仅这一行,这行代码,使用正则的非贪婪匹配规则,

引用

Wildcard Operator and Nongreedy Subrules

正则表达式贪婪与非贪婪模式

1.什么是正则表达式的贪婪与非贪婪匹配

　　如：String str="abcaxc";

　　　　Patter p="ab.*c";

　　贪婪匹配:正则表达式一般趋向于最大长度匹配，也就是所谓的贪婪匹配。如上面使用模式p匹配字符串str，结果就是匹配到：abcaxc(ab.*c)。

　　非贪婪匹配：就是匹配到结果就好，就少的匹配字符。如上面使用模式p匹配字符串str，结果就是匹配到：abc(ab.*c)。

2.编程中如何区分两种模式

　　默认是贪婪模式；在量词后面直接加上一个问号？就是非贪婪模式。

　　量词：{m,n}：m到n个

　　　　　*：任意多个

　　　　　+：一个到多个

　　　　　？：0或一个

	
grammar SqlExtrator; 



SqlFrg :'INSERT INTO SYSA.' ID '(' ID ',' ID ')' WS 'VALUES' '(\'' ID '\',\'' INT '\')' ;
 


fragment WS : (' ' |'\t' |'\r' |'\n' )+  ; 
 
fragment ID: ('a'..'z' |'A'..'Z' |'_' ) ('a'..'z' |'A'..'Z' |'_' |'0'..'9' )*; 

fragment INT: '0'..'9' + ;   

fragment EOL: '\n' | '\r' | '\r\n';

 SUCCESS:'DB20000I  The SQL command completed successfully.' ;



all: (SqlFrg    SUCCESS  {System.out.println($SqlFrg.text);})+ ; 

FILTER: .? -> skip;

SqlExtrator.clj 测试文件

INSERT INTO SYSA.IF_EMPUSRRLA(USRNUM,EMPNUM) VALUES('U037508','275159') 
DB20000I  The SQL command completed successfully. 

document.write(v+' test is '+result+'</br>');//该行代码在第一个版本的语法中会报错

INSERT INTO SYSA.IF_USRSTNRLA(USRNUM,STNNUM) VALUES('U037710','00026') 
DB20000I  The SQL command completed successfully. 

Test.java 测试代码

public class Test {

	public static void main(String[] args)  {
		
		 try {
			String filename = "D:\\workplace\\fieldTypeUpdate\\src\\test\\java\\SqlExtrator.clj"; 
			 InputStream in = new FileInputStream(filename); 
			 ANTLRInputStream input = new ANTLRInputStream(in); 

			 SqlExtratorLexer lexer = new SqlExtratorLexer(input); 
			 
			 CommonTokenStream tokens = new CommonTokenStream(lexer); 
			 
			 SqlExtratorParser parser = new SqlExtratorParser(tokens);
			
			 parser.sql();
			System.out.println("done!");
		} catch (FileNotFoundException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
		
	}

}

测试结果控制台输出:

引用

INSERT INTO SYSA.IF_EMPUSRRLA(USRNUM,EMPNUM) VALUES('U037508','275159')

INSERT INTO SYSA.IF_USRSTNRLA(USRNUM,STNNUM) VALUES('U037710','00026')

done!