I've included some examples that may not exactly fit your use case, but show some ways to write code that is a bit easier to maintain and read.
(In answer to: http://stackoverflow.com/questions/19402699/java-unable-to-read-full-file/19417882#19417882)
Code that is hard to read, it hard to debug and maintain.
- Objects need to validate their input (constructor args).
- Reject bad data. Fail fast not later when it is harder to debug.
- Never catch exceptions unless you can recover. Either soften it (wrap in a runtime and re-throw it), or add it to your throws clause. If you don't know what to do with it, do nothing.
- Your code holds on to state it does not need.
- Classes are a bit more self describing than two arrays of gak.
- Avoid public vars. Unless they are final. Protect the state of your objects.
- Think about how your methods are going to get called, and avoid side effects. Calling readData twice would cause weird hard-to-debug side effects
- Memory is cheap but not free
- If you open it, you have to close it
Java 7 and 8 allow you to read lines from the FileSystem so there is no need to write most of this code to start with:
Path thePath = FileSystems.getDefault().getPath(location);
return Files.readAllLines(thePath, Charset.forName("UTF-8"));
If you had to read a lot of small files into lines and did not want to use the FileSystem or you were using Java 6 or Java 5, then you would create a utility class as follows:
public class IOUtils {
public final static String CHARSET = "UTF-8";
...
public static List<String> readLines(File file) {
try (FileReader reader = new FileReader(file)) {
return readLines(reader);
} catch (Exception ex) {
return Exceptions.handle(List.class, ex);
}
}
which calls:
public static List<String> readLines(Reader reader) {
try (BufferedReader bufferedReader = new BufferedReader(reader)) {
return readLines(bufferedReader);
} catch (Exception ex) {
return Exceptions.handle(List.class, ex);
}
}
which calls:
public static List<String> readLines(BufferedReader reader) {
List<String> lines = new ArrayList<>(80);
try (BufferedReader bufferedReader = reader) {
String line = null;
while ( (line = bufferedReader.readLine()) != null) {
lines.add(line);
}
} catch (Exception ex) {
return Exceptions.handle(List.class, ex);
}
return lines;
}
Apache has a set of utilities call Apache commons. It includes lang and it includes IO utils. Either of these would be good to have if you are using Java 5 or Java 6.
Going back to our example, you could turn any location into a list of lines:
public static List<String> readLines(String location) {
URI uri = URI.create(location);
try {
if ( uri.getScheme()==null ) {
Path thePath = FileSystems.getDefault().getPath(location);
return Files.readAllLines(thePath, Charset.forName("UTF-8"));
} else if ( uri.getScheme().equals("file") ) {
Path thePath = FileSystems.getDefault().getPath(uri.getPath());
return Files.readAllLines(thePath, Charset.forName("UTF-8"));
} else {
return readLines(location, uri);
}
} catch (Exception ex) {
return Exceptions.handle(List.class, ex);
}
}
FileSystem, Path, URI, etc. are all in the JDK.
Continuing the example:
private static List<String> readLines(String location, URI uri) throws Exception {
try {
FileSystem fileSystem = FileSystems.getFileSystem(uri);
Path fsPath = fileSystem.getPath(location);
return Files.readAllLines(fsPath, Charset.forName("UTF-8"));
} catch (ProviderNotFoundException ex) {
return readLines(uri.toURL().openStream());
}
}
The above tries to read the uri from the FileSystem and if it can't load it, then it looks it up via a URL stream. URL, URI, File, FileSystem, etc. are all part of the JDK.
To turn the URL stream into a Reader and then into a string we use:
public static List<String> readLines(InputStream is) {
try (Reader reader = new InputStreamReader(is, CHARSET)) {
return readLines(reader);
} catch (Exception ex) {
return Exceptions.handle(List.class, ex);
}
}
:)
Now let's return to our example (we can now read lines from anywhere including files):
public static final class Proxy {
private final String address;
private final int port;
private static final String DATA_FILE = "./files/proxy.txt";
private static final Pattern addressPattern = Pattern.compile("^(\\d{1,3}[.]{1}){3}[0-9]{1,3}$");
private Proxy(String address, int port) {
/* Validate address in not null.*/
Objects.requireNonNull(address, "address should not be null");
/* Validate port is in range. */
if (port < 1 || port > 65535) {
throw new IllegalArgumentException("Port is not in range port=" + port);
}
/* Validate address is of the form 123.12.1.5 .*/
if (!addressPattern.matcher(address).matches()) {
throw new IllegalArgumentException("Invalid Inet address");
}
/* Now initialize our address and port. */
this.address = address;
this.port = port;
}
private static Proxy createProxy(String line) {
String[] lineSplit = line.split(":");
String address = lineSplit[0];
int port = parseInt(lineSplit[1]);
return new Proxy(address, port);
}
public final String getAddress() {
return address;
}
public final int getPort() {
return port;
}
public static List<Proxy> loadProxies() {
List <String> lines = IOUtils.readLines(DATA_FILE);
List<Proxy> proxyList = new ArrayList<>(lines.size());
for (String line : lines) {
proxyList.add(createProxy(line));
}
return proxyList;
}
}
Notice we don't have any immutable state. This prevents bugs. And it makes your code easier to debug and support.
Notice our IOUtils.readLines which reads the lines from the file system.
Notice the extra work in the constructor to make sure no one initializes an instance of Proxy with bad state. These are all in the JDK Objects, Pattern, etc.
If you wanted a ProxyLoader that was reusable it would look something like this:
public static class ProxyLoader {
private static final String DATA_FILE = "./files/proxy.txt";
private List<Proxy> proxyList = Collections.EMPTY_LIST;
private final String dataFile;
public ProxyLoader() {
this.dataFile = DATA_FILE;
init();
}
public ProxyLoader(String dataFile) {
this.dataFile = DATA_FILE;
init();
}
private void init() {
List <String> lines = IO.readLines(dataFile);
proxyList = new ArrayList<>(lines.size());
for (String line : lines) {
proxyList.add(Proxy.createProxy(line));
}
}
public String getDataFile() {
return this.dataFile;
}
public static List<Proxy> loadProxies() {
return new ProxyLoader().getProxyList();
}
public List<Proxy> getProxyList() {
return proxyList;
}
}
public static class Proxy {
private final String address;
private final int port;
...
public Proxy(String address, int port) {
...
this.address = address;
this.port = port;
}
public static Proxy createProxy(String line) {
String[] lineSplit = line.split(":");
String address = lineSplit[0];
int port = parseInt(lineSplit[1]);
return new Proxy(address, port);
}
public String getAddress() {
return address;
}
public int getPort() {
return port;
}
}
Coding is great. Testing is divine!
public static class ProxyLoader {
private static final String DATA_FILE = "./files/proxy.txt";
private List<Proxy> proxyList = Collections.EMPTY_LIST;
private final String dataFile;
public ProxyLoader() {
this.dataFile = DATA_FILE;
init();
}
public ProxyLoader(String dataFile) {
this.dataFile = DATA_FILE;
init();
}
private void init() {
List <String> lines = IO.readLines(dataFile);
proxyList = new ArrayList<>(lines.size());
for (String line : lines) {
proxyList.add(Proxy.createProxy(line));
}
}
public String getDataFile() {
return this.dataFile;
}
public static List<Proxy> loadProxies() {
return new ProxyLoader().getProxyList();
}
public List<Proxy> getProxyList() {
return proxyList;
}
}
public static class Proxy {
private final String address;
private final int port;
public Proxy(String address, int port) {
this.address = address;
this.port = port;
}
public static Proxy createProxy(String line) {
String[] lineSplit = line.split(":");
String address = lineSplit[0];
int port = parseInt(lineSplit[1]);
return new Proxy(address, port);
}
public String getAddress() {
return address;
}
public int getPort() {
return port;
}
}
public static final class Proxy2 {
private final String address;
private final int port;
private static final String DATA_FILE = "./files/proxy.txt";
private static final Pattern addressPattern = Pattern.compile("^(\\d{1,3}[.]{1}){3}[0-9]{1,3}$");
private Proxy2(String address, int port) {
/* Validate address in not null.*/
Objects.requireNonNull(address, "address should not be null");
/* Validate port is in range. */
if (port < 1 || port > 65535) {
throw new IllegalArgumentException("Port is not in range port=" + port);
}
/* Validate address is of the form 123.12.1.5 .*/
if (!addressPattern.matcher(address).matches()) {
throw new IllegalArgumentException("Invalid Inet address");
}
/* Now initialize our address and port. */
this.address = address;
this.port = port;
}
private static Proxy2 createProxy(String line) {
String[] lineSplit = line.split(":");
String address = lineSplit[0];
int port = parseInt(lineSplit[1]);
return new Proxy2(address, port);
}
public final String getAddress() {
return address;
}
public final int getPort() {
return port;
}
public static List<Proxy2> loadProxies() {
List <String> lines = IO.readLines(DATA_FILE);
List<Proxy2> proxyList = new ArrayList<>(lines.size());
for (String line : lines) {
proxyList.add(createProxy(line));
}
return proxyList;
}
}
@Test public void proxyTest() {
List<Proxy> proxyList = ProxyLoader.loadProxies();
assertEquals(
5, len(proxyList)
);
assertEquals(
"127.0.0.1", idx(proxyList, 0).getAddress()
);
assertEquals(
8080, idx(proxyList, 0).getPort()
);
//192.55.55.57:9091
assertEquals(
"192.55.55.57", idx(proxyList, -1).getAddress()
);
assertEquals(
9091, idx(proxyList, -1).getPort()
);
}
@Test public void proxyTest2() {
List<Proxy2> proxyList = Proxy2.loadProxies();
assertEquals(
5, len(proxyList)
);
assertEquals(
"127.0.0.1", idx(proxyList, 0).getAddress()
);
assertEquals(
8080, idx(proxyList, 0).getPort()
);
//192.55.55.57:9091
assertEquals(
"192.55.55.57", idx(proxyList, -1).getAddress()
);
assertEquals(
9091, idx(proxyList, -1).getPort()
);
}
My input file
127.0.0.1:8080
192.55.55.55:9090
127.0.0.2:8080
192.55.55.56:9090
192.55.55.57:9091
And what about my IOUtils (which is actually called IO):
Here is the test for those who care for the IO (utils):
package org.boon.utils;
import com.sun.net.httpserver.Headers;
import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpHandler;
import com.sun.net.httpserver.HttpServer;
import org.junit.Test;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.net.URI;
import java.util.*;
import java.util.regex.Pattern;
import static javax.lang.Integer.parseInt;
import static org.boon.utils.Lists.idx;
import static org.boon.utils.Lists.len;
import static org.boon.utils.Maps.copy;
import static org.boon.utils.Maps.map;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
public class IOTest {
@Test
public void testReadLines() {
File testDir = new File("src/test/resources");
File testFile = new File(testDir, "testfile.txt");
List<String> lines = IO.readLines(testFile);
assertLines(lines);
}
private void assertLines(List<String> lines) {
assertEquals(
4, len(lines)
);
assertEquals(
"line 1", idx(lines, 0)
);
assertEquals(
"grapes", idx(lines, 3)
);
}
@Test
public void testReadLinesFromPath() {
List<String> lines = IO.readLines("src/test/resources/testfile.txt");
assertLines(lines);
}
@Test
public void testReadLinesURI() {
File testDir = new File("src/test/resources");
File testFile = new File(testDir, "testfile.txt");
URI uri = testFile.toURI();
System.out.println(uri);
//"file:///....src/test/resources/testfile.txt"
List<String> lines = IO.readLines(uri.toString());
assertLines(lines);
}
static class MyHandler implements HttpHandler {
public void handle(HttpExchange t) throws IOException {
File testDir = new File("src/test/resources");
File testFile = new File(testDir, "testfile.txt");
String body = IO.read(testFile);
t.sendResponseHeaders(200, body.length());
OutputStream os = t.getResponseBody();
os.write(body.getBytes(IO.CHARSET));
os.close();
}
}
@Test
public void testReadFromHttp() throws Exception {
HttpServer server = HttpServer.create(new InetSocketAddress(9666), 0);
server.createContext("/test", new MyHandler());
server.setExecutor(null); // creates a default executor
server.start();
Thread.sleep(1000);
List<String> lines = IO.readLines("http://localhost:9666/test");
assertLines(lines);
}
public static class ProxyLoader {
private static final String DATA_FILE = "./files/proxy.txt";
private List<Proxy> proxyList = Collections.EMPTY_LIST;
private final String dataFile;
public ProxyLoader() {
this.dataFile = DATA_FILE;
init();
}
public ProxyLoader(String dataFile) {
this.dataFile = DATA_FILE;
init();
}
private void init() {
List <String> lines = IO.readLines(dataFile);
proxyList = new ArrayList<>(lines.size());
for (String line : lines) {
proxyList.add(Proxy.createProxy(line));
}
}
public String getDataFile() {
return this.dataFile;
}
public static List<Proxy> loadProxies() {
return new ProxyLoader().getProxyList();
}
public List<Proxy> getProxyList() {
return proxyList;
}
}
public static class Proxy {
private final String address;
private final int port;
public Proxy(String address, int port) {
this.address = address;
this.port = port;
}
public static Proxy createProxy(String line) {
String[] lineSplit = line.split(":");
String address = lineSplit[0];
int port = parseInt(lineSplit[1]);
return new Proxy(address, port);
}
public String getAddress() {
return address;
}
public int getPort() {
return port;
}
}
public static final class Proxy2 {
private final String address;
private final int port;
private static final String DATA_FILE = "./files/proxy.txt";
private static final Pattern addressPattern = Pattern.compile("^(\\d{1,3}[.]{1}){3}[0-9]{1,3}$");
private Proxy2(String address, int port) {
/* Validate address in not null.*/
Objects.requireNonNull(address, "address should not be null");
/* Validate port is in range. */
if (port < 1 || port > 65535) {
throw new IllegalArgumentException("Port is not in range port=" + port);
}
/* Validate address is of the form 123.12.1.5 .*/
if (!addressPattern.matcher(address).matches()) {
throw new IllegalArgumentException("Invalid Inet address");
}
/* Now initialize our address and port. */
this.address = address;
this.port = port;
}
private static Proxy2 createProxy(String line) {
String[] lineSplit = line.split(":");
String address = lineSplit[0];
int port = parseInt(lineSplit[1]);
return new Proxy2(address, port);
}
public final String getAddress() {
return address;
}
public final int getPort() {
return port;
}
public static List<Proxy2> loadProxies() {
List <String> lines = IO.readLines(DATA_FILE);
List<Proxy2> proxyList = new ArrayList<>(lines.size());
for (String line : lines) {
proxyList.add(createProxy(line));
}
return proxyList;
}
}
@Test public void proxyTest() {
List<Proxy> proxyList = ProxyLoader.loadProxies();
assertEquals(
5, len(proxyList)
);
assertEquals(
"127.0.0.1", idx(proxyList, 0).getAddress()
);
assertEquals(
8080, idx(proxyList, 0).getPort()
);
//192.55.55.57:9091
assertEquals(
"192.55.55.57", idx(proxyList, -1).getAddress()
);
assertEquals(
9091, idx(proxyList, -1).getPort()
);
}
@Test public void proxyTest2() {
List<Proxy2> proxyList = Proxy2.loadProxies();
assertEquals(
5, len(proxyList)
);
assertEquals(
"127.0.0.1", idx(proxyList, 0).getAddress()
);
assertEquals(
8080, idx(proxyList, 0).getPort()
);
//192.55.55.57:9091
assertEquals(
"192.55.55.57", idx(proxyList, -1).getAddress()
);
assertEquals(
9091, idx(proxyList, -1).getPort()
);
}
}
You can see all of the source code for this example and this utility classes here:
Hi Riak,
ReplyDeleteThanks for the article.
Few suggestions/corrections
Path thePath = FileSystems.getDefault().getPath(location);
can be replaced with
Paths.get(location); (concise :))
Related to Proxy class there is a typo 'Notice we don't have any [im]mutable state'
It should be 'Notice we don't have any mutable state'
It would be worth mentioning StandardCharsets. No-one should be hard coding the "UTF-8" string anymore...
ReplyDeletethanks guys!
ReplyDeleteI updated the example in github to use StandardCharsets and Paths.
ReplyDeleteThe sample that comes with NIO / zip file system has lots of good ideas for utilities with new FileSystem gak. (Note to self)
ReplyDelete