JAVA实现镜像tar包上传到harbor镜像仓库

harbor

Harbor 是一个开源的容器镜像仓库，用于存储和管理 Docker 镜像和其他容器镜像。 * 容器镜像仓库、存储和管理 Docker 镜像和其他容器镜像 * 有什么特点：支持多种镜像格式、易于使用、安全性和访问控制

项目地址：https://gitcode.com/gh_mirrors/ha/harbor

免费下载资源

白日做梦_

1712人浏览 · 2024-03-02 16:12:51

白日做梦_ · 2024-03-02 16:12:51 发布

前言

使用harbor过程中，一直想使用harbor的api实现镜像的上传功能，但是实际上harbor是直接调用了docker registry的api，harbor层只是做了一个透传的功能，这个可以参考《harbor权威指南》这本书，参考官网接口以及网上大佬的思想，实现了一个Java版本，主要是实现了docker daemon上传的逻辑。

docker镜像tar包结构

实现上传首先需要将镜像的tar包解压，读取目录结构，一个典型的docker镜像包（使用docker save命令）结构如下：

.
├── 1ecf8bc84a7c3d60c0a6bbdd294f12a6b0e17a8269616fc9bdbedd926f74f50c
│   ├── VERSION
│   ├── json
│   └── layer.tar
├── 6f4ec1f3d7ea33646d491a705f94442f5a706e9ac9acbca22fa9b117094eb720.json
├── aaac5bde2c2bcb6cc28b1e6d3f29fe13efce6d6b669300cc2c6bfab96b942af4
│   ├── VERSION
│   ├── json
│   └── layer.tar
├── b63363f0d2ac8b3dca6f903bb5a7301bf497b1e5be8dc4f57a14e4dc649ef9bb
│   ├── VERSION
│   ├── json
│   └── layer.tar
├── c453224a84b8318b0a09a83052314dd876899d3a1a1cf2379e74bba410415059
│   ├── VERSION
│   ├── json
│   └── layer.tar
├── dd8ef1d42fbcccc87927eee94e57519c401b84437b98fcf35505fb6b7267a375
│   ├── VERSION
│   ├── json
│   └── layer.tar
├── manifest.json
└── repositories

清单文件manifet.json结构

[
    {
        "Config":"6f4ec1f3d7ea33646d491a705f94442f5a706e9ac9acbca22fa9b117094eb720.json",
        "RepoTags":[
            "alpine:filebeat-6.8.7-arm64"
        ],
        "Layers":[
            "aaac5bde2c2bcb6cc28b1e6d3f29fe13efce6d6b669300cc2c6bfab96b942af4/layer.tar",
            "dd8ef1d42fbcccc87927eee94e57519c401b84437b98fcf35505fb6b7267a375/layer.tar",
            "c453224a84b8318b0a09a83052314dd876899d3a1a1cf2379e74bba410415059/layer.tar",
            "b63363f0d2ac8b3dca6f903bb5a7301bf497b1e5be8dc4f57a14e4dc649ef9bb/layer.tar",
            "1ecf8bc84a7c3d60c0a6bbdd294f12a6b0e17a8269616fc9bdbedd926f74f50c/layer.tar"
        ]
    }
]

manifest.json 包含了对这个tar包的描述信息，比如image config文件地址，tags说明，镜像layer信息，在解析的时候也是根据这个文件去获取关联的文件

上传流程

获取鉴权信息
检查layer.tar是否已经存在
上传layer.tar
上传image config
上传manifest（非包中的manifest.json而是Manifest struct）

核心实现类DockerImageUploadBiz.java

/**
 * @author Administrator
 */
@Service
@Slf4j
public class DockerImageUploadBiz {
    
    //远程仓库
    @Value("${docker.remote.repo}")
    private String targetRepoAddress;
    
    //本地解压路径
    @Value("${docker.upload.extractPath}")
    private String tarPath;
    
    //harbor用户名
    @Value("${docker.harbor.userName}")
    private String userName;
    
    //密码
    @Value("${docker.harbor.password}")
    private String password;
    
    /**
    * @param sourceTar 上传的镜像文件
    * @param project 项目
    */
    public void push(File sourceTar, String project) {
        if (!sourceTar.exists()) {
            log.warn("Error!file is not exist!path:{}", sourceTar);
            return;
        }
        try {
            String unTarPath = FileUtil.doUnArchiver(sourceTar, tarPath);
            String manifest = FileUtil.readJsonFile(unTarPath + File.separator + "manifest.json");
            JSONArray jsonArray = JSONObject.parseArray(manifest);
            if (Objects.isNull(jsonArray)) {
                log.warn("manifest convert error!path:{},content:{}", unTarPath + File.separator + "manifest.json", manifest);
                return;
            }
            for (Object arr : jsonArray) {
                JSONObject jsonObject = (JSONObject) arr;
                JSONArray repoTags = jsonObject.getJSONArray("RepoTags");
                for (Object repoTag : repoTags) {
                    String repo = repoTag.toString();
                    String substring = repo.substring(repo.lastIndexOf('/') + 1);
                    String[] split = substring.split(":");
                    String imageName = split[0];
                    String tag = split[1];
                    log.info("imageName:{},tag:{}", imageName, tag);
                    JSONArray layers = jsonObject.getJSONArray("Layers");
                    //1.上传layer
                    log.info("========================STEP:1/3===============================");
                    log.info("PUSHING LAYERS STARTING...");
                    List<String> layerPathList = new ArrayList<>(layers.size());
                    int i = 1;
                    for (Object layer : layers) {
                        String layerPath = unTarPath + File.separator + layer.toString();
                        log.info("PUSHING LAYER:{}-{} ...", i, layers.size());
                        layerPathList.add(layerPath);
                        pushLayer(project, layerPath, imageName);
                        i++;
                    }
                    log.info("PUSHING LAYERS ENDED...");

                    log.info("========================STEP:2/3===============================");
                    //2.上传config
                    log.info("PUSHING CONFIG STARTING...");
                    String config = jsonObject.getString("Config");
                    String configPath = unTarPath + File.separator + config;
                    pushingConfig(project, configPath, imageName);
                    log.info("PUSHING CONFIG ENDED...");

                    log.info("========================STEP:3/3===============================");
                    //3.上传manifest
                    log.info("PUSHING MANIFEST STARTING...");
                    pushingManifest(project, layerPathList, configPath, imageName, tag);
                    log.info("PUSHING MANIFEST ENDED...");
                    log.info("PUSHING {} COMPLETED!", repo);
                }
            }

        } catch (Exception e) {
            log.error("", e);
        }
    }

    /**
     * 上传镜像层
     *
     * @param layerPath 层路径
     * @param imageName 镜像名称
     */
    private void pushLayer(String project, String layerPath, String imageName) throws Exception {
        File layerFile = new File(layerPath);
        boolean layerExist = checkLayerExist(project, layerFile, imageName);
        if (layerExist) {
            log.info("LAYER ALREADY EXISTS! LAYER PATH:{}", layerPath);
            return;
        }
        String location = startingPush(project, imageName);
        chunkPush(layerFile, location);
//        monolithicPush(layerFile,location);
    }

    /**
     * 判断层是否存在
     *
     * @param layer     层
     * @param imageName 镜像名称
     * @return true:存在，false:不存在
     */
    private boolean checkLayerExist(String project, File layer, String imageName) throws Exception {
        String hash256 = FileUtil.hash256(layer);
        String url = String
                .format("%s/v2/%s/blobs/%s", targetRepoAddress, project + "/" + imageName, "sha256:" + hash256);
        Response response = OkHttpClientUtil.headOkHttp(url);
        return response.code() == HttpStatus.OK.value();
    }

    /**
     * 开始上传
     *
     * @param imageName 镜像名称
     */
    private String startingPush(String project, String imageName) throws IOException {
        String url = String.format("%s/v2/%s/blobs/uploads/", targetRepoAddress, project + "/" + imageName);
        Response response = OkHttpClientUtil.postOkHttp(url, RequestBody.create(null, ""));
        if (response.code() == HttpStatus.ACCEPTED.value()) {
            return response.header("location");
        }
        return "";
    }

    /**
     * 分块上传
     */
    private void chunkPush(File layerFile, String url) throws Exception {
        long length = layerFile.length();
        log.info("file size:{}", length);
        //10M
        int len = 1024 * 1024 * 5;
        byte[] chunk = new byte[len];
        int offset = 0;
        int index = 0;
        MessageDigest messageDigest = MessageDigest.getInstance("SHA-256");
        while (true) {
            byte[] blocks = FileUtil.getBlock(offset, layerFile, chunk.length);
            if (Objects.isNull(blocks)) {
                log.warn("File block is null!");
                break;
            }
            offset += blocks.length;
            messageDigest.update(blocks);
            log.info("pushing range:[{}-{}]... {}%", index, offset, String.format("%.2f", (float) offset / (float) length * 100));
            if (offset == length) {
                String hash256 = FileUtil.byte2Hex(messageDigest.digest());
                url = String.format("%s&digest=sha256:%s", url, hash256);
                Response response = OkHttpClientUtil.putOkHttp(url, index, offset, blocks);
                if (response.code() != HttpStatus.CREATED.value()) {
                    log.error("chunk push error!code:{},digest:{},{}", response.code(), hash256, response.body().string());
                    throw new RuntimeException("chunk push error");
                }
                response.close();
                break;
            } else {
                Response response = OkHttpClientUtil.patchOkHttp(url, index, offset, blocks);
                if (response.code() != HttpStatus.ACCEPTED.value()) {
                    log.error("patch error!code:{},response:{}", response.code(), response.body().string());
                    throw new RuntimeException("patch error!");
                }
                url = response.header("location");
            }
            index = offset;
        }
    }

    /**
     * 整块上传
     *
     * @param layer
     */
    private void monolithicPush(File layer, String url) throws Exception {
        byte[] contents = FileUtils.readFileToByteArray(layer);
        String hash256 = FileUtil.hash256(layer);
        url = url + "&digest=sha256:" + hash256;
        Response response = OkHttpClientUtil.putOkHttp(url, contents);
        if (response.code() != HttpStatus.CREATED.value()) {
            log.error("monolithicPush error!code:{},{}", response.code(), response.body().string());
            throw new RuntimeException("monolithicPush error!");
        }
    }

    /**
     * 上传config
     *
     * @param configPath 路径
     * @param imageName  镜像名称
     * @throws Exception 异常
     */
    private void pushingConfig(String project, String configPath, String imageName) throws Exception {
        File file = new File(configPath);
        if (checkLayerExist(project, file, imageName)) {
            log.warn("{} exists!", configPath);
            return;
        }
        log.info("start pushing config...");
        String url = startingPush(project, imageName);
        monolithicPush(file, url);
        log.info("config:{} upload success!", configPath);
    }

    /**
     * 上传manifest清单
     *
     * @param layerArrays
     * @param configPath
     * @param tag
     * @throws Exception
     */
    private void pushingManifest(String project, List<String> layerArrays, String configPath, String imageName, String tag) throws Exception {
        ManifestV2 manifestV2 = new ManifestV2()
                .setMediaType("application/vnd.docker.distribution.manifest.v2+json")
                .setSchemaVersion(2);
        File configFile = new File(configPath);
        String hash256 = FileUtil.hash256(configFile);
        Config config = new Config()
                .setMediaType("application/vnd.docker.container.image.v1+json")
                .setDigest("sha256:" + hash256)
                .setSize((int) configFile.length());
        manifestV2.setConfig(config);
        List<Layer> layers = layerArrays.stream()
                .map(layerPath -> {
                    File layerFile = new File(layerPath);
                    Layer layer = new Layer();
                    String hash2561 = FileUtil.hash256(layerFile);
                    layer.setDigest("sha256:" + hash2561);
                    layer.setMediaType("application/vnd.docker.image.rootfs.diff.tar");
                    layer.setSize((int) layerFile.length());
                    return layer;
                }).collect(Collectors.toList());
        manifestV2.setLayers(layers);
        String manifestStr = JSON.toJSONString(manifestV2);
//        System.out.println(manifestStr);
        String url = String.format("%s/v2/%s/manifests/%s", targetRepoAddress, project + "/" + imageName, tag);
        Response response = OkHttpClientUtil.putManifestOkHttp(url, manifestStr.getBytes(StandardCharsets.UTF_8));
        if (response.code() != HttpStatus.CREATED.value()) {
            log.error("upload manifest error!,code:{},response:{}", response.code(), response.body().string());
            return;
        }
//        response.close();
        log.info("manifest upload success!");
    }
}

FileUtil.java实现了文件的解压以及获取文件sha256等方法

/**
 * @author Administrator
 */
public class FileUtil {

    /**
     * 解压tar
     *
     * @param sourceFile
     * @param destPath
     * @throws Exception
     */
    public static String doUnArchiver(File sourceFile, String destPath)
            throws Exception {
        byte[] buf = new byte[1024];
        FileInputStream fis = new FileInputStream(sourceFile);
        BufferedInputStream bis = new BufferedInputStream(fis);
        TarArchiveInputStream tais = new TarArchiveInputStream(bis);
        TarArchiveEntry tae = null;
        destPath = createTempDirIfNotExist(sourceFile.getName(), destPath);
        while ((tae = tais.getNextTarEntry()) != null) {
            File f = new File(destPath + "/" + tae.getName());
            if (tae.isDirectory()) {
                f.mkdirs();
            } else {
                /*
                 * 父目录不存在则创建
                 */
                File parent = f.getParentFile();
                if (!parent.exists()) {
                    parent.mkdirs();
                }

                FileOutputStream fos = new FileOutputStream(f);
                BufferedOutputStream bos = new BufferedOutputStream(fos);
                int len;
                while ((len = tais.read(buf)) != -1) {
                    bos.write(buf, 0, len);
                }
                bos.flush();
                bos.close();
            }
        }
        tais.close();
        return destPath;
    }

    /**
     * 创建临时目录
     *
     * @param pathName
     * @param basePath
     */
    private static synchronized String createTempDirIfNotExist(String pathName, String basePath) {
        String dir;
        if (pathName.contains(".")) {
            String[] split = pathName.split("\\.");
            dir = basePath + File.separator + split[0];
        } else {
            dir = basePath + File.separator + pathName;
        }
        File file = new File(dir);
        if (!file.exists()) {
            file.mkdirs();
        }
        return dir;
    }

    /**
     * 读取json文件
     *
     * @param fileName
     * @return
     */
    public static String readJsonFile(String fileName) {
        try {
            File jsonFile = new File(fileName);
            Reader reader = new InputStreamReader(new FileInputStream(jsonFile), StandardCharsets.UTF_8);
            return IOUtils.toString(reader);
        } catch (IOException e) {
            e.printStackTrace();
            return null;
        }
    }

    public static String hash256(File file) {

        try (InputStream fis = new FileInputStream(file)) {
            byte[] buffer = new byte[4096];
            MessageDigest md5 = MessageDigest.getInstance("SHA-256");
            for (int numRead = 0; (numRead = fis.read(buffer)) > 0; ) {
                md5.update(buffer, 0, numRead);
            }
            return byte2Hex(md5.digest());
        } catch (Exception e) {
            e.printStackTrace();
        }
        return "";
    }

    /**
     * 将byte转为16进制
     *
     * @param bytes 要转换的bytes
     * @return 16进制String
     */
    public static String byte2Hex(byte[] bytes) {
        StringBuilder stringBuffer = new StringBuilder();
        String temp;
        for (byte b : bytes) {
            temp = Integer.toHexString(b & 0xFF);
            if (temp.length() == 1) {
                // 1得到一位的进行补0操作
                stringBuffer.append("0");
            }
            stringBuffer.append(temp);
        }
        return stringBuffer.toString();
    }

    /**
     * 文件分块工具
     *
     * @param offset    起始偏移位置
     * @param file      文件
     * @param blockSize 分块大小
     * @return 分块数据
     */

    public static byte[] getBlock(long offset, File file, int blockSize) {
        byte[] result = new byte[blockSize];
        try (RandomAccessFile accessFile = new RandomAccessFile(file, "r")) {
            accessFile.seek(offset);
            int readSize = accessFile.read(result);
            if (readSize == -1) {
                return null;
            } else if (readSize == blockSize) {
                return result;
            } else {
                byte[] tmpByte = new byte[readSize];
                System.arraycopy(result, 0, tmpByte, 0, readSize);
                return tmpByte;
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
        return null;
    }
}

OkHttpClientUtil.java实现了http相关方法

/**
 * @author Administrator
 */
@Slf4j
public class OkHttpClientUtil {

    private static final TrustAllManager trustAllManager = new TrustAllManager();

    private static final OkHttpClient okHttpClient = new OkHttpClient.Builder()
            .authenticator((route, response) -> {
                //用户名、密码
                String credential = Credentials.basic("admin", "password", StandardCharsets.UTF_8);
                return response.request().newBuilder()
                        .header("Authorization", credential)
                        .build();
            })
            .connectionPool(new ConnectionPool(10,1, TimeUnit.MINUTES))
            .sslSocketFactory(createTrustAllSSLFactory(trustAllManager),trustAllManager)
//            .proxy(new Proxy(Proxy.Type.HTTP,new InetSocketAddress("127.0.0.1",8888)))
            .writeTimeout(60,TimeUnit.MINUTES)
            .build();

    protected static SSLSocketFactory createTrustAllSSLFactory(TrustAllManager trustAllManager) {
        SSLSocketFactory ssfFactory = null;
        try {
            SSLContext sc = SSLContext.getInstance("TLS");
            sc.init(null, new TrustManager[]{trustAllManager}, new SecureRandom());
            ssfFactory = sc.getSocketFactory();
        } catch (Exception ignored) {
            ignored.printStackTrace();
        }

        return ssfFactory;
    }

    /**
     * get请求
     *
     * @param url
     */
    public static Response getOkHttp(String url) throws IOException {
        Request request = new Request.Builder()
                .url(url)
                .get()
                .build();
        return okHttpClient.newCall(request).execute();
    }

    /**
     * get请求
     *
     * @param url
     */
    public static Response headOkHttp(String url) throws IOException {
        Request request = new Request.Builder()
                .url(url)
                .head()
                .build();
        return okHttpClient.newCall(request).execute();
    }

    /**
     * post请求
     *
     * @param url
     * @param body
     */
    public static Response postOkHttp(String url, RequestBody body)
            throws IOException {
        Request request = new Request.Builder()
                .url(url)
                .post(body)
                .build();
        return okHttpClient.newCall(request).execute();
    }

    /**
     * patch方式
     *
     * @param url
     * @return
     * @throws IOException
     */
    public static Response patchOkHttp(String url, int index, int offset, byte[] buffer)
            throws IOException {
        MediaType mediaType = MediaType.parse("application/octet-stream");
        RequestBody body = RequestBody.create(mediaType, buffer);
        Request request = new Builder()
                .url(url)
                .patch(body)
                .header("Content-Type", "application/octet-stream")
                .header("Content-Length", String.valueOf(buffer.length))
                .header("Content-Range", String.format("%s-%s", index, offset))
                .build();
        return okHttpClient.newCall(request).execute();
    }

    /**
     * put方式
     *
     * @param url
     * @param index
     * @param offset
     * @param buffer
     * @return
     * @throws IOException
     */
    public static Response putOkHttp(String url, int index, int offset, byte[] buffer)
            throws IOException {
        MediaType mediaType = MediaType.parse("application/octet-stream");
        RequestBody body = RequestBody.create(mediaType, buffer);
        Request request = new Builder()
                .url(url)
                .put(body)
                .header("Content-Type", "application/octet-stream")
                .header("Content-Length", String.valueOf(buffer.length))
                .header("Content-Range", String.format("%s-%s", index, offset))
                .build();
        return okHttpClient.newCall(request).execute();
    }

    /**
     * put方式
     *
     * @param url
     * @param buffer
     * @return
     * @throws IOException
     */
    public static Response putOkHttp(String url, byte[] buffer)
            throws IOException {
        MediaType mediaType = MediaType.parse("application/octet-stream");
        RequestBody body = RequestBody.create(mediaType, buffer);
        Request request = new Builder()
                .url(url)
                .put(body)
                .header("Content-Type", "application/octet-stream")
                .header("Content-Length", String.valueOf(buffer.length))
                .build();
        return okHttpClient.newCall(request).execute();
    }

    /**
     * put方式
     *
     * @param url
     * @param buffers
     * @return
     * @throws IOException
     */
    public static Response putManifestOkHttp(String url, byte[] buffers)
            throws IOException {
        MediaType mediaType = MediaType.parse("application/vnd.docker.distribution.manifest.v2+json");
        RequestBody body = RequestBody.create(mediaType, buffers);
        Request request = new Builder()
                .url(url)
                .put(body)
                .header("Content-Type", "application/vnd.docker.distribution.manifest.v2+json")
                .build();
        return okHttpClient.newCall(request).execute();
    }
}

class TrustAllManager implements X509TrustManager{
    @Override
    public void checkClientTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {

    }

    @Override
    public void checkServerTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {

    }

    @Override
    public X509Certificate[] getAcceptedIssuers() {
        return new X509Certificate[0];
    }
}

清单文件结构ManifestV2.java

@Data
@JsonIgnoreProperties(ignoreUnknown = true)
@Accessors(chain = true)
public class ManifestV2 {
    private Integer schemaVersion;
    private String mediaType;
    private Config config;
    private List<Layer> layers;
}

@Data
@JsonIgnoreProperties(ignoreUnknown = true)
@Accessors(chain = true)
public class Config {
    private String mediaType;
    private Integer size;
    private String digest;
}

@Data
@JsonIgnoreProperties(ignoreUnknown = true)
public class Layer {
    private String mediaType;
    private Integer size;
    private String digest;
}

controller层

@PostMapping("fileUpload")
    public String fileUpload2(@RequestParam("file") MultipartFile file, String project) throws IOException {
        long startTime = System.currentTimeMillis();
        String path = "C:\\test" + File.separator + file.getOriginalFilename();

        File newFile = new File(path);
        //通过CommonsMultipartFile的方法直接写文件（注意这个时候）
        file.transferTo(newFile);
        long endTime = System.currentTimeMillis();
        log.info("file:{} upload success,size:{} byte,spend time:{} ms", newFile.getName(), newFile.length(), endTime - startTime);
        log.info("start to push registry...");
        dockerImageUploadBiz.push(newFile, project);
        return "/success";
    }

GitHub 加速计划 / ha / harbor

23.24 K

4.68 K

下载

最近提交(Master分支：2 个月前 )

9e55afbb pull image from registry.goharbor.io instead of dockerhub Update testcase to support Docker Image Can Be Pulled With Credential Change gitlab project name when user changed. Update permissions count and permission count total Change webhook_endpoint_ui Signed-off-by: stonezdj <stone.zhang@broadcom.com> Co-authored-by: Wang Yan <wangyan@vmware.com> 8 天前

3dbfd422 Signed-off-by: wang yan <wangyan@vmware.com> 8 天前

GitCode 开源社区

旨在为数千万中国开发者提供一个无缝且高效的云端环境，以支持学习、使用和贡献开源项目。

更多推荐

[转载]在Windows环境下安装GNU Radio

转自：在Windows环境下安装GNURadio_恐弱智_新浪博客GNU Radio是用Python开发的，大部分开源的工程能够在Linux环境下运行良好，而Windows下却运行的很勉强，而且安装配置都很复杂。GNU Radio算是个例外了，不光提供了Windows的二进制安装，还有比较详细的说明。我是Python小白，所以折腾了好久才弄好，特意记录下来，免得以后再装还折腾。GNU Radio的

GitCode 开源社区

centOS 8 使用dnf安装Docker

DNF是什么？CentOS 8使用YUM软件包管理器版本v4.0.4。现在，该版本使用DNF(已删除YUM)。DNF是软件包管理器。它会在Linux发行版上安装，执行更新并删除软件包。使用DNF安装Docker跳过具有损坏依赖性的程序包一个有效的解决方案是使您的CentOS 8系统使用以下--nobest命令安装最符合条件的版本：sudo dnf install docker...

GitCode 开源社区

定时同步数据库表(mysql+linux+crontab)

sync.sh里面的参数需要改变，ip/username/password/database/tablesync.sh#!/bin/sh# Please change the IP and password of the data source db.# Then change the table name.filename=/home/nington/db/$(date +%Y-%m