Harbor CLI

Name: Mehul Kumar
Email: mehul2002kumar@gmail.com
GitHub:github.com/Mehul-Kumar-27

This blog contains my proposal for the Harbor CLI project for LFX mentorship program 2024. 🚀

This approach will not only empower users with streamlined workflows but also pave the way for automated deployments and a more efficient container management experience.

Past Contributions to Harbor

Pull Requests in Review -:

Added summary, candidate, and scanner command under project command.

This pull request introduces three new subcommands (summary, candidate, and scanner) under the existing project command. The addition of these subcommands aims to provide users with more granular control and functionality when managing projects, thereby enhancing the overall user experience.

Related PR#47
bug: harbor project create command panics due to unhandled error.

During testing, I uncovered a bug that caused the application to panic when users executed the harbor project create command. This pull request resolves the issue by properly handling the error case, ensuring a stable and reliable experience for users when creating new projects.

Related PR#60
feat: Added test cases for the project package.

To improve the codebase's maintainability and ensure its long-term quality, this pull request introduces a comprehensive suite of test cases for the project package. These tests, written using the popular stretchr/testify package, cover various scenarios and edge cases, helping to catch potential issues early in the development process. Furthermore, to facilitate the writing of these test cases, I refactored the project's structure, employing interfaces to improve code modularity and testability.

Related PR#61

Pull Requests Closed -:

added the credential-name flag to the registry command

I reported a bug in the harbor registry command related to an unregistered flag in the Cobra command. This unregistered flag caused the application to panic. In addition to the create subcommand, the same issue was present in other subcommands like list and view. To address this, I registered the credential-name flag as a persistent flag in the registry command, ensuring its availability across all child commands.

Fortunately this issue was solved by merging another PR#40 .

Related PR#46

Goals

The primary goals of this project are -:

Developing Robust Harbor CLI :
- Create a command line interface using Golang that integrates seamlessly with Harbor.
- Ensure the CLI supports the typical workflows of Harbor administrators and users.
- Adding additional commands that would make the CLI application more convenient for the users to manage their projects.
Comprehensive Documentation:
- Well-documented instructions and guidelines for using the CLI.
- Documentation available to the user from command line in form of manpages as well as hosted online.
CI/CD Pipeline Implementation:
- A working CI/CD pipeline using GitHub Actions.
- The pipeline should be capable of creating multi-architecture binaries and containers to ensure broad compatibility.

By delivering these components, the project will ensure a robust, user-friendly, and well-documented CLI that enhances the Harbor experience for administrators and users alike. The integration of a CI/CD pipeline will further ensure efficient and versatile deployment processes.

Development Roadmap:

Harbor CLI

Current implementation of Harbor CLI, allows user to perform tasks like login, creating projects, registries , repositories and manage users. But there are several other features that are offered by the Golang client library that can be added in the harbor cli application some of them are -:

Artifact Command

This command would help the users to manage the artifacts inside a particular repository of a project.

Subcommands under this would be -:

List artifacts

List the artifacts (files, packages, etc.) for a specific project and repository. In addition to the basic properties, we can use additional filters in the "q" parameter:
1. "tags=*" lists only artifacts that have tags.
2. "tags=nil" lists only artifacts without any tags.
3. "tags=~v" lists artifacts whose tags contain the letter "v".
4. "tags=v" lists artifacts with an exact tag of "v".
5. "labels=(id1, id2)" lists artifacts that have both the label with id1 and the label with id2.

These filters would allow us to narrow down the list of artifacts based on their tags or assigned labels.

Copy Artifacts

The copy artifacts interface as per the go-sdk takes the following parameters.
1. project-name
2. repository-name
3. from - "The artifact from which the new artifact is copied from, the format should be "project/repository:tag" or "project/repository:from"
  
  This functionality to list artifacts is provided by the go-sdk through the given implementation of the artifact client interface below.
Get Artifacts

This command would help the users to retrieve the artifact specified by the reference within the project and repository.
Delete Artifacts
This command would help users to delete the artifact specified by the reference under the project and repository. The reference can be either a digest or a tag.
Creating Tags

This command would help in creating tag of a specified artifact.
Listing Tags
This command would list all the tags available.
Delete Tags
This command would delete a particular tag.

List accessories
This command would help user to list the accessories of a particular artifact.

The response of this command would contain the following information about the artifact.

 [
   {
     "id": 0,
     "artifact_id": 0,
     "subject_artifact_id": 0,
     "subject_artifact_digest": "string",
     "subject_artifact_repo": "string",
     "size": 0,
     "digest": "string",
     "type": "string",
     "icon": "string",
     "creation_time": "2024-05-22T10:06:47.421Z"
   }
 ]

In a single request we can fetch more that one artifact accessories. We can also filter the result using queries.

Get vulnerabilities of a specified artifact
This command would help the user to fetch the vulnerabilities addition of the artifact specified by the reference under the project and repository.

I plan to implement all these commands under the artifact command using the the artifact interface provided by the go-sdk.

Scan Command

This command would help the user to scan artifacts, get logs of the scan of artifacts and controlling the scan job of an artifact.

Subcommands under this would -:

Scan
This command would help the user to scan a particular artifact. The reference of an artifact could be digest or tag.
Stop Scan
This command would help the user to cancel the scan job for a particular artifact.
Get Logs

This command would help the user to get the scan logs of the artifact.

I plan to implement all these commands under the scan using the scan interface provided by the go sdk.

Scanner Commands

Scanners are one of the most vital component offered by the Harbor go-sdk. Scanners in Harbor are vulnerability scanners that can scan container images for vulnerabilities.

The subcommands under this would be -:

Get all scanners
This command would return a list of currently configured scanner registration.
Create Scanner
This command would help the users to create a new scanner registration with the given data.
Getting registration details
This command would help the user to get the registration details of the of a specified scanner registration.

Update a particular scanner

This command would help the user update a particular scanner registration. The update request would contain the following fields to update.

 {
   "name": "Trivy",
   "description": "A free-to-use tool that scans container images for package vulnerabilities.\n",
   "url": "http://harbor-scanner-trivy:8080",
   "auth": "Bearer",
   "access_credential": "Bearer: JWTTOKENGOESHERE",
   "skip_certVerify": false,
   "use_internal_addr": false,
   "disabled": false
 }

Delete a scanner registration
This command would help the user to delete a particular scanner registration.
Set default
This command would help the user to manage the default scanner registration.
Getting metadata

This command would help the user to get the metadata of the scanner registration.

Testing

A well-tested Harbor CLI would really help its users. To make this happen, I aim to boost test coverage for both the existing code and any new features I add.

I'll test individual commands using the stretchr/testify package, mocking the implementation with interfaces. This approach allows us to easily mock functions and thoroughly test the entire CLI application.

Here is an example how the testing would be done -:

Create an Interface

First, we would create an interface to define the functionalities provided by the Go SDK. This interface abstracts the operations related to projects, making it easier to mock them during testing.

Example -:

  type ProjectInterface interface {
      CreateProject(context.Context, *project.CreateProjectParams) (*project.CreateProjectCreated, error)
      DeleteProject(context.Context, *project.DeleteProjectParams) (*project.DeleteProjectOK, error)
      ListProjects(context.Context, *project.ListProjectsParams) (*project.ListProjectsOK, error)
      GetLogs(context.Context, *project.GetLogsParams) (*project.GetLogsOK, error)
  }

Implement CLI Command Using the Interface

Next, we implement the CLI command to use the above interface. This ensures that the CLI logic remains decoupled from the actual implementation, facilitating easier testing and future maintenance.

For example - :

func DeleteProjectCommand() *cobra.Command {

    cmd := &cobra.Command{
        Use:   "delete",
        Short: "delete project by name or id",
        Args:  cobra.MaximumNArgs(1),
        Run: func(cmd *cobra.Command, args []string) {
            var err error
            credentialName := viper.GetString("current-credential-name")
            client := utils.GetClientByCredentialName(credentialName)
            ctx := context.Background()
            if len(args) > 0 {
                err = RunDeleteProject(args[0], ctx, client.Project)
            } else {
                projectName := utils.GetProjectNameFromUser()
                err = RunDeleteProject(projectName, ctx, client.Project)
            }
            if err != nil {
                log.Errorf("failed to delete project: %v", err)
            }
        },
    }

    return cmd
}

func RunDeleteProject(projectName string, ctx context.Context, projectInterface ProjectInterface) error {

    _, err := projectInterface.DeleteProject(ctx, &project.DeleteProjectParams{ProjectNameOrID: projectName})

    if err != nil {
        return err
    }

    log.Info("project deleted successfully")
    return nil
}

In this example, DeleteProjectCommand is a CLI command that deletes a project by its name or ID. The RunDeleteProject function uses the ProjectInterface to perform the deletion, allowing you to mock this interface in tests.

Implement the Mock Client for Testing

For testing, implement the interface using a mock client. This approach allows us to simulate various scenarios and test how the CLI handles different responses.

Here is how you can create a mock implementation of ProjectInterface:

For example we can implement the mock client as follows -:


type MockProject struct {
    mock.Mock
}

func (m *MockProject) CreateProject(ctx context.Context, params *project.CreateProjectParams) (*project.CreateProjectCreated, error) {
    args := m.Called(ctx, params)
    if params.Project.Public == nil {
        return nil, errors.New("public field is missing")
    }
    return &project.CreateProjectCreated{}, args.Error(1)
}
func (m *MockProject) DeleteProject(ctx context.Context, params *project.DeleteProjectParams) (*project.DeleteProjectOK, error) {
    args := m.Called(ctx, params)
    return &project.DeleteProjectOK{}, args.Error(1)
}

func (m *MockProject) ListProjects(ctx context.Context, params *project.ListProjectsParams) (*project.ListProjectsOK, error) {
    args := m.Called(ctx, params)
    return &project.ListProjectsOK{}, args.Error(1)
}

func (m *MockProject) GetLogs(ctx context.Context, params *project.GetLogsParams) (*project.GetLogsOK, error) {
    args := m.Called(ctx, params)
    return &project.GetLogsOK{}, args.Error(1)
}

In this mock implementation:

MockProject embeds mock.Mock from the testify package, enabling it to record function calls and their arguments.
Each method in MockProject calls m.Called to record the invocation and returns either predefined results or errors.
Writing Test Cases

Now we can write test cases using the mock client to ensure the CLI commands behave as expected.

Automating Documentation Generation for Harbor CLI

As mentioned in the issue CLI Documentation we need to have way to have documentation available to the user in online form as well as available from the CLI.

To achieve the functionality of making the documentation available from both the CLI and online hosting, I plan to write the documentation in Markdown and then convert it into HTML pages and man pages.

Man pages are one of the best methods through which we can ensure that our documentation is accessible to users through the command line. Once our documentation is converted into man pages, we can attach them to our Go binary releases. This would also help to keep the documentation updated.

We can achieve this using Pandoc, which is a Haskell library for converting from one markup format to another, and a command-line tool that uses this library.

Here are the steps which we can follow to achieve this -:

Creating the GitHub Actions Workflow
I can can create a GitHub Actions workflow to automate the generation of your documentation. We can create a file named generate-docs.yml in the .github/workflows/ directory of our repository with the following content:

name: Generate Documentation

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout repository
      uses: actions/checkout@v3

    - name: Set up Go
      uses: actions/setup-go@v4
      with:
        go-version: 'go-version'

    - name: Install pandoc
      run: sudo apt-get install -y pandoc

    - name: Generate manpages with pandoc
      run: |
        for file in $(find docs -name '*.md'); do
          base=$(basename "$file" .md)
          pandoc -s -t man "$file" -o "man/man1/${base}.1"
        done

    - name: Generate HTML documentation with pandoc
      run: |
        for file in $(find docs -name '*.md'); do
          base=$(basename "$file" .md)
          pandoc -s -t html "$file" -o "docs/${base}.html"
        done

    - name: Embed manpages in Go binary
      run: go build -o mycli .

This GitHub action would help us in generating the HTML files and the man pages in the CI.

Embedding Man pages in Harbor CLI Application
To ensure that our generated man pages are available through the command line application, we can embed these generated files inside the Harbor CLI Go binary using the embed package in Go.

Similarly, the generated HTML pages would provide the documentation available to the user in online mode.

Note: Man pages are not available for Windows. However, they could be made available if the user installs any third-party tool that enables them to view man pages. I am still looking for any workaround to solve this.

CI/CD Pipeline Implementation

I plan to setup GitHub action workflows for the following automation.

Generating Multi-Architecture Binaries and Containers
We can generate multi-architecture binaries by setting up the following .github/workflows workflows.

name: Build and Release Binaries

on:
  push:
    tags:
      - 'v*.*.*'

jobs:
  build:
    runs-on: ubuntu-latest

    strategy:
      matrix:
        go-version: [1.20.x]
        os: [ubuntu-latest, windows-latest, macos-latest]
        arch: [amd64, arm64]

    steps:
    - name: Set up Go
      uses: actions/setup-go@v3
      with:
        go-version: ${{ matrix.go-version }}

    - name: Check out the code
      uses: actions/checkout@v3

    - name: Build binary
      run: |
        mkdir -p dist
        GOOS=$(echo ${{ matrix.os }} | sed -e 's/-latest//' | tr '[:upper:]' '[:lower:]')
        GOARCH=${{ matrix.arch }}
        go build -o dist/harbor-cli-${GOOS}-${GOARCH}

    - name: Upload binary
      uses: actions/upload-artifact@v3
      with:
        name: harbor-cli-${{ matrix.os }}-${{ matrix.arch }}
        path: dist/

The above workflow would help use build multi-architecture binaries for our harbor cli.

Generating Multiarchitecture Containers

We can build multi-architecture container images for harbor cli with the help of the following workflow.

name: Build and Push Docker Image

on:
  push:
    tags:
      - 'v*.*.*'

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v2

    - name: Login to Docker Hub
      uses: docker/login-action@v2
      with:
        username: ${{ secrets.DOCKER_USERNAME }}
        password: ${{ secrets.DOCKER_PASSWORD }}

    - name: Check out the code
      uses: actions/checkout@v3

    - name: Build and push Docker images
      uses: docker/build-push-action@v5
      with:
        context: .
        push: true
        tags: |
          <habor docker username>/harbor-cli:latest
          <harbor docker username>/harbor-cli:${{ github.ref_name }}
        platforms: linux/amd64,linux/arm64

The above workflow would help us in building docker images for various platform using docker Buildx. This work flow would also help us in publishing our docker images on docker hub so that it would be available for user to use in the CI/CD for managing user etc.

Releasing Binaries

To release the binaries, we can use the GitHub Release action to create a new release and upload the binaries to it.

name: Create GitHub Release

on:
  push:
    tags:
      - 'v*.*.*'

jobs:
  release:
    runs-on: ubuntu-latest
    needs: build

    steps:
    - name: Check out the code
      uses: actions/checkout@v3

    - name: Download binaries
      uses: actions/download-artifact@v3
      with:
        name: harbor-cli-${{ matrix.os }}-${{ matrix.arch }}
        path: ./dist

    - name: Create GitHub Release
      id: create_release
      uses: actions/create-release@v1
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        tag_name: ${{ github.ref_name }}
        release_name: Release ${{ github.ref_name }}
        draft: false
        prerelease: false

    - name: Upload Release Asset
      uses: actions/upload-release-asset@v1
      with:
        upload_url: ${{ steps.create_release.outputs.upload_url }}
        asset_path: ./dist/harbor-cli-${{ matrix.os }}-${{ matrix.arch }}
        asset_name: harbor-cli-${{ matrix.os }}-${{ matrix.arch }}
        asset_content_type: application/octet-stream

Project Timeline :

The below section contains details of the timeline of the project for the 12 weeks program.

Week 1
- Implementing list artifact command.
- Implementing get artifact command.
- Implementing delete artifact command.
- Implementing creating tags command.
- Implementing delete tags.
Week 2
- Implementing list accessories command.
- Implementing get vulnerabilities command.
- Writing test cases for all the commands under the artifact command.
Week 3
- Implementing scan command.
- Implementing stop scan command.
- Implementing get logs command.
- Writing test cases for all the commands under the scan commands.
Week 4
- Implementing get all scanner command under the scan command.
- Implementing create scanner command.
- Implementing registration command.
- Implementing update scanner command.
- Implementing delete scanner command.
Week 5
- Implementing set default scanner command.
- Implementing getting metadata command.
Week 6
- Writing test cases for the scanner command.
- Complementing any pending tasks.
Mid Term Evaluation
- Major Deliverables
  - Complete functionality to handle artifacts from the Harbor CLI.
  - Complete functionality to scan artifacts.
  - Complete functionality to handle scanner from the Harbor CLI.
  - Full test coverage of the Harbor CLI, for the above commands.
Week 7
- Writing markdown formatted documentation.
Week 8
- Using Pandoc for generating man pages and HTML pages for documentation.
Week 9
- Implementing GitHub Actions actions for generating multi-architectural binaries.
Week 10
- Implementing GitHub Actions for generating multi-architectural containers.
Week 11
- Implementing GitHub Actions for releasing binaries.
Week 12
- Finishing any pending tasks.
Final Evaluation
- Major Deliverables
  - Complete documentation for the Harbor CLI.
  - Functionality to release multiarchitecture binaries and containers.

Commitments

With a genuine passion for this field, I am excited about the prospect of contributing to Harbor's mission and bringing my skills to the table. I understand the importance of meeting deadlines and maintaining open communication throughout the duration of the program.
To ensure that we achieve success, I am prepared to devote 40 hours per week, to the project. I am confident that with my dedication and hard work, I can make a valuable contribution to Harbor team during LFX mentorship program.

WHY ME ?

Past Contributions to Harbor CLI
I have consistently contributed to the Harbor CLI project, ranging from adding more features to writing test cases and reporting bugs. My experience and knowledge of the existing codebase make me a well-suited candidate for the fellowship program.
Google Summer Of Code' 23 Scholar
I've been an active contributor to numerous open-source projects and was selected for the prestigious Google Summer of Code, where I collaborated with the LibreHealth community to develop open-source software.

Google Summer of Project Page Link
Furthermore, I believe that effective collaboration and communication are key to any successful project, and I am committed to keeping an open dialogue with my mentors and fellow contributors. I am eager to learn from others and to contribute my own insights and ideas, so that we can work together to create impactful solutions.
As a self-starter who is comfortable working independently, I am confident in my ability to take ownership of tasks and see them through to completion. I am also a quick learner and enjoy tackling new challenges, so I am excited about the opportunity to work on projects that will stretch my abilities and allow me to develop new skills.

Relevant Personal Projects

GoTrader :
GoTrader is a robust, extensible, and reliable backend written in Golang, designed to provide real-time stock prices. It adopts a microservices architecture interconnected through gRPC, facilitating fast and fault-tolerant communication between the microservices. GoTrader can be deployed in swarm mode using Docker, ensuring scalability.

Technologies Used: Golang, Grpc, Docker, Apache Kafka, Microservices

GitHub Link :https://github.com/Mehul-Kumar-27/GoTrader
Hotel Reservation :
Hotel Reservation is a replica of a highly scalable hotel reservation system designed to handle a database of 20,000 users. It offers functionalities such as hotel booking, cancellation, email sending, and authentication.

Technologies Used: Golang, Grpc, Docker, SQL, SQL Debezium connector.

GitHub Link :https://github.com/Mehul-Kumar-27/HotelReservation

Harbor CLI - Expanding CLI Functionality and Implementing CI/CD

Table of contents