Jul 01 18:35:14.483512 ip-10-0-129-245 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 01 18:35:14.483522 ip-10-0-129-245 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 01 18:35:14.483529 ip-10-0-129-245 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 01 18:35:14.483745 ip-10-0-129-245 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 01 18:35:24.698433 ip-10-0-129-245 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 01 18:35:24.698452 ip-10-0-129-245 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f91370d3a320472693aac4f55365416e --
Jul 01 18:37:27.023152 ip-10-0-129-245 systemd[1]: Starting Kubernetes Kubelet...
Jul 01 18:37:27.406272 ip-10-0-129-245 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 18:37:27.406272 ip-10-0-129-245 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 01 18:37:27.406272 ip-10-0-129-245 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 18:37:27.406272 ip-10-0-129-245 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 01 18:37:27.406272 ip-10-0-129-245 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 18:37:27.408013 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.407922    2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 01 18:37:27.414528 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414508    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:37:27.414528 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414525    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:37:27.414528 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414530    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:37:27.414528 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414533    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:37:27.414528 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414538    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:37:27.414817 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414542    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:37:27.414817 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414546    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:37:27.414817 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414552    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:37:27.414817 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414555    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:37:27.414817 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414559    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:37:27.414817 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414563    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:37:27.414817 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414567    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:37:27.414817 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414570    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:37:27.414817 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414574    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:37:27.414817 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414588    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:37:27.414817 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414592    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:37:27.414817 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414596    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:37:27.414817 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414600    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:37:27.414817 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414604    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:37:27.414817 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414608    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:37:27.414817 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414612    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:37:27.414817 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414617    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:37:27.414817 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414621    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:37:27.414817 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414625    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:37:27.414817 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414629    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:37:27.415626 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414633    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:37:27.415626 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414639    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:37:27.415626 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414644    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:37:27.415626 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414649    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:37:27.415626 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414653    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:37:27.415626 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414657    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:37:27.415626 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414662    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:37:27.415626 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414666    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:37:27.415626 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414670    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:37:27.415626 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414674    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:37:27.415626 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414678    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:37:27.415626 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414682    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:37:27.415626 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414686    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:37:27.415626 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414692    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:37:27.415626 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414696    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:37:27.415626 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414700    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:37:27.415626 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414705    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:37:27.415626 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414709    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:37:27.415626 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414713    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:37:27.415626 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414718    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:37:27.416455 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414722    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:37:27.416455 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414728    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:37:27.416455 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414736    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:37:27.416455 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414742    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:37:27.416455 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414747    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:37:27.416455 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414752    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:37:27.416455 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414757    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:37:27.416455 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414761    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:37:27.416455 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414765    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:37:27.416455 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414769    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:37:27.416455 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414774    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:37:27.416455 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414778    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:37:27.416455 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414783    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:37:27.416455 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414788    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:37:27.416455 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414794    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:37:27.416455 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414798    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:37:27.416455 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414802    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:37:27.416455 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414806    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:37:27.416455 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414811    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:37:27.417021 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414815    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:37:27.417021 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414819    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:37:27.417021 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414823    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:37:27.417021 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414827    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:37:27.417021 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414833    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:37:27.417021 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414837    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:37:27.417021 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414842    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:37:27.417021 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414847    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:37:27.417021 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414852    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:37:27.417021 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414856    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:37:27.417021 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414861    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:37:27.417021 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414865    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:37:27.417021 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414869    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:37:27.417021 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414874    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:37:27.417021 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414892    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:37:27.417021 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414897    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:37:27.417021 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414901    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:37:27.417021 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414905    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:37:27.417021 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414910    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:37:27.417482 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414914    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:37:27.417482 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414919    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:37:27.417482 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414923    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:37:27.417482 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414927    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:37:27.417482 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414931    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:37:27.417482 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414935    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:37:27.417482 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414939    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:37:27.417482 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414944    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:37:27.417482 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414948    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:37:27.417482 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414952    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:37:27.417482 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414956    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:37:27.417482 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414961    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:37:27.417482 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414965    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:37:27.417482 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414972    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:37:27.417482 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414978    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:37:27.417482 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.414983    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:37:27.417482 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416149    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:37:27.417482 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416159    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:37:27.417482 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416164    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:37:27.417482 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416169    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:37:27.417973 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416174    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:37:27.417973 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416179    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:37:27.417973 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416183    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:37:27.417973 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416187    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:37:27.417973 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416192    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:37:27.417973 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416196    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:37:27.417973 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416200    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:37:27.417973 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416204    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:37:27.417973 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416208    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:37:27.417973 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416214    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:37:27.417973 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416218    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:37:27.417973 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416223    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:37:27.417973 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416227    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:37:27.417973 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416231    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:37:27.417973 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416237    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:37:27.417973 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416241    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:37:27.417973 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416245    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:37:27.417973 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416252    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:37:27.417973 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416259    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:37:27.418436 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416264    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:37:27.418436 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416269    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:37:27.418436 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416273    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:37:27.418436 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416278    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:37:27.418436 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416284    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:37:27.418436 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416290    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:37:27.418436 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416295    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:37:27.418436 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416300    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:37:27.418436 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416305    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:37:27.418436 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416309    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:37:27.418436 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416314    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:37:27.418436 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416318    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:37:27.418436 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416322    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:37:27.418436 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416326    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:37:27.418436 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416331    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:37:27.418436 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416335    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:37:27.418436 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416339    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:37:27.418436 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416344    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:37:27.418436 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416348    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:37:27.418916 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416353    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:37:27.418916 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416357    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:37:27.418916 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416361    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:37:27.418916 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416365    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:37:27.418916 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416370    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:37:27.418916 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416376    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:37:27.418916 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416380    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:37:27.418916 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416385    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:37:27.418916 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416389    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:37:27.418916 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416393    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:37:27.418916 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416397    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:37:27.418916 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416402    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:37:27.418916 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416406    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:37:27.418916 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416410    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:37:27.418916 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416415    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:37:27.418916 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416419    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:37:27.418916 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416424    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:37:27.418916 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416429    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:37:27.418916 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416433    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:37:27.418916 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416438    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:37:27.419411 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416443    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:37:27.419411 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416447    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:37:27.419411 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416451    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:37:27.419411 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416454    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:37:27.419411 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416459    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:37:27.419411 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416463    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:37:27.419411 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416467    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:37:27.419411 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416471    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:37:27.419411 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416477    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:37:27.419411 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416481    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:37:27.419411 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416485    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:37:27.419411 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416490    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:37:27.419411 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416494    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:37:27.419411 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416499    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:37:27.419411 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416505    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:37:27.419411 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416509    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:37:27.419411 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416514    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:37:27.419411 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416519    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:37:27.419411 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416523    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:37:27.419891 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416527    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:37:27.419891 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416532    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:37:27.419891 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416536    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:37:27.419891 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416540    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:37:27.419891 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416544    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:37:27.419891 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416548    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:37:27.419891 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416553    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:37:27.419891 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416557    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:37:27.419891 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416561    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:37:27.419891 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416565    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:37:27.419891 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416570    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:37:27.419891 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416574    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:37:27.419891 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416578    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:37:27.419891 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416583    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:37:27.419891 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416587    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:37:27.419891 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416591    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:37:27.419891 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416595    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:37:27.419891 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.416600    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:37:27.419891 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417076    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:37:27.419891 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417089    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:37:27.420390 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417092    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:37:27.420390 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417095    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:37:27.420390 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417098    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:37:27.420390 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417102    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:37:27.420390 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417107    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:37:27.420390 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417110    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:37:27.420390 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417113    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:37:27.420390 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417116    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:37:27.420390 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417119    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:37:27.420390 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417122    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:37:27.420390 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417125    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:37:27.420390 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417127    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:37:27.420390 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417130    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:37:27.420390 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417133    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:37:27.420390 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417135    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:37:27.420390 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417138    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:37:27.420390 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417141    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:37:27.420390 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417143    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:37:27.420390 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417146    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:37:27.420834 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417148    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:37:27.420834 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417151    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:37:27.420834 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417153    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:37:27.420834 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417156    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:37:27.420834 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417158    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:37:27.420834 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417161    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:37:27.420834 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417164    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:37:27.420834 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417166    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:37:27.420834 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417169    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:37:27.420834 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417171    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:37:27.420834 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417173    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:37:27.420834 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417177    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:37:27.420834 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417183    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:37:27.420834 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417186    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:37:27.420834 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417188    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:37:27.420834 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417191    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:37:27.420834 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417193    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:37:27.420834 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417196    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:37:27.420834 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417198    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:37:27.420834 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417201    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:37:27.421323 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417203    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:37:27.421323 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417207    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:37:27.421323 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417210    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:37:27.421323 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417214    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:37:27.421323 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417218    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:37:27.421323 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417221    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:37:27.421323 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417223    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:37:27.421323 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417226    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:37:27.421323 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417228    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:37:27.421323 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417231    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:37:27.421323 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417233    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:37:27.421323 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417236    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:37:27.421323 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417239    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:37:27.421323 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417242    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:37:27.421323 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417245    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:37:27.421323 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417247    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:37:27.421323 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417251    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:37:27.421323 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417255    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:37:27.421323 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417257    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:37:27.421829 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417260    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:37:27.421829 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417262    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:37:27.421829 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417265    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:37:27.421829 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417267    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:37:27.421829 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417270    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:37:27.421829 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417275    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:37:27.421829 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417278    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:37:27.421829 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417280    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:37:27.421829 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417283    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:37:27.421829 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417285    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:37:27.421829 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417288    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:37:27.421829 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417290    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:37:27.421829 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417293    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:37:27.421829 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417295    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:37:27.421829 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417297    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:37:27.421829 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417300    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:37:27.421829 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417303    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:37:27.421829 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417305    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:37:27.421829 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417308    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:37:27.421829 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417310    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:37:27.422345 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417312    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:37:27.422345 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417315    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:37:27.422345 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417317    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:37:27.422345 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417320    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:37:27.422345 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417322    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:37:27.422345 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417325    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:37:27.422345 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417327    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:37:27.422345 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417330    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:37:27.422345 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417332    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:37:27.422345 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417335    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:37:27.422345 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417337    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:37:27.422345 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417340    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:37:27.422345 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417342    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:37:27.422345 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417345    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:37:27.422345 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417347    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:37:27.422345 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417349    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:37:27.422345 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417352    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:37:27.422345 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417356    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:37:27.422345 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.417367    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:37:27.422345 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417432    2573 flags.go:64] FLAG: --address="0.0.0.0"
Jul 01 18:37:27.422847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417439    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 01 18:37:27.422847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417447    2573 flags.go:64] FLAG: --anonymous-auth="true"
Jul 01 18:37:27.422847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417451    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 01 18:37:27.422847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417455    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 01 18:37:27.422847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417459    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 01 18:37:27.422847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417463    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 01 18:37:27.422847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417467    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 01 18:37:27.422847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417470    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 01 18:37:27.422847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417474    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 01 18:37:27.422847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417477    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 01 18:37:27.422847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417480    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 01 18:37:27.422847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417484    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 01 18:37:27.422847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417487    2573 flags.go:64] FLAG: --cgroup-root=""
Jul 01 18:37:27.422847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417490    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 01 18:37:27.422847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417493    2573 flags.go:64] FLAG: --client-ca-file=""
Jul 01 18:37:27.422847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417496    2573 flags.go:64] FLAG: --cloud-provider="external"
Jul 01 18:37:27.422847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417499    2573 flags.go:64] FLAG: --cluster-dns="[]"
Jul 01 18:37:27.422847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417504    2573 flags.go:64] FLAG: --cluster-domain=""
Jul 01 18:37:27.422847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417506    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 01 18:37:27.422847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417509    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 01 18:37:27.422847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417513    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 01 18:37:27.422847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417516    2573 flags.go:64] FLAG: --container-log-max-files="5"
Jul 01 18:37:27.422847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417520    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 01 18:37:27.422847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417523    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417526    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417529    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417532    2573 flags.go:64] FLAG: --contention-profiling="false"
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417535    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417538    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417543    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417546    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417551    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417554    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417557    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417559    2573 flags.go:64] FLAG: --enable-load-reader="false"
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417562    2573 flags.go:64] FLAG: --enable-server="true"
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417565    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417570    2573 flags.go:64] FLAG: --event-burst="100"
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417573    2573 flags.go:64] FLAG: --event-qps="50"
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417576    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417579    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417582    2573 flags.go:64] FLAG: --eviction-hard=""
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417586    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417589    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417592    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417595    2573 flags.go:64] FLAG: --eviction-soft=""
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417598    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417601    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 01 18:37:27.423426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417604    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 01 18:37:27.424060 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417607    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 01 18:37:27.424060 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417609    2573 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 01 18:37:27.424060 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417612    2573 flags.go:64] FLAG: --fail-swap-on="true"
Jul 01 18:37:27.424060 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417615    2573 flags.go:64] FLAG: --feature-gates=""
Jul 01 18:37:27.424060 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417618    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 01 18:37:27.424060 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417621    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 01 18:37:27.424060 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417625    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 01 18:37:27.424060 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417628    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 01 18:37:27.424060 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417631    2573 flags.go:64] FLAG: --healthz-port="10248"
Jul 01 18:37:27.424060 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417634    2573 flags.go:64] FLAG: --help="false"
Jul 01 18:37:27.424060 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417636    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-129-245.ec2.internal"
Jul 01 18:37:27.424060 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417639    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 01 18:37:27.424060 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417645    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 01 18:37:27.424060 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417648    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 01 18:37:27.424060 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417651    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 01 18:37:27.424060 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417656    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 01 18:37:27.424060 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417659    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 01 18:37:27.424060 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417661    2573 flags.go:64] FLAG: --image-service-endpoint=""
Jul 01 18:37:27.424060 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417664    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 01 18:37:27.424060 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417667    2573 flags.go:64] FLAG: --kube-api-burst="100"
Jul 01 18:37:27.424060 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417670    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 01 18:37:27.424060 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417673    2573 flags.go:64] FLAG: --kube-api-qps="50"
Jul 01 18:37:27.424060 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417676    2573 flags.go:64] FLAG: --kube-reserved=""
Jul 01 18:37:27.424060 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417678    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417681    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417684    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417687    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417690    2573 flags.go:64] FLAG: --lock-file=""
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417692    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417695    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417699    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417704    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417708    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417711    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417713    2573 flags.go:64] FLAG: --logging-format="text"
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417716    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417719    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417722    2573 flags.go:64] FLAG: --manifest-url=""
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417725    2573 flags.go:64] FLAG: --manifest-url-header=""
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417729    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417732    2573 flags.go:64] FLAG: --max-open-files="1000000"
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417736    2573 flags.go:64] FLAG: --max-pods="110"
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417739    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417742    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417747    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417750    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417753    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417756    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 01 18:37:27.424636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417760    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417766    2573 flags.go:64] FLAG: --node-status-max-images="50"
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417770    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417772    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417776    2573 flags.go:64] FLAG: --pod-cidr=""
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417778    2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417785    2573 flags.go:64] FLAG: --pod-manifest-path=""
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417788    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417791    2573 flags.go:64] FLAG: --pods-per-core="0"
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417794    2573 flags.go:64] FLAG: --port="10250"
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417797    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417800    2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-076db97d091f7681a"
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417803    2573 flags.go:64] FLAG: --qos-reserved=""
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417806    2573 flags.go:64] FLAG: --read-only-port="10255"
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417808    2573 flags.go:64] FLAG: --register-node="true"
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417811    2573 flags.go:64] FLAG: --register-with-taints=""
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417815    2573 flags.go:64] FLAG: --registry-burst="10"
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417817    2573 flags.go:64] FLAG: --registry-qps="5"
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417820    2573 flags.go:64] FLAG: --reserved-cpus=""
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417823    2573 flags.go:64] FLAG: --reserved-memory=""
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417826    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417829    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417832    2573 flags.go:64] FLAG: --rotate-certificates="false"
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417835    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417838    2573 flags.go:64] FLAG: --runonce="false"
Jul 01 18:37:27.425477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417841    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417844    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417846    2573 flags.go:64] FLAG: --seccomp-default="false"
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417850    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417854    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417857    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417860    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417864    2573 flags.go:64] FLAG: --storage-driver-password="root"
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417867    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417870    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417873    2573 flags.go:64] FLAG: --storage-driver-user="root"
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417876    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417892    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417895    2573 flags.go:64] FLAG: --system-cgroups=""
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417898    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417903    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417906    2573 flags.go:64] FLAG: --tls-cert-file=""
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417909    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417913    2573 flags.go:64] FLAG: --tls-min-version=""
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417916    2573 flags.go:64] FLAG: --tls-private-key-file=""
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417919    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417921    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417924    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417927    2573 flags.go:64] FLAG: --v="2"
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417930    2573 flags.go:64] FLAG: --version="false"
Jul 01 18:37:27.426341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417934    2573 flags.go:64] FLAG: --vmodule=""
Jul 01 18:37:27.427020 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417938    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 01 18:37:27.427020 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.417941    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 01 18:37:27.427020 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418054    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:37:27.427020 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418057    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:37:27.427020 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418062    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:37:27.427020 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418066    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:37:27.427020 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418069    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:37:27.427020 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418072    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:37:27.427020 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418075    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:37:27.427020 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418079    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:37:27.427020 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418084    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:37:27.427020 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418086    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:37:27.427020 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418089    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:37:27.427020 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418091    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:37:27.427020 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418094    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:37:27.427020 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418097    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:37:27.427020 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418100    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:37:27.427020 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418102    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:37:27.427020 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418105    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:37:27.427020 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418108    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:37:27.427500 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418112    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:37:27.427500 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418115    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:37:27.427500 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418118    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:37:27.427500 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418120    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:37:27.427500 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418123    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:37:27.427500 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418126    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:37:27.427500 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418129    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:37:27.427500 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418131    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:37:27.427500 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418134    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:37:27.427500 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418136    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:37:27.427500 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418139    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:37:27.427500 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418142    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:37:27.427500 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418145    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:37:27.427500 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418147    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:37:27.427500 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418150    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:37:27.427500 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418152    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:37:27.427500 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418155    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:37:27.427500 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418157    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:37:27.427500 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418160    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:37:27.427966 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418162    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:37:27.427966 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418165    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:37:27.427966 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418168    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:37:27.427966 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418172    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:37:27.427966 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418175    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:37:27.427966 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418177    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:37:27.427966 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418180    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:37:27.427966 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418182    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:37:27.427966 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418185    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:37:27.427966 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418191    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:37:27.427966 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418194    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:37:27.427966 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418197    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:37:27.427966 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418199    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:37:27.427966 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418202    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:37:27.427966 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418204    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:37:27.427966 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418207    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:37:27.427966 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418209    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:37:27.427966 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418212    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:37:27.427966 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418215    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:37:27.428513 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418217    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:37:27.428513 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418220    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:37:27.428513 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418222    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:37:27.428513 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418225    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:37:27.428513 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418227    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:37:27.428513 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418230    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:37:27.428513 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418232    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:37:27.428513 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418235    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:37:27.428513 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418237    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:37:27.428513 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418240    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:37:27.428513 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418242    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:37:27.428513 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418245    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:37:27.428513 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418247    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:37:27.428513 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418250    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:37:27.428513 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418252    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:37:27.428513 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418256    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:37:27.428513 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418260    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:37:27.428513 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418262    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:37:27.428513 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418265    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:37:27.428513 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418267    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:37:27.429066 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418270    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:37:27.429066 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418272    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:37:27.429066 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418274    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:37:27.429066 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418278    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:37:27.429066 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418280    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:37:27.429066 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418283    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:37:27.429066 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418286    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:37:27.429066 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418288    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:37:27.429066 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418291    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:37:27.429066 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418293    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:37:27.429066 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418296    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:37:27.429066 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418299    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:37:27.429066 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418301    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:37:27.429066 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418304    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:37:27.429066 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418306    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:37:27.429066 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418309    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:37:27.429066 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418311    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:37:27.429066 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418314    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:37:27.429066 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418316    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:37:27.429066 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418319    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:37:27.429541 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418322    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:37:27.429541 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418324    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:37:27.429541 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.418328    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:37:27.429541 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.419052    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 18:37:27.429541 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.427172    2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 01 18:37:27.429541 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.427191    2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 01 18:37:27.429541 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.427725    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 01 18:37:27.429541 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.427732    2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 01 18:37:27.429541 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427790    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:37:27.429541 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427796    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:37:27.429541 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427799    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:37:27.429541 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427802    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:37:27.429541 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427805    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:37:27.429541 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427810    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:37:27.429899 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427814    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:37:27.429899 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427817    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:37:27.429899 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427820    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:37:27.429899 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427823    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:37:27.429899 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427826    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:37:27.429899 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427829    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:37:27.429899 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427831    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:37:27.429899 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427834    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:37:27.429899 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427837    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:37:27.429899 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427839    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:37:27.429899 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427842    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:37:27.429899 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427845    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:37:27.429899 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427848    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:37:27.429899 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427851    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:37:27.429899 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427853    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:37:27.429899 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427857    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:37:27.429899 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427860    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:37:27.429899 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427862    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:37:27.429899 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427865    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:37:27.429899 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427867    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:37:27.430414 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427870    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:37:27.430414 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427872    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:37:27.430414 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427875    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:37:27.430414 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427894    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:37:27.430414 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427897    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:37:27.430414 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427900    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:37:27.430414 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427903    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:37:27.430414 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427905    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:37:27.430414 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427908    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:37:27.430414 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427911    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:37:27.430414 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427913    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:37:27.430414 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427916    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:37:27.430414 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427919    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:37:27.430414 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427921    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:37:27.430414 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427924    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:37:27.430414 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427926    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:37:27.430414 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427930    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:37:27.430414 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427934    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:37:27.430414 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427937    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:37:27.430414 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427940    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:37:27.430903 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427944    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:37:27.430903 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427947    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:37:27.430903 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427950    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:37:27.430903 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427952    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:37:27.430903 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427955    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:37:27.430903 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427958    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:37:27.430903 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427961    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:37:27.430903 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427963    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:37:27.430903 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427966    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:37:27.430903 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427968    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:37:27.430903 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427971    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:37:27.430903 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427974    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:37:27.430903 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427977    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:37:27.430903 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427980    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:37:27.430903 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427983    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:37:27.430903 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427986    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:37:27.430903 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427989    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:37:27.430903 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427992    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:37:27.430903 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427994    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:37:27.430903 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.427997    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:37:27.431376 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428000    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:37:27.431376 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428003    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:37:27.431376 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428005    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:37:27.431376 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428008    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:37:27.431376 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428010    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:37:27.431376 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428013    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:37:27.431376 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428015    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:37:27.431376 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428018    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:37:27.431376 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428020    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:37:27.431376 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428023    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:37:27.431376 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428026    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:37:27.431376 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428028    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:37:27.431376 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428031    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:37:27.431376 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428035    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:37:27.431376 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428037    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:37:27.431376 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428040    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:37:27.431376 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428043    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:37:27.431376 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428045    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:37:27.431376 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428048    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:37:27.431376 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428050    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:37:27.431845 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428053    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:37:27.431845 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428057    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:37:27.431845 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428060    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:37:27.431845 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428062    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:37:27.431845 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428065    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:37:27.431845 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428068    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:37:27.431845 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428071    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:37:27.431845 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428074    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:37:27.431845 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428076    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:37:27.431845 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428079    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:37:27.431845 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428081    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:37:27.431845 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428084    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:37:27.431845 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428086    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:37:27.431845 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.428091    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 18:37:27.432227 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428202    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:37:27.432227 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428207    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:37:27.432227 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428210    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:37:27.432227 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428213    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:37:27.432227 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428216    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:37:27.432227 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428219    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:37:27.432227 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428223    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:37:27.432227 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428226    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:37:27.432227 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428229    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:37:27.432227 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428232    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:37:27.432227 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428235    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:37:27.432227 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428238    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:37:27.432227 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428242    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:37:27.432227 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428244    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:37:27.432227 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428247    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:37:27.432227 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428249    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:37:27.432227 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428252    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:37:27.432227 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428254    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:37:27.432227 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428257    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:37:27.432227 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428260    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:37:27.432725 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428262    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:37:27.432725 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428264    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:37:27.432725 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428267    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:37:27.432725 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428270    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:37:27.432725 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428272    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:37:27.432725 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428275    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:37:27.432725 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428278    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:37:27.432725 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428281    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:37:27.432725 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428283    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:37:27.432725 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428286    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:37:27.432725 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428288    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:37:27.432725 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428291    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:37:27.432725 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428294    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:37:27.432725 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428296    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:37:27.432725 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428299    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:37:27.432725 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428301    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:37:27.432725 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428304    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:37:27.432725 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428306    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:37:27.432725 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428309    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:37:27.432725 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428311    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:37:27.433244 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428314    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:37:27.433244 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428316    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:37:27.433244 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428318    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:37:27.433244 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428321    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:37:27.433244 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428324    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:37:27.433244 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428327    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:37:27.433244 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428330    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:37:27.433244 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428332    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:37:27.433244 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428335    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:37:27.433244 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428337    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:37:27.433244 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428340    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:37:27.433244 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428342    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:37:27.433244 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428345    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:37:27.433244 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428347    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:37:27.433244 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428350    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:37:27.433244 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428352    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:37:27.433244 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428355    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:37:27.433244 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428360    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:37:27.433244 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428363    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:37:27.433707 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428366    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:37:27.433707 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428369    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:37:27.433707 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428371    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:37:27.433707 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428374    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:37:27.433707 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428377    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:37:27.433707 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428380    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:37:27.433707 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428382    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:37:27.433707 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428386    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:37:27.433707 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428388    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:37:27.433707 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428391    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:37:27.433707 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428393    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:37:27.433707 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428396    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:37:27.433707 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428398    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:37:27.433707 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428401    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:37:27.433707 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428403    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:37:27.433707 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428406    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:37:27.433707 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428408    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:37:27.433707 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428410    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:37:27.433707 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428413    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:37:27.433707 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428416    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:37:27.434228 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428418    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:37:27.434228 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428421    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:37:27.434228 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428424    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:37:27.434228 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428426    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:37:27.434228 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428428    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:37:27.434228 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428431    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:37:27.434228 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428433    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:37:27.434228 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428436    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:37:27.434228 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428439    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:37:27.434228 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428441    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:37:27.434228 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428444    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:37:27.434228 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428447    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:37:27.434228 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428449    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:37:27.434228 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428456    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:37:27.434228 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428458    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:37:27.434228 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428461    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:37:27.434228 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428463    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:37:27.434228 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428467    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:37:27.434228 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428470    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:37:27.434688 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:27.428473    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:37:27.434688 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.428478    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 18:37:27.434688 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.428588    2573 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 01 18:37:27.434688 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.431745    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 01 18:37:27.434688 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.433131    2573 server.go:1019] "Starting client certificate rotation"
Jul 01 18:37:27.434688 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.433224    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 18:37:27.434688 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.433256    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 18:37:27.435791 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.435778    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 18:37:27.441233 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.441212    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 18:37:27.456474 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.456458    2573 log.go:25] "Validated CRI v1 runtime API"
Jul 01 18:37:27.458581 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.458567    2573 log.go:25] "Validated CRI v1 image API"
Jul 01 18:37:27.459388 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.459375    2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 01 18:37:27.462488 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.462468    2573 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9b417270-a44c-4260-b225-91e6f58f0da1:/dev/nvme0n1p4 a3debf11-73db-49ee-b11f-9463ff9704b5:/dev/nvme0n1p3]
Jul 01 18:37:27.462543 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.462488    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 01 18:37:27.466945 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.466927    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 18:37:27.467961 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.467830    2573 manager.go:216] Machine: {Timestamp:2026-07-01 18:37:27.466598564 +0000 UTC m=+0.330267902 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3111748 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ca474c5d6fc273af0121c864c3cb4 SystemUUID:ec2ca474-c5d6-fc27-3af0-121c864c3cb4 BootID:f91370d3-a320-4726-93aa-c4f55365416e Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:25:fa:66:76:57 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:25:fa:66:76:57 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:9f:44:a2:04:07 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 01 18:37:27.467961 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.467959    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 01 18:37:27.468091 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.468076    2573 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 01 18:37:27.470594 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.470399    2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 01 18:37:27.470746 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.470601    2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-245.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 01 18:37:27.472099 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.472088    2573 topology_manager.go:138] "Creating topology manager with none policy"
Jul 01 18:37:27.472132 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.472103    2573 container_manager_linux.go:309] "Creating device plugin manager"
Jul 01 18:37:27.472132 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.472110    2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 18:37:27.473073 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.473063    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 18:37:27.473953 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.473943    2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 01 18:37:27.476808 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.476797    2573 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 18:37:27.476943 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.476934    2573 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 01 18:37:27.476972 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.476966    2573 kubelet.go:486] "Attempting to sync node with API server"
Jul 01 18:37:27.476999 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.476984    2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 01 18:37:27.476999 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.476998    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 01 18:37:27.477055 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.477007    2573 kubelet.go:398] "Adding apiserver pod source"
Jul 01 18:37:27.477055 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.477016    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 01 18:37:27.477992 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.477980    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 18:37:27.478057 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.478011    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 18:37:27.480458 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.480436    2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 01 18:37:27.481799 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.481784    2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 01 18:37:27.481922 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.481815    2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 01 18:37:27.482487 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.482475    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 01 18:37:27.482530 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.482494    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 01 18:37:27.482530 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.482500    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 01 18:37:27.482530 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.482506    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 01 18:37:27.482530 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.482514    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 01 18:37:27.482530 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.482520    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 01 18:37:27.482657 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.482551    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 01 18:37:27.482657 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.482558    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 01 18:37:27.482657 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.482565    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 01 18:37:27.482657 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.482570    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 01 18:37:27.482657 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.482579    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 01 18:37:27.482657 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.482587    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 01 18:37:27.484053 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.484043    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 01 18:37:27.484053 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.484053    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 01 18:37:27.486761 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.486739    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-245.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 01 18:37:27.486854 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:27.486779    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 01 18:37:27.486854 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:27.486786    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-245.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 01 18:37:27.487153 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.487142    2573 server.go:1268] "Started kubelet"
Jul 01 18:37:27.487341 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.487223    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 01 18:37:27.487488 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.487442    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 01 18:37:27.487852 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.487824    2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 01 18:37:27.488031 ip-10-0-129-245 systemd[1]: Started Kubernetes Kubelet.
Jul 01 18:37:27.488733 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.488622    2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 01 18:37:27.489743 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.489732    2573 server.go:310] "Adding debug handlers to kubelet server"
Jul 01 18:37:27.493374 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:27.493346    2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 01 18:37:27.495259 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:27.494379    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-245.ec2.internal.18be3e57607c092e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-245.ec2.internal,UID:ip-10-0-129-245.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-245.ec2.internal,},FirstTimestamp:2026-07-01 18:37:27.487113518 +0000 UTC m=+0.350782855,LastTimestamp:2026-07-01 18:37:27.487113518 +0000 UTC m=+0.350782855,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-245.ec2.internal,}"
Jul 01 18:37:27.495782 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.495763    2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 01 18:37:27.495782 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.495771    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 01 18:37:27.495945 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:27.495914    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 01 18:37:27.496556 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.496539    2573 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 01 18:37:27.496637 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.496567    2573 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 01 18:37:27.496637 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.496539    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 01 18:37:27.496830 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.496811    2573 reconstruct.go:97] "Volume reconstruction finished"
Jul 01 18:37:27.496830 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.496829    2573 reconciler.go:29] "Reconciler: start to sync state"
Jul 01 18:37:27.496970 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.496835    2573 factory.go:153] Registering CRI-O factory
Jul 01 18:37:27.496970 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.496847    2573 factory.go:223] Registration of the crio container factory successfully
Jul 01 18:37:27.496970 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.496916    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 01 18:37:27.496970 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.496926    2573 factory.go:55] Registering systemd factory
Jul 01 18:37:27.496970 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.496934    2573 factory.go:223] Registration of the systemd container factory successfully
Jul 01 18:37:27.496970 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.496964    2573 factory.go:103] Registering Raw factory
Jul 01 18:37:27.497184 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.496995    2573 manager.go:1196] Started watching for new ooms in manager
Jul 01 18:37:27.497633 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.497618    2573 manager.go:349] Starting recovery of all containers
Jul 01 18:37:27.500543 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.500500    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k5n9t"
Jul 01 18:37:27.503475 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:27.503441    2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-245.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 01 18:37:27.503642 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:27.503616    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 01 18:37:27.510764 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.510745    2573 manager.go:354] Recovery completed
Jul 01 18:37:27.515194 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.515182    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:37:27.516796 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.516778    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k5n9t"
Jul 01 18:37:27.518255 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.518239    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:37:27.518319 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.518271    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:37:27.518319 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.518297    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:37:27.518914 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.518898    2573 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 01 18:37:27.518914 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.518909    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:37:27.519031 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.518912    2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 01 18:37:27.519031 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.518998    2573 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 18:37:27.519651 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.519636    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:37:27.519706 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.519666    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:37:27.519706 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.519680    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:37:27.520165 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.520151    2573 policy_none.go:49] "None policy: Start"
Jul 01 18:37:27.520232 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.520172    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 01 18:37:27.520232 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.520184    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 01 18:37:27.520423 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:27.520323    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-245.ec2.internal.18be3e5762573989  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-245.ec2.internal,UID:ip-10-0-129-245.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-245.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-245.ec2.internal,},FirstTimestamp:2026-07-01 18:37:27.518255497 +0000 UTC m=+0.381924837,LastTimestamp:2026-07-01 18:37:27.518255497 +0000 UTC m=+0.381924837,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-245.ec2.internal,}"
Jul 01 18:37:27.521270 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.521257    2573 policy_none.go:47] "Start"
Jul 01 18:37:27.560716 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.560530    2573 manager.go:337] "Starting Device Plugin manager"
Jul 01 18:37:27.568514 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:27.560743    2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 01 18:37:27.568514 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.560754    2573 server.go:87] "Starting device plugin registration server"
Jul 01 18:37:27.568514 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.561037    2573 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 01 18:37:27.568514 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.561046    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 01 18:37:27.568514 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.561164    2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 01 18:37:27.568514 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.561242    2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 01 18:37:27.568514 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.561249    2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 01 18:37:27.568514 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:27.562111    2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 01 18:37:27.568514 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:27.562163    2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 01 18:37:27.568514 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.563753    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 01 18:37:27.568514 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.564983    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 01 18:37:27.568514 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.565018    2573 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 01 18:37:27.568514 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.565043    2573 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 01 18:37:27.568514 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:27.565077    2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 01 18:37:27.568514 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.568038    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:37:27.661790 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.661715    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:37:27.662705 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.662686    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:37:27.662802 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.662715    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:37:27.662802 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.662737    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:37:27.662802 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.662768    2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-245.ec2.internal"
Jul 01 18:37:27.665400 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.665377    2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-245.ec2.internal"]
Jul 01 18:37:27.665453 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.665438    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:37:27.666946 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.666931    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:37:27.667043 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.666959    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:37:27.667043 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.666971    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:37:27.668313 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.668299    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:37:27.668427 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.668412    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal"
Jul 01 18:37:27.668473 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.668451    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:37:27.669560 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.669547    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:37:27.669645 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.669583    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:37:27.669645 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.669598    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:37:27.669645 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.669603    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:37:27.669645 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.669627    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:37:27.669645 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.669636    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:37:27.671002 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.670983    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-245.ec2.internal"
Jul 01 18:37:27.671074 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.671027    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:37:27.671757 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.671732    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:37:27.671830 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.671767    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:37:27.671830 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.671783    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:37:27.673020 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.673003    2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-245.ec2.internal"
Jul 01 18:37:27.673095 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:27.673030    2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-245.ec2.internal\": node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 01 18:37:27.696482 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:27.696464    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-245.ec2.internal\" not found" node="ip-10-0-129-245.ec2.internal"
Jul 01 18:37:27.697913 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.697894    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b568c82d895ac42942951c7fc4260965-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal\" (UID: \"b568c82d895ac42942951c7fc4260965\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal"
Jul 01 18:37:27.697976 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.697944    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b568c82d895ac42942951c7fc4260965-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal\" (UID: \"b568c82d895ac42942951c7fc4260965\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal"
Jul 01 18:37:27.698028 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.698011    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8a07c43805922ed70622b7aec23d988f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-245.ec2.internal\" (UID: \"8a07c43805922ed70622b7aec23d988f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-245.ec2.internal"
Jul 01 18:37:27.699837 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:27.699823    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-245.ec2.internal\" not found" node="ip-10-0-129-245.ec2.internal"
Jul 01 18:37:27.711077 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:27.711053    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 01 18:37:27.798451 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.798423    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b568c82d895ac42942951c7fc4260965-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal\" (UID: \"b568c82d895ac42942951c7fc4260965\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal"
Jul 01 18:37:27.798640 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.798457    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b568c82d895ac42942951c7fc4260965-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal\" (UID: \"b568c82d895ac42942951c7fc4260965\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal"
Jul 01 18:37:27.798640 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.798482    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8a07c43805922ed70622b7aec23d988f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-245.ec2.internal\" (UID: \"8a07c43805922ed70622b7aec23d988f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-245.ec2.internal"
Jul 01 18:37:27.798640 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.798551    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8a07c43805922ed70622b7aec23d988f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-245.ec2.internal\" (UID: \"8a07c43805922ed70622b7aec23d988f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-245.ec2.internal"
Jul 01 18:37:27.798640 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.798552    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b568c82d895ac42942951c7fc4260965-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal\" (UID: \"b568c82d895ac42942951c7fc4260965\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal"
Jul 01 18:37:27.798640 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:27.798552    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b568c82d895ac42942951c7fc4260965-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal\" (UID: \"b568c82d895ac42942951c7fc4260965\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal"
Jul 01 18:37:27.811590 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:27.811558    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 01 18:37:27.912435 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:27.912334    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 01 18:37:28.000370 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:28.000342    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal"
Jul 01 18:37:28.003653 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:28.003436    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-245.ec2.internal"
Jul 01 18:37:28.012932 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:28.012915    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 01 18:37:28.113451 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:28.113421    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 01 18:37:28.213966 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:28.213895    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 01 18:37:28.314463 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:28.314434    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 01 18:37:28.414943 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:28.414912    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 01 18:37:28.433066 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:28.433041    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 01 18:37:28.433206 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:28.433182    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 18:37:28.496757 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:28.496700    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:37:28.515574 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:28.515544    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 01 18:37:28.521690 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:28.521666    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-30 18:32:29 +0000 UTC" deadline="2028-01-27 11:02:39.522162393 +0000 UTC"
Jul 01 18:37:28.521760 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:28.521691    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13792h25m11.00047464s"
Jul 01 18:37:28.532426 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:28.532397    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a07c43805922ed70622b7aec23d988f.slice/crio-661a5d844e44827e17d69b5e1ede54d25d2c440c685c4fe4be5e62a04784169d WatchSource:0}: Error finding container 661a5d844e44827e17d69b5e1ede54d25d2c440c685c4fe4be5e62a04784169d: Status 404 returned error can't find the container with id 661a5d844e44827e17d69b5e1ede54d25d2c440c685c4fe4be5e62a04784169d
Jul 01 18:37:28.536735 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:28.536708    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 18:37:28.568311 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:28.568274    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-245.ec2.internal" event={"ID":"8a07c43805922ed70622b7aec23d988f","Type":"ContainerStarted","Data":"661a5d844e44827e17d69b5e1ede54d25d2c440c685c4fe4be5e62a04784169d"}
Jul 01 18:37:28.569276 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:28.569250    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal" event={"ID":"b568c82d895ac42942951c7fc4260965","Type":"ContainerStarted","Data":"a6502a021fb21fa603c1ef2ec33d8e3703c41081971f4c6654b484d47d2553bd"}
Jul 01 18:37:28.615928 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:28.615907    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 01 18:37:28.619719 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:28.619702    2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:37:30.811325 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:30.811305    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 01 18:37:30.911839 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:30.911756    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 01 18:37:30.950581 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:30.950555    2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:37:30.991451 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:30.991433    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal"
Jul 01 18:37:31.013981 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.013958    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 18:37:31.014923 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.014911    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-245.ec2.internal"
Jul 01 18:37:31.040608 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.040587    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 18:37:31.572653 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.572621    2573 apiserver.go:52] "Watching apiserver"
Jul 01 18:37:31.581507 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.581484    2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 01 18:37:31.581914 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.581891    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-4z9r8","openshift-multus/multus-additional-cni-plugins-nx7jq","openshift-network-operator/iptables-alerter-prmsf","openshift-ovn-kubernetes/ovnkube-node-tjxh2","kube-system/global-pull-secret-syncer-mrx57","kube-system/konnectivity-agent-4pmfd","openshift-cluster-node-tuning-operator/tuned-frw2w","openshift-dns/node-resolver-l5m2d","openshift-image-registry/node-ca-dctc8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal","openshift-multus/network-metrics-daemon-plc42","openshift-network-diagnostics/network-check-target-9t8tv","kube-system/kube-apiserver-proxy-ip-10-0-129-245.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds"]
Jul 01 18:37:31.583510 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.583493    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.584759 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.584734    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:31.584854 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:31.584811    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plc42" podUID="5180da14-bd5e-4777-b34c-4041f623113b"
Jul 01 18:37:31.585931 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.585909    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-prmsf"
Jul 01 18:37:31.586579 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.586278    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 01 18:37:31.586579 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.586360    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 01 18:37:31.586579 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.586504    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bj5lc\""
Jul 01 18:37:31.587287 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.587266    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.589726 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.589689    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:31.589827 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:31.589798    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrx57" podUID="8bd0c6cc-1f64-4be9-8b7e-31fe9357b388"
Jul 01 18:37:31.589827 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.589805    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 01 18:37:31.590016 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.589996    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kxmk9\""
Jul 01 18:37:31.590360 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.590343    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 01 18:37:31.590763 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.590747    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 01 18:37:31.590854 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.590839    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 01 18:37:31.591163 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.591149    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 01 18:37:31.591658 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.591641    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-knxvr\""
Jul 01 18:37:31.592203 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.592180    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 01 18:37:31.592285 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.592266    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 01 18:37:31.594058 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.592475    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 01 18:37:31.594058 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.592533    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 01 18:37:31.594058 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.593332    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 01 18:37:31.595354 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.595330    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4pmfd"
Jul 01 18:37:31.596657 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.596639    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dctc8"
Jul 01 18:37:31.597919 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.597901    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds"
Jul 01 18:37:31.598016 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.597917    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 01 18:37:31.598016 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.597995    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cn5xm\""
Jul 01 18:37:31.598340 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.598321    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 01 18:37:31.598944 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.598922    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 01 18:37:31.599015 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.598952    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 01 18:37:31.599015 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.598958    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xf8qd\""
Jul 01 18:37:31.599501 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.599415    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.599585 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.599568    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 01 18:37:31.600332 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.600315    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 01 18:37:31.600429 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.600349    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 01 18:37:31.600672 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.600574    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 01 18:37:31.600941 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.600800    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lzgn6\""
Jul 01 18:37:31.601018 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.600945    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l5m2d"
Jul 01 18:37:31.601669 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.601650    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 01 18:37:31.601803 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.601780    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 01 18:37:31.601952 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.601926    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 01 18:37:31.601952 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.601926    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 01 18:37:31.602080 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.602034    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hwttj\""
Jul 01 18:37:31.602209 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.602180    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 01 18:37:31.603026 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.603004    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2k6qn\""
Jul 01 18:37:31.603146 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.603132    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 01 18:37:31.603479 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.603462    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 01 18:37:31.603479 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.603470    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-etc-kubernetes\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.603610 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.603498    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-etc-sysctl-d\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.603610 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.603524    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-var-lib-kubelet\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.603610 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.603548    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-etc-tuned\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.603610 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.603590    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-host-run-ovn-kubernetes\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.603828 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.603615    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-original-pull-secret\") pod \"global-pull-secret-syncer-mrx57\" (UID: \"8bd0c6cc-1f64-4be9-8b7e-31fe9357b388\") " pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:31.603828 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.603640    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-lib-modules\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.603828 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.603664    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-var-lib-openvswitch\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.603828 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.603729    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-host-cni-bin\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.603828 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.603816    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.604053 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.603851    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b97a39a8-98bb-4dd5-88b6-c256ebb3c08e-agent-certs\") pod \"konnectivity-agent-4pmfd\" (UID: \"b97a39a8-98bb-4dd5-88b6-c256ebb3c08e\") " pod="kube-system/konnectivity-agent-4pmfd"
Jul 01 18:37:31.604053 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.603902    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5vjq\" (UniqueName: \"kubernetes.io/projected/6d3508bc-a5db-48ad-96f1-e7663193fe59-kube-api-access-k5vjq\") pod \"iptables-alerter-prmsf\" (UID: \"6d3508bc-a5db-48ad-96f1-e7663193fe59\") " pod="openshift-network-operator/iptables-alerter-prmsf"
Jul 01 18:37:31.604053 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.603967    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-host-cni-netd\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.604053 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604023    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-etc-systemd\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.604257 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604066    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-tmp\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.604257 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604092    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fv78\" (UniqueName: \"kubernetes.io/projected/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-kube-api-access-5fv78\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.604257 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604142    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-log-socket\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.604257 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604184    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d3508bc-a5db-48ad-96f1-e7663193fe59-host-slash\") pod \"iptables-alerter-prmsf\" (UID: \"6d3508bc-a5db-48ad-96f1-e7663193fe59\") " pod="openshift-network-operator/iptables-alerter-prmsf"
Jul 01 18:37:31.604257 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604223    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-host-slash\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.604257 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604258    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-host-run-netns\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.604539 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604281    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-etc-sysctl-conf\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.604539 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604333    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6d3508bc-a5db-48ad-96f1-e7663193fe59-iptables-alerter-script\") pod \"iptables-alerter-prmsf\" (UID: \"6d3508bc-a5db-48ad-96f1-e7663193fe59\") " pod="openshift-network-operator/iptables-alerter-prmsf"
Jul 01 18:37:31.604539 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604357    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-run-openvswitch\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.604539 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604382    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtf7g\" (UniqueName: \"kubernetes.io/projected/3709542d-b4c1-4e02-ba6e-8d917651974d-kube-api-access-gtf7g\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.604539 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604402    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-etc-modprobe-d\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.604539 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604432    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-sys\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.604539 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604463    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-run-systemd\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.604539 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604501    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-etc-openvswitch\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.604539 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604518    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3709542d-b4c1-4e02-ba6e-8d917651974d-ovnkube-config\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.604539 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604538    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3709542d-b4c1-4e02-ba6e-8d917651974d-env-overrides\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.604952 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604572    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-dbus\") pod \"global-pull-secret-syncer-mrx57\" (UID: \"8bd0c6cc-1f64-4be9-8b7e-31fe9357b388\") " pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:31.604952 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604605    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b97a39a8-98bb-4dd5-88b6-c256ebb3c08e-konnectivity-ca\") pod \"konnectivity-agent-4pmfd\" (UID: \"b97a39a8-98bb-4dd5-88b6-c256ebb3c08e\") " pod="kube-system/konnectivity-agent-4pmfd"
Jul 01 18:37:31.604952 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604631    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-run-ovn\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.604952 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604658    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.604952 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604685    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3709542d-b4c1-4e02-ba6e-8d917651974d-ovn-node-metrics-cert\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.604952 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604727    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3709542d-b4c1-4e02-ba6e-8d917651974d-ovnkube-script-lib\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.604952 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604759    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-systemd-units\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.604952 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604781    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-etc-sysconfig\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.604952 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604803    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-run\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.604952 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604825    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-host\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.604952 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604862    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs\") pod \"network-metrics-daemon-plc42\" (UID: \"5180da14-bd5e-4777-b34c-4041f623113b\") " pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:31.604952 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604885    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsvjq\" (UniqueName: \"kubernetes.io/projected/5180da14-bd5e-4777-b34c-4041f623113b-kube-api-access-nsvjq\") pod \"network-metrics-daemon-plc42\" (UID: \"5180da14-bd5e-4777-b34c-4041f623113b\") " pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:31.604952 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604921    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-node-log\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.604952 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604943    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:37:31.604952 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604955    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-kubelet-config\") pod \"global-pull-secret-syncer-mrx57\" (UID: \"8bd0c6cc-1f64-4be9-8b7e-31fe9357b388\") " pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:31.605552 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.604977    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-host-kubelet\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.605552 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:31.604998    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9t8tv" podUID="b105299c-7212-43c4-bb52-34aaaa2253f0"
Jul 01 18:37:31.606186 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.606136    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 01 18:37:31.606270 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.606194    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8sjzs\""
Jul 01 18:37:31.621811 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.621790    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 18:37:31.637161 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.637141    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9k9bt"
Jul 01 18:37:31.650834 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.650815    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9k9bt"
Jul 01 18:37:31.692977 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.692956    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 01 18:37:31.705505 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.705477    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-host-run-k8s-cni-cncf-io\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.705612 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.705511    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j7bd\" (UniqueName: \"kubernetes.io/projected/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-kube-api-access-8j7bd\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.705612 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.705539    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.705612 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.705563    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3709542d-b4c1-4e02-ba6e-8d917651974d-ovn-node-metrics-cert\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.705612 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.705585    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3709542d-b4c1-4e02-ba6e-8d917651974d-ovnkube-script-lib\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.705612 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.705605    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/92fa86b8-a872-4317-9b0a-c8452bec1f34-serviceca\") pod \"node-ca-dctc8\" (UID: \"92fa86b8-a872-4317-9b0a-c8452bec1f34\") " pod="openshift-image-registry/node-ca-dctc8"
Jul 01 18:37:31.705902 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.705613    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.705902 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.705625    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n4qd\" (UniqueName: \"kubernetes.io/projected/b105299c-7212-43c4-bb52-34aaaa2253f0-kube-api-access-2n4qd\") pod \"network-check-target-9t8tv\" (UID: \"b105299c-7212-43c4-bb52-34aaaa2253f0\") " pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:37:31.705902 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.705647    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-systemd-units\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.705902 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.705758    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-node-log\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.705902 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.705781    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0282f93f-4b16-4820-8b1e-c0f7c010fa82-system-cni-dir\") pod \"multus-additional-cni-plugins-nx7jq\" (UID: \"0282f93f-4b16-4820-8b1e-c0f7c010fa82\") " pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.705902 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.705839    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-systemd-units\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.705902 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.705842    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-node-log\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.706228 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.705911    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-etc-kubernetes\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.706228 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.705954    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-original-pull-secret\") pod \"global-pull-secret-syncer-mrx57\" (UID: \"8bd0c6cc-1f64-4be9-8b7e-31fe9357b388\") " pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:31.706228 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.705984    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-host-var-lib-cni-bin\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.706228 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.705991    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-etc-kubernetes\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.706228 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706093    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-host-var-lib-kubelet\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.706228 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706121    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-lib-modules\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.706228 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706140    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b97a39a8-98bb-4dd5-88b6-c256ebb3c08e-agent-certs\") pod \"konnectivity-agent-4pmfd\" (UID: \"b97a39a8-98bb-4dd5-88b6-c256ebb3c08e\") " pod="kube-system/konnectivity-agent-4pmfd"
Jul 01 18:37:31.706228 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706182    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-multus-cni-dir\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.706228 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706207    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3709542d-b4c1-4e02-ba6e-8d917651974d-ovnkube-script-lib\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.706678 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706209    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-cni-binary-copy\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.706678 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706281    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-host-run-multus-certs\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.706678 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706291    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-lib-modules\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.706678 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:31.706299    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:31.706678 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706310    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5vjq\" (UniqueName: \"kubernetes.io/projected/6d3508bc-a5db-48ad-96f1-e7663193fe59-kube-api-access-k5vjq\") pod \"iptables-alerter-prmsf\" (UID: \"6d3508bc-a5db-48ad-96f1-e7663193fe59\") " pod="openshift-network-operator/iptables-alerter-prmsf"
Jul 01 18:37:31.706678 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:31.706412    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-original-pull-secret podName:8bd0c6cc-1f64-4be9-8b7e-31fe9357b388 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:32.206375798 +0000 UTC m=+2.975213398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-original-pull-secret") pod "global-pull-secret-syncer-mrx57" (UID: "8bd0c6cc-1f64-4be9-8b7e-31fe9357b388") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:31.706678 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706433    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-host-run-netns\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.706678 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706488    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-etc-systemd\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.706678 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706516    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-tmp\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.706678 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706540    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fv78\" (UniqueName: \"kubernetes.io/projected/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-kube-api-access-5fv78\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.706678 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706552    2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 01 18:37:31.706678 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706563    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-log-socket\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.706678 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706589    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fv4f\" (UniqueName: \"kubernetes.io/projected/92fa86b8-a872-4317-9b0a-c8452bec1f34-kube-api-access-7fv4f\") pod \"node-ca-dctc8\" (UID: \"92fa86b8-a872-4317-9b0a-c8452bec1f34\") " pod="openshift-image-registry/node-ca-dctc8"
Jul 01 18:37:31.706678 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706560    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-etc-systemd\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.706678 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706613    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d6898f6-ff4c-4ea5-a95a-bc04245f72e2-registration-dir\") pod \"aws-ebs-csi-driver-node-d2zds\" (UID: \"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds"
Jul 01 18:37:31.706678 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706630    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-log-socket\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.706678 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706636    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d6898f6-ff4c-4ea5-a95a-bc04245f72e2-device-dir\") pod \"aws-ebs-csi-driver-node-d2zds\" (UID: \"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds"
Jul 01 18:37:31.707382 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706660    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d3508bc-a5db-48ad-96f1-e7663193fe59-host-slash\") pod \"iptables-alerter-prmsf\" (UID: \"6d3508bc-a5db-48ad-96f1-e7663193fe59\") " pod="openshift-network-operator/iptables-alerter-prmsf"
Jul 01 18:37:31.707382 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706693    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-host-slash\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.707382 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706746    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d3508bc-a5db-48ad-96f1-e7663193fe59-host-slash\") pod \"iptables-alerter-prmsf\" (UID: \"6d3508bc-a5db-48ad-96f1-e7663193fe59\") " pod="openshift-network-operator/iptables-alerter-prmsf"
Jul 01 18:37:31.707382 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706749    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-host-run-netns\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.707382 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706775    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-host-slash\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.707382 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706807    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6d3508bc-a5db-48ad-96f1-e7663193fe59-iptables-alerter-script\") pod \"iptables-alerter-prmsf\" (UID: \"6d3508bc-a5db-48ad-96f1-e7663193fe59\") " pod="openshift-network-operator/iptables-alerter-prmsf"
Jul 01 18:37:31.707382 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706897    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-run-openvswitch\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.707382 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.706879    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-host-run-netns\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.707382 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707005    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtf7g\" (UniqueName: \"kubernetes.io/projected/3709542d-b4c1-4e02-ba6e-8d917651974d-kube-api-access-gtf7g\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.707382 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707017    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-run-openvswitch\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.707382 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707056    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d6898f6-ff4c-4ea5-a95a-bc04245f72e2-sys-fs\") pod \"aws-ebs-csi-driver-node-d2zds\" (UID: \"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds"
Jul 01 18:37:31.707382 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707081    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0282f93f-4b16-4820-8b1e-c0f7c010fa82-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nx7jq\" (UID: \"0282f93f-4b16-4820-8b1e-c0f7c010fa82\") " pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.707382 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707102    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-etc-modprobe-d\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.707382 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707121    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3709542d-b4c1-4e02-ba6e-8d917651974d-ovnkube-config\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.707382 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707139    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3709542d-b4c1-4e02-ba6e-8d917651974d-env-overrides\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.707382 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707157    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b97a39a8-98bb-4dd5-88b6-c256ebb3c08e-konnectivity-ca\") pod \"konnectivity-agent-4pmfd\" (UID: \"b97a39a8-98bb-4dd5-88b6-c256ebb3c08e\") " pod="kube-system/konnectivity-agent-4pmfd"
Jul 01 18:37:31.707382 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707176    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0282f93f-4b16-4820-8b1e-c0f7c010fa82-cnibin\") pod \"multus-additional-cni-plugins-nx7jq\" (UID: \"0282f93f-4b16-4820-8b1e-c0f7c010fa82\") " pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.708196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707196    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0282f93f-4b16-4820-8b1e-c0f7c010fa82-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nx7jq\" (UID: \"0282f93f-4b16-4820-8b1e-c0f7c010fa82\") " pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.708196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707216    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-os-release\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.708196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707234    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-hostroot\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.708196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707254    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-run-ovn\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.708196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707275    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-etc-modprobe-d\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.708196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707286    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d6898f6-ff4c-4ea5-a95a-bc04245f72e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-d2zds\" (UID: \"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds"
Jul 01 18:37:31.708196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707307    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4ttl\" (UniqueName: \"kubernetes.io/projected/0282f93f-4b16-4820-8b1e-c0f7c010fa82-kube-api-access-w4ttl\") pod \"multus-additional-cni-plugins-nx7jq\" (UID: \"0282f93f-4b16-4820-8b1e-c0f7c010fa82\") " pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.708196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707335    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6d3508bc-a5db-48ad-96f1-e7663193fe59-iptables-alerter-script\") pod \"iptables-alerter-prmsf\" (UID: \"6d3508bc-a5db-48ad-96f1-e7663193fe59\") " pod="openshift-network-operator/iptables-alerter-prmsf"
Jul 01 18:37:31.708196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707336    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a42d5d-0f95-4aed-94f3-ea0b79e4e326-tmp-dir\") pod \"node-resolver-l5m2d\" (UID: \"f3a42d5d-0f95-4aed-94f3-ea0b79e4e326\") " pod="openshift-dns/node-resolver-l5m2d"
Jul 01 18:37:31.708196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707396    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-run-ovn\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.708196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707398    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbvw6\" (UniqueName: \"kubernetes.io/projected/f3a42d5d-0f95-4aed-94f3-ea0b79e4e326-kube-api-access-dbvw6\") pod \"node-resolver-l5m2d\" (UID: \"f3a42d5d-0f95-4aed-94f3-ea0b79e4e326\") " pod="openshift-dns/node-resolver-l5m2d"
Jul 01 18:37:31.708196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707429    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-multus-daemon-config\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.708196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707453    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-etc-kubernetes\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.708196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707515    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-etc-sysconfig\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.708196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707557    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-run\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.708196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707574    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-host\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.708196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707594    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs\") pod \"network-metrics-daemon-plc42\" (UID: \"5180da14-bd5e-4777-b34c-4041f623113b\") " pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:31.708990 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707642    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsvjq\" (UniqueName: \"kubernetes.io/projected/5180da14-bd5e-4777-b34c-4041f623113b-kube-api-access-nsvjq\") pod \"network-metrics-daemon-plc42\" (UID: \"5180da14-bd5e-4777-b34c-4041f623113b\") " pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:31.708990 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707663    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-kubelet-config\") pod \"global-pull-secret-syncer-mrx57\" (UID: \"8bd0c6cc-1f64-4be9-8b7e-31fe9357b388\") " pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:31.708990 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707681    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3709542d-b4c1-4e02-ba6e-8d917651974d-env-overrides\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.708990 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707685    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-host-kubelet\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.708990 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707732    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-host-kubelet\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.708990 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707744    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-etc-sysconfig\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.708990 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707748    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-etc-sysctl-d\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.708990 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707801    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-var-lib-kubelet\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.708990 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707817    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-run\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.708990 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707827    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-etc-tuned\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.708990 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707853    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-host-run-ovn-kubernetes\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.708990 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707883    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-kubelet-config\") pod \"global-pull-secret-syncer-mrx57\" (UID: \"8bd0c6cc-1f64-4be9-8b7e-31fe9357b388\") " pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:31.708990 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707878    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f3a42d5d-0f95-4aed-94f3-ea0b79e4e326-hosts-file\") pod \"node-resolver-l5m2d\" (UID: \"f3a42d5d-0f95-4aed-94f3-ea0b79e4e326\") " pod="openshift-dns/node-resolver-l5m2d"
Jul 01 18:37:31.708990 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707827    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-etc-sysctl-d\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.708990 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707920    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-multus-socket-dir-parent\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.708990 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:31.707886    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:31.708990 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707974    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-var-lib-kubelet\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.708990 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.707967    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b97a39a8-98bb-4dd5-88b6-c256ebb3c08e-konnectivity-ca\") pod \"konnectivity-agent-4pmfd\" (UID: \"b97a39a8-98bb-4dd5-88b6-c256ebb3c08e\") " pod="kube-system/konnectivity-agent-4pmfd"
Jul 01 18:37:31.709771 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708006    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-host-var-lib-cni-multus\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.709771 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708033    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3709542d-b4c1-4e02-ba6e-8d917651974d-ovnkube-config\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.709771 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708035    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-var-lib-openvswitch\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.709771 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:31.708057    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs podName:5180da14-bd5e-4777-b34c-4041f623113b nodeName:}" failed. No retries permitted until 2026-07-01 18:37:32.208041128 +0000 UTC m=+2.976878711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs") pod "network-metrics-daemon-plc42" (UID: "5180da14-bd5e-4777-b34c-4041f623113b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:31.709771 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708068    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-var-lib-openvswitch\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.709771 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708086    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-host-run-ovn-kubernetes\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.709771 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708088    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-host-cni-bin\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.709771 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708096    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-host\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.709771 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708124    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d6898f6-ff4c-4ea5-a95a-bc04245f72e2-socket-dir\") pod \"aws-ebs-csi-driver-node-d2zds\" (UID: \"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds"
Jul 01 18:37:31.709771 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708125    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-host-cni-bin\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.709771 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708157    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0282f93f-4b16-4820-8b1e-c0f7c010fa82-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nx7jq\" (UID: \"0282f93f-4b16-4820-8b1e-c0f7c010fa82\") " pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.709771 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708202    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-multus-conf-dir\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.709771 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708236    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-host-cni-netd\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.709771 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708284    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-host-cni-netd\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.709771 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708325    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0282f93f-4b16-4820-8b1e-c0f7c010fa82-cni-binary-copy\") pod \"multus-additional-cni-plugins-nx7jq\" (UID: \"0282f93f-4b16-4820-8b1e-c0f7c010fa82\") " pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.709771 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708352    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-system-cni-dir\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.710443 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708407    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-etc-sysctl-conf\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.710443 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708489    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/92fa86b8-a872-4317-9b0a-c8452bec1f34-host\") pod \"node-ca-dctc8\" (UID: \"92fa86b8-a872-4317-9b0a-c8452bec1f34\") " pod="openshift-image-registry/node-ca-dctc8"
Jul 01 18:37:31.710443 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708503    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-etc-sysctl-conf\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.710443 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708523    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-277t9\" (UniqueName: \"kubernetes.io/projected/1d6898f6-ff4c-4ea5-a95a-bc04245f72e2-kube-api-access-277t9\") pod \"aws-ebs-csi-driver-node-d2zds\" (UID: \"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds"
Jul 01 18:37:31.710443 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708564    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0282f93f-4b16-4820-8b1e-c0f7c010fa82-os-release\") pod \"multus-additional-cni-plugins-nx7jq\" (UID: \"0282f93f-4b16-4820-8b1e-c0f7c010fa82\") " pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.710443 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708591    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-sys\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.710443 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708620    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-run-systemd\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.710443 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708643    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-etc-openvswitch\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.710443 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708666    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-dbus\") pod \"global-pull-secret-syncer-mrx57\" (UID: \"8bd0c6cc-1f64-4be9-8b7e-31fe9357b388\") " pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:31.710443 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708685    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-sys\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.710443 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708691    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d6898f6-ff4c-4ea5-a95a-bc04245f72e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d2zds\" (UID: \"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds"
Jul 01 18:37:31.710443 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708699    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-run-systemd\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.710443 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708730    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-cnibin\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.710443 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708750    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3709542d-b4c1-4e02-ba6e-8d917651974d-etc-openvswitch\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.710443 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.708819    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-dbus\") pod \"global-pull-secret-syncer-mrx57\" (UID: \"8bd0c6cc-1f64-4be9-8b7e-31fe9357b388\") " pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:31.711128 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.710603    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-etc-tuned\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.711128 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.710684    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-tmp\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.711128 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.710742    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b97a39a8-98bb-4dd5-88b6-c256ebb3c08e-agent-certs\") pod \"konnectivity-agent-4pmfd\" (UID: \"b97a39a8-98bb-4dd5-88b6-c256ebb3c08e\") " pod="kube-system/konnectivity-agent-4pmfd"
Jul 01 18:37:31.711128 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.711011    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3709542d-b4c1-4e02-ba6e-8d917651974d-ovn-node-metrics-cert\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.719545 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.719518    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fv78\" (UniqueName: \"kubernetes.io/projected/99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25-kube-api-access-5fv78\") pod \"tuned-frw2w\" (UID: \"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25\") " pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.720407 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.720361    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsvjq\" (UniqueName: \"kubernetes.io/projected/5180da14-bd5e-4777-b34c-4041f623113b-kube-api-access-nsvjq\") pod \"network-metrics-daemon-plc42\" (UID: \"5180da14-bd5e-4777-b34c-4041f623113b\") " pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:31.721172 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.721137    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtf7g\" (UniqueName: \"kubernetes.io/projected/3709542d-b4c1-4e02-ba6e-8d917651974d-kube-api-access-gtf7g\") pod \"ovnkube-node-tjxh2\" (UID: \"3709542d-b4c1-4e02-ba6e-8d917651974d\") " pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.721418 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.721398    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5vjq\" (UniqueName: \"kubernetes.io/projected/6d3508bc-a5db-48ad-96f1-e7663193fe59-kube-api-access-k5vjq\") pod \"iptables-alerter-prmsf\" (UID: \"6d3508bc-a5db-48ad-96f1-e7663193fe59\") " pod="openshift-network-operator/iptables-alerter-prmsf"
Jul 01 18:37:31.809406 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.809375    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d6898f6-ff4c-4ea5-a95a-bc04245f72e2-sys-fs\") pod \"aws-ebs-csi-driver-node-d2zds\" (UID: \"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds"
Jul 01 18:37:31.809572 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.809412    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0282f93f-4b16-4820-8b1e-c0f7c010fa82-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nx7jq\" (UID: \"0282f93f-4b16-4820-8b1e-c0f7c010fa82\") " pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.809572 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.809438    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0282f93f-4b16-4820-8b1e-c0f7c010fa82-cnibin\") pod \"multus-additional-cni-plugins-nx7jq\" (UID: \"0282f93f-4b16-4820-8b1e-c0f7c010fa82\") " pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.809572 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.809489    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0282f93f-4b16-4820-8b1e-c0f7c010fa82-cnibin\") pod \"multus-additional-cni-plugins-nx7jq\" (UID: \"0282f93f-4b16-4820-8b1e-c0f7c010fa82\") " pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.809572 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.809506    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d6898f6-ff4c-4ea5-a95a-bc04245f72e2-sys-fs\") pod \"aws-ebs-csi-driver-node-d2zds\" (UID: \"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds"
Jul 01 18:37:31.809572 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.809521    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0282f93f-4b16-4820-8b1e-c0f7c010fa82-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nx7jq\" (UID: \"0282f93f-4b16-4820-8b1e-c0f7c010fa82\") " pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.809572 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.809543    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-os-release\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.809919 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.809616    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-os-release\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.809919 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.809620    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-hostroot\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.809919 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.809651    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d6898f6-ff4c-4ea5-a95a-bc04245f72e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-d2zds\" (UID: \"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds"
Jul 01 18:37:31.809919 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.809676    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4ttl\" (UniqueName: \"kubernetes.io/projected/0282f93f-4b16-4820-8b1e-c0f7c010fa82-kube-api-access-w4ttl\") pod \"multus-additional-cni-plugins-nx7jq\" (UID: \"0282f93f-4b16-4820-8b1e-c0f7c010fa82\") " pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.809919 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.809683    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-hostroot\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.809919 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.809700    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a42d5d-0f95-4aed-94f3-ea0b79e4e326-tmp-dir\") pod \"node-resolver-l5m2d\" (UID: \"f3a42d5d-0f95-4aed-94f3-ea0b79e4e326\") " pod="openshift-dns/node-resolver-l5m2d"
Jul 01 18:37:31.809919 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.809696    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0282f93f-4b16-4820-8b1e-c0f7c010fa82-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nx7jq\" (UID: \"0282f93f-4b16-4820-8b1e-c0f7c010fa82\") " pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.809919 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.809781    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbvw6\" (UniqueName: \"kubernetes.io/projected/f3a42d5d-0f95-4aed-94f3-ea0b79e4e326-kube-api-access-dbvw6\") pod \"node-resolver-l5m2d\" (UID: \"f3a42d5d-0f95-4aed-94f3-ea0b79e4e326\") " pod="openshift-dns/node-resolver-l5m2d"
Jul 01 18:37:31.809919 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.809811    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-multus-daemon-config\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.809919 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.809833    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d6898f6-ff4c-4ea5-a95a-bc04245f72e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-d2zds\" (UID: \"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds"
Jul 01 18:37:31.809919 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.809839    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-etc-kubernetes\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.809919 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.809870    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-etc-kubernetes\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.809919 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.809908    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f3a42d5d-0f95-4aed-94f3-ea0b79e4e326-hosts-file\") pod \"node-resolver-l5m2d\" (UID: \"f3a42d5d-0f95-4aed-94f3-ea0b79e4e326\") " pod="openshift-dns/node-resolver-l5m2d"
Jul 01 18:37:31.810533 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.809932    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-multus-socket-dir-parent\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.810533 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.809961    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-host-var-lib-cni-multus\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.810533 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.809992    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d6898f6-ff4c-4ea5-a95a-bc04245f72e2-socket-dir\") pod \"aws-ebs-csi-driver-node-d2zds\" (UID: \"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds"
Jul 01 18:37:31.810533 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810011    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a42d5d-0f95-4aed-94f3-ea0b79e4e326-tmp-dir\") pod \"node-resolver-l5m2d\" (UID: \"f3a42d5d-0f95-4aed-94f3-ea0b79e4e326\") " pod="openshift-dns/node-resolver-l5m2d"
Jul 01 18:37:31.810533 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810017    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0282f93f-4b16-4820-8b1e-c0f7c010fa82-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nx7jq\" (UID: \"0282f93f-4b16-4820-8b1e-c0f7c010fa82\") " pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.810533 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810020    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f3a42d5d-0f95-4aed-94f3-ea0b79e4e326-hosts-file\") pod \"node-resolver-l5m2d\" (UID: \"f3a42d5d-0f95-4aed-94f3-ea0b79e4e326\") " pod="openshift-dns/node-resolver-l5m2d"
Jul 01 18:37:31.810533 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810041    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-multus-conf-dir\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.810533 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810071    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-host-var-lib-cni-multus\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.810533 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810072    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0282f93f-4b16-4820-8b1e-c0f7c010fa82-cni-binary-copy\") pod \"multus-additional-cni-plugins-nx7jq\" (UID: \"0282f93f-4b16-4820-8b1e-c0f7c010fa82\") " pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.810533 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810095    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0282f93f-4b16-4820-8b1e-c0f7c010fa82-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nx7jq\" (UID: \"0282f93f-4b16-4820-8b1e-c0f7c010fa82\") " pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.810533 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810101    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-multus-socket-dir-parent\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.810533 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810128    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d6898f6-ff4c-4ea5-a95a-bc04245f72e2-socket-dir\") pod \"aws-ebs-csi-driver-node-d2zds\" (UID: \"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds"
Jul 01 18:37:31.810533 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810129    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-system-cni-dir\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.810533 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810177    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-multus-conf-dir\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.810533 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810179    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-system-cni-dir\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.810533 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810216    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/92fa86b8-a872-4317-9b0a-c8452bec1f34-host\") pod \"node-ca-dctc8\" (UID: \"92fa86b8-a872-4317-9b0a-c8452bec1f34\") " pod="openshift-image-registry/node-ca-dctc8"
Jul 01 18:37:31.810533 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810248    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-277t9\" (UniqueName: \"kubernetes.io/projected/1d6898f6-ff4c-4ea5-a95a-bc04245f72e2-kube-api-access-277t9\") pod \"aws-ebs-csi-driver-node-d2zds\" (UID: \"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds"
Jul 01 18:37:31.811245 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810273    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0282f93f-4b16-4820-8b1e-c0f7c010fa82-os-release\") pod \"multus-additional-cni-plugins-nx7jq\" (UID: \"0282f93f-4b16-4820-8b1e-c0f7c010fa82\") " pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.811245 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810282    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/92fa86b8-a872-4317-9b0a-c8452bec1f34-host\") pod \"node-ca-dctc8\" (UID: \"92fa86b8-a872-4317-9b0a-c8452bec1f34\") " pod="openshift-image-registry/node-ca-dctc8"
Jul 01 18:37:31.811245 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810300    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d6898f6-ff4c-4ea5-a95a-bc04245f72e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d2zds\" (UID: \"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds"
Jul 01 18:37:31.811245 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810324    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-cnibin\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.811245 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810347    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-host-run-k8s-cni-cncf-io\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.811245 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810372    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j7bd\" (UniqueName: \"kubernetes.io/projected/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-kube-api-access-8j7bd\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.811245 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810384    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d6898f6-ff4c-4ea5-a95a-bc04245f72e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d2zds\" (UID: \"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds"
Jul 01 18:37:31.811245 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810397    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/92fa86b8-a872-4317-9b0a-c8452bec1f34-serviceca\") pod \"node-ca-dctc8\" (UID: \"92fa86b8-a872-4317-9b0a-c8452bec1f34\") " pod="openshift-image-registry/node-ca-dctc8"
Jul 01 18:37:31.811245 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810348    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0282f93f-4b16-4820-8b1e-c0f7c010fa82-os-release\") pod \"multus-additional-cni-plugins-nx7jq\" (UID: \"0282f93f-4b16-4820-8b1e-c0f7c010fa82\") " pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.811245 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810422    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n4qd\" (UniqueName: \"kubernetes.io/projected/b105299c-7212-43c4-bb52-34aaaa2253f0-kube-api-access-2n4qd\") pod \"network-check-target-9t8tv\" (UID: \"b105299c-7212-43c4-bb52-34aaaa2253f0\") " pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:37:31.811245 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810440    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-host-run-k8s-cni-cncf-io\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.811245 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810484    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-cnibin\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.811245 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810491    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-multus-daemon-config\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.811245 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810553    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0282f93f-4b16-4820-8b1e-c0f7c010fa82-cni-binary-copy\") pod \"multus-additional-cni-plugins-nx7jq\" (UID: \"0282f93f-4b16-4820-8b1e-c0f7c010fa82\") " pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.811245 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810561    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0282f93f-4b16-4820-8b1e-c0f7c010fa82-system-cni-dir\") pod \"multus-additional-cni-plugins-nx7jq\" (UID: \"0282f93f-4b16-4820-8b1e-c0f7c010fa82\") " pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.811245 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810567    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0282f93f-4b16-4820-8b1e-c0f7c010fa82-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nx7jq\" (UID: \"0282f93f-4b16-4820-8b1e-c0f7c010fa82\") " pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.811245 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810602    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0282f93f-4b16-4820-8b1e-c0f7c010fa82-system-cni-dir\") pod \"multus-additional-cni-plugins-nx7jq\" (UID: \"0282f93f-4b16-4820-8b1e-c0f7c010fa82\") " pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.811758 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810627    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-host-var-lib-cni-bin\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.811758 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810644    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-host-var-lib-kubelet\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.811758 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810661    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-multus-cni-dir\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.811758 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810675    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-cni-binary-copy\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.811758 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810702    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-host-var-lib-cni-bin\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.811758 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810727    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-host-var-lib-kubelet\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.811758 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810732    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-host-run-multus-certs\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.811758 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810769    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-multus-cni-dir\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.811758 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810770    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-host-run-netns\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.811758 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810802    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-host-run-netns\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.811758 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810811    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fv4f\" (UniqueName: \"kubernetes.io/projected/92fa86b8-a872-4317-9b0a-c8452bec1f34-kube-api-access-7fv4f\") pod \"node-ca-dctc8\" (UID: \"92fa86b8-a872-4317-9b0a-c8452bec1f34\") " pod="openshift-image-registry/node-ca-dctc8"
Jul 01 18:37:31.811758 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810772    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-host-run-multus-certs\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.811758 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810839    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d6898f6-ff4c-4ea5-a95a-bc04245f72e2-registration-dir\") pod \"aws-ebs-csi-driver-node-d2zds\" (UID: \"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds"
Jul 01 18:37:31.811758 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810867    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d6898f6-ff4c-4ea5-a95a-bc04245f72e2-device-dir\") pod \"aws-ebs-csi-driver-node-d2zds\" (UID: \"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds"
Jul 01 18:37:31.811758 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810875    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/92fa86b8-a872-4317-9b0a-c8452bec1f34-serviceca\") pod \"node-ca-dctc8\" (UID: \"92fa86b8-a872-4317-9b0a-c8452bec1f34\") " pod="openshift-image-registry/node-ca-dctc8"
Jul 01 18:37:31.811758 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.810983    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d6898f6-ff4c-4ea5-a95a-bc04245f72e2-registration-dir\") pod \"aws-ebs-csi-driver-node-d2zds\" (UID: \"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds"
Jul 01 18:37:31.811758 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.811006    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d6898f6-ff4c-4ea5-a95a-bc04245f72e2-device-dir\") pod \"aws-ebs-csi-driver-node-d2zds\" (UID: \"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds"
Jul 01 18:37:31.811758 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.811171    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-cni-binary-copy\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.824732 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:31.824651    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:37:31.824732 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:31.824674    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:37:31.824732 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:31.824687    2573 projected.go:196] Error preparing data for projected volume kube-api-access-2n4qd for pod openshift-network-diagnostics/network-check-target-9t8tv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:31.825016 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:31.824801    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b105299c-7212-43c4-bb52-34aaaa2253f0-kube-api-access-2n4qd podName:b105299c-7212-43c4-bb52-34aaaa2253f0 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:32.32478368 +0000 UTC m=+3.093621279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2n4qd" (UniqueName: "kubernetes.io/projected/b105299c-7212-43c4-bb52-34aaaa2253f0-kube-api-access-2n4qd") pod "network-check-target-9t8tv" (UID: "b105299c-7212-43c4-bb52-34aaaa2253f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:31.826351 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.826298    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4ttl\" (UniqueName: \"kubernetes.io/projected/0282f93f-4b16-4820-8b1e-c0f7c010fa82-kube-api-access-w4ttl\") pod \"multus-additional-cni-plugins-nx7jq\" (UID: \"0282f93f-4b16-4820-8b1e-c0f7c010fa82\") " pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.826351 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.826318    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fv4f\" (UniqueName: \"kubernetes.io/projected/92fa86b8-a872-4317-9b0a-c8452bec1f34-kube-api-access-7fv4f\") pod \"node-ca-dctc8\" (UID: \"92fa86b8-a872-4317-9b0a-c8452bec1f34\") " pod="openshift-image-registry/node-ca-dctc8"
Jul 01 18:37:31.826534 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.826373    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbvw6\" (UniqueName: \"kubernetes.io/projected/f3a42d5d-0f95-4aed-94f3-ea0b79e4e326-kube-api-access-dbvw6\") pod \"node-resolver-l5m2d\" (UID: \"f3a42d5d-0f95-4aed-94f3-ea0b79e4e326\") " pod="openshift-dns/node-resolver-l5m2d"
Jul 01 18:37:31.826750 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.826705    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-277t9\" (UniqueName: \"kubernetes.io/projected/1d6898f6-ff4c-4ea5-a95a-bc04245f72e2-kube-api-access-277t9\") pod \"aws-ebs-csi-driver-node-d2zds\" (UID: \"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds"
Jul 01 18:37:31.828108 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.828093    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j7bd\" (UniqueName: \"kubernetes.io/projected/ab6bcc12-5bb6-4d85-8e61-cb188b50b51e-kube-api-access-8j7bd\") pod \"multus-4z9r8\" (UID: \"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e\") " pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:31.895928 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.895897    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-frw2w"
Jul 01 18:37:31.904071 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.904049    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-prmsf"
Jul 01 18:37:31.911636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.911616    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:31.916612 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.916593    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4pmfd"
Jul 01 18:37:31.924358 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.924329    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dctc8"
Jul 01 18:37:31.931881 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.931865    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds"
Jul 01 18:37:31.939428 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.939410    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nx7jq"
Jul 01 18:37:31.945946 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.945927    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l5m2d"
Jul 01 18:37:31.950530 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:31.950515    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4z9r8"
Jul 01 18:37:32.113573 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:32.113494    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:37:32.214121 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:32.214085    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-original-pull-secret\") pod \"global-pull-secret-syncer-mrx57\" (UID: \"8bd0c6cc-1f64-4be9-8b7e-31fe9357b388\") " pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:32.214271 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:32.214149    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs\") pod \"network-metrics-daemon-plc42\" (UID: \"5180da14-bd5e-4777-b34c-4041f623113b\") " pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:32.214327 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:32.214282    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:32.214379 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:32.214329    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:32.214379 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:32.214354    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs podName:5180da14-bd5e-4777-b34c-4041f623113b nodeName:}" failed. No retries permitted until 2026-07-01 18:37:33.214334452 +0000 UTC m=+3.983172033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs") pod "network-metrics-daemon-plc42" (UID: "5180da14-bd5e-4777-b34c-4041f623113b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:32.214499 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:32.214396    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-original-pull-secret podName:8bd0c6cc-1f64-4be9-8b7e-31fe9357b388 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:33.214382363 +0000 UTC m=+3.983219944 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-original-pull-secret") pod "global-pull-secret-syncer-mrx57" (UID: "8bd0c6cc-1f64-4be9-8b7e-31fe9357b388") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:32.415641 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:32.415560    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n4qd\" (UniqueName: \"kubernetes.io/projected/b105299c-7212-43c4-bb52-34aaaa2253f0-kube-api-access-2n4qd\") pod \"network-check-target-9t8tv\" (UID: \"b105299c-7212-43c4-bb52-34aaaa2253f0\") " pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:37:32.415833 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:32.415744    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:37:32.415833 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:32.415767    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:37:32.415833 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:32.415777    2573 projected.go:196] Error preparing data for projected volume kube-api-access-2n4qd for pod openshift-network-diagnostics/network-check-target-9t8tv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:32.415997 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:32.415839    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b105299c-7212-43c4-bb52-34aaaa2253f0-kube-api-access-2n4qd podName:b105299c-7212-43c4-bb52-34aaaa2253f0 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:33.415820629 +0000 UTC m=+4.184658218 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2n4qd" (UniqueName: "kubernetes.io/projected/b105299c-7212-43c4-bb52-34aaaa2253f0-kube-api-access-2n4qd") pod "network-check-target-9t8tv" (UID: "b105299c-7212-43c4-bb52-34aaaa2253f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:32.558483 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:32.558454    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3508bc_a5db_48ad_96f1_e7663193fe59.slice/crio-d0e68d65c9498857eb9f5ee51ec40b12a17b9a9cfa9e96c2898253c629462c21 WatchSource:0}: Error finding container d0e68d65c9498857eb9f5ee51ec40b12a17b9a9cfa9e96c2898253c629462c21: Status 404 returned error can't find the container with id d0e68d65c9498857eb9f5ee51ec40b12a17b9a9cfa9e96c2898253c629462c21
Jul 01 18:37:32.560759 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:32.560658    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3709542d_b4c1_4e02_ba6e_8d917651974d.slice/crio-1feae04d9527848126230f08b03c9b4b326f8a83e5b36dc3d5134e27b5030245 WatchSource:0}: Error finding container 1feae04d9527848126230f08b03c9b4b326f8a83e5b36dc3d5134e27b5030245: Status 404 returned error can't find the container with id 1feae04d9527848126230f08b03c9b4b326f8a83e5b36dc3d5134e27b5030245
Jul 01 18:37:32.562154 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:32.561925    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0282f93f_4b16_4820_8b1e_c0f7c010fa82.slice/crio-3ca1d4643c39e379c2fe066066198ba2163b276af783f196a8c24b2e9f640738 WatchSource:0}: Error finding container 3ca1d4643c39e379c2fe066066198ba2163b276af783f196a8c24b2e9f640738: Status 404 returned error can't find the container with id 3ca1d4643c39e379c2fe066066198ba2163b276af783f196a8c24b2e9f640738
Jul 01 18:37:32.564563 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:32.564540    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97a39a8_98bb_4dd5_88b6_c256ebb3c08e.slice/crio-7c77fea3ced5a16f216cefd9d8d7e88801f23e0f2c2d9e89092a66376762ebab WatchSource:0}: Error finding container 7c77fea3ced5a16f216cefd9d8d7e88801f23e0f2c2d9e89092a66376762ebab: Status 404 returned error can't find the container with id 7c77fea3ced5a16f216cefd9d8d7e88801f23e0f2c2d9e89092a66376762ebab
Jul 01 18:37:32.565315 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:32.565297    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab6bcc12_5bb6_4d85_8e61_cb188b50b51e.slice/crio-a2dfeaab3e3a0ce4c2ed6776723520fdd49a147322fd8e8a9b239329b7af522d WatchSource:0}: Error finding container a2dfeaab3e3a0ce4c2ed6776723520fdd49a147322fd8e8a9b239329b7af522d: Status 404 returned error can't find the container with id a2dfeaab3e3a0ce4c2ed6776723520fdd49a147322fd8e8a9b239329b7af522d
Jul 01 18:37:32.567384 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:32.567363    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d6898f6_ff4c_4ea5_a95a_bc04245f72e2.slice/crio-438988de5e8344ac9ab1f59dda82c5bd537ac4008eea55efa2deb3bf658c29a0 WatchSource:0}: Error finding container 438988de5e8344ac9ab1f59dda82c5bd537ac4008eea55efa2deb3bf658c29a0: Status 404 returned error can't find the container with id 438988de5e8344ac9ab1f59dda82c5bd537ac4008eea55efa2deb3bf658c29a0
Jul 01 18:37:32.568171 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:32.568079    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99fa0e9b_7b1f_4dd9_8029_c6d90d6dfb25.slice/crio-1b6f52d4371085fc8c13c0b2dcca73be6ba7e4154e8a68c2423e1ee13aefeee9 WatchSource:0}: Error finding container 1b6f52d4371085fc8c13c0b2dcca73be6ba7e4154e8a68c2423e1ee13aefeee9: Status 404 returned error can't find the container with id 1b6f52d4371085fc8c13c0b2dcca73be6ba7e4154e8a68c2423e1ee13aefeee9
Jul 01 18:37:32.569109 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:32.569066    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92fa86b8_a872_4317_9b0a_c8452bec1f34.slice/crio-4cd1eb348130c107c88ba86d483432273532d850d663d6c4e2c08862395e9c57 WatchSource:0}: Error finding container 4cd1eb348130c107c88ba86d483432273532d850d663d6c4e2c08862395e9c57: Status 404 returned error can't find the container with id 4cd1eb348130c107c88ba86d483432273532d850d663d6c4e2c08862395e9c57
Jul 01 18:37:32.571247 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:37:32.571212    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a42d5d_0f95_4aed_94f3_ea0b79e4e326.slice/crio-9a6fbb89b74f311d5e1f77fe56d5ee4bb40d1555ddacaa1489d2db84c230ad7a WatchSource:0}: Error finding container 9a6fbb89b74f311d5e1f77fe56d5ee4bb40d1555ddacaa1489d2db84c230ad7a: Status 404 returned error can't find the container with id 9a6fbb89b74f311d5e1f77fe56d5ee4bb40d1555ddacaa1489d2db84c230ad7a
Jul 01 18:37:32.652494 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:32.652455    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 18:32:31 +0000 UTC" deadline="2028-02-20 14:19:09.979874983 +0000 UTC"
Jul 01 18:37:32.652494 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:32.652493    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14371h41m37.327384269s"
Jul 01 18:37:32.669440 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:32.669385    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-245.ec2.internal" event={"ID":"8a07c43805922ed70622b7aec23d988f","Type":"ContainerStarted","Data":"030519c2f4cc6f33fb94500c3f76b2e881d92dbca1be4c9d8a6c4690c3c6cf5b"}
Jul 01 18:37:32.670425 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:32.670403    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l5m2d" event={"ID":"f3a42d5d-0f95-4aed-94f3-ea0b79e4e326","Type":"ContainerStarted","Data":"9a6fbb89b74f311d5e1f77fe56d5ee4bb40d1555ddacaa1489d2db84c230ad7a"}
Jul 01 18:37:32.671335 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:32.671312    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dctc8" event={"ID":"92fa86b8-a872-4317-9b0a-c8452bec1f34","Type":"ContainerStarted","Data":"4cd1eb348130c107c88ba86d483432273532d850d663d6c4e2c08862395e9c57"}
Jul 01 18:37:32.677757 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:32.677733    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4z9r8" event={"ID":"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e","Type":"ContainerStarted","Data":"a2dfeaab3e3a0ce4c2ed6776723520fdd49a147322fd8e8a9b239329b7af522d"}
Jul 01 18:37:32.680678 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:32.680657    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nx7jq" event={"ID":"0282f93f-4b16-4820-8b1e-c0f7c010fa82","Type":"ContainerStarted","Data":"3ca1d4643c39e379c2fe066066198ba2163b276af783f196a8c24b2e9f640738"}
Jul 01 18:37:32.685066 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:32.685046    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-frw2w" event={"ID":"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25","Type":"ContainerStarted","Data":"1b6f52d4371085fc8c13c0b2dcca73be6ba7e4154e8a68c2423e1ee13aefeee9"}
Jul 01 18:37:32.686859 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:32.686840    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds" event={"ID":"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2","Type":"ContainerStarted","Data":"438988de5e8344ac9ab1f59dda82c5bd537ac4008eea55efa2deb3bf658c29a0"}
Jul 01 18:37:32.688013 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:32.687993    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4pmfd" event={"ID":"b97a39a8-98bb-4dd5-88b6-c256ebb3c08e","Type":"ContainerStarted","Data":"7c77fea3ced5a16f216cefd9d8d7e88801f23e0f2c2d9e89092a66376762ebab"}
Jul 01 18:37:32.689908 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:32.689885    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2" event={"ID":"3709542d-b4c1-4e02-ba6e-8d917651974d","Type":"ContainerStarted","Data":"1feae04d9527848126230f08b03c9b4b326f8a83e5b36dc3d5134e27b5030245"}
Jul 01 18:37:32.690754 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:32.690736    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-prmsf" event={"ID":"6d3508bc-a5db-48ad-96f1-e7663193fe59","Type":"ContainerStarted","Data":"d0e68d65c9498857eb9f5ee51ec40b12a17b9a9cfa9e96c2898253c629462c21"}
Jul 01 18:37:32.693437 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:32.693401    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-245.ec2.internal" podStartSLOduration=1.6933894619999998 podStartE2EDuration="1.693389462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:32.693023954 +0000 UTC m=+3.461861557" watchObservedRunningTime="2026-07-01 18:37:32.693389462 +0000 UTC m=+3.462227064"
Jul 01 18:37:33.220924 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:33.220658    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs\") pod \"network-metrics-daemon-plc42\" (UID: \"5180da14-bd5e-4777-b34c-4041f623113b\") " pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:33.221088 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:33.221002    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-original-pull-secret\") pod \"global-pull-secret-syncer-mrx57\" (UID: \"8bd0c6cc-1f64-4be9-8b7e-31fe9357b388\") " pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:33.221167 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:33.220861    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:33.221216 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:33.221205    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs podName:5180da14-bd5e-4777-b34c-4041f623113b nodeName:}" failed. No retries permitted until 2026-07-01 18:37:35.221185706 +0000 UTC m=+5.990023307 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs") pod "network-metrics-daemon-plc42" (UID: "5180da14-bd5e-4777-b34c-4041f623113b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:33.221539 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:33.221140    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:33.221692 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:33.221579    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-original-pull-secret podName:8bd0c6cc-1f64-4be9-8b7e-31fe9357b388 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:35.221562991 +0000 UTC m=+5.990400573 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-original-pull-secret") pod "global-pull-secret-syncer-mrx57" (UID: "8bd0c6cc-1f64-4be9-8b7e-31fe9357b388") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:33.423081 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:33.422959    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n4qd\" (UniqueName: \"kubernetes.io/projected/b105299c-7212-43c4-bb52-34aaaa2253f0-kube-api-access-2n4qd\") pod \"network-check-target-9t8tv\" (UID: \"b105299c-7212-43c4-bb52-34aaaa2253f0\") " pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:37:33.423247 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:33.423144    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:37:33.423247 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:33.423186    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:37:33.423247 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:33.423200    2573 projected.go:196] Error preparing data for projected volume kube-api-access-2n4qd for pod openshift-network-diagnostics/network-check-target-9t8tv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:33.423404 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:33.423259    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b105299c-7212-43c4-bb52-34aaaa2253f0-kube-api-access-2n4qd podName:b105299c-7212-43c4-bb52-34aaaa2253f0 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:35.423241186 +0000 UTC m=+6.192078784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2n4qd" (UniqueName: "kubernetes.io/projected/b105299c-7212-43c4-bb52-34aaaa2253f0-kube-api-access-2n4qd") pod "network-check-target-9t8tv" (UID: "b105299c-7212-43c4-bb52-34aaaa2253f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:33.653407 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:33.653337    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 18:32:31 +0000 UTC" deadline="2028-02-10 22:11:29.878697588 +0000 UTC"
Jul 01 18:37:33.653407 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:33.653375    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14139h33m56.225326528s"
Jul 01 18:37:33.663284 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:33.662373    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:33.663284 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:33.662497    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrx57" podUID="8bd0c6cc-1f64-4be9-8b7e-31fe9357b388"
Jul 01 18:37:33.663284 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:33.662930    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:33.663284 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:33.663033    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plc42" podUID="5180da14-bd5e-4777-b34c-4041f623113b"
Jul 01 18:37:33.663284 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:33.663091    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:37:33.663284 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:33.663158    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9t8tv" podUID="b105299c-7212-43c4-bb52-34aaaa2253f0"
Jul 01 18:37:33.702810 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:33.702779    2573 generic.go:356] "Generic (PLEG): container finished" podID="b568c82d895ac42942951c7fc4260965" containerID="1cc88c1da12341838209f4d938551b0d0488bb0595275a313226e55b8ca4ff08" exitCode=0
Jul 01 18:37:33.703358 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:33.703331    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal" event={"ID":"b568c82d895ac42942951c7fc4260965","Type":"ContainerDied","Data":"1cc88c1da12341838209f4d938551b0d0488bb0595275a313226e55b8ca4ff08"}
Jul 01 18:37:34.713241 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:34.713201    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal" event={"ID":"b568c82d895ac42942951c7fc4260965","Type":"ContainerStarted","Data":"9369d5240c7993eed57a025d0226e5a53088e5e9a83670509faeb45ea0a7af20"}
Jul 01 18:37:35.239787 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:35.238920    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs\") pod \"network-metrics-daemon-plc42\" (UID: \"5180da14-bd5e-4777-b34c-4041f623113b\") " pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:35.239787 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:35.239009    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-original-pull-secret\") pod \"global-pull-secret-syncer-mrx57\" (UID: \"8bd0c6cc-1f64-4be9-8b7e-31fe9357b388\") " pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:35.239787 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:35.239155    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:35.239787 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:35.239217    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-original-pull-secret podName:8bd0c6cc-1f64-4be9-8b7e-31fe9357b388 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:39.239199469 +0000 UTC m=+10.008037072 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-original-pull-secret") pod "global-pull-secret-syncer-mrx57" (UID: "8bd0c6cc-1f64-4be9-8b7e-31fe9357b388") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:35.239787 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:35.239304    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:35.239787 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:35.239342    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs podName:5180da14-bd5e-4777-b34c-4041f623113b nodeName:}" failed. No retries permitted until 2026-07-01 18:37:39.239329231 +0000 UTC m=+10.008166817 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs") pod "network-metrics-daemon-plc42" (UID: "5180da14-bd5e-4777-b34c-4041f623113b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:35.440111 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:35.440076    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n4qd\" (UniqueName: \"kubernetes.io/projected/b105299c-7212-43c4-bb52-34aaaa2253f0-kube-api-access-2n4qd\") pod \"network-check-target-9t8tv\" (UID: \"b105299c-7212-43c4-bb52-34aaaa2253f0\") " pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:37:35.440286 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:35.440268    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:37:35.440286 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:35.440286    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:37:35.440392 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:35.440300    2573 projected.go:196] Error preparing data for projected volume kube-api-access-2n4qd for pod openshift-network-diagnostics/network-check-target-9t8tv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:35.440392 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:35.440356    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b105299c-7212-43c4-bb52-34aaaa2253f0-kube-api-access-2n4qd podName:b105299c-7212-43c4-bb52-34aaaa2253f0 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:39.44033805 +0000 UTC m=+10.209175650 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2n4qd" (UniqueName: "kubernetes.io/projected/b105299c-7212-43c4-bb52-34aaaa2253f0-kube-api-access-2n4qd") pod "network-check-target-9t8tv" (UID: "b105299c-7212-43c4-bb52-34aaaa2253f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:35.660276 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:35.660202    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:35.660444 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:35.660343    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plc42" podUID="5180da14-bd5e-4777-b34c-4041f623113b"
Jul 01 18:37:35.660891 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:35.660753    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:37:35.660891 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:35.660853    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9t8tv" podUID="b105299c-7212-43c4-bb52-34aaaa2253f0"
Jul 01 18:37:35.661082 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:35.660904    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:35.661082 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:35.660981    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrx57" podUID="8bd0c6cc-1f64-4be9-8b7e-31fe9357b388"
Jul 01 18:37:37.661143 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:37.661084    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:37.661143 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:37.661121    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:37:37.661655 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:37.661082    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:37.661655 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:37.661224    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plc42" podUID="5180da14-bd5e-4777-b34c-4041f623113b"
Jul 01 18:37:37.661655 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:37.661311    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrx57" podUID="8bd0c6cc-1f64-4be9-8b7e-31fe9357b388"
Jul 01 18:37:37.661655 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:37.661394    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9t8tv" podUID="b105299c-7212-43c4-bb52-34aaaa2253f0"
Jul 01 18:37:39.273184 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:39.273137    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-original-pull-secret\") pod \"global-pull-secret-syncer-mrx57\" (UID: \"8bd0c6cc-1f64-4be9-8b7e-31fe9357b388\") " pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:39.273633 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:39.273218    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs\") pod \"network-metrics-daemon-plc42\" (UID: \"5180da14-bd5e-4777-b34c-4041f623113b\") " pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:39.273633 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:39.273362    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:39.273633 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:39.273424    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs podName:5180da14-bd5e-4777-b34c-4041f623113b nodeName:}" failed. No retries permitted until 2026-07-01 18:37:47.27340448 +0000 UTC m=+18.042242076 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs") pod "network-metrics-daemon-plc42" (UID: "5180da14-bd5e-4777-b34c-4041f623113b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:39.273940 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:39.273851    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:39.273940 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:39.273905    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-original-pull-secret podName:8bd0c6cc-1f64-4be9-8b7e-31fe9357b388 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:47.273889273 +0000 UTC m=+18.042726869 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-original-pull-secret") pod "global-pull-secret-syncer-mrx57" (UID: "8bd0c6cc-1f64-4be9-8b7e-31fe9357b388") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:39.474850 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:39.474812    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n4qd\" (UniqueName: \"kubernetes.io/projected/b105299c-7212-43c4-bb52-34aaaa2253f0-kube-api-access-2n4qd\") pod \"network-check-target-9t8tv\" (UID: \"b105299c-7212-43c4-bb52-34aaaa2253f0\") " pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:37:39.475047 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:39.474991    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:37:39.475047 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:39.475011    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:37:39.475047 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:39.475025    2573 projected.go:196] Error preparing data for projected volume kube-api-access-2n4qd for pod openshift-network-diagnostics/network-check-target-9t8tv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:39.475208 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:39.475089    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b105299c-7212-43c4-bb52-34aaaa2253f0-kube-api-access-2n4qd podName:b105299c-7212-43c4-bb52-34aaaa2253f0 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:47.475069391 +0000 UTC m=+18.243906974 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2n4qd" (UniqueName: "kubernetes.io/projected/b105299c-7212-43c4-bb52-34aaaa2253f0-kube-api-access-2n4qd") pod "network-check-target-9t8tv" (UID: "b105299c-7212-43c4-bb52-34aaaa2253f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:39.661469 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:39.661397    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:39.661627 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:39.661499    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrx57" podUID="8bd0c6cc-1f64-4be9-8b7e-31fe9357b388"
Jul 01 18:37:39.661627 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:39.661562    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:39.661754 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:39.661634    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plc42" podUID="5180da14-bd5e-4777-b34c-4041f623113b"
Jul 01 18:37:39.661754 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:39.661682    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:37:39.661869 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:39.661769    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9t8tv" podUID="b105299c-7212-43c4-bb52-34aaaa2253f0"
Jul 01 18:37:41.660562 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:41.660531    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:41.661041 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:41.660682    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:37:41.661041 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:41.660774    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9t8tv" podUID="b105299c-7212-43c4-bb52-34aaaa2253f0"
Jul 01 18:37:41.661041 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:41.660679    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plc42" podUID="5180da14-bd5e-4777-b34c-4041f623113b"
Jul 01 18:37:41.661041 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:41.660806    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:41.661041 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:41.660877    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrx57" podUID="8bd0c6cc-1f64-4be9-8b7e-31fe9357b388"
Jul 01 18:37:43.663698 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:43.663599    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:37:43.664200 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:43.663732    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:43.664200 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:43.663739    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9t8tv" podUID="b105299c-7212-43c4-bb52-34aaaa2253f0"
Jul 01 18:37:43.664200 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:43.663772    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:43.664200 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:43.663873    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plc42" podUID="5180da14-bd5e-4777-b34c-4041f623113b"
Jul 01 18:37:43.664200 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:43.663937    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrx57" podUID="8bd0c6cc-1f64-4be9-8b7e-31fe9357b388"
Jul 01 18:37:45.661133 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:45.661097    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:37:45.661133 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:45.661127    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:45.661669 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:45.661134    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:45.661669 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:45.661205    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9t8tv" podUID="b105299c-7212-43c4-bb52-34aaaa2253f0"
Jul 01 18:37:45.661669 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:45.661309    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrx57" podUID="8bd0c6cc-1f64-4be9-8b7e-31fe9357b388"
Jul 01 18:37:45.661669 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:45.661386    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plc42" podUID="5180da14-bd5e-4777-b34c-4041f623113b"
Jul 01 18:37:47.335526 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:47.335478    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs\") pod \"network-metrics-daemon-plc42\" (UID: \"5180da14-bd5e-4777-b34c-4041f623113b\") " pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:47.336015 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:47.335575    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-original-pull-secret\") pod \"global-pull-secret-syncer-mrx57\" (UID: \"8bd0c6cc-1f64-4be9-8b7e-31fe9357b388\") " pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:47.336015 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:47.335659    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:47.336015 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:47.335724    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:47.336015 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:47.335750    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs podName:5180da14-bd5e-4777-b34c-4041f623113b nodeName:}" failed. No retries permitted until 2026-07-01 18:38:03.335728792 +0000 UTC m=+34.104566394 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs") pod "network-metrics-daemon-plc42" (UID: "5180da14-bd5e-4777-b34c-4041f623113b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:47.336015 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:47.335778    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-original-pull-secret podName:8bd0c6cc-1f64-4be9-8b7e-31fe9357b388 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:03.335760859 +0000 UTC m=+34.104598507 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-original-pull-secret") pod "global-pull-secret-syncer-mrx57" (UID: "8bd0c6cc-1f64-4be9-8b7e-31fe9357b388") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:47.537661 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:47.537630    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n4qd\" (UniqueName: \"kubernetes.io/projected/b105299c-7212-43c4-bb52-34aaaa2253f0-kube-api-access-2n4qd\") pod \"network-check-target-9t8tv\" (UID: \"b105299c-7212-43c4-bb52-34aaaa2253f0\") " pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:37:47.537863 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:47.537842    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:37:47.537921 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:47.537870    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:37:47.537921 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:47.537883    2573 projected.go:196] Error preparing data for projected volume kube-api-access-2n4qd for pod openshift-network-diagnostics/network-check-target-9t8tv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:47.537985 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:47.537947    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b105299c-7212-43c4-bb52-34aaaa2253f0-kube-api-access-2n4qd podName:b105299c-7212-43c4-bb52-34aaaa2253f0 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:03.537928641 +0000 UTC m=+34.306766235 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2n4qd" (UniqueName: "kubernetes.io/projected/b105299c-7212-43c4-bb52-34aaaa2253f0-kube-api-access-2n4qd") pod "network-check-target-9t8tv" (UID: "b105299c-7212-43c4-bb52-34aaaa2253f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:47.663961 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:47.663884    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:37:47.663961 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:47.663915    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:47.663961 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:47.663922    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:47.664196 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:47.664002    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9t8tv" podUID="b105299c-7212-43c4-bb52-34aaaa2253f0"
Jul 01 18:37:47.664196 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:47.664104    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrx57" podUID="8bd0c6cc-1f64-4be9-8b7e-31fe9357b388"
Jul 01 18:37:47.664196 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:47.664172    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plc42" podUID="5180da14-bd5e-4777-b34c-4041f623113b"
Jul 01 18:37:49.660923 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:49.660899    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:49.661383 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:49.661015    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrx57" podUID="8bd0c6cc-1f64-4be9-8b7e-31fe9357b388"
Jul 01 18:37:49.661383 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:49.661093    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:37:49.661383 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:49.661201    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9t8tv" podUID="b105299c-7212-43c4-bb52-34aaaa2253f0"
Jul 01 18:37:49.661383 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:49.661206    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:49.661383 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:49.661308    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plc42" podUID="5180da14-bd5e-4777-b34c-4041f623113b"
Jul 01 18:37:51.660889 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:51.660508    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:51.661568 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:51.660538    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:37:51.661568 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:51.661009    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plc42" podUID="5180da14-bd5e-4777-b34c-4041f623113b"
Jul 01 18:37:51.661568 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:51.660589    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:51.661568 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:51.661067    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9t8tv" podUID="b105299c-7212-43c4-bb52-34aaaa2253f0"
Jul 01 18:37:51.661568 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:51.661160    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrx57" podUID="8bd0c6cc-1f64-4be9-8b7e-31fe9357b388"
Jul 01 18:37:51.745564 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:51.745530    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4pmfd" event={"ID":"b97a39a8-98bb-4dd5-88b6-c256ebb3c08e","Type":"ContainerStarted","Data":"23aaba22dfe98417f98e44a28a8d021c93be41b213af1db922a98ab6837e6825"}
Jul 01 18:37:51.748040 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:51.748016    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2" event={"ID":"3709542d-b4c1-4e02-ba6e-8d917651974d","Type":"ContainerStarted","Data":"85980f1773b8fa39d97216e0882f73cec9d57cd37ae8b336d147807d9dd0eebd"}
Jul 01 18:37:51.748146 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:51.748082    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2" event={"ID":"3709542d-b4c1-4e02-ba6e-8d917651974d","Type":"ContainerStarted","Data":"380719ea355f55b5942605578f4bd9f30a649d55aa181f2108068abef46b2ddc"}
Jul 01 18:37:51.748146 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:51.748117    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2" event={"ID":"3709542d-b4c1-4e02-ba6e-8d917651974d","Type":"ContainerStarted","Data":"b5250899f50dfc4789e69a6c1e390a53dd73e98f804f102f80f9b2297eff08a2"}
Jul 01 18:37:51.748239 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:51.748155    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2" event={"ID":"3709542d-b4c1-4e02-ba6e-8d917651974d","Type":"ContainerStarted","Data":"93947f9a606f293f9698a9554e6ae833180855934f4ffcf1e35b279b8436277f"}
Jul 01 18:37:51.748239 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:51.748195    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2" event={"ID":"3709542d-b4c1-4e02-ba6e-8d917651974d","Type":"ContainerStarted","Data":"6c4077f9d9e4c1fcf2156f81aca6d74b718e0792c2c2f4e6b9e81960c0d5a765"}
Jul 01 18:37:51.749221 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:51.749199    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l5m2d" event={"ID":"f3a42d5d-0f95-4aed-94f3-ea0b79e4e326","Type":"ContainerStarted","Data":"700bf1846122b06da65c1832adc0658a058e6d39549dddff8b272f9d10553e59"}
Jul 01 18:37:51.750422 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:51.750401    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dctc8" event={"ID":"92fa86b8-a872-4317-9b0a-c8452bec1f34","Type":"ContainerStarted","Data":"c2372d7562ebb1812e167dc974ce6ba65dd15508bde7fb04f1ee57a8b97ce9fb"}
Jul 01 18:37:51.751875 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:51.751851    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4z9r8" event={"ID":"ab6bcc12-5bb6-4d85-8e61-cb188b50b51e","Type":"ContainerStarted","Data":"52044334dec67b4693cc34de10b8a840184e3813e5d000ce5267893782f437df"}
Jul 01 18:37:51.753438 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:51.753419    2573 generic.go:356] "Generic (PLEG): container finished" podID="0282f93f-4b16-4820-8b1e-c0f7c010fa82" containerID="72ad6d814f7adfabce82c103216d827d68d10c5d481734b74ec121574126b65a" exitCode=0
Jul 01 18:37:51.753531 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:51.753482    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nx7jq" event={"ID":"0282f93f-4b16-4820-8b1e-c0f7c010fa82","Type":"ContainerDied","Data":"72ad6d814f7adfabce82c103216d827d68d10c5d481734b74ec121574126b65a"}
Jul 01 18:37:51.755055 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:51.755034    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-frw2w" event={"ID":"99fa0e9b-7b1f-4dd9-8029-c6d90d6dfb25","Type":"ContainerStarted","Data":"e2c36a39321ab90052864f2b74030370f9d2d50ad417f5fd6b5220a1eedf6ff7"}
Jul 01 18:37:51.756509 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:51.756481    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds" event={"ID":"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2","Type":"ContainerStarted","Data":"88531d21a59225f6c9257b213dc1b376a0608343a6dd80cc1757ebbf8fe9aab8"}
Jul 01 18:37:51.764974 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:51.764936    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4pmfd" podStartSLOduration=4.15743597 podStartE2EDuration="22.764925442s" totalImagesPullingTime="18.607489472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:51.764640659 +0000 UTC m=+22.533478257" watchObservedRunningTime="2026-07-01 18:37:51.764925442 +0000 UTC m=+22.533763045"
Jul 01 18:37:51.765185 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:51.765159    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal" podStartSLOduration=21.765152289 podStartE2EDuration="21.765152289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:34.738866401 +0000 UTC m=+5.507704010" watchObservedRunningTime="2026-07-01 18:37:51.765152289 +0000 UTC m=+22.533989892"
Jul 01 18:37:51.807163 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:51.807125    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-frw2w" podStartSLOduration=4.2025233029999995 podStartE2EDuration="22.807110665s" totalImagesPullingTime="18.604587362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:51.785096312 +0000 UTC m=+22.553933915" watchObservedRunningTime="2026-07-01 18:37:51.807110665 +0000 UTC m=+22.575948299"
Jul 01 18:37:51.841036 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:51.840962    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dctc8" podStartSLOduration=4.232628021 podStartE2EDuration="22.840942019s" totalImagesPullingTime="18.608313998s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:51.807526272 +0000 UTC m=+22.576363871" watchObservedRunningTime="2026-07-01 18:37:51.840942019 +0000 UTC m=+22.609779624"
Jul 01 18:37:51.859348 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:51.859293    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-l5m2d" podStartSLOduration=4.25753478 podStartE2EDuration="22.859277834s" totalImagesPullingTime="18.601743054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:51.858836707 +0000 UTC m=+22.627674311" watchObservedRunningTime="2026-07-01 18:37:51.859277834 +0000 UTC m=+22.628115438"
Jul 01 18:37:51.888919 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:51.888865    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4z9r8" podStartSLOduration=4.227069814 podStartE2EDuration="22.888851384s" totalImagesPullingTime="18.66178157s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:51.888597982 +0000 UTC m=+22.657435586" watchObservedRunningTime="2026-07-01 18:37:51.888851384 +0000 UTC m=+22.657688987"
Jul 01 18:37:51.943000 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:51.942921    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:37:52.028638 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:52.028596    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:37:52.126153 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:52.126122    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:37:52.682539 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:52.682502    2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 01 18:37:52.760731 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:52.760688    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds" event={"ID":"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2","Type":"ContainerStarted","Data":"e5ab04b720b29fe23873af98472251e247e31c43dc1b76198270270094cb3fb9"}
Jul 01 18:37:52.764947 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:52.764662    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2" event={"ID":"3709542d-b4c1-4e02-ba6e-8d917651974d","Type":"ContainerStarted","Data":"61fb400dc51261c21feb546076ebfb2bdea90f0953084c56a38c63088adf836c"}
Jul 01 18:37:53.660562 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:53.660510    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:53.660764 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:53.660525    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:37:53.660764 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:53.660673    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:53.660764 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:53.660667    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plc42" podUID="5180da14-bd5e-4777-b34c-4041f623113b"
Jul 01 18:37:53.660936 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:53.660783    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9t8tv" podUID="b105299c-7212-43c4-bb52-34aaaa2253f0"
Jul 01 18:37:53.660936 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:53.660869    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrx57" podUID="8bd0c6cc-1f64-4be9-8b7e-31fe9357b388"
Jul 01 18:37:53.668443 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:53.668350    2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-01T18:37:52.682521428Z","UUID":"9e13bfc4-e5a4-4ca5-8da3-eabc0444dc98","Handler":null,"Name":"","Endpoint":""}
Jul 01 18:37:53.670143 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:53.670121    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 01 18:37:53.670267 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:53.670154    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 01 18:37:53.771609 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:53.771358    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-prmsf" event={"ID":"6d3508bc-a5db-48ad-96f1-e7663193fe59","Type":"ContainerStarted","Data":"4c192904382bc107d2bcda86c5a62773432876aa6717ac15d0212e3693134de8"}
Jul 01 18:37:53.791624 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:53.791569    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-prmsf" podStartSLOduration=6.159464849 podStartE2EDuration="24.791549068s" totalImagesPullingTime="18.632084219s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:53.791217175 +0000 UTC m=+24.560054779" watchObservedRunningTime="2026-07-01 18:37:53.791549068 +0000 UTC m=+24.560386671"
Jul 01 18:37:53.918082 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:53.918054    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:37:54.022452 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:54.022421    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:37:54.776396 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:54.776145    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds" event={"ID":"1d6898f6-ff4c-4ea5-a95a-bc04245f72e2","Type":"ContainerStarted","Data":"fb5e7fc640b551c6998cc44f041f181e961337ba31117121f94e433547ec14aa"}
Jul 01 18:37:54.779768 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:54.779743    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2" event={"ID":"3709542d-b4c1-4e02-ba6e-8d917651974d","Type":"ContainerStarted","Data":"50b8d134e634855848097b3570bb058a3073175fe514e4249d6551ba64ae9a58"}
Jul 01 18:37:54.801404 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:54.801353    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2zds" podStartSLOduration=4.7280696429999995 podStartE2EDuration="25.801335528s" totalImagesPullingTime="21.073265885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:54.801115678 +0000 UTC m=+25.569953281" watchObservedRunningTime="2026-07-01 18:37:54.801335528 +0000 UTC m=+25.570173157"
Jul 01 18:37:55.083395 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:55.083303    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4pmfd"
Jul 01 18:37:55.084003 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:55.083984    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4pmfd"
Jul 01 18:37:55.660729 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:55.660679    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:55.660908 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:55.660679    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:37:55.660908 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:55.660833    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:55.660908 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:55.660832    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrx57" podUID="8bd0c6cc-1f64-4be9-8b7e-31fe9357b388"
Jul 01 18:37:55.661085 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:55.660934    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9t8tv" podUID="b105299c-7212-43c4-bb52-34aaaa2253f0"
Jul 01 18:37:55.661085 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:55.661036    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plc42" podUID="5180da14-bd5e-4777-b34c-4041f623113b"
Jul 01 18:37:55.782488 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:55.782453    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4pmfd"
Jul 01 18:37:55.783103 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:55.782913    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4pmfd"
Jul 01 18:37:55.871212 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:55.871185    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:37:56.787502 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:56.787275    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2" event={"ID":"3709542d-b4c1-4e02-ba6e-8d917651974d","Type":"ContainerStarted","Data":"f53f6c861ea16c9eb6daf61489b4c080b380bc95034714a2d205820f1234684a"}
Jul 01 18:37:56.788236 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:56.787561    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:56.789248 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:56.789213    2573 generic.go:356] "Generic (PLEG): container finished" podID="0282f93f-4b16-4820-8b1e-c0f7c010fa82" containerID="3af524b439b18d40a8cb9cdeb1facc692e64e0647a8bb9a3bb88f41d78fc382d" exitCode=0
Jul 01 18:37:56.789387 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:56.789263    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nx7jq" event={"ID":"0282f93f-4b16-4820-8b1e-c0f7c010fa82","Type":"ContainerDied","Data":"3af524b439b18d40a8cb9cdeb1facc692e64e0647a8bb9a3bb88f41d78fc382d"}
Jul 01 18:37:56.803402 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:56.803382    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:56.829276 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:56.829235    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2" podStartSLOduration=9.14204813 podStartE2EDuration="27.829222477s" totalImagesPullingTime="18.687174347s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:56.827835681 +0000 UTC m=+27.596673285" watchObservedRunningTime="2026-07-01 18:37:56.829222477 +0000 UTC m=+27.598060080"
Jul 01 18:37:57.660762 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:57.660729    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:57.661008 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:57.660774    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:37:57.661008 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:57.660731    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:57.661008 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:57.660845    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plc42" podUID="5180da14-bd5e-4777-b34c-4041f623113b"
Jul 01 18:37:57.661008 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:57.660924    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrx57" podUID="8bd0c6cc-1f64-4be9-8b7e-31fe9357b388"
Jul 01 18:37:57.661008 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:57.660975    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9t8tv" podUID="b105299c-7212-43c4-bb52-34aaaa2253f0"
Jul 01 18:37:57.791156 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:57.791131    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:57.791579 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:57.791209    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:57.805545 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:57.805522    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:37:57.930337 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:57.930301    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:37:58.017249 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:58.017216    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:37:58.237046 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:58.236871    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mrx57"]
Jul 01 18:37:58.237207 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:58.237122    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:37:58.237250 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:58.237217    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrx57" podUID="8bd0c6cc-1f64-4be9-8b7e-31fe9357b388"
Jul 01 18:37:58.240675 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:58.240646    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9t8tv"]
Jul 01 18:37:58.240840 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:58.240763    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:37:58.240902 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:58.240863    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9t8tv" podUID="b105299c-7212-43c4-bb52-34aaaa2253f0"
Jul 01 18:37:58.241314 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:58.241294    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-plc42"]
Jul 01 18:37:58.241440 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:58.241387    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:58.241520 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:58.241502    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plc42" podUID="5180da14-bd5e-4777-b34c-4041f623113b"
Jul 01 18:37:58.794669 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:58.794640    2573 generic.go:356] "Generic (PLEG): container finished" podID="0282f93f-4b16-4820-8b1e-c0f7c010fa82" containerID="75c51b606142b96d4304d427f4df787a7e9e94da15df14c3c36062f1fbe79fbd" exitCode=0
Jul 01 18:37:58.795046 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:58.794743    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nx7jq" event={"ID":"0282f93f-4b16-4820-8b1e-c0f7c010fa82","Type":"ContainerDied","Data":"75c51b606142b96d4304d427f4df787a7e9e94da15df14c3c36062f1fbe79fbd"}
Jul 01 18:37:58.983960 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:58.983931    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:37:59.062003 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:59.061935    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:37:59.663053 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:59.663024    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:37:59.663267 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:59.663167    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9t8tv" podUID="b105299c-7212-43c4-bb52-34aaaa2253f0"
Jul 01 18:37:59.663832 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:37:59.663813    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:37:59.664106 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:37:59.663964    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plc42" podUID="5180da14-bd5e-4777-b34c-4041f623113b"
Jul 01 18:38:00.660653 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:00.660625    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:38:00.661112 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:00.660739    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrx57" podUID="8bd0c6cc-1f64-4be9-8b7e-31fe9357b388"
Jul 01 18:38:00.801854 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:00.801816    2573 generic.go:356] "Generic (PLEG): container finished" podID="0282f93f-4b16-4820-8b1e-c0f7c010fa82" containerID="61c5aa7ee8cbd6b029ca14da560cf16c1d8cc211f4dbd484b8a8d30b264e765f" exitCode=0
Jul 01 18:38:00.802001 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:00.801877    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nx7jq" event={"ID":"0282f93f-4b16-4820-8b1e-c0f7c010fa82","Type":"ContainerDied","Data":"61c5aa7ee8cbd6b029ca14da560cf16c1d8cc211f4dbd484b8a8d30b264e765f"}
Jul 01 18:38:01.660307 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:01.660274    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:38:01.660489 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:01.660415    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plc42" podUID="5180da14-bd5e-4777-b34c-4041f623113b"
Jul 01 18:38:01.660489 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:01.660466    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:38:01.660609 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:01.660581    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9t8tv" podUID="b105299c-7212-43c4-bb52-34aaaa2253f0"
Jul 01 18:38:02.330066 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:02.330024    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:02.423909 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:02.423874    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:02.660953 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:02.660878    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:38:02.661109 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:02.660999    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrx57" podUID="8bd0c6cc-1f64-4be9-8b7e-31fe9357b388"
Jul 01 18:38:03.217623 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:03.217569    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:03.321324 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:03.321290    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:03.361437 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:03.361404    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-original-pull-secret\") pod \"global-pull-secret-syncer-mrx57\" (UID: \"8bd0c6cc-1f64-4be9-8b7e-31fe9357b388\") " pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:38:03.362008 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:03.361462    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs\") pod \"network-metrics-daemon-plc42\" (UID: \"5180da14-bd5e-4777-b34c-4041f623113b\") " pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:38:03.362008 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:03.361556    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:38:03.362008 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:03.361584    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:38:03.362008 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:03.361632    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-original-pull-secret podName:8bd0c6cc-1f64-4be9-8b7e-31fe9357b388 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:35.36161099 +0000 UTC m=+66.130448571 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-original-pull-secret") pod "global-pull-secret-syncer-mrx57" (UID: "8bd0c6cc-1f64-4be9-8b7e-31fe9357b388") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:38:03.362008 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:03.361652    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs podName:5180da14-bd5e-4777-b34c-4041f623113b nodeName:}" failed. No retries permitted until 2026-07-01 18:38:35.361641908 +0000 UTC m=+66.130479494 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs") pod "network-metrics-daemon-plc42" (UID: "5180da14-bd5e-4777-b34c-4041f623113b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:38:03.562908 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:03.562816    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n4qd\" (UniqueName: \"kubernetes.io/projected/b105299c-7212-43c4-bb52-34aaaa2253f0-kube-api-access-2n4qd\") pod \"network-check-target-9t8tv\" (UID: \"b105299c-7212-43c4-bb52-34aaaa2253f0\") " pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:38:03.563062 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:03.563005    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:38:03.563062 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:03.563024    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:38:03.563062 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:03.563033    2573 projected.go:196] Error preparing data for projected volume kube-api-access-2n4qd for pod openshift-network-diagnostics/network-check-target-9t8tv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:38:03.563202 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:03.563094    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b105299c-7212-43c4-bb52-34aaaa2253f0-kube-api-access-2n4qd podName:b105299c-7212-43c4-bb52-34aaaa2253f0 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:35.563078507 +0000 UTC m=+66.331916091 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2n4qd" (UniqueName: "kubernetes.io/projected/b105299c-7212-43c4-bb52-34aaaa2253f0-kube-api-access-2n4qd") pod "network-check-target-9t8tv" (UID: "b105299c-7212-43c4-bb52-34aaaa2253f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:38:03.660817 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:03.660778    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:38:03.660817 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:03.660807    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:38:03.661055 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:03.660913    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plc42" podUID="5180da14-bd5e-4777-b34c-4041f623113b"
Jul 01 18:38:03.661110 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:03.661052    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9t8tv" podUID="b105299c-7212-43c4-bb52-34aaaa2253f0"
Jul 01 18:38:04.660071 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:04.660040    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:38:04.660488 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:04.660162    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrx57" podUID="8bd0c6cc-1f64-4be9-8b7e-31fe9357b388"
Jul 01 18:38:04.979002 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:04.978979    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeReady"
Jul 01 18:38:04.979106 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:04.979095    2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 01 18:38:05.058160 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.058130    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-n5dbd"]
Jul 01 18:38:05.060124 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.060104    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n5dbd"
Jul 01 18:38:05.060622 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.060601    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jndl8"]
Jul 01 18:38:05.062885 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.062446    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ttz69\""
Jul 01 18:38:05.062885 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.062625    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 01 18:38:05.062885 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.062667    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 01 18:38:05.062885 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.062787    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 01 18:38:05.065118 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.065099    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jndl8"
Jul 01 18:38:05.067285 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.067264    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 01 18:38:05.067381 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.067365    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7sn7m\""
Jul 01 18:38:05.067628 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.067606    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 01 18:38:05.080892 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.080868    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n5dbd"]
Jul 01 18:38:05.086987 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.086967    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jndl8"]
Jul 01 18:38:05.173381 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.173332    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f9cr\" (UniqueName: \"kubernetes.io/projected/e1af1a8d-9770-41da-b5f0-a4ad75852811-kube-api-access-6f9cr\") pod \"ingress-canary-n5dbd\" (UID: \"e1af1a8d-9770-41da-b5f0-a4ad75852811\") " pod="openshift-ingress-canary/ingress-canary-n5dbd"
Jul 01 18:38:05.173381 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.173386    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-metrics-tls\") pod \"dns-default-jndl8\" (UID: \"d2ee7ba9-21f2-4630-ae35-b9f65ec8de37\") " pod="openshift-dns/dns-default-jndl8"
Jul 01 18:38:05.173632 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.173450    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-tmp-dir\") pod \"dns-default-jndl8\" (UID: \"d2ee7ba9-21f2-4630-ae35-b9f65ec8de37\") " pod="openshift-dns/dns-default-jndl8"
Jul 01 18:38:05.173632 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.173473    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-config-volume\") pod \"dns-default-jndl8\" (UID: \"d2ee7ba9-21f2-4630-ae35-b9f65ec8de37\") " pod="openshift-dns/dns-default-jndl8"
Jul 01 18:38:05.173632 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.173496    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqdkp\" (UniqueName: \"kubernetes.io/projected/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-kube-api-access-rqdkp\") pod \"dns-default-jndl8\" (UID: \"d2ee7ba9-21f2-4630-ae35-b9f65ec8de37\") " pod="openshift-dns/dns-default-jndl8"
Jul 01 18:38:05.173632 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.173565    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1af1a8d-9770-41da-b5f0-a4ad75852811-cert\") pod \"ingress-canary-n5dbd\" (UID: \"e1af1a8d-9770-41da-b5f0-a4ad75852811\") " pod="openshift-ingress-canary/ingress-canary-n5dbd"
Jul 01 18:38:05.272461 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.272430    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:05.274431 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.274374    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f9cr\" (UniqueName: \"kubernetes.io/projected/e1af1a8d-9770-41da-b5f0-a4ad75852811-kube-api-access-6f9cr\") pod \"ingress-canary-n5dbd\" (UID: \"e1af1a8d-9770-41da-b5f0-a4ad75852811\") " pod="openshift-ingress-canary/ingress-canary-n5dbd"
Jul 01 18:38:05.274535 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.274453    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-metrics-tls\") pod \"dns-default-jndl8\" (UID: \"d2ee7ba9-21f2-4630-ae35-b9f65ec8de37\") " pod="openshift-dns/dns-default-jndl8"
Jul 01 18:38:05.274535 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.274489    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-tmp-dir\") pod \"dns-default-jndl8\" (UID: \"d2ee7ba9-21f2-4630-ae35-b9f65ec8de37\") " pod="openshift-dns/dns-default-jndl8"
Jul 01 18:38:05.274535 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.274512    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-config-volume\") pod \"dns-default-jndl8\" (UID: \"d2ee7ba9-21f2-4630-ae35-b9f65ec8de37\") " pod="openshift-dns/dns-default-jndl8"
Jul 01 18:38:05.274695 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.274534    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqdkp\" (UniqueName: \"kubernetes.io/projected/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-kube-api-access-rqdkp\") pod \"dns-default-jndl8\" (UID: \"d2ee7ba9-21f2-4630-ae35-b9f65ec8de37\") " pod="openshift-dns/dns-default-jndl8"
Jul 01 18:38:05.274695 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.274577    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1af1a8d-9770-41da-b5f0-a4ad75852811-cert\") pod \"ingress-canary-n5dbd\" (UID: \"e1af1a8d-9770-41da-b5f0-a4ad75852811\") " pod="openshift-ingress-canary/ingress-canary-n5dbd"
Jul 01 18:38:05.274695 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:05.274590    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:38:05.274695 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:05.274665    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-metrics-tls podName:d2ee7ba9-21f2-4630-ae35-b9f65ec8de37 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:05.774645614 +0000 UTC m=+36.543483208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-metrics-tls") pod "dns-default-jndl8" (UID: "d2ee7ba9-21f2-4630-ae35-b9f65ec8de37") : secret "dns-default-metrics-tls" not found
Jul 01 18:38:05.274913 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.274843    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-tmp-dir\") pod \"dns-default-jndl8\" (UID: \"d2ee7ba9-21f2-4630-ae35-b9f65ec8de37\") " pod="openshift-dns/dns-default-jndl8"
Jul 01 18:38:05.275167 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.275148    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-config-volume\") pod \"dns-default-jndl8\" (UID: \"d2ee7ba9-21f2-4630-ae35-b9f65ec8de37\") " pod="openshift-dns/dns-default-jndl8"
Jul 01 18:38:05.278882 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.278763    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1af1a8d-9770-41da-b5f0-a4ad75852811-cert\") pod \"ingress-canary-n5dbd\" (UID: \"e1af1a8d-9770-41da-b5f0-a4ad75852811\") " pod="openshift-ingress-canary/ingress-canary-n5dbd"
Jul 01 18:38:05.294417 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.294394    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqdkp\" (UniqueName: \"kubernetes.io/projected/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-kube-api-access-rqdkp\") pod \"dns-default-jndl8\" (UID: \"d2ee7ba9-21f2-4630-ae35-b9f65ec8de37\") " pod="openshift-dns/dns-default-jndl8"
Jul 01 18:38:05.294417 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.294407    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f9cr\" (UniqueName: \"kubernetes.io/projected/e1af1a8d-9770-41da-b5f0-a4ad75852811-kube-api-access-6f9cr\") pod \"ingress-canary-n5dbd\" (UID: \"e1af1a8d-9770-41da-b5f0-a4ad75852811\") " pod="openshift-ingress-canary/ingress-canary-n5dbd"
Jul 01 18:38:05.369833 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.369791    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:05.371698 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.371684    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n5dbd"
Jul 01 18:38:05.504504 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.504473    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n5dbd"]
Jul 01 18:38:05.508638 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:38:05.508612    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1af1a8d_9770_41da_b5f0_a4ad75852811.slice/crio-60ce8e0425026562cd6ae9e310345f8472770e4585f5af4b3b9b45060c160faf WatchSource:0}: Error finding container 60ce8e0425026562cd6ae9e310345f8472770e4585f5af4b3b9b45060c160faf: Status 404 returned error can't find the container with id 60ce8e0425026562cd6ae9e310345f8472770e4585f5af4b3b9b45060c160faf
Jul 01 18:38:05.660606 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.660488    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:38:05.660606 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.660533    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:38:05.665477 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.665458    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 18:38:05.665867 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.665850    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g2vz9\""
Jul 01 18:38:05.665970 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.665881    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lwv7s\""
Jul 01 18:38:05.666639 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.666621    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 18:38:05.666707 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.666648    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 18:38:05.778842 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.778816    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-metrics-tls\") pod \"dns-default-jndl8\" (UID: \"d2ee7ba9-21f2-4630-ae35-b9f65ec8de37\") " pod="openshift-dns/dns-default-jndl8"
Jul 01 18:38:05.779009 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:05.778934    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:38:05.779009 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:05.778984    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-metrics-tls podName:d2ee7ba9-21f2-4630-ae35-b9f65ec8de37 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:06.778970779 +0000 UTC m=+37.547808359 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-metrics-tls") pod "dns-default-jndl8" (UID: "d2ee7ba9-21f2-4630-ae35-b9f65ec8de37") : secret "dns-default-metrics-tls" not found
Jul 01 18:38:05.834806 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.834768    2573 generic.go:356] "Generic (PLEG): container finished" podID="0282f93f-4b16-4820-8b1e-c0f7c010fa82" containerID="764bb17eb1aae8a81e899881d3d71c4db1a94f61ed9e22ecce955e760c06b797" exitCode=0
Jul 01 18:38:05.834928 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.834841    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nx7jq" event={"ID":"0282f93f-4b16-4820-8b1e-c0f7c010fa82","Type":"ContainerDied","Data":"764bb17eb1aae8a81e899881d3d71c4db1a94f61ed9e22ecce955e760c06b797"}
Jul 01 18:38:05.836113 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:05.835986    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n5dbd" event={"ID":"e1af1a8d-9770-41da-b5f0-a4ad75852811","Type":"ContainerStarted","Data":"60ce8e0425026562cd6ae9e310345f8472770e4585f5af4b3b9b45060c160faf"}
Jul 01 18:38:06.371047 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:06.371013    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:06.433300 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:06.433268    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:06.660145 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:06.660067    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:38:06.663037 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:06.663013    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 18:38:06.787904 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:06.787873    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-metrics-tls\") pod \"dns-default-jndl8\" (UID: \"d2ee7ba9-21f2-4630-ae35-b9f65ec8de37\") " pod="openshift-dns/dns-default-jndl8"
Jul 01 18:38:06.788076 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:06.788037    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:38:06.788129 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:06.788117    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-metrics-tls podName:d2ee7ba9-21f2-4630-ae35-b9f65ec8de37 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:08.788095529 +0000 UTC m=+39.556933122 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-metrics-tls") pod "dns-default-jndl8" (UID: "d2ee7ba9-21f2-4630-ae35-b9f65ec8de37") : secret "dns-default-metrics-tls" not found
Jul 01 18:38:06.845630 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:06.845593    2573 generic.go:356] "Generic (PLEG): container finished" podID="0282f93f-4b16-4820-8b1e-c0f7c010fa82" containerID="f98f8d67d29c00b23623e6c6a6edcbf4e6d5d8fe7bfe92263420e90768f5d4dd" exitCode=0
Jul 01 18:38:06.845813 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:06.845674    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nx7jq" event={"ID":"0282f93f-4b16-4820-8b1e-c0f7c010fa82","Type":"ContainerDied","Data":"f98f8d67d29c00b23623e6c6a6edcbf4e6d5d8fe7bfe92263420e90768f5d4dd"}
Jul 01 18:38:07.853127 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:07.852947    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nx7jq" event={"ID":"0282f93f-4b16-4820-8b1e-c0f7c010fa82","Type":"ContainerStarted","Data":"12c3eb3445a2f7c9610a4c56c875d712536ddb2dd8e9ec2aba994fdb462d61f9"}
Jul 01 18:38:07.854314 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:07.854286    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n5dbd" event={"ID":"e1af1a8d-9770-41da-b5f0-a4ad75852811","Type":"ContainerStarted","Data":"5865fc6eb0fe4b310ea0d2eeacd3463b8fe5b91e73607e89609b91a6dade409a"}
Jul 01 18:38:07.887514 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:07.887472    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nx7jq" podStartSLOduration=9.831231527 podStartE2EDuration="38.887458701s" totalImagesPullingTime="29.056227174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:07.886403505 +0000 UTC m=+38.655241106" watchObservedRunningTime="2026-07-01 18:38:07.887458701 +0000 UTC m=+38.656296302"
Jul 01 18:38:07.915376 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:07.915340    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-n5dbd" podStartSLOduration=0.923339901 podStartE2EDuration="2.915326749s" totalImagesPullingTime="1.991986848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:07.914662429 +0000 UTC m=+38.683500031" watchObservedRunningTime="2026-07-01 18:38:07.915326749 +0000 UTC m=+38.684164348"
Jul 01 18:38:08.034448 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:08.034375    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:08.113301 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:08.113270    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:08.198322 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:08.198289    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:08.608231 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:08.608182    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:08.679106 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:08.679078    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:08.801532 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:08.801500    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-metrics-tls\") pod \"dns-default-jndl8\" (UID: \"d2ee7ba9-21f2-4630-ae35-b9f65ec8de37\") " pod="openshift-dns/dns-default-jndl8"
Jul 01 18:38:08.801673 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:08.801652    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:38:08.801745 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:08.801735    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-metrics-tls podName:d2ee7ba9-21f2-4630-ae35-b9f65ec8de37 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:12.801697066 +0000 UTC m=+43.570534647 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-metrics-tls") pod "dns-default-jndl8" (UID: "d2ee7ba9-21f2-4630-ae35-b9f65ec8de37") : secret "dns-default-metrics-tls" not found
Jul 01 18:38:12.826920 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:12.826881    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-metrics-tls\") pod \"dns-default-jndl8\" (UID: \"d2ee7ba9-21f2-4630-ae35-b9f65ec8de37\") " pod="openshift-dns/dns-default-jndl8"
Jul 01 18:38:12.827511 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:12.827018    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:38:12.827511 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:12.827082    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-metrics-tls podName:d2ee7ba9-21f2-4630-ae35-b9f65ec8de37 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:20.82706484 +0000 UTC m=+51.595902420 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-metrics-tls") pod "dns-default-jndl8" (UID: "d2ee7ba9-21f2-4630-ae35-b9f65ec8de37") : secret "dns-default-metrics-tls" not found
Jul 01 18:38:20.876837 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:20.876802    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-metrics-tls\") pod \"dns-default-jndl8\" (UID: \"d2ee7ba9-21f2-4630-ae35-b9f65ec8de37\") " pod="openshift-dns/dns-default-jndl8"
Jul 01 18:38:20.877293 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:20.876950    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:38:20.877293 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:20.877027    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-metrics-tls podName:d2ee7ba9-21f2-4630-ae35-b9f65ec8de37 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:36.877010035 +0000 UTC m=+67.645847615 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-metrics-tls") pod "dns-default-jndl8" (UID: "d2ee7ba9-21f2-4630-ae35-b9f65ec8de37") : secret "dns-default-metrics-tls" not found
Jul 01 18:38:23.212487 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:23.212453    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:23.314703 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:23.314677    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:23.413845 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:23.413816    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:24.289323 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.289285    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5f67b55555-4lw92"]
Jul 01 18:38:24.291448 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.291426    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5f67b55555-4lw92"
Jul 01 18:38:24.293543 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.293520    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-cbcfcfdbf-r98lq"]
Jul 01 18:38:24.293976 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.293956    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 01 18:38:24.294075 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.294026    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 01 18:38:24.294361 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.294349    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 01 18:38:24.294780 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.294766    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5v8zb\""
Jul 01 18:38:24.295214 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.295198    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 01 18:38:24.295353 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.295340    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:24.297575 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.297559    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 01 18:38:24.297975 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.297961    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-nrphr\""
Jul 01 18:38:24.298290 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.298273    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 01 18:38:24.298290 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.298287    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 01 18:38:24.298507 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.298490    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0cbfce-e39e-41c5-a6f8-43aea34bdc3c-service-ca-bundle\") pod \"insights-operator-5f67b55555-4lw92\" (UID: \"af0cbfce-e39e-41c5-a6f8-43aea34bdc3c\") " pod="openshift-insights/insights-operator-5f67b55555-4lw92"
Jul 01 18:38:24.298565 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.298515    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zb8d\" (UniqueName: \"kubernetes.io/projected/af0cbfce-e39e-41c5-a6f8-43aea34bdc3c-kube-api-access-6zb8d\") pod \"insights-operator-5f67b55555-4lw92\" (UID: \"af0cbfce-e39e-41c5-a6f8-43aea34bdc3c\") " pod="openshift-insights/insights-operator-5f67b55555-4lw92"
Jul 01 18:38:24.298565 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.298545    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/af0cbfce-e39e-41c5-a6f8-43aea34bdc3c-snapshots\") pod \"insights-operator-5f67b55555-4lw92\" (UID: \"af0cbfce-e39e-41c5-a6f8-43aea34bdc3c\") " pod="openshift-insights/insights-operator-5f67b55555-4lw92"
Jul 01 18:38:24.298643 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.298595    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 01 18:38:24.298643 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.298636    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/af0cbfce-e39e-41c5-a6f8-43aea34bdc3c-serving-cert\") pod \"insights-operator-5f67b55555-4lw92\" (UID: \"af0cbfce-e39e-41c5-a6f8-43aea34bdc3c\") " pod="openshift-insights/insights-operator-5f67b55555-4lw92"
Jul 01 18:38:24.298737 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.298650    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 01 18:38:24.298737 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.298682    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0cbfce-e39e-41c5-a6f8-43aea34bdc3c-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-4lw92\" (UID: \"af0cbfce-e39e-41c5-a6f8-43aea34bdc3c\") " pod="openshift-insights/insights-operator-5f67b55555-4lw92"
Jul 01 18:38:24.298737 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.298726    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af0cbfce-e39e-41c5-a6f8-43aea34bdc3c-tmp\") pod \"insights-operator-5f67b55555-4lw92\" (UID: \"af0cbfce-e39e-41c5-a6f8-43aea34bdc3c\") " pod="openshift-insights/insights-operator-5f67b55555-4lw92"
Jul 01 18:38:24.301050 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.301034    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 01 18:38:24.302129 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.302100    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 01 18:38:24.308533 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.308515    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5f67b55555-4lw92"]
Jul 01 18:38:24.315965 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.315947    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-cbcfcfdbf-r98lq"]
Jul 01 18:38:24.399955 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.399930    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0cbfce-e39e-41c5-a6f8-43aea34bdc3c-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-4lw92\" (UID: \"af0cbfce-e39e-41c5-a6f8-43aea34bdc3c\") " pod="openshift-insights/insights-operator-5f67b55555-4lw92"
Jul 01 18:38:24.400154 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.399960    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b901864b-08c4-4e62-ab99-71715e994501-service-ca-bundle\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:24.400154 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.399978    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph7wc\" (UniqueName: \"kubernetes.io/projected/b901864b-08c4-4e62-ab99-71715e994501-kube-api-access-ph7wc\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:24.400154 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.400001    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af0cbfce-e39e-41c5-a6f8-43aea34bdc3c-tmp\") pod \"insights-operator-5f67b55555-4lw92\" (UID: \"af0cbfce-e39e-41c5-a6f8-43aea34bdc3c\") " pod="openshift-insights/insights-operator-5f67b55555-4lw92"
Jul 01 18:38:24.400154 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.400024    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b901864b-08c4-4e62-ab99-71715e994501-metrics-certs\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:24.400154 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.400085    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0cbfce-e39e-41c5-a6f8-43aea34bdc3c-service-ca-bundle\") pod \"insights-operator-5f67b55555-4lw92\" (UID: \"af0cbfce-e39e-41c5-a6f8-43aea34bdc3c\") " pod="openshift-insights/insights-operator-5f67b55555-4lw92"
Jul 01 18:38:24.400154 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.400103    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zb8d\" (UniqueName: \"kubernetes.io/projected/af0cbfce-e39e-41c5-a6f8-43aea34bdc3c-kube-api-access-6zb8d\") pod \"insights-operator-5f67b55555-4lw92\" (UID: \"af0cbfce-e39e-41c5-a6f8-43aea34bdc3c\") " pod="openshift-insights/insights-operator-5f67b55555-4lw92"
Jul 01 18:38:24.400154 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.400128    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/af0cbfce-e39e-41c5-a6f8-43aea34bdc3c-snapshots\") pod \"insights-operator-5f67b55555-4lw92\" (UID: \"af0cbfce-e39e-41c5-a6f8-43aea34bdc3c\") " pod="openshift-insights/insights-operator-5f67b55555-4lw92"
Jul 01 18:38:24.400154 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.400150    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b901864b-08c4-4e62-ab99-71715e994501-default-certificate\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:24.400584 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.400198    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b901864b-08c4-4e62-ab99-71715e994501-stats-auth\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:24.400584 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.400289    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/af0cbfce-e39e-41c5-a6f8-43aea34bdc3c-serving-cert\") pod \"insights-operator-5f67b55555-4lw92\" (UID: \"af0cbfce-e39e-41c5-a6f8-43aea34bdc3c\") " pod="openshift-insights/insights-operator-5f67b55555-4lw92"
Jul 01 18:38:24.400584 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.400481    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af0cbfce-e39e-41c5-a6f8-43aea34bdc3c-tmp\") pod \"insights-operator-5f67b55555-4lw92\" (UID: \"af0cbfce-e39e-41c5-a6f8-43aea34bdc3c\") " pod="openshift-insights/insights-operator-5f67b55555-4lw92"
Jul 01 18:38:24.400867 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.400850    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0cbfce-e39e-41c5-a6f8-43aea34bdc3c-service-ca-bundle\") pod \"insights-operator-5f67b55555-4lw92\" (UID: \"af0cbfce-e39e-41c5-a6f8-43aea34bdc3c\") " pod="openshift-insights/insights-operator-5f67b55555-4lw92"
Jul 01 18:38:24.400949 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.400933    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af0cbfce-e39e-41c5-a6f8-43aea34bdc3c-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-4lw92\" (UID: \"af0cbfce-e39e-41c5-a6f8-43aea34bdc3c\") " pod="openshift-insights/insights-operator-5f67b55555-4lw92"
Jul 01 18:38:24.400983 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.400965    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/af0cbfce-e39e-41c5-a6f8-43aea34bdc3c-snapshots\") pod \"insights-operator-5f67b55555-4lw92\" (UID: \"af0cbfce-e39e-41c5-a6f8-43aea34bdc3c\") " pod="openshift-insights/insights-operator-5f67b55555-4lw92"
Jul 01 18:38:24.403908 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.403893    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/af0cbfce-e39e-41c5-a6f8-43aea34bdc3c-serving-cert\") pod \"insights-operator-5f67b55555-4lw92\" (UID: \"af0cbfce-e39e-41c5-a6f8-43aea34bdc3c\") " pod="openshift-insights/insights-operator-5f67b55555-4lw92"
Jul 01 18:38:24.426796 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.426774    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zb8d\" (UniqueName: \"kubernetes.io/projected/af0cbfce-e39e-41c5-a6f8-43aea34bdc3c-kube-api-access-6zb8d\") pod \"insights-operator-5f67b55555-4lw92\" (UID: \"af0cbfce-e39e-41c5-a6f8-43aea34bdc3c\") " pod="openshift-insights/insights-operator-5f67b55555-4lw92"
Jul 01 18:38:24.489822 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.489794    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zz5z7"]
Jul 01 18:38:24.492240 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.492225    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zz5z7"
Jul 01 18:38:24.495291 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.495269    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 01 18:38:24.495406 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.495314    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 01 18:38:24.495406 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.495324    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 01 18:38:24.495406 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.495378    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-n4vj9\""
Jul 01 18:38:24.501172 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.501152    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b901864b-08c4-4e62-ab99-71715e994501-service-ca-bundle\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:24.501274 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.501184    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph7wc\" (UniqueName: \"kubernetes.io/projected/b901864b-08c4-4e62-ab99-71715e994501-kube-api-access-ph7wc\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:24.501274 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.501214    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgsgd\" (UniqueName: \"kubernetes.io/projected/f31fae04-dc5d-4a96-8f60-79f6dd398b83-kube-api-access-xgsgd\") pod \"cluster-samples-operator-9ddc5ff76-zz5z7\" (UID: \"f31fae04-dc5d-4a96-8f60-79f6dd398b83\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zz5z7"
Jul 01 18:38:24.501274 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.501250    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b901864b-08c4-4e62-ab99-71715e994501-metrics-certs\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:24.501412 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.501276    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f31fae04-dc5d-4a96-8f60-79f6dd398b83-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-zz5z7\" (UID: \"f31fae04-dc5d-4a96-8f60-79f6dd398b83\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zz5z7"
Jul 01 18:38:24.501412 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:24.501298    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b901864b-08c4-4e62-ab99-71715e994501-service-ca-bundle podName:b901864b-08c4-4e62-ab99-71715e994501 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:25.001283615 +0000 UTC m=+55.770121195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b901864b-08c4-4e62-ab99-71715e994501-service-ca-bundle") pod "router-default-cbcfcfdbf-r98lq" (UID: "b901864b-08c4-4e62-ab99-71715e994501") : configmap references non-existent config key: service-ca.crt
Jul 01 18:38:24.501412 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:24.501346    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 18:38:24.501412 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.501379    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b901864b-08c4-4e62-ab99-71715e994501-default-certificate\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:24.501553 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:24.501417    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b901864b-08c4-4e62-ab99-71715e994501-metrics-certs podName:b901864b-08c4-4e62-ab99-71715e994501 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:25.001390644 +0000 UTC m=+55.770228231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b901864b-08c4-4e62-ab99-71715e994501-metrics-certs") pod "router-default-cbcfcfdbf-r98lq" (UID: "b901864b-08c4-4e62-ab99-71715e994501") : secret "router-metrics-certs-default" not found
Jul 01 18:38:24.501553 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.501468    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b901864b-08c4-4e62-ab99-71715e994501-stats-auth\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:24.503551 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.503529    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b901864b-08c4-4e62-ab99-71715e994501-default-certificate\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:24.503679 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.503660    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b901864b-08c4-4e62-ab99-71715e994501-stats-auth\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:24.507208 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.507186    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zz5z7"]
Jul 01 18:38:24.534330 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.534310    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph7wc\" (UniqueName: \"kubernetes.io/projected/b901864b-08c4-4e62-ab99-71715e994501-kube-api-access-ph7wc\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:24.602766 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.602698    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgsgd\" (UniqueName: \"kubernetes.io/projected/f31fae04-dc5d-4a96-8f60-79f6dd398b83-kube-api-access-xgsgd\") pod \"cluster-samples-operator-9ddc5ff76-zz5z7\" (UID: \"f31fae04-dc5d-4a96-8f60-79f6dd398b83\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zz5z7"
Jul 01 18:38:24.602766 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.602752    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f31fae04-dc5d-4a96-8f60-79f6dd398b83-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-zz5z7\" (UID: \"f31fae04-dc5d-4a96-8f60-79f6dd398b83\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zz5z7"
Jul 01 18:38:24.602901 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.602886    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5f67b55555-4lw92"
Jul 01 18:38:24.602901 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:24.602891    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 18:38:24.602973 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:24.602953    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f31fae04-dc5d-4a96-8f60-79f6dd398b83-samples-operator-tls podName:f31fae04-dc5d-4a96-8f60-79f6dd398b83 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:25.102936331 +0000 UTC m=+55.871773929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f31fae04-dc5d-4a96-8f60-79f6dd398b83-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-zz5z7" (UID: "f31fae04-dc5d-4a96-8f60-79f6dd398b83") : secret "samples-operator-tls" not found
Jul 01 18:38:24.622727 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.622695    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgsgd\" (UniqueName: \"kubernetes.io/projected/f31fae04-dc5d-4a96-8f60-79f6dd398b83-kube-api-access-xgsgd\") pod \"cluster-samples-operator-9ddc5ff76-zz5z7\" (UID: \"f31fae04-dc5d-4a96-8f60-79f6dd398b83\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zz5z7"
Jul 01 18:38:24.634192 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.634171    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:24.709799 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.709771    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:24.725296 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.725272    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5f67b55555-4lw92"]
Jul 01 18:38:24.728073 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:38:24.728050    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0cbfce_e39e_41c5_a6f8_43aea34bdc3c.slice/crio-312de1e1c9489cadfe1a99ba539ce09efd76097bb0439d44c251b3bcb4c13489 WatchSource:0}: Error finding container 312de1e1c9489cadfe1a99ba539ce09efd76097bb0439d44c251b3bcb4c13489: Status 404 returned error can't find the container with id 312de1e1c9489cadfe1a99ba539ce09efd76097bb0439d44c251b3bcb4c13489
Jul 01 18:38:24.892816 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:24.892708    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-4lw92" event={"ID":"af0cbfce-e39e-41c5-a6f8-43aea34bdc3c","Type":"ContainerStarted","Data":"312de1e1c9489cadfe1a99ba539ce09efd76097bb0439d44c251b3bcb4c13489"}
Jul 01 18:38:25.005832 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:25.005803    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b901864b-08c4-4e62-ab99-71715e994501-metrics-certs\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:25.005982 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:25.005944    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b901864b-08c4-4e62-ab99-71715e994501-service-ca-bundle\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:25.005982 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:25.005954    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 18:38:25.006059 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:25.006019    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b901864b-08c4-4e62-ab99-71715e994501-metrics-certs podName:b901864b-08c4-4e62-ab99-71715e994501 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:26.006000246 +0000 UTC m=+56.774837846 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b901864b-08c4-4e62-ab99-71715e994501-metrics-certs") pod "router-default-cbcfcfdbf-r98lq" (UID: "b901864b-08c4-4e62-ab99-71715e994501") : secret "router-metrics-certs-default" not found
Jul 01 18:38:25.006059 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:25.006039    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b901864b-08c4-4e62-ab99-71715e994501-service-ca-bundle podName:b901864b-08c4-4e62-ab99-71715e994501 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:26.006029808 +0000 UTC m=+56.774867391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b901864b-08c4-4e62-ab99-71715e994501-service-ca-bundle") pod "router-default-cbcfcfdbf-r98lq" (UID: "b901864b-08c4-4e62-ab99-71715e994501") : configmap references non-existent config key: service-ca.crt
Jul 01 18:38:25.107177 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:25.107149    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f31fae04-dc5d-4a96-8f60-79f6dd398b83-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-zz5z7\" (UID: \"f31fae04-dc5d-4a96-8f60-79f6dd398b83\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zz5z7"
Jul 01 18:38:25.107328 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:25.107261    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 18:38:25.107328 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:25.107309    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f31fae04-dc5d-4a96-8f60-79f6dd398b83-samples-operator-tls podName:f31fae04-dc5d-4a96-8f60-79f6dd398b83 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:26.107296955 +0000 UTC m=+56.876134536 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f31fae04-dc5d-4a96-8f60-79f6dd398b83-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-zz5z7" (UID: "f31fae04-dc5d-4a96-8f60-79f6dd398b83") : secret "samples-operator-tls" not found
Jul 01 18:38:26.013304 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:26.013266    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b901864b-08c4-4e62-ab99-71715e994501-service-ca-bundle\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:26.013785 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:26.013314    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b901864b-08c4-4e62-ab99-71715e994501-metrics-certs\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:26.013785 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:26.013455    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b901864b-08c4-4e62-ab99-71715e994501-service-ca-bundle podName:b901864b-08c4-4e62-ab99-71715e994501 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:28.013430383 +0000 UTC m=+58.782267964 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b901864b-08c4-4e62-ab99-71715e994501-service-ca-bundle") pod "router-default-cbcfcfdbf-r98lq" (UID: "b901864b-08c4-4e62-ab99-71715e994501") : configmap references non-existent config key: service-ca.crt
Jul 01 18:38:26.013785 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:26.013496    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 18:38:26.013785 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:26.013560    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b901864b-08c4-4e62-ab99-71715e994501-metrics-certs podName:b901864b-08c4-4e62-ab99-71715e994501 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:28.01354346 +0000 UTC m=+58.782381055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b901864b-08c4-4e62-ab99-71715e994501-metrics-certs") pod "router-default-cbcfcfdbf-r98lq" (UID: "b901864b-08c4-4e62-ab99-71715e994501") : secret "router-metrics-certs-default" not found
Jul 01 18:38:26.114822 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:26.114781    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f31fae04-dc5d-4a96-8f60-79f6dd398b83-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-zz5z7\" (UID: \"f31fae04-dc5d-4a96-8f60-79f6dd398b83\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zz5z7"
Jul 01 18:38:26.115016 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:26.114956    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 18:38:26.115082 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:26.115030    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f31fae04-dc5d-4a96-8f60-79f6dd398b83-samples-operator-tls podName:f31fae04-dc5d-4a96-8f60-79f6dd398b83 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:28.115010228 +0000 UTC m=+58.883847867 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f31fae04-dc5d-4a96-8f60-79f6dd398b83-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-zz5z7" (UID: "f31fae04-dc5d-4a96-8f60-79f6dd398b83") : secret "samples-operator-tls" not found
Jul 01 18:38:26.424224 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:26.424190    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:26.498762 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:26.498733    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:26.898807 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:26.898756    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-4lw92" event={"ID":"af0cbfce-e39e-41c5-a6f8-43aea34bdc3c","Type":"ContainerStarted","Data":"ccb40ad9f7e1872f5bdd1015cce823b778fc3520687516cefeb0b3158883a6f1"}
Jul 01 18:38:26.923064 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:26.922998    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5f67b55555-4lw92" podStartSLOduration=1.322975675 podStartE2EDuration="2.922977934s" totalImagesPullingTime="1.600002259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:26.922373729 +0000 UTC m=+57.691211332" watchObservedRunningTime="2026-07-01 18:38:26.922977934 +0000 UTC m=+57.691815538"
Jul 01 18:38:27.258152 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:27.258117    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-x2bsq"]
Jul 01 18:38:27.260455 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:27.260435    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:27.263693 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:27.263672    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 01 18:38:27.263796 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:27.263734    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 01 18:38:27.264046 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:27.264028    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q4zk9\""
Jul 01 18:38:27.278980 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:27.278954    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x2bsq"]
Jul 01 18:38:27.329614 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:27.329589    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdadaf45-8a9b-4695-851d-c0b8297e545e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:27.329778 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:27.329618    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr82n\" (UniqueName: \"kubernetes.io/projected/fdadaf45-8a9b-4695-851d-c0b8297e545e-kube-api-access-vr82n\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:27.329778 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:27.329659    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdadaf45-8a9b-4695-851d-c0b8297e545e-data-volume\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:27.329778 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:27.329752    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdadaf45-8a9b-4695-851d-c0b8297e545e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:27.329906 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:27.329797    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fdadaf45-8a9b-4695-851d-c0b8297e545e-crio-socket\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:27.340797 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:27.340773    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:27.431170 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:27.431132    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdadaf45-8a9b-4695-851d-c0b8297e545e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:27.431304 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:27.431178    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr82n\" (UniqueName: \"kubernetes.io/projected/fdadaf45-8a9b-4695-851d-c0b8297e545e-kube-api-access-vr82n\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:27.431304 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:27.431275    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:27.431304 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:27.431289    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdadaf45-8a9b-4695-851d-c0b8297e545e-data-volume\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:27.431400 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:27.431332    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdadaf45-8a9b-4695-851d-c0b8297e545e-insights-runtime-extractor-tls podName:fdadaf45-8a9b-4695-851d-c0b8297e545e nodeName:}" failed. No retries permitted until 2026-07-01 18:38:27.931317655 +0000 UTC m=+58.700155236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdadaf45-8a9b-4695-851d-c0b8297e545e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x2bsq" (UID: "fdadaf45-8a9b-4695-851d-c0b8297e545e") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:27.431400 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:27.431373    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdadaf45-8a9b-4695-851d-c0b8297e545e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:27.431475 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:27.431412    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fdadaf45-8a9b-4695-851d-c0b8297e545e-crio-socket\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:27.431507 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:27.431472    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 18:38:27.431541 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:27.431513    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdadaf45-8a9b-4695-851d-c0b8297e545e-kube-rbac-proxy-cm podName:fdadaf45-8a9b-4695-851d-c0b8297e545e nodeName:}" failed. No retries permitted until 2026-07-01 18:38:27.931500843 +0000 UTC m=+58.700338424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fdadaf45-8a9b-4695-851d-c0b8297e545e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x2bsq" (UID: "fdadaf45-8a9b-4695-851d-c0b8297e545e") : configmap "kube-rbac-proxy" not found
Jul 01 18:38:27.431541 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:27.431510    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fdadaf45-8a9b-4695-851d-c0b8297e545e-crio-socket\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:27.431606 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:27.431585    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdadaf45-8a9b-4695-851d-c0b8297e545e-data-volume\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:27.435808 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:27.435777    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:27.448008 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:27.447986    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr82n\" (UniqueName: \"kubernetes.io/projected/fdadaf45-8a9b-4695-851d-c0b8297e545e-kube-api-access-vr82n\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:27.935168 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:27.935136    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdadaf45-8a9b-4695-851d-c0b8297e545e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:27.935342 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:27.935226    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdadaf45-8a9b-4695-851d-c0b8297e545e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:27.935342 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:27.935331    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:27.935342 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:27.935339    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 18:38:27.935448 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:27.935388    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdadaf45-8a9b-4695-851d-c0b8297e545e-insights-runtime-extractor-tls podName:fdadaf45-8a9b-4695-851d-c0b8297e545e nodeName:}" failed. No retries permitted until 2026-07-01 18:38:28.935374918 +0000 UTC m=+59.704212503 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdadaf45-8a9b-4695-851d-c0b8297e545e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x2bsq" (UID: "fdadaf45-8a9b-4695-851d-c0b8297e545e") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:27.935448 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:27.935403    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdadaf45-8a9b-4695-851d-c0b8297e545e-kube-rbac-proxy-cm podName:fdadaf45-8a9b-4695-851d-c0b8297e545e nodeName:}" failed. No retries permitted until 2026-07-01 18:38:28.935396201 +0000 UTC m=+59.704233782 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fdadaf45-8a9b-4695-851d-c0b8297e545e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x2bsq" (UID: "fdadaf45-8a9b-4695-851d-c0b8297e545e") : configmap "kube-rbac-proxy" not found
Jul 01 18:38:28.035944 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:28.035914    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b901864b-08c4-4e62-ab99-71715e994501-service-ca-bundle\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:28.036106 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:28.035956    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b901864b-08c4-4e62-ab99-71715e994501-metrics-certs\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:28.036106 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:28.036081    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b901864b-08c4-4e62-ab99-71715e994501-service-ca-bundle podName:b901864b-08c4-4e62-ab99-71715e994501 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:32.036062961 +0000 UTC m=+62.804900541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b901864b-08c4-4e62-ab99-71715e994501-service-ca-bundle") pod "router-default-cbcfcfdbf-r98lq" (UID: "b901864b-08c4-4e62-ab99-71715e994501") : configmap references non-existent config key: service-ca.crt
Jul 01 18:38:28.036209 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:28.036124    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 18:38:28.036209 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:28.036189    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b901864b-08c4-4e62-ab99-71715e994501-metrics-certs podName:b901864b-08c4-4e62-ab99-71715e994501 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:32.036165895 +0000 UTC m=+62.805003496 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b901864b-08c4-4e62-ab99-71715e994501-metrics-certs") pod "router-default-cbcfcfdbf-r98lq" (UID: "b901864b-08c4-4e62-ab99-71715e994501") : secret "router-metrics-certs-default" not found
Jul 01 18:38:28.137312 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:28.137283    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f31fae04-dc5d-4a96-8f60-79f6dd398b83-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-zz5z7\" (UID: \"f31fae04-dc5d-4a96-8f60-79f6dd398b83\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zz5z7"
Jul 01 18:38:28.137481 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:28.137420    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 18:38:28.137531 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:28.137489    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f31fae04-dc5d-4a96-8f60-79f6dd398b83-samples-operator-tls podName:f31fae04-dc5d-4a96-8f60-79f6dd398b83 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:32.137471078 +0000 UTC m=+62.906308675 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f31fae04-dc5d-4a96-8f60-79f6dd398b83-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-zz5z7" (UID: "f31fae04-dc5d-4a96-8f60-79f6dd398b83") : secret "samples-operator-tls" not found
Jul 01 18:38:28.943064 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:28.943031    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdadaf45-8a9b-4695-851d-c0b8297e545e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:28.943418 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:28.943093    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdadaf45-8a9b-4695-851d-c0b8297e545e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:28.943418 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:28.943168    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 18:38:28.943418 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:28.943178    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:28.943418 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:28.943219    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdadaf45-8a9b-4695-851d-c0b8297e545e-kube-rbac-proxy-cm podName:fdadaf45-8a9b-4695-851d-c0b8297e545e nodeName:}" failed. No retries permitted until 2026-07-01 18:38:30.943205821 +0000 UTC m=+61.712043401 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fdadaf45-8a9b-4695-851d-c0b8297e545e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x2bsq" (UID: "fdadaf45-8a9b-4695-851d-c0b8297e545e") : configmap "kube-rbac-proxy" not found
Jul 01 18:38:28.943418 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:28.943232    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdadaf45-8a9b-4695-851d-c0b8297e545e-insights-runtime-extractor-tls podName:fdadaf45-8a9b-4695-851d-c0b8297e545e nodeName:}" failed. No retries permitted until 2026-07-01 18:38:30.943226375 +0000 UTC m=+61.712063956 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdadaf45-8a9b-4695-851d-c0b8297e545e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x2bsq" (UID: "fdadaf45-8a9b-4695-851d-c0b8297e545e") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:29.213771 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:29.213748    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l5m2d_f3a42d5d-0f95-4aed-94f3-ea0b79e4e326/dns-node-resolver/0.log"
Jul 01 18:38:29.814348 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:29.814322    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tjxh2"
Jul 01 18:38:29.895355 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:29.895325    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:30.013331 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:30.013300    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:30.019356 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:30.019333    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dctc8_92fa86b8-a872-4317-9b0a-c8452bec1f34/node-ca/0.log"
Jul 01 18:38:30.107116 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:30.107035    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:30.201334 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:30.201308    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:30.315343 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:30.315295    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:30.410535 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:30.410466    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:30.603685 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:30.603652    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:30.699775 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:30.699684    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:30.958236 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:30.958207    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdadaf45-8a9b-4695-851d-c0b8297e545e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:30.958375 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:30.958322    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdadaf45-8a9b-4695-851d-c0b8297e545e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:30.958375 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:30.958333    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 18:38:30.958461 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:30.958401    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdadaf45-8a9b-4695-851d-c0b8297e545e-kube-rbac-proxy-cm podName:fdadaf45-8a9b-4695-851d-c0b8297e545e nodeName:}" failed. No retries permitted until 2026-07-01 18:38:34.958382772 +0000 UTC m=+65.727220375 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fdadaf45-8a9b-4695-851d-c0b8297e545e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x2bsq" (UID: "fdadaf45-8a9b-4695-851d-c0b8297e545e") : configmap "kube-rbac-proxy" not found
Jul 01 18:38:30.958502 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:30.958466    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:30.958545 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:30.958534    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdadaf45-8a9b-4695-851d-c0b8297e545e-insights-runtime-extractor-tls podName:fdadaf45-8a9b-4695-851d-c0b8297e545e nodeName:}" failed. No retries permitted until 2026-07-01 18:38:34.958520705 +0000 UTC m=+65.727358285 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdadaf45-8a9b-4695-851d-c0b8297e545e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x2bsq" (UID: "fdadaf45-8a9b-4695-851d-c0b8297e545e") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:31.009439 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:31.009405    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n5dbd_e1af1a8d-9770-41da-b5f0-a4ad75852811/serve-healthcheck-canary/0.log"
Jul 01 18:38:32.067253 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.067208    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b901864b-08c4-4e62-ab99-71715e994501-service-ca-bundle\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:32.067774 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.067267    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b901864b-08c4-4e62-ab99-71715e994501-metrics-certs\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:32.067774 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:32.067367    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b901864b-08c4-4e62-ab99-71715e994501-service-ca-bundle podName:b901864b-08c4-4e62-ab99-71715e994501 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:40.06735019 +0000 UTC m=+70.836187774 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b901864b-08c4-4e62-ab99-71715e994501-service-ca-bundle") pod "router-default-cbcfcfdbf-r98lq" (UID: "b901864b-08c4-4e62-ab99-71715e994501") : configmap references non-existent config key: service-ca.crt
Jul 01 18:38:32.067774 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:32.067451    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 18:38:32.067774 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:32.067525    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b901864b-08c4-4e62-ab99-71715e994501-metrics-certs podName:b901864b-08c4-4e62-ab99-71715e994501 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:40.067509699 +0000 UTC m=+70.836347284 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b901864b-08c4-4e62-ab99-71715e994501-metrics-certs") pod "router-default-cbcfcfdbf-r98lq" (UID: "b901864b-08c4-4e62-ab99-71715e994501") : secret "router-metrics-certs-default" not found
Jul 01 18:38:32.168107 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.168067    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f31fae04-dc5d-4a96-8f60-79f6dd398b83-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-zz5z7\" (UID: \"f31fae04-dc5d-4a96-8f60-79f6dd398b83\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zz5z7"
Jul 01 18:38:32.168281 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:32.168230    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 18:38:32.168323 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:32.168291    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f31fae04-dc5d-4a96-8f60-79f6dd398b83-samples-operator-tls podName:f31fae04-dc5d-4a96-8f60-79f6dd398b83 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:40.168274819 +0000 UTC m=+70.937112414 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f31fae04-dc5d-4a96-8f60-79f6dd398b83-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-zz5z7" (UID: "f31fae04-dc5d-4a96-8f60-79f6dd398b83") : secret "samples-operator-tls" not found
Jul 01 18:38:32.185162 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.185130    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-g7qtm"]
Jul 01 18:38:32.188327 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.188312    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7bb4f4bcc7-g7qtm"
Jul 01 18:38:32.191701 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.191679    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 01 18:38:32.191822 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.191747    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 01 18:38:32.192669 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.192648    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fdqvm\""
Jul 01 18:38:32.192778 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.192653    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 01 18:38:32.192778 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.192741    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 01 18:38:32.204884 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.204866    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-g7qtm"]
Jul 01 18:38:32.243545 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.243516    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:32.269330 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.269302    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/67605e58-5743-4ee4-b0c8-a497a48f8b4e-signing-key\") pod \"service-ca-7bb4f4bcc7-g7qtm\" (UID: \"67605e58-5743-4ee4-b0c8-a497a48f8b4e\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-g7qtm"
Jul 01 18:38:32.269494 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.269374    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwvcn\" (UniqueName: \"kubernetes.io/projected/67605e58-5743-4ee4-b0c8-a497a48f8b4e-kube-api-access-pwvcn\") pod \"service-ca-7bb4f4bcc7-g7qtm\" (UID: \"67605e58-5743-4ee4-b0c8-a497a48f8b4e\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-g7qtm"
Jul 01 18:38:32.269494 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.269414    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/67605e58-5743-4ee4-b0c8-a497a48f8b4e-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-g7qtm\" (UID: \"67605e58-5743-4ee4-b0c8-a497a48f8b4e\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-g7qtm"
Jul 01 18:38:32.321927 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.321850    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:32.370582 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.370554    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/67605e58-5743-4ee4-b0c8-a497a48f8b4e-signing-key\") pod \"service-ca-7bb4f4bcc7-g7qtm\" (UID: \"67605e58-5743-4ee4-b0c8-a497a48f8b4e\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-g7qtm"
Jul 01 18:38:32.370687 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.370633    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwvcn\" (UniqueName: \"kubernetes.io/projected/67605e58-5743-4ee4-b0c8-a497a48f8b4e-kube-api-access-pwvcn\") pod \"service-ca-7bb4f4bcc7-g7qtm\" (UID: \"67605e58-5743-4ee4-b0c8-a497a48f8b4e\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-g7qtm"
Jul 01 18:38:32.370687 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.370663    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/67605e58-5743-4ee4-b0c8-a497a48f8b4e-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-g7qtm\" (UID: \"67605e58-5743-4ee4-b0c8-a497a48f8b4e\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-g7qtm"
Jul 01 18:38:32.371318 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.371296    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/67605e58-5743-4ee4-b0c8-a497a48f8b4e-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-g7qtm\" (UID: \"67605e58-5743-4ee4-b0c8-a497a48f8b4e\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-g7qtm"
Jul 01 18:38:32.373081 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.373059    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/67605e58-5743-4ee4-b0c8-a497a48f8b4e-signing-key\") pod \"service-ca-7bb4f4bcc7-g7qtm\" (UID: \"67605e58-5743-4ee4-b0c8-a497a48f8b4e\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-g7qtm"
Jul 01 18:38:32.386289 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.386257    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwvcn\" (UniqueName: \"kubernetes.io/projected/67605e58-5743-4ee4-b0c8-a497a48f8b4e-kube-api-access-pwvcn\") pod \"service-ca-7bb4f4bcc7-g7qtm\" (UID: \"67605e58-5743-4ee4-b0c8-a497a48f8b4e\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-g7qtm"
Jul 01 18:38:32.444687 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.444654    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:32.496944 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.496922    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7bb4f4bcc7-g7qtm"
Jul 01 18:38:32.628757 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.628729    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-g7qtm"]
Jul 01 18:38:32.632632 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:38:32.632607    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67605e58_5743_4ee4_b0c8_a497a48f8b4e.slice/crio-e0f465fff669f5f0425ecddbd30e4b2fa3f27978af6e57ede4a3465c47ed3198 WatchSource:0}: Error finding container e0f465fff669f5f0425ecddbd30e4b2fa3f27978af6e57ede4a3465c47ed3198: Status 404 returned error can't find the container with id e0f465fff669f5f0425ecddbd30e4b2fa3f27978af6e57ede4a3465c47ed3198
Jul 01 18:38:32.913490 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:32.913403    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7bb4f4bcc7-g7qtm" event={"ID":"67605e58-5743-4ee4-b0c8-a497a48f8b4e","Type":"ContainerStarted","Data":"e0f465fff669f5f0425ecddbd30e4b2fa3f27978af6e57ede4a3465c47ed3198"}
Jul 01 18:38:34.919494 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:34.919460    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7bb4f4bcc7-g7qtm" event={"ID":"67605e58-5743-4ee4-b0c8-a497a48f8b4e","Type":"ContainerStarted","Data":"3223373ca02c32037ac3b25baf82133b9c6869e1d38d07acc76feca01d7ee2c2"}
Jul 01 18:38:34.954070 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:34.954027    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-7bb4f4bcc7-g7qtm" podStartSLOduration=1.143206315 podStartE2EDuration="2.95401206s" totalImagesPullingTime="1.810805745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:34.953544615 +0000 UTC m=+65.722382217" watchObservedRunningTime="2026-07-01 18:38:34.95401206 +0000 UTC m=+65.722849663"
Jul 01 18:38:34.993457 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:34.993430    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdadaf45-8a9b-4695-851d-c0b8297e545e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:34.993590 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:34.993550    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdadaf45-8a9b-4695-851d-c0b8297e545e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:34.993660 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:34.993600    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:34.993660 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:34.993646    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 18:38:34.993776 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:34.993674    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdadaf45-8a9b-4695-851d-c0b8297e545e-insights-runtime-extractor-tls podName:fdadaf45-8a9b-4695-851d-c0b8297e545e nodeName:}" failed. No retries permitted until 2026-07-01 18:38:42.993654265 +0000 UTC m=+73.762491850 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdadaf45-8a9b-4695-851d-c0b8297e545e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x2bsq" (UID: "fdadaf45-8a9b-4695-851d-c0b8297e545e") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:34.993776 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:34.993695    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdadaf45-8a9b-4695-851d-c0b8297e545e-kube-rbac-proxy-cm podName:fdadaf45-8a9b-4695-851d-c0b8297e545e nodeName:}" failed. No retries permitted until 2026-07-01 18:38:42.993684903 +0000 UTC m=+73.762522488 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fdadaf45-8a9b-4695-851d-c0b8297e545e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x2bsq" (UID: "fdadaf45-8a9b-4695-851d-c0b8297e545e") : configmap "kube-rbac-proxy" not found
Jul 01 18:38:35.396837 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:35.396802    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs\") pod \"network-metrics-daemon-plc42\" (UID: \"5180da14-bd5e-4777-b34c-4041f623113b\") " pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:38:35.397048 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:35.396865    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-original-pull-secret\") pod \"global-pull-secret-syncer-mrx57\" (UID: \"8bd0c6cc-1f64-4be9-8b7e-31fe9357b388\") " pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:38:35.399560 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:35.399538    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 18:38:35.399688 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:35.399607    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 18:38:35.407750 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:35.407731    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 01 18:38:35.407831 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:35.407798    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs podName:5180da14-bd5e-4777-b34c-4041f623113b nodeName:}" failed. No retries permitted until 2026-07-01 18:39:39.407781219 +0000 UTC m=+130.176618818 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs") pod "network-metrics-daemon-plc42" (UID: "5180da14-bd5e-4777-b34c-4041f623113b") : secret "metrics-daemon-secret" not found
Jul 01 18:38:35.409887 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:35.409870    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bd0c6cc-1f64-4be9-8b7e-31fe9357b388-original-pull-secret\") pod \"global-pull-secret-syncer-mrx57\" (UID: \"8bd0c6cc-1f64-4be9-8b7e-31fe9357b388\") " pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:38:35.470967 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:35.470937    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrx57"
Jul 01 18:38:35.532722 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:35.532676    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:35.598685 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:35.598631    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n4qd\" (UniqueName: \"kubernetes.io/projected/b105299c-7212-43c4-bb52-34aaaa2253f0-kube-api-access-2n4qd\") pod \"network-check-target-9t8tv\" (UID: \"b105299c-7212-43c4-bb52-34aaaa2253f0\") " pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:38:35.601574 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:35.601553    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 18:38:35.605027 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:35.605002    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mrx57"]
Jul 01 18:38:35.608239 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:38:35.608211    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd0c6cc_1f64_4be9_8b7e_31fe9357b388.slice/crio-15e9b97a55a35cb89ca8081771421c7c6267b0d808f1e8f9a5632d6e8796bfd4 WatchSource:0}: Error finding container 15e9b97a55a35cb89ca8081771421c7c6267b0d808f1e8f9a5632d6e8796bfd4: Status 404 returned error can't find the container with id 15e9b97a55a35cb89ca8081771421c7c6267b0d808f1e8f9a5632d6e8796bfd4
Jul 01 18:38:35.611843 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:35.611820    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 18:38:35.622158 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:35.622132    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n4qd\" (UniqueName: \"kubernetes.io/projected/b105299c-7212-43c4-bb52-34aaaa2253f0-kube-api-access-2n4qd\") pod \"network-check-target-9t8tv\" (UID: \"b105299c-7212-43c4-bb52-34aaaa2253f0\") " pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:38:35.678591 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:35.678329    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lwv7s\""
Jul 01 18:38:35.686295 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:35.686231    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:38:35.758554 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:35.758468    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:35.876739 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:35.876677    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9t8tv"]
Jul 01 18:38:35.880033 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:38:35.880002    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb105299c_7212_43c4_bb52_34aaaa2253f0.slice/crio-797db95cfb2ee2df3521ed5d45e38668d971f12a6815af20e7c800826d0d476a WatchSource:0}: Error finding container 797db95cfb2ee2df3521ed5d45e38668d971f12a6815af20e7c800826d0d476a: Status 404 returned error can't find the container with id 797db95cfb2ee2df3521ed5d45e38668d971f12a6815af20e7c800826d0d476a
Jul 01 18:38:35.911178 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:35.911149    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:35.924211 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:35.924129    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9t8tv" event={"ID":"b105299c-7212-43c4-bb52-34aaaa2253f0","Type":"ContainerStarted","Data":"797db95cfb2ee2df3521ed5d45e38668d971f12a6815af20e7c800826d0d476a"}
Jul 01 18:38:35.925405 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:35.925376    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mrx57" event={"ID":"8bd0c6cc-1f64-4be9-8b7e-31fe9357b388","Type":"ContainerStarted","Data":"15e9b97a55a35cb89ca8081771421c7c6267b0d808f1e8f9a5632d6e8796bfd4"}
Jul 01 18:38:36.015518 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:36.015475    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:36.124225 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:36.124191    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:36.254641 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:36.254598    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:36.452395 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:36.452341    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:36.641473 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:36.641436    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:36.844481 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:36.844382    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:36.911387 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:36.911302    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-metrics-tls\") pod \"dns-default-jndl8\" (UID: \"d2ee7ba9-21f2-4630-ae35-b9f65ec8de37\") " pod="openshift-dns/dns-default-jndl8"
Jul 01 18:38:36.911568 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:36.911472    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:38:36.911568 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:36.911546    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-metrics-tls podName:d2ee7ba9-21f2-4630-ae35-b9f65ec8de37 nodeName:}" failed. No retries permitted until 2026-07-01 18:39:08.911526667 +0000 UTC m=+99.680364255 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-metrics-tls") pod "dns-default-jndl8" (UID: "d2ee7ba9-21f2-4630-ae35-b9f65ec8de37") : secret "dns-default-metrics-tls" not found
Jul 01 18:38:37.442573 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:37.442538    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:37.637474 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:37.637406    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:38.250598 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:38.250536    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:38.451496 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:38.451455    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:38.582794 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:38.582686    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:38.855587 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:38.855504    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:39.055008 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:39.054975    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:39.449193 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:39.449155    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:39.547810 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:39.547776    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:39.680687 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:39.680651    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:39.996695 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:39.996650    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:40.143958 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:40.143921    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b901864b-08c4-4e62-ab99-71715e994501-service-ca-bundle\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:40.144148 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:40.143967    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b901864b-08c4-4e62-ab99-71715e994501-metrics-certs\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:40.144148 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:40.144111    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b901864b-08c4-4e62-ab99-71715e994501-service-ca-bundle podName:b901864b-08c4-4e62-ab99-71715e994501 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:56.144087553 +0000 UTC m=+86.912925148 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b901864b-08c4-4e62-ab99-71715e994501-service-ca-bundle") pod "router-default-cbcfcfdbf-r98lq" (UID: "b901864b-08c4-4e62-ab99-71715e994501") : configmap references non-existent config key: service-ca.crt
Jul 01 18:38:40.146648 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:40.146626    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b901864b-08c4-4e62-ab99-71715e994501-metrics-certs\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:40.245848 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:40.245811    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f31fae04-dc5d-4a96-8f60-79f6dd398b83-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-zz5z7\" (UID: \"f31fae04-dc5d-4a96-8f60-79f6dd398b83\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zz5z7"
Jul 01 18:38:40.248508 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:40.248451    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f31fae04-dc5d-4a96-8f60-79f6dd398b83-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-zz5z7\" (UID: \"f31fae04-dc5d-4a96-8f60-79f6dd398b83\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zz5z7"
Jul 01 18:38:40.266349 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:40.266315    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:40.405499 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:40.405459    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-n4vj9\""
Jul 01 18:38:40.411117 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:40.411092    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zz5z7"
Jul 01 18:38:40.424486 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:40.424458    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:40.585946 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:40.585834    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zz5z7"]
Jul 01 18:38:40.645555 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:40.645526    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:40.757331 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:40.757247    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:40.877776 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:40.877740    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:40.940778 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:40.940745    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9t8tv" event={"ID":"b105299c-7212-43c4-bb52-34aaaa2253f0","Type":"ContainerStarted","Data":"f52b24776161dd7b43008e6dc91635bfd4ff62c6f7109161e1c61f2cb4b6c0dc"}
Jul 01 18:38:40.940939 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:40.940836    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:38:40.942232 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:40.942209    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mrx57" event={"ID":"8bd0c6cc-1f64-4be9-8b7e-31fe9357b388","Type":"ContainerStarted","Data":"33db82d8047ea3e8c2753dfe6f06035586487fc8feecf64c5b4d11de4da77a0c"}
Jul 01 18:38:40.943268 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:40.943246    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zz5z7" event={"ID":"f31fae04-dc5d-4a96-8f60-79f6dd398b83","Type":"ContainerStarted","Data":"64d69f24edac6bfc831d810e3514c70591e4fb7602f67b3503be56c6b28c4a61"}
Jul 01 18:38:40.962447 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:40.962405    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9t8tv" podStartSLOduration=67.401218745 podStartE2EDuration="1m11.96239377s" totalImagesPullingTime="4.561175025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:40.961824687 +0000 UTC m=+71.730662301" watchObservedRunningTime="2026-07-01 18:38:40.96239377 +0000 UTC m=+71.731231378"
Jul 01 18:38:40.991000 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:40.990945    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mrx57" podStartSLOduration=67.152269316 podStartE2EDuration="1m11.990925551s" totalImagesPullingTime="4.838656235s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:40.989691985 +0000 UTC m=+71.758529588" watchObservedRunningTime="2026-07-01 18:38:40.990925551 +0000 UTC m=+71.759763156"
Jul 01 18:38:41.057105 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:41.057015    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:41.251478 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:41.251419    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:42.042556 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:42.042510    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:42.680510 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:42.680479    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:42.885619 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:42.885591    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:42.951485 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:42.951396    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zz5z7" event={"ID":"f31fae04-dc5d-4a96-8f60-79f6dd398b83","Type":"ContainerStarted","Data":"aeca2da99f8b6396a7d6545c8dc58ef3df1a6137bf3fd2b6e41e791730df87fc"}
Jul 01 18:38:42.951610 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:42.951504    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zz5z7" event={"ID":"f31fae04-dc5d-4a96-8f60-79f6dd398b83","Type":"ContainerStarted","Data":"a30b64585d99264f5872963421490cedc0a6373042b2a1c3c6af0efeb16717c2"}
Jul 01 18:38:43.001378 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:43.001315    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-zz5z7" podStartSLOduration=17.446015353 podStartE2EDuration="19.001295473s" totalImagesPullingTime="1.55528012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:43.000757264 +0000 UTC m=+73.769594867" watchObservedRunningTime="2026-07-01 18:38:43.001295473 +0000 UTC m=+73.770133078"
Jul 01 18:38:43.012824 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:43.012794    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:43.071130 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:43.071102    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdadaf45-8a9b-4695-851d-c0b8297e545e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:43.071508 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:43.071166    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdadaf45-8a9b-4695-851d-c0b8297e545e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:43.071508 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:43.071345    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:43.071508 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:38:43.071413    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdadaf45-8a9b-4695-851d-c0b8297e545e-insights-runtime-extractor-tls podName:fdadaf45-8a9b-4695-851d-c0b8297e545e nodeName:}" failed. No retries permitted until 2026-07-01 18:38:59.07139484 +0000 UTC m=+89.840232425 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdadaf45-8a9b-4695-851d-c0b8297e545e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x2bsq" (UID: "fdadaf45-8a9b-4695-851d-c0b8297e545e") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:43.071679 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:43.071595    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdadaf45-8a9b-4695-851d-c0b8297e545e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:43.244779 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:43.244750    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:43.427677 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:43.427646    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:43.625961 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:43.625878    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:44.247945 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:44.247917    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:44.671633 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:44.671546    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:44.864392 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:44.864360    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:45.058130 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:45.058098    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:45.268255 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:45.268220    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:46.062000 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:46.061962    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:46.252889 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:46.252862    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:46.484116 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:46.484086    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:49.099519 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:49.099482    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:49.226584 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:49.226549    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:49.423922 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:49.423843    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:49.746130 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:49.746103    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:49.861587 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:49.861559    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:50.157390 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:50.157296    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:50.560203 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:50.560170    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:50.660538 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:50.660506    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:50.759552 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:50.759522    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:50.950592 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:50.950513    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:51.630964 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:51.630930    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:51.820973 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:51.820942    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:51.964460 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:51.964411    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:55.290241 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.290198    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj"]
Jul 01 18:38:55.293102 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.293082    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj"
Jul 01 18:38:55.295745 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.295707    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 01 18:38:55.296823 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.296799    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 01 18:38:55.296935 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.296922    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 01 18:38:55.297000 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.296953    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 01 18:38:55.297063 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.296922    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 01 18:38:55.297063 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.297010    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 01 18:38:55.297063 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.296956    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 01 18:38:55.330942 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.330914    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj"]
Jul 01 18:38:55.373321 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.373291    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/25611c86-04d6-4623-87b3-671d3631a687-ca\") pod \"cluster-proxy-proxy-agent-7798c9f66b-qjhqj\" (UID: \"25611c86-04d6-4623-87b3-671d3631a687\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj"
Jul 01 18:38:55.373321 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.373319    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/25611c86-04d6-4623-87b3-671d3631a687-hub\") pod \"cluster-proxy-proxy-agent-7798c9f66b-qjhqj\" (UID: \"25611c86-04d6-4623-87b3-671d3631a687\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj"
Jul 01 18:38:55.373517 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.373355    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/25611c86-04d6-4623-87b3-671d3631a687-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7798c9f66b-qjhqj\" (UID: \"25611c86-04d6-4623-87b3-671d3631a687\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj"
Jul 01 18:38:55.373517 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.373402    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44rgt\" (UniqueName: \"kubernetes.io/projected/25611c86-04d6-4623-87b3-671d3631a687-kube-api-access-44rgt\") pod \"cluster-proxy-proxy-agent-7798c9f66b-qjhqj\" (UID: \"25611c86-04d6-4623-87b3-671d3631a687\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj"
Jul 01 18:38:55.373517 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.373453    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/25611c86-04d6-4623-87b3-671d3631a687-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7798c9f66b-qjhqj\" (UID: \"25611c86-04d6-4623-87b3-671d3631a687\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj"
Jul 01 18:38:55.373517 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.373474    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/25611c86-04d6-4623-87b3-671d3631a687-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7798c9f66b-qjhqj\" (UID: \"25611c86-04d6-4623-87b3-671d3631a687\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj"
Jul 01 18:38:55.474400 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.474369    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/25611c86-04d6-4623-87b3-671d3631a687-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7798c9f66b-qjhqj\" (UID: \"25611c86-04d6-4623-87b3-671d3631a687\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj"
Jul 01 18:38:55.474400 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.474401    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44rgt\" (UniqueName: \"kubernetes.io/projected/25611c86-04d6-4623-87b3-671d3631a687-kube-api-access-44rgt\") pod \"cluster-proxy-proxy-agent-7798c9f66b-qjhqj\" (UID: \"25611c86-04d6-4623-87b3-671d3631a687\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj"
Jul 01 18:38:55.474614 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.474449    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/25611c86-04d6-4623-87b3-671d3631a687-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7798c9f66b-qjhqj\" (UID: \"25611c86-04d6-4623-87b3-671d3631a687\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj"
Jul 01 18:38:55.474614 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.474487    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/25611c86-04d6-4623-87b3-671d3631a687-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7798c9f66b-qjhqj\" (UID: \"25611c86-04d6-4623-87b3-671d3631a687\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj"
Jul 01 18:38:55.474614 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.474574    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/25611c86-04d6-4623-87b3-671d3631a687-ca\") pod \"cluster-proxy-proxy-agent-7798c9f66b-qjhqj\" (UID: \"25611c86-04d6-4623-87b3-671d3631a687\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj"
Jul 01 18:38:55.474614 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.474597    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/25611c86-04d6-4623-87b3-671d3631a687-hub\") pod \"cluster-proxy-proxy-agent-7798c9f66b-qjhqj\" (UID: \"25611c86-04d6-4623-87b3-671d3631a687\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj"
Jul 01 18:38:55.475332 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.475306    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/25611c86-04d6-4623-87b3-671d3631a687-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7798c9f66b-qjhqj\" (UID: \"25611c86-04d6-4623-87b3-671d3631a687\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj"
Jul 01 18:38:55.476958 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.476937    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/25611c86-04d6-4623-87b3-671d3631a687-ca\") pod \"cluster-proxy-proxy-agent-7798c9f66b-qjhqj\" (UID: \"25611c86-04d6-4623-87b3-671d3631a687\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj"
Jul 01 18:38:55.477116 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.477097    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/25611c86-04d6-4623-87b3-671d3631a687-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7798c9f66b-qjhqj\" (UID: \"25611c86-04d6-4623-87b3-671d3631a687\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj"
Jul 01 18:38:55.477172 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.477137    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/25611c86-04d6-4623-87b3-671d3631a687-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7798c9f66b-qjhqj\" (UID: \"25611c86-04d6-4623-87b3-671d3631a687\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj"
Jul 01 18:38:55.477236 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.477221    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/25611c86-04d6-4623-87b3-671d3631a687-hub\") pod \"cluster-proxy-proxy-agent-7798c9f66b-qjhqj\" (UID: \"25611c86-04d6-4623-87b3-671d3631a687\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj"
Jul 01 18:38:55.514141 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.514117    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44rgt\" (UniqueName: \"kubernetes.io/projected/25611c86-04d6-4623-87b3-671d3631a687-kube-api-access-44rgt\") pod \"cluster-proxy-proxy-agent-7798c9f66b-qjhqj\" (UID: \"25611c86-04d6-4623-87b3-671d3631a687\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj"
Jul 01 18:38:55.527625 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.527589    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:55.612956 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.612879    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj"
Jul 01 18:38:55.631776 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.631738    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:55.773247 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.773218    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj"]
Jul 01 18:38:55.775365 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:38:55.775339    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25611c86_04d6_4623_87b3_671d3631a687.slice/crio-d5c1efc0f4c60185047ee537e4a91a84193dbbdc02360c9347bfd8f3bf7c1a94 WatchSource:0}: Error finding container d5c1efc0f4c60185047ee537e4a91a84193dbbdc02360c9347bfd8f3bf7c1a94: Status 404 returned error can't find the container with id d5c1efc0f4c60185047ee537e4a91a84193dbbdc02360c9347bfd8f3bf7c1a94
Jul 01 18:38:55.991202 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:55.991164    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj" event={"ID":"25611c86-04d6-4623-87b3-671d3631a687","Type":"ContainerStarted","Data":"d5c1efc0f4c60185047ee537e4a91a84193dbbdc02360c9347bfd8f3bf7c1a94"}
Jul 01 18:38:56.180319 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:56.180282    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b901864b-08c4-4e62-ab99-71715e994501-service-ca-bundle\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:56.181122 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:56.181095    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b901864b-08c4-4e62-ab99-71715e994501-service-ca-bundle\") pod \"router-default-cbcfcfdbf-r98lq\" (UID: \"b901864b-08c4-4e62-ab99-71715e994501\") " pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:56.413309 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:56.413243    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-nrphr\""
Jul 01 18:38:56.419623 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:56.419595    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:56.579305 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:56.579262    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-cbcfcfdbf-r98lq"]
Jul 01 18:38:56.583036 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:38:56.583004    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb901864b_08c4_4e62_ab99_71715e994501.slice/crio-e8bae0b123001d45aba5a9e22669be883dbf00ab006053b4e66d247ee61dd6ec WatchSource:0}: Error finding container e8bae0b123001d45aba5a9e22669be883dbf00ab006053b4e66d247ee61dd6ec: Status 404 returned error can't find the container with id e8bae0b123001d45aba5a9e22669be883dbf00ab006053b4e66d247ee61dd6ec
Jul 01 18:38:56.996636 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:56.996597    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-cbcfcfdbf-r98lq" event={"ID":"b901864b-08c4-4e62-ab99-71715e994501","Type":"ContainerStarted","Data":"05ecd52042f1102bdb64f2317534bf606809a1486f81b8aa0735b8925381539c"}
Jul 01 18:38:56.996833 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:56.996655    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-cbcfcfdbf-r98lq" event={"ID":"b901864b-08c4-4e62-ab99-71715e994501","Type":"ContainerStarted","Data":"e8bae0b123001d45aba5a9e22669be883dbf00ab006053b4e66d247ee61dd6ec"}
Jul 01 18:38:57.034297 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:57.034246    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-cbcfcfdbf-r98lq" podStartSLOduration=33.034231019 podStartE2EDuration="33.034231019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:57.032798576 +0000 UTC m=+87.801636178" watchObservedRunningTime="2026-07-01 18:38:57.034231019 +0000 UTC m=+87.803068622"
Jul 01 18:38:57.420744 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:57.420661    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:57.423591 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:57.423571    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:57.654286 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:57.654248    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:57.762302 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:57.762269    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:58.000273 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:58.000244    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:58.001748 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:58.001707    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-cbcfcfdbf-r98lq"
Jul 01 18:38:58.429271 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:58.429195    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:58.554187 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:58.554153    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:58.658647 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:58.658617    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:58.767617 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:58.767583    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:59.034190 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:59.034105    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:59.105231 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:59.105200    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdadaf45-8a9b-4695-851d-c0b8297e545e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:59.107985 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:59.107961    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdadaf45-8a9b-4695-851d-c0b8297e545e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x2bsq\" (UID: \"fdadaf45-8a9b-4695-851d-c0b8297e545e\") " pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:59.371675 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:59.371598    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q4zk9\""
Jul 01 18:38:59.379098 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:59.379075    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x2bsq"
Jul 01 18:38:59.829626 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:38:59.829594    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x2bsq"]
Jul 01 18:38:59.834142 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:38:59.834114    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdadaf45_8a9b_4695_851d_c0b8297e545e.slice/crio-647f847ca7ed412226c56e9bbc8497dde82bd50701ac6d2bb1ddab0d8d39c9db WatchSource:0}: Error finding container 647f847ca7ed412226c56e9bbc8497dde82bd50701ac6d2bb1ddab0d8d39c9db: Status 404 returned error can't find the container with id 647f847ca7ed412226c56e9bbc8497dde82bd50701ac6d2bb1ddab0d8d39c9db
Jul 01 18:39:00.009658 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:00.009623    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj" event={"ID":"25611c86-04d6-4623-87b3-671d3631a687","Type":"ContainerStarted","Data":"f75356b9c726ad2f8d638c50c7b5c261aa05d5d193dfb1e2a7a67e8a87dbc782"}
Jul 01 18:39:00.009865 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:00.009667    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj" event={"ID":"25611c86-04d6-4623-87b3-671d3631a687","Type":"ContainerStarted","Data":"90de8c5cfeae678544950c1e6a5f3cd9bef33eb3a477f4485272b14c20f6a498"}
Jul 01 18:39:00.009865 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:00.009683    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj" event={"ID":"25611c86-04d6-4623-87b3-671d3631a687","Type":"ContainerStarted","Data":"71eaf3c611b3e90d8eeef6c4b1864d3c800fd30cf5915ba6b2ab014b539dbcad"}
Jul 01 18:39:00.010845 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:00.010820    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x2bsq" event={"ID":"fdadaf45-8a9b-4695-851d-c0b8297e545e","Type":"ContainerStarted","Data":"647f847ca7ed412226c56e9bbc8497dde82bd50701ac6d2bb1ddab0d8d39c9db"}
Jul 01 18:39:00.034160 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:00.034111    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7798c9f66b-qjhqj" podStartSLOduration=1.009376869 podStartE2EDuration="5.034096789s" totalImagesPullingTime="4.02471992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:00.033293812 +0000 UTC m=+90.802131413" watchObservedRunningTime="2026-07-01 18:39:00.034096789 +0000 UTC m=+90.802934391"
Jul 01 18:39:01.379487 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:01.379446    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:01.467158 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:01.467122    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:04.026213 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:04.026179    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x2bsq" event={"ID":"fdadaf45-8a9b-4695-851d-c0b8297e545e","Type":"ContainerStarted","Data":"e845ad71ae7782ec75217ad885f68d17477105d4f36257f6734ffd9726655db7"}
Jul 01 18:39:04.428944 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:04.428860    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:04.539450 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:04.539410    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:05.032008 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:05.031961    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x2bsq" event={"ID":"fdadaf45-8a9b-4695-851d-c0b8297e545e","Type":"ContainerStarted","Data":"11b5767a369166c735e837ecdfd74bb8e6347caa7c2b783b274f914e4ed3f9b2"}
Jul 01 18:39:05.623625 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:05.623590    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:05.731506 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:05.731476    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:07.039979 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:07.039940    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x2bsq" event={"ID":"fdadaf45-8a9b-4695-851d-c0b8297e545e","Type":"ContainerStarted","Data":"d1f318abbcc3972da39d638b236320cdd5382d84fd2f87c0c47db1a28d808845"}
Jul 01 18:39:07.063597 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:07.063539    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-x2bsq" podStartSLOduration=33.490577041 podStartE2EDuration="40.063520328s" totalImagesPullingTime="6.572943287s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:38:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:07.062857947 +0000 UTC m=+97.831695550" watchObservedRunningTime="2026-07-01 18:39:07.063520328 +0000 UTC m=+97.832357943"
Jul 01 18:39:07.762050 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:07.762015    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:07.885915 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:07.885883    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:08.003243 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:08.003210    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:08.989464 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:08.989427    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-metrics-tls\") pod \"dns-default-jndl8\" (UID: \"d2ee7ba9-21f2-4630-ae35-b9f65ec8de37\") " pod="openshift-dns/dns-default-jndl8"
Jul 01 18:39:08.991651 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:08.991633    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2ee7ba9-21f2-4630-ae35-b9f65ec8de37-metrics-tls\") pod \"dns-default-jndl8\" (UID: \"d2ee7ba9-21f2-4630-ae35-b9f65ec8de37\") " pod="openshift-dns/dns-default-jndl8"
Jul 01 18:39:09.278808 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:09.278737    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7sn7m\""
Jul 01 18:39:09.286581 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:09.286560    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jndl8"
Jul 01 18:39:09.427807 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:09.427779    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jndl8"]
Jul 01 18:39:09.430864 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:39:09.430836    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ee7ba9_21f2_4630_ae35_b9f65ec8de37.slice/crio-e8f7aa06babdbfa0a35bef4c0592f6d296883e12126196b0809643979f590bb4 WatchSource:0}: Error finding container e8f7aa06babdbfa0a35bef4c0592f6d296883e12126196b0809643979f590bb4: Status 404 returned error can't find the container with id e8f7aa06babdbfa0a35bef4c0592f6d296883e12126196b0809643979f590bb4
Jul 01 18:39:10.052119 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:10.052081    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jndl8" event={"ID":"d2ee7ba9-21f2-4630-ae35-b9f65ec8de37","Type":"ContainerStarted","Data":"e8f7aa06babdbfa0a35bef4c0592f6d296883e12126196b0809643979f590bb4"}
Jul 01 18:39:10.804199 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:10.804157    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:10.957487 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:10.957458    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:11.057406 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:11.057374    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jndl8" event={"ID":"d2ee7ba9-21f2-4630-ae35-b9f65ec8de37","Type":"ContainerStarted","Data":"15a34a4fd9d0419f1ec6bc6f78832f68dd0172461d7466f9d2c457ba32c5a1ca"}
Jul 01 18:39:11.099692 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:11.099651    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:11.948421 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:11.948391    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9t8tv"
Jul 01 18:39:12.062023 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:12.061974    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jndl8" event={"ID":"d2ee7ba9-21f2-4630-ae35-b9f65ec8de37","Type":"ContainerStarted","Data":"2c8a87b3c0a17d40b0c00de030bbfaf2ee74bc675735daf1941584d5c00c9ba8"}
Jul 01 18:39:12.062415 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:12.062097    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jndl8"
Jul 01 18:39:12.088047 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:12.088002    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jndl8" podStartSLOduration=65.638245304 podStartE2EDuration="1m7.087988393s" totalImagesPullingTime="1.449743089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:12.086836762 +0000 UTC m=+102.855674366" watchObservedRunningTime="2026-07-01 18:39:12.087988393 +0000 UTC m=+102.856825997"
Jul 01 18:39:13.055964 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.055925    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:13.220806 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.220767    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:13.373329 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.373254    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:13.532960 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.532927    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:13.559109 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.559076    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gtr56"]
Jul 01 18:39:13.579765 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.579733    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.582576 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.582549    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 01 18:39:13.582824 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.582810    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 01 18:39:13.583509 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.583491    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 01 18:39:13.583657 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.583638    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 01 18:39:13.583753 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.583672    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 01 18:39:13.583825 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.583754    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mtwh7\""
Jul 01 18:39:13.583825 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.583782    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 01 18:39:13.624681 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.624584    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b199819c-4d58-44f6-9abc-5096b64e4bbd-node-exporter-tls\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.624681 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.624650    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b199819c-4d58-44f6-9abc-5096b64e4bbd-node-exporter-accelerators-collector-config\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.624923 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.624755    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b199819c-4d58-44f6-9abc-5096b64e4bbd-node-exporter-textfile\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.624923 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.624788    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b199819c-4d58-44f6-9abc-5096b64e4bbd-metrics-client-ca\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.624923 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.624821    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b199819c-4d58-44f6-9abc-5096b64e4bbd-node-exporter-wtmp\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.625061 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.624928    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b199819c-4d58-44f6-9abc-5096b64e4bbd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.625061 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.624970    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqjr4\" (UniqueName: \"kubernetes.io/projected/b199819c-4d58-44f6-9abc-5096b64e4bbd-kube-api-access-tqjr4\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.625061 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.625013    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b199819c-4d58-44f6-9abc-5096b64e4bbd-root\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.625061 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.625041    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b199819c-4d58-44f6-9abc-5096b64e4bbd-sys\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.691727 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.691676    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:13.726028 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.726000    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b199819c-4d58-44f6-9abc-5096b64e4bbd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.726196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.726032    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqjr4\" (UniqueName: \"kubernetes.io/projected/b199819c-4d58-44f6-9abc-5096b64e4bbd-kube-api-access-tqjr4\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.726196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.726058    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b199819c-4d58-44f6-9abc-5096b64e4bbd-root\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.726196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.726078    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b199819c-4d58-44f6-9abc-5096b64e4bbd-sys\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.726196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.726103    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b199819c-4d58-44f6-9abc-5096b64e4bbd-node-exporter-tls\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.726196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.726152    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b199819c-4d58-44f6-9abc-5096b64e4bbd-root\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.726196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.726176    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b199819c-4d58-44f6-9abc-5096b64e4bbd-sys\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.726482 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.726238    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b199819c-4d58-44f6-9abc-5096b64e4bbd-node-exporter-accelerators-collector-config\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.726482 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.726288    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b199819c-4d58-44f6-9abc-5096b64e4bbd-node-exporter-textfile\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.726482 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.726312    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b199819c-4d58-44f6-9abc-5096b64e4bbd-metrics-client-ca\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.726482 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.726342    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b199819c-4d58-44f6-9abc-5096b64e4bbd-node-exporter-wtmp\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.726679 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.726497    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b199819c-4d58-44f6-9abc-5096b64e4bbd-node-exporter-wtmp\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.726679 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.726630    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b199819c-4d58-44f6-9abc-5096b64e4bbd-node-exporter-textfile\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.726798 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.726787    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b199819c-4d58-44f6-9abc-5096b64e4bbd-node-exporter-accelerators-collector-config\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.726998 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.726976    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b199819c-4d58-44f6-9abc-5096b64e4bbd-metrics-client-ca\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.728399 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.728379    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b199819c-4d58-44f6-9abc-5096b64e4bbd-node-exporter-tls\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.728502 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.728487    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b199819c-4d58-44f6-9abc-5096b64e4bbd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.740581 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.740556    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqjr4\" (UniqueName: \"kubernetes.io/projected/b199819c-4d58-44f6-9abc-5096b64e4bbd-kube-api-access-tqjr4\") pod \"node-exporter-gtr56\" (UID: \"b199819c-4d58-44f6-9abc-5096b64e4bbd\") " pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.829580 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.829551    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:13.890605 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:13.890526    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gtr56"
Jul 01 18:39:13.912519 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:39:13.912492    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb199819c_4d58_44f6_9abc_5096b64e4bbd.slice/crio-ac81573d404da6f9f5431e4302a18b78051c76cd65ba8d2098c79031937c5f20 WatchSource:0}: Error finding container ac81573d404da6f9f5431e4302a18b78051c76cd65ba8d2098c79031937c5f20: Status 404 returned error can't find the container with id ac81573d404da6f9f5431e4302a18b78051c76cd65ba8d2098c79031937c5f20
Jul 01 18:39:14.014898 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:14.014868    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:14.070640 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:14.070603    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gtr56" event={"ID":"b199819c-4d58-44f6-9abc-5096b64e4bbd","Type":"ContainerStarted","Data":"ac81573d404da6f9f5431e4302a18b78051c76cd65ba8d2098c79031937c5f20"}
Jul 01 18:39:14.279514 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:14.279483    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:14.561446 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:14.561349    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:14.765251 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:14.765215    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:14.993139 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:14.993110    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:15.196620 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:15.196587    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:16.028442 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:16.028380    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:16.078595 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:16.078563    2573 generic.go:356] "Generic (PLEG): container finished" podID="b199819c-4d58-44f6-9abc-5096b64e4bbd" containerID="79c3560907adb7fde119b7a2b5d393a31a15c2875974469fe7df9c6e3e9a8497" exitCode=0
Jul 01 18:39:16.078809 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:16.078606    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gtr56" event={"ID":"b199819c-4d58-44f6-9abc-5096b64e4bbd","Type":"ContainerDied","Data":"79c3560907adb7fde119b7a2b5d393a31a15c2875974469fe7df9c6e3e9a8497"}
Jul 01 18:39:16.155847 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:16.155821    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:16.637404 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:16.637366    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:16.795536 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:16.795500    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:17.075520 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:17.075481    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:17.083896 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:17.083872    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gtr56" event={"ID":"b199819c-4d58-44f6-9abc-5096b64e4bbd","Type":"ContainerStarted","Data":"437dd267f5b107e1b41cea73dc89511c3a7346e88703bb6e1780dc9bcaa0f6b2"}
Jul 01 18:39:17.084040 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:17.083916    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gtr56" event={"ID":"b199819c-4d58-44f6-9abc-5096b64e4bbd","Type":"ContainerStarted","Data":"98f443100bbb8f968391402b753fdee69efc4e7ad6168b4b79820b3a8af95ef2"}
Jul 01 18:39:17.118211 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:17.118157    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gtr56" podStartSLOduration=2.620378886 podStartE2EDuration="4.118142173s" totalImagesPullingTime="1.431905823s" totalInitContainerRuntime="65.857464ms" isStatefulPod=true podCreationTimestamp="2026-07-01 18:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:17.116412184 +0000 UTC m=+107.885249788" watchObservedRunningTime="2026-07-01 18:39:17.118142173 +0000 UTC m=+107.886979775"
Jul 01 18:39:17.451548 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:17.451456    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:17.577063 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:17.577029    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:18.092775 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:18.092737    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:18.238546 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:18.238510    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:18.595230 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:18.595197    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:19.703056 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.703019    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-569b8dddf7-rfnp7"]
Jul 01 18:39:19.705051 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.705035    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:19.710844 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.710815    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 01 18:39:19.710844 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.710842    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 01 18:39:19.711067 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.710880    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 01 18:39:19.711067 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.710815    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 01 18:39:19.712102 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.712085    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 01 18:39:19.713637 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.713622    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 01 18:39:19.713740 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.713691    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 01 18:39:19.714018 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.714004    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-47s2p\""
Jul 01 18:39:19.719376 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.719357    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 01 18:39:19.730759 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.730729    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-569b8dddf7-rfnp7"]
Jul 01 18:39:19.777162 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.777126    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-oauth-serving-cert\") pod \"console-569b8dddf7-rfnp7\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") " pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:19.777335 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.777184    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-console-config\") pod \"console-569b8dddf7-rfnp7\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") " pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:19.777335 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.777264    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-console-oauth-config\") pod \"console-569b8dddf7-rfnp7\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") " pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:19.777335 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.777304    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-trusted-ca-bundle\") pod \"console-569b8dddf7-rfnp7\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") " pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:19.777335 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.777325    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-console-serving-cert\") pod \"console-569b8dddf7-rfnp7\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") " pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:19.777504 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.777343    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-service-ca\") pod \"console-569b8dddf7-rfnp7\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") " pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:19.777504 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.777359    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfc4k\" (UniqueName: \"kubernetes.io/projected/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-kube-api-access-lfc4k\") pod \"console-569b8dddf7-rfnp7\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") " pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:19.877666 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.877637    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-trusted-ca-bundle\") pod \"console-569b8dddf7-rfnp7\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") " pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:19.877870 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.877674    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-console-serving-cert\") pod \"console-569b8dddf7-rfnp7\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") " pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:19.877870 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.877691    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-service-ca\") pod \"console-569b8dddf7-rfnp7\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") " pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:19.877870 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.877706    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfc4k\" (UniqueName: \"kubernetes.io/projected/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-kube-api-access-lfc4k\") pod \"console-569b8dddf7-rfnp7\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") " pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:19.877870 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.877756    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-oauth-serving-cert\") pod \"console-569b8dddf7-rfnp7\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") " pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:19.877870 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.877815    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-console-config\") pod \"console-569b8dddf7-rfnp7\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") " pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:19.877870 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.877865    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-console-oauth-config\") pod \"console-569b8dddf7-rfnp7\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") " pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:19.878642 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.878617    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-oauth-serving-cert\") pod \"console-569b8dddf7-rfnp7\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") " pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:19.878770 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.878669    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-trusted-ca-bundle\") pod \"console-569b8dddf7-rfnp7\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") " pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:19.878770 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.878701    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-service-ca\") pod \"console-569b8dddf7-rfnp7\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") " pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:19.878770 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.878732    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-console-config\") pod \"console-569b8dddf7-rfnp7\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") " pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:19.880802 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.880779    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-console-serving-cert\") pod \"console-569b8dddf7-rfnp7\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") " pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:19.880892 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.880854    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-console-oauth-config\") pod \"console-569b8dddf7-rfnp7\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") " pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:19.889073 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:19.889052    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfc4k\" (UniqueName: \"kubernetes.io/projected/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-kube-api-access-lfc4k\") pod \"console-569b8dddf7-rfnp7\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") " pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:20.014284 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:20.014185    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:20.130475 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:20.130422    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:20.187580 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:20.187545    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-569b8dddf7-rfnp7"]
Jul 01 18:39:20.193587 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:39:20.193563    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f2cfb3_aa91_4695_bba9_495a83bf4d9d.slice/crio-213595be89ac29d793fe1346f82683b133a8836ba236912fa2a34309e4c83157 WatchSource:0}: Error finding container 213595be89ac29d793fe1346f82683b133a8836ba236912fa2a34309e4c83157: Status 404 returned error can't find the container with id 213595be89ac29d793fe1346f82683b133a8836ba236912fa2a34309e4c83157
Jul 01 18:39:21.100672 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:21.100637    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-569b8dddf7-rfnp7" event={"ID":"19f2cfb3-aa91-4695-bba9-495a83bf4d9d","Type":"ContainerStarted","Data":"213595be89ac29d793fe1346f82683b133a8836ba236912fa2a34309e4c83157"}
Jul 01 18:39:21.898036 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:21.898002    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:21.993786 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:21.993739    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:22.068959 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:22.068928    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jndl8"
Jul 01 18:39:23.109184 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:23.109097    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-569b8dddf7-rfnp7" event={"ID":"19f2cfb3-aa91-4695-bba9-495a83bf4d9d","Type":"ContainerStarted","Data":"6ce3b736fba120d6feba53479b5cd0d2a8f4adc47536517c8a040a3da2ceef3c"}
Jul 01 18:39:23.132577 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:23.132528    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-569b8dddf7-rfnp7" podStartSLOduration=1.5380572620000001 podStartE2EDuration="4.132510238s" totalImagesPullingTime="2.594452976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:23.13110907 +0000 UTC m=+113.899946671" watchObservedRunningTime="2026-07-01 18:39:23.132510238 +0000 UTC m=+113.901347842"
Jul 01 18:39:28.017798 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:28.017762    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:28.079575 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:28.079545    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-569b8dddf7-rfnp7"]
Jul 01 18:39:28.162538 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:28.162508    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:28.333415 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:28.333327    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:28.445767 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:28.445735    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:29.234351 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:29.234318    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:29.336828 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:29.336798    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:29.538857 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:29.538770    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:29.627018 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:29.626992    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:30.014649 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:30.014613    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:31.379913 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:31.379882    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:31.465272 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:31.465236    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:32.594393 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:32.594348    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:32.679675 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:32.679645    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:37.158614 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:37.158523    2573 generic.go:356] "Generic (PLEG): container finished" podID="af0cbfce-e39e-41c5-a6f8-43aea34bdc3c" containerID="ccb40ad9f7e1872f5bdd1015cce823b778fc3520687516cefeb0b3158883a6f1" exitCode=0
Jul 01 18:39:37.159008 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:37.158602    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-4lw92" event={"ID":"af0cbfce-e39e-41c5-a6f8-43aea34bdc3c","Type":"ContainerDied","Data":"ccb40ad9f7e1872f5bdd1015cce823b778fc3520687516cefeb0b3158883a6f1"}
Jul 01 18:39:37.159008 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:37.158945    2573 scope.go:117] "RemoveContainer" containerID="ccb40ad9f7e1872f5bdd1015cce823b778fc3520687516cefeb0b3158883a6f1"
Jul 01 18:39:37.918010 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:37.917983    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-cbcfcfdbf-r98lq_b901864b-08c4-4e62-ab99-71715e994501/router/0.log"
Jul 01 18:39:37.968381 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:37.968356    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n5dbd_e1af1a8d-9770-41da-b5f0-a4ad75852811/serve-healthcheck-canary/0.log"
Jul 01 18:39:38.031575 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:38.031542    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:38.163669 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:38.163643    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-4lw92" event={"ID":"af0cbfce-e39e-41c5-a6f8-43aea34bdc3c","Type":"ContainerStarted","Data":"9388254508e6a847486ceee7d1b518cd16718a273b985cc597967a8fe39d60e7"}
Jul 01 18:39:38.242363 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:38.242328    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:38.329144 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:38.329110    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:38.612872 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:38.612793    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:38.696401 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:38.696372    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:39.438812 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:39.438775    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs\") pod \"network-metrics-daemon-plc42\" (UID: \"5180da14-bd5e-4777-b34c-4041f623113b\") " pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:39:39.441002 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:39.440982    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5180da14-bd5e-4777-b34c-4041f623113b-metrics-certs\") pod \"network-metrics-daemon-plc42\" (UID: \"5180da14-bd5e-4777-b34c-4041f623113b\") " pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:39:39.574918 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:39.574886    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g2vz9\""
Jul 01 18:39:39.581233 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:39.581217    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plc42"
Jul 01 18:39:39.712270 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:39.712243    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-plc42"]
Jul 01 18:39:39.713153 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:39.713128    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:39.715976 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:39:39.715951    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5180da14_bd5e_4777_b34c_4041f623113b.slice/crio-9b7d86b6082a60dbc6eb6534c6a71fcd1e4d31a1824a0e502bc02fcd5a0fff3b WatchSource:0}: Error finding container 9b7d86b6082a60dbc6eb6534c6a71fcd1e4d31a1824a0e502bc02fcd5a0fff3b: Status 404 returned error can't find the container with id 9b7d86b6082a60dbc6eb6534c6a71fcd1e4d31a1824a0e502bc02fcd5a0fff3b
Jul 01 18:39:39.805597 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:39.805563    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:39.909982 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:39.909935    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:40.005849 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:40.005772    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:40.171232 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:40.171194    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-plc42" event={"ID":"5180da14-bd5e-4777-b34c-4041f623113b","Type":"ContainerStarted","Data":"9b7d86b6082a60dbc6eb6534c6a71fcd1e4d31a1824a0e502bc02fcd5a0fff3b"}
Jul 01 18:39:41.176621 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:41.176583    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-plc42" event={"ID":"5180da14-bd5e-4777-b34c-4041f623113b","Type":"ContainerStarted","Data":"4b04be9dc94f057464e25520f578f7a438723abb3f3ea3b0535e14220b0fb5ab"}
Jul 01 18:39:41.177133 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:41.176670    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-plc42" event={"ID":"5180da14-bd5e-4777-b34c-4041f623113b","Type":"ContainerStarted","Data":"2d026c57c31d93b217514216804d0d9a3898a3fd0c09718cf2743a1cb761f3a7"}
Jul 01 18:39:41.213333 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:41.213289    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-plc42" podStartSLOduration=131.319301876 podStartE2EDuration="2m12.213275514s" totalImagesPullingTime="893.973638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:41.212162177 +0000 UTC m=+131.980999781" watchObservedRunningTime="2026-07-01 18:39:41.213275514 +0000 UTC m=+131.982113117"
Jul 01 18:39:41.371161 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:41.371113    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:41.446409 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:41.446325    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:43.343255 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:43.343225    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:43.466285 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:43.466254    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:46.780145 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:46.780096    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:46.861916 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:46.861877    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:53.099357 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.099310    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-569b8dddf7-rfnp7" podUID="19f2cfb3-aa91-4695-bba9-495a83bf4d9d" containerName="console" containerID="cri-o://6ce3b736fba120d6feba53479b5cd0d2a8f4adc47536517c8a040a3da2ceef3c" gracePeriod=15
Jul 01 18:39:53.218350 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.218324    2573 generic.go:356] "Generic (PLEG): container finished" podID="19f2cfb3-aa91-4695-bba9-495a83bf4d9d" containerID="6ce3b736fba120d6feba53479b5cd0d2a8f4adc47536517c8a040a3da2ceef3c" exitCode=0
Jul 01 18:39:53.218475 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.218403    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-569b8dddf7-rfnp7" event={"ID":"19f2cfb3-aa91-4695-bba9-495a83bf4d9d","Type":"ContainerDied","Data":"6ce3b736fba120d6feba53479b5cd0d2a8f4adc47536517c8a040a3da2ceef3c"}
Jul 01 18:39:53.343665 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.343643    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:53.441455 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.441394    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-console-oauth-config\") pod \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") "
Jul 01 18:39:53.441455 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.441420    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-service-ca\") pod \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") "
Jul 01 18:39:53.441455 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.441447    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-trusted-ca-bundle\") pod \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") "
Jul 01 18:39:53.441705 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.441467    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfc4k\" (UniqueName: \"kubernetes.io/projected/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-kube-api-access-lfc4k\") pod \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") "
Jul 01 18:39:53.441705 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.441502    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-console-config\") pod \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") "
Jul 01 18:39:53.441705 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.441541    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-console-serving-cert\") pod \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") "
Jul 01 18:39:53.441705 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.441557    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-oauth-serving-cert\") pod \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\" (UID: \"19f2cfb3-aa91-4695-bba9-495a83bf4d9d\") "
Jul 01 18:39:53.441968 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.441944    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "19f2cfb3-aa91-4695-bba9-495a83bf4d9d" (UID: "19f2cfb3-aa91-4695-bba9-495a83bf4d9d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:39:53.442031 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.441968    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-console-config" (OuterVolumeSpecName: "console-config") pod "19f2cfb3-aa91-4695-bba9-495a83bf4d9d" (UID: "19f2cfb3-aa91-4695-bba9-495a83bf4d9d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:39:53.442031 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.441955    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-service-ca" (OuterVolumeSpecName: "service-ca") pod "19f2cfb3-aa91-4695-bba9-495a83bf4d9d" (UID: "19f2cfb3-aa91-4695-bba9-495a83bf4d9d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:39:53.442146 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.442042    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "19f2cfb3-aa91-4695-bba9-495a83bf4d9d" (UID: "19f2cfb3-aa91-4695-bba9-495a83bf4d9d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:39:53.443642 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.443616    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "19f2cfb3-aa91-4695-bba9-495a83bf4d9d" (UID: "19f2cfb3-aa91-4695-bba9-495a83bf4d9d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:39:53.443795 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.443768    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-kube-api-access-lfc4k" (OuterVolumeSpecName: "kube-api-access-lfc4k") pod "19f2cfb3-aa91-4695-bba9-495a83bf4d9d" (UID: "19f2cfb3-aa91-4695-bba9-495a83bf4d9d"). InnerVolumeSpecName "kube-api-access-lfc4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:39:53.443868 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.443818    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "19f2cfb3-aa91-4695-bba9-495a83bf4d9d" (UID: "19f2cfb3-aa91-4695-bba9-495a83bf4d9d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:39:53.542928 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.542905    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfc4k\" (UniqueName: \"kubernetes.io/projected/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-kube-api-access-lfc4k\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:39:53.542928 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.542927    2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-console-config\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:39:53.543058 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.542937    2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-console-serving-cert\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:39:53.543058 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.542945    2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-oauth-serving-cert\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:39:53.543058 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.542954    2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-console-oauth-config\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:39:53.543058 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.542963    2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-service-ca\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:39:53.543058 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:53.542991    2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19f2cfb3-aa91-4695-bba9-495a83bf4d9d-trusted-ca-bundle\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:39:54.223201 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:54.223170    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-569b8dddf7-rfnp7"
Jul 01 18:39:54.223639 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:54.223170    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-569b8dddf7-rfnp7" event={"ID":"19f2cfb3-aa91-4695-bba9-495a83bf4d9d","Type":"ContainerDied","Data":"213595be89ac29d793fe1346f82683b133a8836ba236912fa2a34309e4c83157"}
Jul 01 18:39:54.223639 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:54.223309    2573 scope.go:117] "RemoveContainer" containerID="6ce3b736fba120d6feba53479b5cd0d2a8f4adc47536517c8a040a3da2ceef3c"
Jul 01 18:39:54.245969 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:54.245948    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-569b8dddf7-rfnp7"]
Jul 01 18:39:54.251823 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:54.251805    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-569b8dddf7-rfnp7"]
Jul 01 18:39:55.664327 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:39:55.664285    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19f2cfb3-aa91-4695-bba9-495a83bf4d9d" path="/var/lib/kubelet/pods/19f2cfb3-aa91-4695-bba9-495a83bf4d9d/volumes"
Jul 01 18:40:29.970921 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:40:29.970884    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:30.037625 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:40:30.037593    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:32.303061 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:40:32.303023    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:33.972063 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:40:33.972028    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:34.071471 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:40:34.071439    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:34.171585 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:40:34.171559    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:34.667853 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:40:34.667808    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:34.768704 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:40:34.768666    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:41.869030 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:40:41.868999    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:41.954129 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:40:41.954088    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:44.924042 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:40:44.924000    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:45.048460 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:40:45.048410    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:45.118784 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:40:45.118750    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:58.320641 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:40:58.320607    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:58.419927 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:40:58.419898    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:58.494021 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:40:58.493964    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:41:08.018222 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:41:08.018181    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:02.263496 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:02.263416    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:02.362618 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:02.362574    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:02.662161 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:02.662071    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:02.770671 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:02.770642    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:02.868370 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:02.868338    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:03.719347 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:03.719317    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s"]
Jul 01 18:42:03.719765 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:03.719544    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19f2cfb3-aa91-4695-bba9-495a83bf4d9d" containerName="console"
Jul 01 18:42:03.719765 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:03.719556    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="19f2cfb3-aa91-4695-bba9-495a83bf4d9d" containerName="console"
Jul 01 18:42:03.719765 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:03.719608    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="19f2cfb3-aa91-4695-bba9-495a83bf4d9d" containerName="console"
Jul 01 18:42:03.722395 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:03.722373    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s"
Jul 01 18:42:03.726197 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:03.726176    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-df62h\""
Jul 01 18:42:03.726197 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:03.726192    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 01 18:42:03.727069 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:03.727047    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 01 18:42:03.735901 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:03.735877    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s"]
Jul 01 18:42:03.787374 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:03.787340    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smh4w\" (UniqueName: \"kubernetes.io/projected/913957ed-150a-447c-ae4e-74e2c3285c24-kube-api-access-smh4w\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s\" (UID: \"913957ed-150a-447c-ae4e-74e2c3285c24\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s"
Jul 01 18:42:03.787555 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:03.787391    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/913957ed-150a-447c-ae4e-74e2c3285c24-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s\" (UID: \"913957ed-150a-447c-ae4e-74e2c3285c24\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s"
Jul 01 18:42:03.787555 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:03.787418    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/913957ed-150a-447c-ae4e-74e2c3285c24-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s\" (UID: \"913957ed-150a-447c-ae4e-74e2c3285c24\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s"
Jul 01 18:42:03.888707 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:03.888677    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smh4w\" (UniqueName: \"kubernetes.io/projected/913957ed-150a-447c-ae4e-74e2c3285c24-kube-api-access-smh4w\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s\" (UID: \"913957ed-150a-447c-ae4e-74e2c3285c24\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s"
Jul 01 18:42:03.888848 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:03.888748    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/913957ed-150a-447c-ae4e-74e2c3285c24-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s\" (UID: \"913957ed-150a-447c-ae4e-74e2c3285c24\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s"
Jul 01 18:42:03.888848 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:03.888773    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/913957ed-150a-447c-ae4e-74e2c3285c24-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s\" (UID: \"913957ed-150a-447c-ae4e-74e2c3285c24\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s"
Jul 01 18:42:03.889114 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:03.889097    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/913957ed-150a-447c-ae4e-74e2c3285c24-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s\" (UID: \"913957ed-150a-447c-ae4e-74e2c3285c24\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s"
Jul 01 18:42:03.889165 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:03.889149    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/913957ed-150a-447c-ae4e-74e2c3285c24-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s\" (UID: \"913957ed-150a-447c-ae4e-74e2c3285c24\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s"
Jul 01 18:42:03.900483 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:03.900447    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smh4w\" (UniqueName: \"kubernetes.io/projected/913957ed-150a-447c-ae4e-74e2c3285c24-kube-api-access-smh4w\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s\" (UID: \"913957ed-150a-447c-ae4e-74e2c3285c24\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s"
Jul 01 18:42:04.031875 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:04.031789    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s"
Jul 01 18:42:04.172937 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:04.172905    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s"]
Jul 01 18:42:04.176033 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:42:04.175999    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod913957ed_150a_447c_ae4e_74e2c3285c24.slice/crio-94c35d3d8d10e15d8b6b718da3f789c0e41fb5de0837085bd6d340b8e19d62b4 WatchSource:0}: Error finding container 94c35d3d8d10e15d8b6b718da3f789c0e41fb5de0837085bd6d340b8e19d62b4: Status 404 returned error can't find the container with id 94c35d3d8d10e15d8b6b718da3f789c0e41fb5de0837085bd6d340b8e19d62b4
Jul 01 18:42:04.658832 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:04.658797    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s" event={"ID":"913957ed-150a-447c-ae4e-74e2c3285c24","Type":"ContainerStarted","Data":"94c35d3d8d10e15d8b6b718da3f789c0e41fb5de0837085bd6d340b8e19d62b4"}
Jul 01 18:42:13.692227 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:13.692191    2573 generic.go:356] "Generic (PLEG): container finished" podID="913957ed-150a-447c-ae4e-74e2c3285c24" containerID="6127952dcaaffd6311e46571f710f3f41c5eb1ab82edf9532a0d61e9b9a17b07" exitCode=0
Jul 01 18:42:13.692653 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:13.692294    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s" event={"ID":"913957ed-150a-447c-ae4e-74e2c3285c24","Type":"ContainerDied","Data":"6127952dcaaffd6311e46571f710f3f41c5eb1ab82edf9532a0d61e9b9a17b07"}
Jul 01 18:42:16.345143 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:16.345111    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:16.706691 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:16.706657    2573 generic.go:356] "Generic (PLEG): container finished" podID="913957ed-150a-447c-ae4e-74e2c3285c24" containerID="0bd370b0026c075553d89083836c83ca1d6afa9c23930122f8a24b22332729a2" exitCode=0
Jul 01 18:42:16.706871 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:16.706724    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s" event={"ID":"913957ed-150a-447c-ae4e-74e2c3285c24","Type":"ContainerDied","Data":"0bd370b0026c075553d89083836c83ca1d6afa9c23930122f8a24b22332729a2"}
Jul 01 18:42:25.738252 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:25.738221    2573 generic.go:356] "Generic (PLEG): container finished" podID="913957ed-150a-447c-ae4e-74e2c3285c24" containerID="171fede308c2a6354939642c3e40bb79c48f7b4d3d44cca8b2517f41e9f3673d" exitCode=0
Jul 01 18:42:25.738646 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:25.738268    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s" event={"ID":"913957ed-150a-447c-ae4e-74e2c3285c24","Type":"ContainerDied","Data":"171fede308c2a6354939642c3e40bb79c48f7b4d3d44cca8b2517f41e9f3673d"}
Jul 01 18:42:26.869057 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:26.869030    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s"
Jul 01 18:42:26.969677 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:26.969633    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smh4w\" (UniqueName: \"kubernetes.io/projected/913957ed-150a-447c-ae4e-74e2c3285c24-kube-api-access-smh4w\") pod \"913957ed-150a-447c-ae4e-74e2c3285c24\" (UID: \"913957ed-150a-447c-ae4e-74e2c3285c24\") "
Jul 01 18:42:26.969876 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:26.969762    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/913957ed-150a-447c-ae4e-74e2c3285c24-bundle\") pod \"913957ed-150a-447c-ae4e-74e2c3285c24\" (UID: \"913957ed-150a-447c-ae4e-74e2c3285c24\") "
Jul 01 18:42:26.969876 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:26.969785    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/913957ed-150a-447c-ae4e-74e2c3285c24-util\") pod \"913957ed-150a-447c-ae4e-74e2c3285c24\" (UID: \"913957ed-150a-447c-ae4e-74e2c3285c24\") "
Jul 01 18:42:26.970408 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:26.970378    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913957ed-150a-447c-ae4e-74e2c3285c24-bundle" (OuterVolumeSpecName: "bundle") pod "913957ed-150a-447c-ae4e-74e2c3285c24" (UID: "913957ed-150a-447c-ae4e-74e2c3285c24"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:42:26.971832 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:26.971804    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/913957ed-150a-447c-ae4e-74e2c3285c24-kube-api-access-smh4w" (OuterVolumeSpecName: "kube-api-access-smh4w") pod "913957ed-150a-447c-ae4e-74e2c3285c24" (UID: "913957ed-150a-447c-ae4e-74e2c3285c24"). InnerVolumeSpecName "kube-api-access-smh4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:42:26.974043 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:26.974021    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913957ed-150a-447c-ae4e-74e2c3285c24-util" (OuterVolumeSpecName: "util") pod "913957ed-150a-447c-ae4e-74e2c3285c24" (UID: "913957ed-150a-447c-ae4e-74e2c3285c24"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:42:27.070517 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:27.070420    2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/913957ed-150a-447c-ae4e-74e2c3285c24-bundle\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:42:27.070517 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:27.070458    2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/913957ed-150a-447c-ae4e-74e2c3285c24-util\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:42:27.070517 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:27.070468    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smh4w\" (UniqueName: \"kubernetes.io/projected/913957ed-150a-447c-ae4e-74e2c3285c24-kube-api-access-smh4w\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:42:27.746445 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:27.746406    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s" event={"ID":"913957ed-150a-447c-ae4e-74e2c3285c24","Type":"ContainerDied","Data":"94c35d3d8d10e15d8b6b718da3f789c0e41fb5de0837085bd6d340b8e19d62b4"}
Jul 01 18:42:27.746598 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:27.746463    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94c35d3d8d10e15d8b6b718da3f789c0e41fb5de0837085bd6d340b8e19d62b4"
Jul 01 18:42:27.746598 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:27.746420    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589gn55s"
Jul 01 18:42:29.611970 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:29.611944    2573 kubelet.go:1648] "Image garbage collection succeeded"
Jul 01 18:42:30.049824 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:30.049791    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:30.159199 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:30.159163    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:36.454336 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:36.454297    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:36.740399 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:36.740312    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:36.993786 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:36.993680    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:37.091177 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.091147    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:37.454266 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.454233    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:37.585776 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.585705    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:37.611140 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.611104    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-m2c9q"]
Jul 01 18:42:37.611360 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.611348    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="913957ed-150a-447c-ae4e-74e2c3285c24" containerName="util"
Jul 01 18:42:37.611403 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.611362    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="913957ed-150a-447c-ae4e-74e2c3285c24" containerName="util"
Jul 01 18:42:37.611403 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.611380    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="913957ed-150a-447c-ae4e-74e2c3285c24" containerName="pull"
Jul 01 18:42:37.611403 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.611385    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="913957ed-150a-447c-ae4e-74e2c3285c24" containerName="pull"
Jul 01 18:42:37.611403 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.611392    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="913957ed-150a-447c-ae4e-74e2c3285c24" containerName="extract"
Jul 01 18:42:37.611403 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.611398    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="913957ed-150a-447c-ae4e-74e2c3285c24" containerName="extract"
Jul 01 18:42:37.611546 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.611441    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="913957ed-150a-447c-ae4e-74e2c3285c24" containerName="extract"
Jul 01 18:42:37.666062 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.666033    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-m2c9q"]
Jul 01 18:42:37.666216 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.666149    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-m2c9q"
Jul 01 18:42:37.673795 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.673768    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 01 18:42:37.673931 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.673887    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 01 18:42:37.674822 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.674803    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 01 18:42:37.674924 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.674833    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 01 18:42:37.675126 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.675107    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-w4hbd\""
Jul 01 18:42:37.686360 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.686328    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:37.753345 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.753246    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln7bn\" (UniqueName: \"kubernetes.io/projected/d51de53b-30f2-4573-945b-dcccb06c3022-kube-api-access-ln7bn\") pod \"keda-admission-5fbd5c4644-m2c9q\" (UID: \"d51de53b-30f2-4573-945b-dcccb06c3022\") " pod="openshift-keda/keda-admission-5fbd5c4644-m2c9q"
Jul 01 18:42:37.753345 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.753318    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d51de53b-30f2-4573-945b-dcccb06c3022-certificates\") pod \"keda-admission-5fbd5c4644-m2c9q\" (UID: \"d51de53b-30f2-4573-945b-dcccb06c3022\") " pod="openshift-keda/keda-admission-5fbd5c4644-m2c9q"
Jul 01 18:42:37.784243 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.784211    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:37.854039 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.854001    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln7bn\" (UniqueName: \"kubernetes.io/projected/d51de53b-30f2-4573-945b-dcccb06c3022-kube-api-access-ln7bn\") pod \"keda-admission-5fbd5c4644-m2c9q\" (UID: \"d51de53b-30f2-4573-945b-dcccb06c3022\") " pod="openshift-keda/keda-admission-5fbd5c4644-m2c9q"
Jul 01 18:42:37.854254 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.854067    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d51de53b-30f2-4573-945b-dcccb06c3022-certificates\") pod \"keda-admission-5fbd5c4644-m2c9q\" (UID: \"d51de53b-30f2-4573-945b-dcccb06c3022\") " pod="openshift-keda/keda-admission-5fbd5c4644-m2c9q"
Jul 01 18:42:37.856500 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.856478    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d51de53b-30f2-4573-945b-dcccb06c3022-certificates\") pod \"keda-admission-5fbd5c4644-m2c9q\" (UID: \"d51de53b-30f2-4573-945b-dcccb06c3022\") " pod="openshift-keda/keda-admission-5fbd5c4644-m2c9q"
Jul 01 18:42:37.863613 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.863583    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln7bn\" (UniqueName: \"kubernetes.io/projected/d51de53b-30f2-4573-945b-dcccb06c3022-kube-api-access-ln7bn\") pod \"keda-admission-5fbd5c4644-m2c9q\" (UID: \"d51de53b-30f2-4573-945b-dcccb06c3022\") " pod="openshift-keda/keda-admission-5fbd5c4644-m2c9q"
Jul 01 18:42:37.977322 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.977292    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-m2c9q"
Jul 01 18:42:37.988433 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:37.988405    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:38.133851 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:38.133824    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-m2c9q"]
Jul 01 18:42:38.136565 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:42:38.136534    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd51de53b_30f2_4573_945b_dcccb06c3022.slice/crio-236eb76953d2aa1e97507985e0ef5f2c4d8f682e9656af3814dc565ee356a10c WatchSource:0}: Error finding container 236eb76953d2aa1e97507985e0ef5f2c4d8f682e9656af3814dc565ee356a10c: Status 404 returned error can't find the container with id 236eb76953d2aa1e97507985e0ef5f2c4d8f682e9656af3814dc565ee356a10c
Jul 01 18:42:38.137903 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:38.137882    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 18:42:38.185456 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:38.185418    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:38.389588 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:38.389499    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:38.490217 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:38.490185    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:38.785619 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:38.785585    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-m2c9q" event={"ID":"d51de53b-30f2-4573-945b-dcccb06c3022","Type":"ContainerStarted","Data":"236eb76953d2aa1e97507985e0ef5f2c4d8f682e9656af3814dc565ee356a10c"}
Jul 01 18:42:40.793961 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:40.793932    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-m2c9q" event={"ID":"d51de53b-30f2-4573-945b-dcccb06c3022","Type":"ContainerStarted","Data":"2375284ae68d8f798523205bfd8a9d36570d9bc628d285e73ec53b22f3e4c797"}
Jul 01 18:42:40.794321 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:40.793992    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-m2c9q"
Jul 01 18:42:40.814783 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:42:40.814731    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-m2c9q" podStartSLOduration=1.5712595070000002 podStartE2EDuration="3.814697519s" totalImagesPullingTime="2.243438012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:42:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:42:40.813442406 +0000 UTC m=+311.582280011" watchObservedRunningTime="2026-07-01 18:42:40.814697519 +0000 UTC m=+311.583535122"
Jul 01 18:43:01.800756 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:01.800705    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-m2c9q"
Jul 01 18:43:37.938547 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:37.938510    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:43:46.723210 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:46.723164    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:43:46.827433 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:46.827371    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:43:46.867209 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:46.867174    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-64868fc46-58wv8"]
Jul 01 18:43:46.869778 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:46.869761    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-64868fc46-58wv8"
Jul 01 18:43:46.873886 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:46.873851    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 01 18:43:46.874032 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:46.873917    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-zm7px\""
Jul 01 18:43:46.874032 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:46.873923    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 18:43:46.874032 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:46.874017    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 18:43:46.885863 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:46.885839    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-64868fc46-58wv8"]
Jul 01 18:43:46.912759 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:46.912701    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rh5d\" (UniqueName: \"kubernetes.io/projected/01934f3b-0da3-4179-864a-1fb749288abd-kube-api-access-4rh5d\") pod \"kserve-controller-manager-64868fc46-58wv8\" (UID: \"01934f3b-0da3-4179-864a-1fb749288abd\") " pod="kserve/kserve-controller-manager-64868fc46-58wv8"
Jul 01 18:43:46.912941 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:46.912867    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01934f3b-0da3-4179-864a-1fb749288abd-cert\") pod \"kserve-controller-manager-64868fc46-58wv8\" (UID: \"01934f3b-0da3-4179-864a-1fb749288abd\") " pod="kserve/kserve-controller-manager-64868fc46-58wv8"
Jul 01 18:43:46.946173 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:46.946136    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:43:46.946406 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:46.946336    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-ggzgd"]
Jul 01 18:43:46.948456 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:46.948436    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-ggzgd"
Jul 01 18:43:46.951620 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:46.951602    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 01 18:43:46.951733 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:46.951653    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-m2nzb\""
Jul 01 18:43:46.978981 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:46.978884    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-ggzgd"]
Jul 01 18:43:47.013544 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:47.013510    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01934f3b-0da3-4179-864a-1fb749288abd-cert\") pod \"kserve-controller-manager-64868fc46-58wv8\" (UID: \"01934f3b-0da3-4179-864a-1fb749288abd\") " pod="kserve/kserve-controller-manager-64868fc46-58wv8"
Jul 01 18:43:47.013761 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:47.013560    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rh5d\" (UniqueName: \"kubernetes.io/projected/01934f3b-0da3-4179-864a-1fb749288abd-kube-api-access-4rh5d\") pod \"kserve-controller-manager-64868fc46-58wv8\" (UID: \"01934f3b-0da3-4179-864a-1fb749288abd\") " pod="kserve/kserve-controller-manager-64868fc46-58wv8"
Jul 01 18:43:47.013761 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:47.013587    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/39f0f120-82c8-4db1-bac3-1753b7b308fa-data\") pod \"seaweedfs-64568bcd49-ggzgd\" (UID: \"39f0f120-82c8-4db1-bac3-1753b7b308fa\") " pod="kserve/seaweedfs-64568bcd49-ggzgd"
Jul 01 18:43:47.013761 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:47.013637    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsqnc\" (UniqueName: \"kubernetes.io/projected/39f0f120-82c8-4db1-bac3-1753b7b308fa-kube-api-access-fsqnc\") pod \"seaweedfs-64568bcd49-ggzgd\" (UID: \"39f0f120-82c8-4db1-bac3-1753b7b308fa\") " pod="kserve/seaweedfs-64568bcd49-ggzgd"
Jul 01 18:43:47.016192 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:47.016171    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01934f3b-0da3-4179-864a-1fb749288abd-cert\") pod \"kserve-controller-manager-64868fc46-58wv8\" (UID: \"01934f3b-0da3-4179-864a-1fb749288abd\") " pod="kserve/kserve-controller-manager-64868fc46-58wv8"
Jul 01 18:43:47.029612 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:47.029585    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rh5d\" (UniqueName: \"kubernetes.io/projected/01934f3b-0da3-4179-864a-1fb749288abd-kube-api-access-4rh5d\") pod \"kserve-controller-manager-64868fc46-58wv8\" (UID: \"01934f3b-0da3-4179-864a-1fb749288abd\") " pod="kserve/kserve-controller-manager-64868fc46-58wv8"
Jul 01 18:43:47.045399 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:47.045358    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:43:47.114771 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:47.114737    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsqnc\" (UniqueName: \"kubernetes.io/projected/39f0f120-82c8-4db1-bac3-1753b7b308fa-kube-api-access-fsqnc\") pod \"seaweedfs-64568bcd49-ggzgd\" (UID: \"39f0f120-82c8-4db1-bac3-1753b7b308fa\") " pod="kserve/seaweedfs-64568bcd49-ggzgd"
Jul 01 18:43:47.114924 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:47.114818    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/39f0f120-82c8-4db1-bac3-1753b7b308fa-data\") pod \"seaweedfs-64568bcd49-ggzgd\" (UID: \"39f0f120-82c8-4db1-bac3-1753b7b308fa\") " pod="kserve/seaweedfs-64568bcd49-ggzgd"
Jul 01 18:43:47.115162 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:47.115146    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/39f0f120-82c8-4db1-bac3-1753b7b308fa-data\") pod \"seaweedfs-64568bcd49-ggzgd\" (UID: \"39f0f120-82c8-4db1-bac3-1753b7b308fa\") " pod="kserve/seaweedfs-64568bcd49-ggzgd"
Jul 01 18:43:47.127048 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:47.127015    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsqnc\" (UniqueName: \"kubernetes.io/projected/39f0f120-82c8-4db1-bac3-1753b7b308fa-kube-api-access-fsqnc\") pod \"seaweedfs-64568bcd49-ggzgd\" (UID: \"39f0f120-82c8-4db1-bac3-1753b7b308fa\") " pod="kserve/seaweedfs-64568bcd49-ggzgd"
Jul 01 18:43:47.179389 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:47.179359    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-64868fc46-58wv8"
Jul 01 18:43:47.197362 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:47.197329    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:43:47.257374 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:47.257169    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-ggzgd"
Jul 01 18:43:47.323814 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:47.323791    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-64868fc46-58wv8"]
Jul 01 18:43:47.326176 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:43:47.326148    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01934f3b_0da3_4179_864a_1fb749288abd.slice/crio-9d1735b564b9584e0f1cc4997576a240761bfa9ce6cf9d37094d0d4177b4504d WatchSource:0}: Error finding container 9d1735b564b9584e0f1cc4997576a240761bfa9ce6cf9d37094d0d4177b4504d: Status 404 returned error can't find the container with id 9d1735b564b9584e0f1cc4997576a240761bfa9ce6cf9d37094d0d4177b4504d
Jul 01 18:43:47.397805 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:47.397776    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-ggzgd"]
Jul 01 18:43:47.400583 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:43:47.400556    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f0f120_82c8_4db1_bac3_1753b7b308fa.slice/crio-3308ab30d9d503e1420c973174cc99190ca58f024ac8491c9dbe8d809df627a5 WatchSource:0}: Error finding container 3308ab30d9d503e1420c973174cc99190ca58f024ac8491c9dbe8d809df627a5: Status 404 returned error can't find the container with id 3308ab30d9d503e1420c973174cc99190ca58f024ac8491c9dbe8d809df627a5
Jul 01 18:43:47.595497 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:47.595370    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:43:47.692525 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:47.692495    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:43:48.024596 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:48.024515    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-ggzgd" event={"ID":"39f0f120-82c8-4db1-bac3-1753b7b308fa","Type":"ContainerStarted","Data":"3308ab30d9d503e1420c973174cc99190ca58f024ac8491c9dbe8d809df627a5"}
Jul 01 18:43:48.026827 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:48.026775    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-64868fc46-58wv8" event={"ID":"01934f3b-0da3-4179-864a-1fb749288abd","Type":"ContainerStarted","Data":"9d1735b564b9584e0f1cc4997576a240761bfa9ce6cf9d37094d0d4177b4504d"}
Jul 01 18:43:51.040492 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:51.040459    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-64868fc46-58wv8" event={"ID":"01934f3b-0da3-4179-864a-1fb749288abd","Type":"ContainerStarted","Data":"2f3d109f8df786c58739e57c933932611f6aa546035019bf37bf502dea717226"}
Jul 01 18:43:51.040893 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:51.040655    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-64868fc46-58wv8"
Jul 01 18:43:51.067126 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:51.067075    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-64868fc46-58wv8" podStartSLOduration=2.251477563 podStartE2EDuration="5.067059087s" totalImagesPullingTime="2.815581524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:43:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:43:51.065786611 +0000 UTC m=+381.834624214" watchObservedRunningTime="2026-07-01 18:43:51.067059087 +0000 UTC m=+381.835896690"
Jul 01 18:43:52.046982 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:52.046935    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-ggzgd" event={"ID":"39f0f120-82c8-4db1-bac3-1753b7b308fa","Type":"ContainerStarted","Data":"ce687101c6b7ee51fa151b24a6f9aafb917f7ca56af5b29a54d318540ef2dd0b"}
Jul 01 18:43:52.047392 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:52.047126    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-ggzgd"
Jul 01 18:43:52.072427 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:52.072378    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-ggzgd" podStartSLOduration=2.141519249 podStartE2EDuration="6.072363003s" totalImagesPullingTime="3.930843754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:43:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:43:52.071127795 +0000 UTC m=+382.839965398" watchObservedRunningTime="2026-07-01 18:43:52.072363003 +0000 UTC m=+382.841200606"
Jul 01 18:43:58.053884 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:43:58.053854    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-ggzgd"
Jul 01 18:44:22.052484 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:22.052453    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-64868fc46-58wv8"
Jul 01 18:44:23.186276 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:23.186244    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-64868fc46-58wv8"]
Jul 01 18:44:23.186708 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:23.186412    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-64868fc46-58wv8" podUID="01934f3b-0da3-4179-864a-1fb749288abd" containerName="manager" containerID="cri-o://2f3d109f8df786c58739e57c933932611f6aa546035019bf37bf502dea717226" gracePeriod=10
Jul 01 18:44:23.270405 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:23.270378    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-64868fc46-ttqhf"]
Jul 01 18:44:23.272596 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:23.272581    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-64868fc46-ttqhf"
Jul 01 18:44:23.290235 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:23.290212    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-64868fc46-ttqhf"]
Jul 01 18:44:23.384196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:23.384156    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2439698b-e24f-4d30-a494-8c44fce48f65-cert\") pod \"kserve-controller-manager-64868fc46-ttqhf\" (UID: \"2439698b-e24f-4d30-a494-8c44fce48f65\") " pod="kserve/kserve-controller-manager-64868fc46-ttqhf"
Jul 01 18:44:23.384353 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:23.384235    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvz8c\" (UniqueName: \"kubernetes.io/projected/2439698b-e24f-4d30-a494-8c44fce48f65-kube-api-access-zvz8c\") pod \"kserve-controller-manager-64868fc46-ttqhf\" (UID: \"2439698b-e24f-4d30-a494-8c44fce48f65\") " pod="kserve/kserve-controller-manager-64868fc46-ttqhf"
Jul 01 18:44:23.426359 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:23.426338    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-64868fc46-58wv8"
Jul 01 18:44:23.484631 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:23.484600    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01934f3b-0da3-4179-864a-1fb749288abd-cert\") pod \"01934f3b-0da3-4179-864a-1fb749288abd\" (UID: \"01934f3b-0da3-4179-864a-1fb749288abd\") "
Jul 01 18:44:23.484834 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:23.484660    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rh5d\" (UniqueName: \"kubernetes.io/projected/01934f3b-0da3-4179-864a-1fb749288abd-kube-api-access-4rh5d\") pod \"01934f3b-0da3-4179-864a-1fb749288abd\" (UID: \"01934f3b-0da3-4179-864a-1fb749288abd\") "
Jul 01 18:44:23.484834 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:23.484748    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvz8c\" (UniqueName: \"kubernetes.io/projected/2439698b-e24f-4d30-a494-8c44fce48f65-kube-api-access-zvz8c\") pod \"kserve-controller-manager-64868fc46-ttqhf\" (UID: \"2439698b-e24f-4d30-a494-8c44fce48f65\") " pod="kserve/kserve-controller-manager-64868fc46-ttqhf"
Jul 01 18:44:23.484834 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:23.484788    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2439698b-e24f-4d30-a494-8c44fce48f65-cert\") pod \"kserve-controller-manager-64868fc46-ttqhf\" (UID: \"2439698b-e24f-4d30-a494-8c44fce48f65\") " pod="kserve/kserve-controller-manager-64868fc46-ttqhf"
Jul 01 18:44:23.486970 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:23.486937    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01934f3b-0da3-4179-864a-1fb749288abd-cert" (OuterVolumeSpecName: "cert") pod "01934f3b-0da3-4179-864a-1fb749288abd" (UID: "01934f3b-0da3-4179-864a-1fb749288abd"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:44:23.486970 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:23.486963    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01934f3b-0da3-4179-864a-1fb749288abd-kube-api-access-4rh5d" (OuterVolumeSpecName: "kube-api-access-4rh5d") pod "01934f3b-0da3-4179-864a-1fb749288abd" (UID: "01934f3b-0da3-4179-864a-1fb749288abd"). InnerVolumeSpecName "kube-api-access-4rh5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:44:23.487286 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:23.487267    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2439698b-e24f-4d30-a494-8c44fce48f65-cert\") pod \"kserve-controller-manager-64868fc46-ttqhf\" (UID: \"2439698b-e24f-4d30-a494-8c44fce48f65\") " pod="kserve/kserve-controller-manager-64868fc46-ttqhf"
Jul 01 18:44:23.500284 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:23.500249    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvz8c\" (UniqueName: \"kubernetes.io/projected/2439698b-e24f-4d30-a494-8c44fce48f65-kube-api-access-zvz8c\") pod \"kserve-controller-manager-64868fc46-ttqhf\" (UID: \"2439698b-e24f-4d30-a494-8c44fce48f65\") " pod="kserve/kserve-controller-manager-64868fc46-ttqhf"
Jul 01 18:44:23.581860 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:23.581819    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-64868fc46-ttqhf"
Jul 01 18:44:23.585975 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:23.585957    2573 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01934f3b-0da3-4179-864a-1fb749288abd-cert\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:44:23.586022 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:23.585980    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rh5d\" (UniqueName: \"kubernetes.io/projected/01934f3b-0da3-4179-864a-1fb749288abd-kube-api-access-4rh5d\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:44:23.715432 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:23.715407    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-64868fc46-ttqhf"]
Jul 01 18:44:23.717664 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:44:23.717641    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2439698b_e24f_4d30_a494_8c44fce48f65.slice/crio-aefc0434770afee3b5f814145da2f73c12347e7f6e2397c40d9cec664f652b72 WatchSource:0}: Error finding container aefc0434770afee3b5f814145da2f73c12347e7f6e2397c40d9cec664f652b72: Status 404 returned error can't find the container with id aefc0434770afee3b5f814145da2f73c12347e7f6e2397c40d9cec664f652b72
Jul 01 18:44:24.158243 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:24.158205    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-64868fc46-ttqhf" event={"ID":"2439698b-e24f-4d30-a494-8c44fce48f65","Type":"ContainerStarted","Data":"aefc0434770afee3b5f814145da2f73c12347e7f6e2397c40d9cec664f652b72"}
Jul 01 18:44:24.159310 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:24.159287    2573 generic.go:356] "Generic (PLEG): container finished" podID="01934f3b-0da3-4179-864a-1fb749288abd" containerID="2f3d109f8df786c58739e57c933932611f6aa546035019bf37bf502dea717226" exitCode=0
Jul 01 18:44:24.159430 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:24.159316    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-64868fc46-58wv8" event={"ID":"01934f3b-0da3-4179-864a-1fb749288abd","Type":"ContainerDied","Data":"2f3d109f8df786c58739e57c933932611f6aa546035019bf37bf502dea717226"}
Jul 01 18:44:24.159430 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:24.159343    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-64868fc46-58wv8" event={"ID":"01934f3b-0da3-4179-864a-1fb749288abd","Type":"ContainerDied","Data":"9d1735b564b9584e0f1cc4997576a240761bfa9ce6cf9d37094d0d4177b4504d"}
Jul 01 18:44:24.159430 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:24.159345    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-64868fc46-58wv8"
Jul 01 18:44:24.159430 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:24.159362    2573 scope.go:117] "RemoveContainer" containerID="2f3d109f8df786c58739e57c933932611f6aa546035019bf37bf502dea717226"
Jul 01 18:44:24.168539 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:24.168519    2573 scope.go:117] "RemoveContainer" containerID="2f3d109f8df786c58739e57c933932611f6aa546035019bf37bf502dea717226"
Jul 01 18:44:24.168854 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:44:24.168825    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f3d109f8df786c58739e57c933932611f6aa546035019bf37bf502dea717226\": container with ID starting with 2f3d109f8df786c58739e57c933932611f6aa546035019bf37bf502dea717226 not found: ID does not exist" containerID="2f3d109f8df786c58739e57c933932611f6aa546035019bf37bf502dea717226"
Jul 01 18:44:24.168897 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:24.168863    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f3d109f8df786c58739e57c933932611f6aa546035019bf37bf502dea717226"} err="failed to get container status \"2f3d109f8df786c58739e57c933932611f6aa546035019bf37bf502dea717226\": rpc error: code = NotFound desc = could not find container \"2f3d109f8df786c58739e57c933932611f6aa546035019bf37bf502dea717226\": container with ID starting with 2f3d109f8df786c58739e57c933932611f6aa546035019bf37bf502dea717226 not found: ID does not exist"
Jul 01 18:44:24.186188 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:24.186166    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-64868fc46-58wv8"]
Jul 01 18:44:24.188397 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:24.188378    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-64868fc46-58wv8"]
Jul 01 18:44:25.164343 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:25.164307    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-64868fc46-ttqhf" event={"ID":"2439698b-e24f-4d30-a494-8c44fce48f65","Type":"ContainerStarted","Data":"17b63057c9c9ab2b0d4045f0bda600b1143dd563037865ff73dceed7f2be012d"}
Jul 01 18:44:25.164539 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:25.164398    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-64868fc46-ttqhf"
Jul 01 18:44:25.191966 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:25.191922    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-64868fc46-ttqhf" podStartSLOduration=1.7425658899999998 podStartE2EDuration="2.191907344s" totalImagesPullingTime="449.341454ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:44:25.190905689 +0000 UTC m=+415.959743292" watchObservedRunningTime="2026-07-01 18:44:25.191907344 +0000 UTC m=+415.960744947"
Jul 01 18:44:25.665097 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:25.665055    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01934f3b-0da3-4179-864a-1fb749288abd" path="/var/lib/kubelet/pods/01934f3b-0da3-4179-864a-1fb749288abd/volumes"
Jul 01 18:44:31.763849 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:31.763809    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:44:32.071246 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:32.071170    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:44:56.174322 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:56.174250    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-64868fc46-ttqhf"
Jul 01 18:44:57.188072 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:57.188015    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:44:57.247862 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:57.247827    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-dskcv"]
Jul 01 18:44:57.248094 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:57.248082    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01934f3b-0da3-4179-864a-1fb749288abd" containerName="manager"
Jul 01 18:44:57.248094 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:57.248095    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="01934f3b-0da3-4179-864a-1fb749288abd" containerName="manager"
Jul 01 18:44:57.248188 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:57.248142    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="01934f3b-0da3-4179-864a-1fb749288abd" containerName="manager"
Jul 01 18:44:57.251066 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:57.251047    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-dskcv"
Jul 01 18:44:57.253730 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:57.253705    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 01 18:44:57.253955 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:57.253937    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-wghn2\""
Jul 01 18:44:57.266155 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:57.266129    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-dskcv"]
Jul 01 18:44:57.281018 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:57.280986    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:44:57.343898 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:57.343854    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2937658c-a602-46c3-964f-44a30da26c28-tls-certs\") pod \"model-serving-api-fd65d7d6-dskcv\" (UID: \"2937658c-a602-46c3-964f-44a30da26c28\") " pod="kserve/model-serving-api-fd65d7d6-dskcv"
Jul 01 18:44:57.344110 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:57.343920    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2qc8\" (UniqueName: \"kubernetes.io/projected/2937658c-a602-46c3-964f-44a30da26c28-kube-api-access-n2qc8\") pod \"model-serving-api-fd65d7d6-dskcv\" (UID: \"2937658c-a602-46c3-964f-44a30da26c28\") " pod="kserve/model-serving-api-fd65d7d6-dskcv"
Jul 01 18:44:57.444988 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:57.444894    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2937658c-a602-46c3-964f-44a30da26c28-tls-certs\") pod \"model-serving-api-fd65d7d6-dskcv\" (UID: \"2937658c-a602-46c3-964f-44a30da26c28\") " pod="kserve/model-serving-api-fd65d7d6-dskcv"
Jul 01 18:44:57.444988 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:57.444947    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2qc8\" (UniqueName: \"kubernetes.io/projected/2937658c-a602-46c3-964f-44a30da26c28-kube-api-access-n2qc8\") pod \"model-serving-api-fd65d7d6-dskcv\" (UID: \"2937658c-a602-46c3-964f-44a30da26c28\") " pod="kserve/model-serving-api-fd65d7d6-dskcv"
Jul 01 18:44:57.447432 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:57.447408    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2937658c-a602-46c3-964f-44a30da26c28-tls-certs\") pod \"model-serving-api-fd65d7d6-dskcv\" (UID: \"2937658c-a602-46c3-964f-44a30da26c28\") " pod="kserve/model-serving-api-fd65d7d6-dskcv"
Jul 01 18:44:57.460901 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:57.460866    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:44:57.461182 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:57.461162    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2qc8\" (UniqueName: \"kubernetes.io/projected/2937658c-a602-46c3-964f-44a30da26c28-kube-api-access-n2qc8\") pod \"model-serving-api-fd65d7d6-dskcv\" (UID: \"2937658c-a602-46c3-964f-44a30da26c28\") " pod="kserve/model-serving-api-fd65d7d6-dskcv"
Jul 01 18:44:57.558881 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:57.558840    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:44:57.561734 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:57.561698    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-dskcv"
Jul 01 18:44:57.687607 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:57.687585    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-dskcv"]
Jul 01 18:44:57.690139 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:44:57.690115    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2937658c_a602_46c3_964f_44a30da26c28.slice/crio-e23213f2cf32528b843f3c53578780be46201eb4921868fff6495ec66ab3fedc WatchSource:0}: Error finding container e23213f2cf32528b843f3c53578780be46201eb4921868fff6495ec66ab3fedc: Status 404 returned error can't find the container with id e23213f2cf32528b843f3c53578780be46201eb4921868fff6495ec66ab3fedc
Jul 01 18:44:58.181792 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:58.181755    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:44:58.263743 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:58.263689    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:44:58.277729 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:44:58.277682    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-dskcv" event={"ID":"2937658c-a602-46c3-964f-44a30da26c28","Type":"ContainerStarted","Data":"e23213f2cf32528b843f3c53578780be46201eb4921868fff6495ec66ab3fedc"}
Jul 01 18:45:00.289087 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:45:00.289046    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-dskcv" event={"ID":"2937658c-a602-46c3-964f-44a30da26c28","Type":"ContainerStarted","Data":"00bb5eb67d885fb64a22793b9ccee6a9f11464eaac9ebfa72233cfc7603eceb3"}
Jul 01 18:45:00.289431 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:45:00.289159    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-dskcv"
Jul 01 18:45:00.315511 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:45:00.315468    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-dskcv" podStartSLOduration=0.873027215 podStartE2EDuration="3.315455382s" totalImagesPullingTime="2.442428167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:45:00.314448965 +0000 UTC m=+451.083286569" watchObservedRunningTime="2026-07-01 18:45:00.315455382 +0000 UTC m=+451.084292984"
Jul 01 18:45:11.296890 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:45:11.296859    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-dskcv"
Jul 01 18:45:38.553966 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:45:38.553930    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:45:56.502490 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:45:56.502434    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:45:57.676819 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:45:57.676785    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:46:32.682124 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:32.682085    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:46:33.206618 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:33.206584    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:46:44.434182 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:44.434146    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:46:44.730893 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:44.730852    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:46:44.756708 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:44.756669    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4"]
Jul 01 18:46:44.760098 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:44.760074    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4"
Jul 01 18:46:44.767262 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:44.767237    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f2cdf-predictor-serving-cert\""
Jul 01 18:46:44.767394 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:44.767354    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 18:46:44.769517 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:44.769500    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qplkr\""
Jul 01 18:46:44.769613 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:44.769591    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 18:46:44.770110 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:44.770092    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f2cdf-kube-rbac-proxy-sar-config\""
Jul 01 18:46:44.776963 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:44.776937    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4"]
Jul 01 18:46:44.836581 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:44.836529    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a903f2e3-d219-46b2-ad31-5a1d31733461-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4\" (UID: \"a903f2e3-d219-46b2-ad31-5a1d31733461\") " pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4"
Jul 01 18:46:44.836798 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:44.836621    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a903f2e3-d219-46b2-ad31-5a1d31733461-proxy-tls\") pod \"success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4\" (UID: \"a903f2e3-d219-46b2-ad31-5a1d31733461\") " pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4"
Jul 01 18:46:44.836798 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:44.836664    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfg8v\" (UniqueName: \"kubernetes.io/projected/a903f2e3-d219-46b2-ad31-5a1d31733461-kube-api-access-gfg8v\") pod \"success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4\" (UID: \"a903f2e3-d219-46b2-ad31-5a1d31733461\") " pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4"
Jul 01 18:46:44.887703 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:44.887667    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:46:44.937431 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:44.937397    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a903f2e3-d219-46b2-ad31-5a1d31733461-proxy-tls\") pod \"success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4\" (UID: \"a903f2e3-d219-46b2-ad31-5a1d31733461\") " pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4"
Jul 01 18:46:44.937605 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:44.937458    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfg8v\" (UniqueName: \"kubernetes.io/projected/a903f2e3-d219-46b2-ad31-5a1d31733461-kube-api-access-gfg8v\") pod \"success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4\" (UID: \"a903f2e3-d219-46b2-ad31-5a1d31733461\") " pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4"
Jul 01 18:46:44.937605 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:44.937518    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a903f2e3-d219-46b2-ad31-5a1d31733461-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4\" (UID: \"a903f2e3-d219-46b2-ad31-5a1d31733461\") " pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4"
Jul 01 18:46:44.938377 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:44.938354    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a903f2e3-d219-46b2-ad31-5a1d31733461-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4\" (UID: \"a903f2e3-d219-46b2-ad31-5a1d31733461\") " pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4"
Jul 01 18:46:44.939861 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:44.939840    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a903f2e3-d219-46b2-ad31-5a1d31733461-proxy-tls\") pod \"success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4\" (UID: \"a903f2e3-d219-46b2-ad31-5a1d31733461\") " pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4"
Jul 01 18:46:44.958760 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:44.958734    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfg8v\" (UniqueName: \"kubernetes.io/projected/a903f2e3-d219-46b2-ad31-5a1d31733461-kube-api-access-gfg8v\") pod \"success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4\" (UID: \"a903f2e3-d219-46b2-ad31-5a1d31733461\") " pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4"
Jul 01 18:46:44.982211 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:44.982134    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:46:45.069934 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:45.069902    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4"
Jul 01 18:46:45.091617 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:45.091588    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:46:45.125623 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:45.125588    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd"]
Jul 01 18:46:45.129802 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:45.129774    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd"
Jul 01 18:46:45.132328 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:45.132303    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f2cdf-predictor-serving-cert\""
Jul 01 18:46:45.132506 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:45.132303    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f2cdf-kube-rbac-proxy-sar-config\""
Jul 01 18:46:45.141254 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:45.139808    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdl8h\" (UniqueName: \"kubernetes.io/projected/1932b4d7-e389-4605-bda4-4728ecf22790-kube-api-access-hdl8h\") pod \"error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd\" (UID: \"1932b4d7-e389-4605-bda4-4728ecf22790\") " pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd"
Jul 01 18:46:45.141254 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:45.139892    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1932b4d7-e389-4605-bda4-4728ecf22790-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd\" (UID: \"1932b4d7-e389-4605-bda4-4728ecf22790\") " pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd"
Jul 01 18:46:45.141254 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:45.139940    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1932b4d7-e389-4605-bda4-4728ecf22790-proxy-tls\") pod \"error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd\" (UID: \"1932b4d7-e389-4605-bda4-4728ecf22790\") " pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd"
Jul 01 18:46:45.141254 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:45.140593    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd"]
Jul 01 18:46:45.188964 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:45.188927    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:46:45.220659 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:45.220632    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4"]
Jul 01 18:46:45.222470 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:46:45.222441    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda903f2e3_d219_46b2_ad31_5a1d31733461.slice/crio-4de0ca029f00be4f9c8ecbbd5c61d848b14b9e577c739fbdc4d4a1037c48a396 WatchSource:0}: Error finding container 4de0ca029f00be4f9c8ecbbd5c61d848b14b9e577c739fbdc4d4a1037c48a396: Status 404 returned error can't find the container with id 4de0ca029f00be4f9c8ecbbd5c61d848b14b9e577c739fbdc4d4a1037c48a396
Jul 01 18:46:45.240628 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:45.240565    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1932b4d7-e389-4605-bda4-4728ecf22790-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd\" (UID: \"1932b4d7-e389-4605-bda4-4728ecf22790\") " pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd"
Jul 01 18:46:45.240628 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:45.240605    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1932b4d7-e389-4605-bda4-4728ecf22790-proxy-tls\") pod \"error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd\" (UID: \"1932b4d7-e389-4605-bda4-4728ecf22790\") " pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd"
Jul 01 18:46:45.240821 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:45.240630    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdl8h\" (UniqueName: \"kubernetes.io/projected/1932b4d7-e389-4605-bda4-4728ecf22790-kube-api-access-hdl8h\") pod \"error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd\" (UID: \"1932b4d7-e389-4605-bda4-4728ecf22790\") " pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd"
Jul 01 18:46:45.240821 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:46:45.240749    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-serving-cert: secret "error-404-isvc-f2cdf-predictor-serving-cert" not found
Jul 01 18:46:45.240821 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:46:45.240815    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1932b4d7-e389-4605-bda4-4728ecf22790-proxy-tls podName:1932b4d7-e389-4605-bda4-4728ecf22790 nodeName:}" failed. No retries permitted until 2026-07-01 18:46:45.74079369 +0000 UTC m=+556.509631286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1932b4d7-e389-4605-bda4-4728ecf22790-proxy-tls") pod "error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd" (UID: "1932b4d7-e389-4605-bda4-4728ecf22790") : secret "error-404-isvc-f2cdf-predictor-serving-cert" not found
Jul 01 18:46:45.241263 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:45.241240    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1932b4d7-e389-4605-bda4-4728ecf22790-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd\" (UID: \"1932b4d7-e389-4605-bda4-4728ecf22790\") " pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd"
Jul 01 18:46:45.261442 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:45.261419    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdl8h\" (UniqueName: \"kubernetes.io/projected/1932b4d7-e389-4605-bda4-4728ecf22790-kube-api-access-hdl8h\") pod \"error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd\" (UID: \"1932b4d7-e389-4605-bda4-4728ecf22790\") " pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd"
Jul 01 18:46:45.296233 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:45.296198    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:46:45.489559 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:45.489527    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:46:45.602090 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:45.601996    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:46:45.655249 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:45.655217    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4" event={"ID":"a903f2e3-d219-46b2-ad31-5a1d31733461","Type":"ContainerStarted","Data":"4de0ca029f00be4f9c8ecbbd5c61d848b14b9e577c739fbdc4d4a1037c48a396"}
Jul 01 18:46:45.744687 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:45.744649    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1932b4d7-e389-4605-bda4-4728ecf22790-proxy-tls\") pod \"error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd\" (UID: \"1932b4d7-e389-4605-bda4-4728ecf22790\") " pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd"
Jul 01 18:46:45.747163 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:45.747137    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1932b4d7-e389-4605-bda4-4728ecf22790-proxy-tls\") pod \"error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd\" (UID: \"1932b4d7-e389-4605-bda4-4728ecf22790\") " pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd"
Jul 01 18:46:46.046091 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:46.046052    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd"
Jul 01 18:46:46.242164 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:46.242134    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd"]
Jul 01 18:46:46.246239 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:46:46.246205    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1932b4d7_e389_4605_bda4_4728ecf22790.slice/crio-a78ba2815cd131484bbb884f36d35174719d9be3eb918477743141c12718f7bb WatchSource:0}: Error finding container a78ba2815cd131484bbb884f36d35174719d9be3eb918477743141c12718f7bb: Status 404 returned error can't find the container with id a78ba2815cd131484bbb884f36d35174719d9be3eb918477743141c12718f7bb
Jul 01 18:46:46.661873 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:46:46.661834    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd" event={"ID":"1932b4d7-e389-4605-bda4-4728ecf22790","Type":"ContainerStarted","Data":"a78ba2815cd131484bbb884f36d35174719d9be3eb918477743141c12718f7bb"}
Jul 01 18:47:02.751318 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:02.751278    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4" event={"ID":"a903f2e3-d219-46b2-ad31-5a1d31733461","Type":"ContainerStarted","Data":"5ec4213465dbbb0fb3f56bb927ab5b57c17cb4320e28643fe263ce6aa4160f9c"}
Jul 01 18:47:02.753204 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:02.753178    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd" event={"ID":"1932b4d7-e389-4605-bda4-4728ecf22790","Type":"ContainerStarted","Data":"64440a49b5d6477632b8e6b13cf4df0f7acaef6d3b2c01360365999b6500845c"}
Jul 01 18:47:04.764753 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:04.764701    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4" event={"ID":"a903f2e3-d219-46b2-ad31-5a1d31733461","Type":"ContainerStarted","Data":"410cb927c621062fcbb4a403e4f1f39959e931750d2a25a04e5ef7d06e2b161a"}
Jul 01 18:47:04.765180 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:04.764812    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4"
Jul 01 18:47:04.765180 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:04.764942    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4"
Jul 01 18:47:04.766226 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:04.766194    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4" podUID="a903f2e3-d219-46b2-ad31-5a1d31733461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 01 18:47:04.766592 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:04.766570    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd" event={"ID":"1932b4d7-e389-4605-bda4-4728ecf22790","Type":"ContainerStarted","Data":"2c76096a497d7a1616160dd265ab270cc95ec1b8892f373d770e7f8475122a69"}
Jul 01 18:47:04.766746 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:04.766731    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd"
Jul 01 18:47:04.799483 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:04.799428    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4" podStartSLOduration=1.872236776 podStartE2EDuration="20.79941637s" totalImagesPullingTime="18.927179594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:46:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:47:04.797853885 +0000 UTC m=+575.566691490" watchObservedRunningTime="2026-07-01 18:47:04.79941637 +0000 UTC m=+575.568253973"
Jul 01 18:47:04.826233 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:04.826194    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd" podStartSLOduration=1.929714213 podStartE2EDuration="19.826181271s" totalImagesPullingTime="17.896467058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:46:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:47:04.824158363 +0000 UTC m=+575.592995965" watchObservedRunningTime="2026-07-01 18:47:04.826181271 +0000 UTC m=+575.595018875"
Jul 01 18:47:05.770989 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:05.770948    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4" podUID="a903f2e3-d219-46b2-ad31-5a1d31733461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 01 18:47:05.771458 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:05.771086    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd"
Jul 01 18:47:05.772223 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:05.772190    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd" podUID="1932b4d7-e389-4605-bda4-4728ecf22790" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 18:47:06.775575 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:06.775529    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd" podUID="1932b4d7-e389-4605-bda4-4728ecf22790" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 18:47:10.775376 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:10.775350    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4"
Jul 01 18:47:10.775951 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:10.775863    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4" podUID="a903f2e3-d219-46b2-ad31-5a1d31733461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 01 18:47:11.779674 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:11.779642    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd"
Jul 01 18:47:11.780201 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:11.780178    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd" podUID="1932b4d7-e389-4605-bda4-4728ecf22790" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 18:47:20.776676 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:20.776637    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4" podUID="a903f2e3-d219-46b2-ad31-5a1d31733461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 01 18:47:21.780351 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:21.780312    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd" podUID="1932b4d7-e389-4605-bda4-4728ecf22790" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 18:47:30.775890 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:30.775848    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4" podUID="a903f2e3-d219-46b2-ad31-5a1d31733461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 01 18:47:31.780392 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:31.780356    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd" podUID="1932b4d7-e389-4605-bda4-4728ecf22790" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 18:47:34.884352 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:34.884317    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:47:40.775792 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:40.775754    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4" podUID="a903f2e3-d219-46b2-ad31-5a1d31733461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 01 18:47:41.780263 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:41.780207    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd" podUID="1932b4d7-e389-4605-bda4-4728ecf22790" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 18:47:50.776908 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:50.776830    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4"
Jul 01 18:47:51.781483 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:47:51.781453    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd"
Jul 01 18:48:04.452933 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:04.452898    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:48:04.674091 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:04.674058    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:48:18.612133 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.612098    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:48:18.768269 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.768235    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:48:18.791815 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.791769    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt"]
Jul 01 18:48:18.795183 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.795162    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt"
Jul 01 18:48:18.797427 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.797405    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9857d-kube-rbac-proxy-sar-config\""
Jul 01 18:48:18.797541 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.797413    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9857d-predictor-serving-cert\""
Jul 01 18:48:18.807688 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.807661    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt"]
Jul 01 18:48:18.837733 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.837686    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4"]
Jul 01 18:48:18.838046 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.838000    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4" podUID="a903f2e3-d219-46b2-ad31-5a1d31733461" containerName="kserve-container" containerID="cri-o://5ec4213465dbbb0fb3f56bb927ab5b57c17cb4320e28643fe263ce6aa4160f9c" gracePeriod=30
Jul 01 18:48:18.838633 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.838026    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4" podUID="a903f2e3-d219-46b2-ad31-5a1d31733461" containerName="kube-rbac-proxy" containerID="cri-o://410cb927c621062fcbb4a403e4f1f39959e931750d2a25a04e5ef7d06e2b161a" gracePeriod=30
Jul 01 18:48:18.864446 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.864382    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:48:18.887138 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.887111    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn7vs\" (UniqueName: \"kubernetes.io/projected/b4f0fd3f-76a8-46ca-857d-8b4e1f662afe-kube-api-access-dn7vs\") pod \"success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt\" (UID: \"b4f0fd3f-76a8-46ca-857d-8b4e1f662afe\") " pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt"
Jul 01 18:48:18.887278 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.887151    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4f0fd3f-76a8-46ca-857d-8b4e1f662afe-proxy-tls\") pod \"success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt\" (UID: \"b4f0fd3f-76a8-46ca-857d-8b4e1f662afe\") " pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt"
Jul 01 18:48:18.887278 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.887181    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4f0fd3f-76a8-46ca-857d-8b4e1f662afe-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt\" (UID: \"b4f0fd3f-76a8-46ca-857d-8b4e1f662afe\") " pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt"
Jul 01 18:48:18.895706 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.895681    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h"]
Jul 01 18:48:18.898957 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.898942    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h"
Jul 01 18:48:18.901530 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.901496    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9857d-predictor-serving-cert\""
Jul 01 18:48:18.901833 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.901818    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9857d-kube-rbac-proxy-sar-config\""
Jul 01 18:48:18.920369 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.920275    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h"]
Jul 01 18:48:18.941071 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.941034    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:48:18.967503 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.967470    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd"]
Jul 01 18:48:18.968045 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.968014    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd" podUID="1932b4d7-e389-4605-bda4-4728ecf22790" containerName="kserve-container" containerID="cri-o://64440a49b5d6477632b8e6b13cf4df0f7acaef6d3b2c01360365999b6500845c" gracePeriod=30
Jul 01 18:48:18.968531 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.968506    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd" podUID="1932b4d7-e389-4605-bda4-4728ecf22790" containerName="kube-rbac-proxy" containerID="cri-o://2c76096a497d7a1616160dd265ab270cc95ec1b8892f373d770e7f8475122a69" gracePeriod=30
Jul 01 18:48:18.987752 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.987692    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnjp5\" (UniqueName: \"kubernetes.io/projected/0d99116f-f1da-4605-b102-292b79631e8f-kube-api-access-pnjp5\") pod \"error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h\" (UID: \"0d99116f-f1da-4605-b102-292b79631e8f\") " pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h"
Jul 01 18:48:18.987873 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.987808    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn7vs\" (UniqueName: \"kubernetes.io/projected/b4f0fd3f-76a8-46ca-857d-8b4e1f662afe-kube-api-access-dn7vs\") pod \"success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt\" (UID: \"b4f0fd3f-76a8-46ca-857d-8b4e1f662afe\") " pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt"
Jul 01 18:48:18.987873 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.987851    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d99116f-f1da-4605-b102-292b79631e8f-proxy-tls\") pod \"error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h\" (UID: \"0d99116f-f1da-4605-b102-292b79631e8f\") " pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h"
Jul 01 18:48:18.987987 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.987879    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4f0fd3f-76a8-46ca-857d-8b4e1f662afe-proxy-tls\") pod \"success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt\" (UID: \"b4f0fd3f-76a8-46ca-857d-8b4e1f662afe\") " pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt"
Jul 01 18:48:18.987987 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.987925    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4f0fd3f-76a8-46ca-857d-8b4e1f662afe-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt\" (UID: \"b4f0fd3f-76a8-46ca-857d-8b4e1f662afe\") " pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt"
Jul 01 18:48:18.987987 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.987952    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d99116f-f1da-4605-b102-292b79631e8f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h\" (UID: \"0d99116f-f1da-4605-b102-292b79631e8f\") " pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h"
Jul 01 18:48:18.988133 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:48:18.988036    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-9857d-predictor-serving-cert: secret "success-200-isvc-9857d-predictor-serving-cert" not found
Jul 01 18:48:18.988190 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:48:18.988136    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4f0fd3f-76a8-46ca-857d-8b4e1f662afe-proxy-tls podName:b4f0fd3f-76a8-46ca-857d-8b4e1f662afe nodeName:}" failed. No retries permitted until 2026-07-01 18:48:19.488095438 +0000 UTC m=+650.256933023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b4f0fd3f-76a8-46ca-857d-8b4e1f662afe-proxy-tls") pod "success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt" (UID: "b4f0fd3f-76a8-46ca-857d-8b4e1f662afe") : secret "success-200-isvc-9857d-predictor-serving-cert" not found
Jul 01 18:48:18.988657 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.988616    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4f0fd3f-76a8-46ca-857d-8b4e1f662afe-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt\" (UID: \"b4f0fd3f-76a8-46ca-857d-8b4e1f662afe\") " pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt"
Jul 01 18:48:18.998431 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:18.998404    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn7vs\" (UniqueName: \"kubernetes.io/projected/b4f0fd3f-76a8-46ca-857d-8b4e1f662afe-kube-api-access-dn7vs\") pod \"success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt\" (UID: \"b4f0fd3f-76a8-46ca-857d-8b4e1f662afe\") " pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt"
Jul 01 18:48:19.017151 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:19.017116    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:48:19.044862 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:19.044826    2573 generic.go:356] "Generic (PLEG): container finished" podID="a903f2e3-d219-46b2-ad31-5a1d31733461" containerID="410cb927c621062fcbb4a403e4f1f39959e931750d2a25a04e5ef7d06e2b161a" exitCode=2
Jul 01 18:48:19.045024 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:19.044932    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4" event={"ID":"a903f2e3-d219-46b2-ad31-5a1d31733461","Type":"ContainerDied","Data":"410cb927c621062fcbb4a403e4f1f39959e931750d2a25a04e5ef7d06e2b161a"}
Jul 01 18:48:19.088919 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:19.088886    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d99116f-f1da-4605-b102-292b79631e8f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h\" (UID: \"0d99116f-f1da-4605-b102-292b79631e8f\") " pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h"
Jul 01 18:48:19.089061 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:19.088961    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnjp5\" (UniqueName: \"kubernetes.io/projected/0d99116f-f1da-4605-b102-292b79631e8f-kube-api-access-pnjp5\") pod \"error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h\" (UID: \"0d99116f-f1da-4605-b102-292b79631e8f\") " pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h"
Jul 01 18:48:19.089061 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:19.089045    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d99116f-f1da-4605-b102-292b79631e8f-proxy-tls\") pod \"error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h\" (UID: \"0d99116f-f1da-4605-b102-292b79631e8f\") " pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h"
Jul 01 18:48:19.089556 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:19.089532    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d99116f-f1da-4605-b102-292b79631e8f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h\" (UID: \"0d99116f-f1da-4605-b102-292b79631e8f\") " pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h"
Jul 01 18:48:19.091417 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:19.091396    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d99116f-f1da-4605-b102-292b79631e8f-proxy-tls\") pod \"error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h\" (UID: \"0d99116f-f1da-4605-b102-292b79631e8f\") " pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h"
Jul 01 18:48:19.100050 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:19.100027    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnjp5\" (UniqueName: \"kubernetes.io/projected/0d99116f-f1da-4605-b102-292b79631e8f-kube-api-access-pnjp5\") pod \"error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h\" (UID: \"0d99116f-f1da-4605-b102-292b79631e8f\") " pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h"
Jul 01 18:48:19.139564 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:19.139461    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:48:19.210707 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:19.210669    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h"
Jul 01 18:48:19.255931 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:19.255896    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:48:19.344693 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:19.344661    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h"]
Jul 01 18:48:19.347593 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:48:19.347568    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d99116f_f1da_4605_b102_292b79631e8f.slice/crio-1748be1bd8a83654bba711f770b973933a61e099df5fe542e6ffc33d818a93a7 WatchSource:0}: Error finding container 1748be1bd8a83654bba711f770b973933a61e099df5fe542e6ffc33d818a93a7: Status 404 returned error can't find the container with id 1748be1bd8a83654bba711f770b973933a61e099df5fe542e6ffc33d818a93a7
Jul 01 18:48:19.493142 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:19.493106    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4f0fd3f-76a8-46ca-857d-8b4e1f662afe-proxy-tls\") pod \"success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt\" (UID: \"b4f0fd3f-76a8-46ca-857d-8b4e1f662afe\") " pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt"
Jul 01 18:48:19.495481 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:19.495461    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4f0fd3f-76a8-46ca-857d-8b4e1f662afe-proxy-tls\") pod \"success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt\" (UID: \"b4f0fd3f-76a8-46ca-857d-8b4e1f662afe\") " pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt"
Jul 01 18:48:19.707327 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:19.707296    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt"
Jul 01 18:48:19.844554 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:19.844250    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt"]
Jul 01 18:48:19.847020 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:48:19.846997    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f0fd3f_76a8_46ca_857d_8b4e1f662afe.slice/crio-de9398fc54d23ff819d61d061f47bffe175a2082121c00b27c3432838d3e443e WatchSource:0}: Error finding container de9398fc54d23ff819d61d061f47bffe175a2082121c00b27c3432838d3e443e: Status 404 returned error can't find the container with id de9398fc54d23ff819d61d061f47bffe175a2082121c00b27c3432838d3e443e
Jul 01 18:48:20.050797 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:20.050762    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt" event={"ID":"b4f0fd3f-76a8-46ca-857d-8b4e1f662afe","Type":"ContainerStarted","Data":"485b1e0b0636353003b4ae188aa3a5df44d30a63a9c04c3bf33252ccf3021260"}
Jul 01 18:48:20.050996 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:20.050825    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt" event={"ID":"b4f0fd3f-76a8-46ca-857d-8b4e1f662afe","Type":"ContainerStarted","Data":"0af6f26463829ef640a40c800250c9539b0c47d855d0c1b73cb06188534b29f5"}
Jul 01 18:48:20.050996 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:20.050854    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt" event={"ID":"b4f0fd3f-76a8-46ca-857d-8b4e1f662afe","Type":"ContainerStarted","Data":"de9398fc54d23ff819d61d061f47bffe175a2082121c00b27c3432838d3e443e"}
Jul 01 18:48:20.050996 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:20.050880    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt"
Jul 01 18:48:20.052667 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:20.052638    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h" event={"ID":"0d99116f-f1da-4605-b102-292b79631e8f","Type":"ContainerStarted","Data":"b9e922bb27422f273019ab279dbc7964afafba9f1eb809898a483ef30510dffa"}
Jul 01 18:48:20.052818 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:20.052702    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h"
Jul 01 18:48:20.052818 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:20.052761    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h"
Jul 01 18:48:20.052818 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:20.052787    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h" event={"ID":"0d99116f-f1da-4605-b102-292b79631e8f","Type":"ContainerStarted","Data":"d78dd0bae1938c6c4487392f5225592d22bab8c672f46d3c80de90dab3198ca7"}
Jul 01 18:48:20.052956 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:20.052820    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h" event={"ID":"0d99116f-f1da-4605-b102-292b79631e8f","Type":"ContainerStarted","Data":"1748be1bd8a83654bba711f770b973933a61e099df5fe542e6ffc33d818a93a7"}
Jul 01 18:48:20.054350 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:20.054319    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h" podUID="0d99116f-f1da-4605-b102-292b79631e8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 18:48:20.054468 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:20.054449    2573 generic.go:356] "Generic (PLEG): container finished" podID="1932b4d7-e389-4605-bda4-4728ecf22790" containerID="2c76096a497d7a1616160dd265ab270cc95ec1b8892f373d770e7f8475122a69" exitCode=2
Jul 01 18:48:20.054525 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:20.054475    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd" event={"ID":"1932b4d7-e389-4605-bda4-4728ecf22790","Type":"ContainerDied","Data":"2c76096a497d7a1616160dd265ab270cc95ec1b8892f373d770e7f8475122a69"}
Jul 01 18:48:20.072036 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:20.071984    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt" podStartSLOduration=2.071969241 podStartE2EDuration="2.071969241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:48:20.070623033 +0000 UTC m=+650.839460637" watchObservedRunningTime="2026-07-01 18:48:20.071969241 +0000 UTC m=+650.840806844"
Jul 01 18:48:20.093482 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:20.093428    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h" podStartSLOduration=2.093412329 podStartE2EDuration="2.093412329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:48:20.091640967 +0000 UTC m=+650.860478571" watchObservedRunningTime="2026-07-01 18:48:20.093412329 +0000 UTC m=+650.862249978"
Jul 01 18:48:20.771538 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:20.771499    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4" podUID="a903f2e3-d219-46b2-ad31-5a1d31733461" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 01 18:48:20.775819 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:20.775789    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4" podUID="a903f2e3-d219-46b2-ad31-5a1d31733461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 01 18:48:21.058612 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:21.058526    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h" podUID="0d99116f-f1da-4605-b102-292b79631e8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 18:48:21.058896 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:21.058650    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt"
Jul 01 18:48:21.059859 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:21.059831    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt" podUID="b4f0fd3f-76a8-46ca-857d-8b4e1f662afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 18:48:21.775822 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:21.775786    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd" podUID="1932b4d7-e389-4605-bda4-4728ecf22790" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 01 18:48:21.781138 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:21.781114    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd" podUID="1932b4d7-e389-4605-bda4-4728ecf22790" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 01 18:48:22.064424 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:22.064399    2573 generic.go:356] "Generic (PLEG): container finished" podID="1932b4d7-e389-4605-bda4-4728ecf22790" containerID="64440a49b5d6477632b8e6b13cf4df0f7acaef6d3b2c01360365999b6500845c" exitCode=0
Jul 01 18:48:22.064557 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:22.064471    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd" event={"ID":"1932b4d7-e389-4605-bda4-4728ecf22790","Type":"ContainerDied","Data":"64440a49b5d6477632b8e6b13cf4df0f7acaef6d3b2c01360365999b6500845c"}
Jul 01 18:48:22.064781 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:22.064755    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt" podUID="b4f0fd3f-76a8-46ca-857d-8b4e1f662afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 18:48:22.115369 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:22.115347    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd"
Jul 01 18:48:22.216673 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:22.216648    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1932b4d7-e389-4605-bda4-4728ecf22790-proxy-tls\") pod \"1932b4d7-e389-4605-bda4-4728ecf22790\" (UID: \"1932b4d7-e389-4605-bda4-4728ecf22790\") "
Jul 01 18:48:22.216844 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:22.216740    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1932b4d7-e389-4605-bda4-4728ecf22790-kube-rbac-proxy-sar-config\") pod \"1932b4d7-e389-4605-bda4-4728ecf22790\" (UID: \"1932b4d7-e389-4605-bda4-4728ecf22790\") "
Jul 01 18:48:22.216844 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:22.216766    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdl8h\" (UniqueName: \"kubernetes.io/projected/1932b4d7-e389-4605-bda4-4728ecf22790-kube-api-access-hdl8h\") pod \"1932b4d7-e389-4605-bda4-4728ecf22790\" (UID: \"1932b4d7-e389-4605-bda4-4728ecf22790\") "
Jul 01 18:48:22.217507 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:22.217464    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1932b4d7-e389-4605-bda4-4728ecf22790-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1932b4d7-e389-4605-bda4-4728ecf22790" (UID: "1932b4d7-e389-4605-bda4-4728ecf22790"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:48:22.221946 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:22.219645    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1932b4d7-e389-4605-bda4-4728ecf22790-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1932b4d7-e389-4605-bda4-4728ecf22790" (UID: "1932b4d7-e389-4605-bda4-4728ecf22790"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:48:22.221946 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:22.219778    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1932b4d7-e389-4605-bda4-4728ecf22790-kube-api-access-hdl8h" (OuterVolumeSpecName: "kube-api-access-hdl8h") pod "1932b4d7-e389-4605-bda4-4728ecf22790" (UID: "1932b4d7-e389-4605-bda4-4728ecf22790"). InnerVolumeSpecName "kube-api-access-hdl8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:48:22.317861 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:22.317781    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1932b4d7-e389-4605-bda4-4728ecf22790-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:48:22.317861 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:22.317810    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdl8h\" (UniqueName: \"kubernetes.io/projected/1932b4d7-e389-4605-bda4-4728ecf22790-kube-api-access-hdl8h\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:48:22.317861 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:22.317823    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1932b4d7-e389-4605-bda4-4728ecf22790-proxy-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:48:22.691603 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:22.691577    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4"
Jul 01 18:48:22.822152 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:22.822119    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a903f2e3-d219-46b2-ad31-5a1d31733461-kube-rbac-proxy-sar-config\") pod \"a903f2e3-d219-46b2-ad31-5a1d31733461\" (UID: \"a903f2e3-d219-46b2-ad31-5a1d31733461\") "
Jul 01 18:48:22.822535 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:22.822204    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfg8v\" (UniqueName: \"kubernetes.io/projected/a903f2e3-d219-46b2-ad31-5a1d31733461-kube-api-access-gfg8v\") pod \"a903f2e3-d219-46b2-ad31-5a1d31733461\" (UID: \"a903f2e3-d219-46b2-ad31-5a1d31733461\") "
Jul 01 18:48:22.822535 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:22.822233    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a903f2e3-d219-46b2-ad31-5a1d31733461-proxy-tls\") pod \"a903f2e3-d219-46b2-ad31-5a1d31733461\" (UID: \"a903f2e3-d219-46b2-ad31-5a1d31733461\") "
Jul 01 18:48:22.822535 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:22.822472    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a903f2e3-d219-46b2-ad31-5a1d31733461-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a903f2e3-d219-46b2-ad31-5a1d31733461" (UID: "a903f2e3-d219-46b2-ad31-5a1d31733461"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:48:22.824283 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:22.824256    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a903f2e3-d219-46b2-ad31-5a1d31733461-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a903f2e3-d219-46b2-ad31-5a1d31733461" (UID: "a903f2e3-d219-46b2-ad31-5a1d31733461"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:48:22.824401 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:22.824366    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a903f2e3-d219-46b2-ad31-5a1d31733461-kube-api-access-gfg8v" (OuterVolumeSpecName: "kube-api-access-gfg8v") pod "a903f2e3-d219-46b2-ad31-5a1d31733461" (UID: "a903f2e3-d219-46b2-ad31-5a1d31733461"). InnerVolumeSpecName "kube-api-access-gfg8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:48:22.922984 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:22.922897    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfg8v\" (UniqueName: \"kubernetes.io/projected/a903f2e3-d219-46b2-ad31-5a1d31733461-kube-api-access-gfg8v\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:48:22.922984 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:22.922929    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a903f2e3-d219-46b2-ad31-5a1d31733461-proxy-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:48:22.922984 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:22.922943    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a903f2e3-d219-46b2-ad31-5a1d31733461-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:48:23.069969 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:23.069937    2573 generic.go:356] "Generic (PLEG): container finished" podID="a903f2e3-d219-46b2-ad31-5a1d31733461" containerID="5ec4213465dbbb0fb3f56bb927ab5b57c17cb4320e28643fe263ce6aa4160f9c" exitCode=0
Jul 01 18:48:23.070150 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:23.069988    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4"
Jul 01 18:48:23.070150 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:23.070019    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4" event={"ID":"a903f2e3-d219-46b2-ad31-5a1d31733461","Type":"ContainerDied","Data":"5ec4213465dbbb0fb3f56bb927ab5b57c17cb4320e28643fe263ce6aa4160f9c"}
Jul 01 18:48:23.070150 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:23.070079    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4" event={"ID":"a903f2e3-d219-46b2-ad31-5a1d31733461","Type":"ContainerDied","Data":"4de0ca029f00be4f9c8ecbbd5c61d848b14b9e577c739fbdc4d4a1037c48a396"}
Jul 01 18:48:23.070150 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:23.070105    2573 scope.go:117] "RemoveContainer" containerID="410cb927c621062fcbb4a403e4f1f39959e931750d2a25a04e5ef7d06e2b161a"
Jul 01 18:48:23.071776 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:23.071749    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd" event={"ID":"1932b4d7-e389-4605-bda4-4728ecf22790","Type":"ContainerDied","Data":"a78ba2815cd131484bbb884f36d35174719d9be3eb918477743141c12718f7bb"}
Jul 01 18:48:23.071876 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:23.071796    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd"
Jul 01 18:48:23.078316 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:23.078290    2573 scope.go:117] "RemoveContainer" containerID="5ec4213465dbbb0fb3f56bb927ab5b57c17cb4320e28643fe263ce6aa4160f9c"
Jul 01 18:48:23.086009 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:23.085994    2573 scope.go:117] "RemoveContainer" containerID="410cb927c621062fcbb4a403e4f1f39959e931750d2a25a04e5ef7d06e2b161a"
Jul 01 18:48:23.086250 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:48:23.086234    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"410cb927c621062fcbb4a403e4f1f39959e931750d2a25a04e5ef7d06e2b161a\": container with ID starting with 410cb927c621062fcbb4a403e4f1f39959e931750d2a25a04e5ef7d06e2b161a not found: ID does not exist" containerID="410cb927c621062fcbb4a403e4f1f39959e931750d2a25a04e5ef7d06e2b161a"
Jul 01 18:48:23.086319 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:23.086255    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"410cb927c621062fcbb4a403e4f1f39959e931750d2a25a04e5ef7d06e2b161a"} err="failed to get container status \"410cb927c621062fcbb4a403e4f1f39959e931750d2a25a04e5ef7d06e2b161a\": rpc error: code = NotFound desc = could not find container \"410cb927c621062fcbb4a403e4f1f39959e931750d2a25a04e5ef7d06e2b161a\": container with ID starting with 410cb927c621062fcbb4a403e4f1f39959e931750d2a25a04e5ef7d06e2b161a not found: ID does not exist"
Jul 01 18:48:23.086319 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:23.086270    2573 scope.go:117] "RemoveContainer" containerID="5ec4213465dbbb0fb3f56bb927ab5b57c17cb4320e28643fe263ce6aa4160f9c"
Jul 01 18:48:23.086486 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:48:23.086472    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ec4213465dbbb0fb3f56bb927ab5b57c17cb4320e28643fe263ce6aa4160f9c\": container with ID starting with 5ec4213465dbbb0fb3f56bb927ab5b57c17cb4320e28643fe263ce6aa4160f9c not found: ID does not exist" containerID="5ec4213465dbbb0fb3f56bb927ab5b57c17cb4320e28643fe263ce6aa4160f9c"
Jul 01 18:48:23.086525 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:23.086489    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec4213465dbbb0fb3f56bb927ab5b57c17cb4320e28643fe263ce6aa4160f9c"} err="failed to get container status \"5ec4213465dbbb0fb3f56bb927ab5b57c17cb4320e28643fe263ce6aa4160f9c\": rpc error: code = NotFound desc = could not find container \"5ec4213465dbbb0fb3f56bb927ab5b57c17cb4320e28643fe263ce6aa4160f9c\": container with ID starting with 5ec4213465dbbb0fb3f56bb927ab5b57c17cb4320e28643fe263ce6aa4160f9c not found: ID does not exist"
Jul 01 18:48:23.086525 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:23.086502    2573 scope.go:117] "RemoveContainer" containerID="2c76096a497d7a1616160dd265ab270cc95ec1b8892f373d770e7f8475122a69"
Jul 01 18:48:23.093489 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:23.093468    2573 scope.go:117] "RemoveContainer" containerID="64440a49b5d6477632b8e6b13cf4df0f7acaef6d3b2c01360365999b6500845c"
Jul 01 18:48:23.094039 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:23.094018    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4"]
Jul 01 18:48:23.098728 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:23.098695    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f2cdf-predictor-64d5b74467-n5sd4"]
Jul 01 18:48:23.110428 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:23.110407    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd"]
Jul 01 18:48:23.116527 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:23.116504    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f2cdf-predictor-58c9fc6cc8-6spwd"]
Jul 01 18:48:23.665557 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:23.665528    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1932b4d7-e389-4605-bda4-4728ecf22790" path="/var/lib/kubelet/pods/1932b4d7-e389-4605-bda4-4728ecf22790/volumes"
Jul 01 18:48:23.665953 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:23.665940    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a903f2e3-d219-46b2-ad31-5a1d31733461" path="/var/lib/kubelet/pods/a903f2e3-d219-46b2-ad31-5a1d31733461/volumes"
Jul 01 18:48:26.063548 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:26.063520    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h"
Jul 01 18:48:26.064086 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:26.064062    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h" podUID="0d99116f-f1da-4605-b102-292b79631e8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 18:48:27.069144 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:27.069118    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt"
Jul 01 18:48:27.069528 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:27.069480    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt" podUID="b4f0fd3f-76a8-46ca-857d-8b4e1f662afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 18:48:36.064567 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:36.064522    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h" podUID="0d99116f-f1da-4605-b102-292b79631e8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 18:48:37.069531 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:37.069490    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt" podUID="b4f0fd3f-76a8-46ca-857d-8b4e1f662afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 18:48:46.064400 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:46.064362    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h" podUID="0d99116f-f1da-4605-b102-292b79631e8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 18:48:47.070267 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:47.070227    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt" podUID="b4f0fd3f-76a8-46ca-857d-8b4e1f662afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 18:48:49.551195 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:49.551165    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:48:56.064998 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:56.064953    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h" podUID="0d99116f-f1da-4605-b102-292b79631e8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 18:48:57.069970 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:48:57.069927    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt" podUID="b4f0fd3f-76a8-46ca-857d-8b4e1f662afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 18:49:04.417489 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:04.417457    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:49:04.972768 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:04.972736    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:49:06.064664 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:06.064634    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h"
Jul 01 18:49:07.070849 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:07.070819    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt"
Jul 01 18:49:11.532758 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:11.532723    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:49:11.606101 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:11.606061    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:49:14.530008 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.529975    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:49:14.668410 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.668366    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:49:14.691031 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.690997    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg"]
Jul 01 18:49:14.691450 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.691435    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a903f2e3-d219-46b2-ad31-5a1d31733461" containerName="kserve-container"
Jul 01 18:49:14.691519 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.691453    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a903f2e3-d219-46b2-ad31-5a1d31733461" containerName="kserve-container"
Jul 01 18:49:14.691519 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.691464    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1932b4d7-e389-4605-bda4-4728ecf22790" containerName="kube-rbac-proxy"
Jul 01 18:49:14.691519 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.691473    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1932b4d7-e389-4605-bda4-4728ecf22790" containerName="kube-rbac-proxy"
Jul 01 18:49:14.691519 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.691496    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a903f2e3-d219-46b2-ad31-5a1d31733461" containerName="kube-rbac-proxy"
Jul 01 18:49:14.691519 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.691505    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a903f2e3-d219-46b2-ad31-5a1d31733461" containerName="kube-rbac-proxy"
Jul 01 18:49:14.691703 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.691527    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1932b4d7-e389-4605-bda4-4728ecf22790" containerName="kserve-container"
Jul 01 18:49:14.691703 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.691535    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1932b4d7-e389-4605-bda4-4728ecf22790" containerName="kserve-container"
Jul 01 18:49:14.691703 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.691596    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1932b4d7-e389-4605-bda4-4728ecf22790" containerName="kube-rbac-proxy"
Jul 01 18:49:14.691703 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.691605    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a903f2e3-d219-46b2-ad31-5a1d31733461" containerName="kube-rbac-proxy"
Jul 01 18:49:14.691703 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.691615    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a903f2e3-d219-46b2-ad31-5a1d31733461" containerName="kserve-container"
Jul 01 18:49:14.691703 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.691624    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1932b4d7-e389-4605-bda4-4728ecf22790" containerName="kserve-container"
Jul 01 18:49:14.695019 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.695001    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg"
Jul 01 18:49:14.698095 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.698072    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-eb211-predictor-serving-cert\""
Jul 01 18:49:14.698206 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.698072    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-eb211-kube-rbac-proxy-sar-config\""
Jul 01 18:49:14.708413 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.708389    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg"]
Jul 01 18:49:14.713397 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.713376    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f74764e-8c3c-4486-a25f-1a5c2126aff6-proxy-tls\") pod \"success-200-isvc-eb211-predictor-69b49c778c-qn8vg\" (UID: \"1f74764e-8c3c-4486-a25f-1a5c2126aff6\") " pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg"
Jul 01 18:49:14.713481 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.713432    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f74764e-8c3c-4486-a25f-1a5c2126aff6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-eb211-predictor-69b49c778c-qn8vg\" (UID: \"1f74764e-8c3c-4486-a25f-1a5c2126aff6\") " pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg"
Jul 01 18:49:14.713540 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.713506    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpk9c\" (UniqueName: \"kubernetes.io/projected/1f74764e-8c3c-4486-a25f-1a5c2126aff6-kube-api-access-cpk9c\") pod \"success-200-isvc-eb211-predictor-69b49c778c-qn8vg\" (UID: \"1f74764e-8c3c-4486-a25f-1a5c2126aff6\") " pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg"
Jul 01 18:49:14.772994 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.772956    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:49:14.814196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.814098    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f74764e-8c3c-4486-a25f-1a5c2126aff6-proxy-tls\") pod \"success-200-isvc-eb211-predictor-69b49c778c-qn8vg\" (UID: \"1f74764e-8c3c-4486-a25f-1a5c2126aff6\") " pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg"
Jul 01 18:49:14.814196 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.814161    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f74764e-8c3c-4486-a25f-1a5c2126aff6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-eb211-predictor-69b49c778c-qn8vg\" (UID: \"1f74764e-8c3c-4486-a25f-1a5c2126aff6\") " pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg"
Jul 01 18:49:14.814397 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.814207    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpk9c\" (UniqueName: \"kubernetes.io/projected/1f74764e-8c3c-4486-a25f-1a5c2126aff6-kube-api-access-cpk9c\") pod \"success-200-isvc-eb211-predictor-69b49c778c-qn8vg\" (UID: \"1f74764e-8c3c-4486-a25f-1a5c2126aff6\") " pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg"
Jul 01 18:49:14.814611 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:49:14.814594    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-eb211-predictor-serving-cert: secret "success-200-isvc-eb211-predictor-serving-cert" not found
Jul 01 18:49:14.814674 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:49:14.814668    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f74764e-8c3c-4486-a25f-1a5c2126aff6-proxy-tls podName:1f74764e-8c3c-4486-a25f-1a5c2126aff6 nodeName:}" failed. No retries permitted until 2026-07-01 18:49:15.314649062 +0000 UTC m=+706.083486648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1f74764e-8c3c-4486-a25f-1a5c2126aff6-proxy-tls") pod "success-200-isvc-eb211-predictor-69b49c778c-qn8vg" (UID: "1f74764e-8c3c-4486-a25f-1a5c2126aff6") : secret "success-200-isvc-eb211-predictor-serving-cert" not found
Jul 01 18:49:14.815539 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.815518    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f74764e-8c3c-4486-a25f-1a5c2126aff6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-eb211-predictor-69b49c778c-qn8vg\" (UID: \"1f74764e-8c3c-4486-a25f-1a5c2126aff6\") " pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg"
Jul 01 18:49:14.827251 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.827225    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p"]
Jul 01 18:49:14.830585 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.830568    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p"
Jul 01 18:49:14.833483 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.833466    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-eb211-predictor-serving-cert\""
Jul 01 18:49:14.833579 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.833465    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-eb211-kube-rbac-proxy-sar-config\""
Jul 01 18:49:14.836180 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.836157    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpk9c\" (UniqueName: \"kubernetes.io/projected/1f74764e-8c3c-4486-a25f-1a5c2126aff6-kube-api-access-cpk9c\") pod \"success-200-isvc-eb211-predictor-69b49c778c-qn8vg\" (UID: \"1f74764e-8c3c-4486-a25f-1a5c2126aff6\") " pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg"
Jul 01 18:49:14.848507 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.848483    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p"]
Jul 01 18:49:14.877989 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.877956    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:49:14.915194 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.915160    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6zrt\" (UniqueName: \"kubernetes.io/projected/8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0-kube-api-access-r6zrt\") pod \"error-404-isvc-eb211-predictor-6695c85886-hvd2p\" (UID: \"8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0\") " pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p"
Jul 01 18:49:14.915361 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.915222    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-eb211-predictor-6695c85886-hvd2p\" (UID: \"8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0\") " pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p"
Jul 01 18:49:14.915361 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:14.915277    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0-proxy-tls\") pod \"error-404-isvc-eb211-predictor-6695c85886-hvd2p\" (UID: \"8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0\") " pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p"
Jul 01 18:49:15.016432 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:15.016396    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0-proxy-tls\") pod \"error-404-isvc-eb211-predictor-6695c85886-hvd2p\" (UID: \"8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0\") " pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p"
Jul 01 18:49:15.016622 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:15.016526    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6zrt\" (UniqueName: \"kubernetes.io/projected/8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0-kube-api-access-r6zrt\") pod \"error-404-isvc-eb211-predictor-6695c85886-hvd2p\" (UID: \"8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0\") " pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p"
Jul 01 18:49:15.016622 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:49:15.016540    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-eb211-predictor-serving-cert: secret "error-404-isvc-eb211-predictor-serving-cert" not found
Jul 01 18:49:15.016622 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:15.016588    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-eb211-predictor-6695c85886-hvd2p\" (UID: \"8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0\") " pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p"
Jul 01 18:49:15.016622 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:49:15.016599    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0-proxy-tls podName:8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0 nodeName:}" failed. No retries permitted until 2026-07-01 18:49:15.516584616 +0000 UTC m=+706.285422198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0-proxy-tls") pod "error-404-isvc-eb211-predictor-6695c85886-hvd2p" (UID: "8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0") : secret "error-404-isvc-eb211-predictor-serving-cert" not found
Jul 01 18:49:15.017230 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:15.017198    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-eb211-predictor-6695c85886-hvd2p\" (UID: \"8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0\") " pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p"
Jul 01 18:49:15.026156 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:15.026135    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6zrt\" (UniqueName: \"kubernetes.io/projected/8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0-kube-api-access-r6zrt\") pod \"error-404-isvc-eb211-predictor-6695c85886-hvd2p\" (UID: \"8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0\") " pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p"
Jul 01 18:49:15.228109 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:15.228083    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:49:15.318210 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:15.318180    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f74764e-8c3c-4486-a25f-1a5c2126aff6-proxy-tls\") pod \"success-200-isvc-eb211-predictor-69b49c778c-qn8vg\" (UID: \"1f74764e-8c3c-4486-a25f-1a5c2126aff6\") " pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg"
Jul 01 18:49:15.320483 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:15.320466    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f74764e-8c3c-4486-a25f-1a5c2126aff6-proxy-tls\") pod \"success-200-isvc-eb211-predictor-69b49c778c-qn8vg\" (UID: \"1f74764e-8c3c-4486-a25f-1a5c2126aff6\") " pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg"
Jul 01 18:49:15.330623 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:15.330600    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:49:15.519064 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:15.518990    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0-proxy-tls\") pod \"error-404-isvc-eb211-predictor-6695c85886-hvd2p\" (UID: \"8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0\") " pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p"
Jul 01 18:49:15.521355 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:15.521338    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0-proxy-tls\") pod \"error-404-isvc-eb211-predictor-6695c85886-hvd2p\" (UID: \"8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0\") " pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p"
Jul 01 18:49:15.606068 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:15.606032    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg"
Jul 01 18:49:15.733246 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:15.733132    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg"]
Jul 01 18:49:15.735408 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:49:15.735376    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f74764e_8c3c_4486_a25f_1a5c2126aff6.slice/crio-5dea3b78ea1ee7480464559d3fc9c8393810bf7ffab452888762989aa8f0f8fb WatchSource:0}: Error finding container 5dea3b78ea1ee7480464559d3fc9c8393810bf7ffab452888762989aa8f0f8fb: Status 404 returned error can't find the container with id 5dea3b78ea1ee7480464559d3fc9c8393810bf7ffab452888762989aa8f0f8fb
Jul 01 18:49:15.749985 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:15.749963    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p"
Jul 01 18:49:15.882663 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:15.882638    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p"]
Jul 01 18:49:15.886012 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:49:15.885992    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8265b84a_2ed0_4bd9_a163_6cd2f9c1b9d0.slice/crio-1145fe4d6783409f3be410da792aeb7c371739cea80035dcde7d583cdd4b36fe WatchSource:0}: Error finding container 1145fe4d6783409f3be410da792aeb7c371739cea80035dcde7d583cdd4b36fe: Status 404 returned error can't find the container with id 1145fe4d6783409f3be410da792aeb7c371739cea80035dcde7d583cdd4b36fe
Jul 01 18:49:16.274093 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:16.274057    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p" event={"ID":"8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0","Type":"ContainerStarted","Data":"b1b672e0d9638e1dab17df87627f4980b5bce57396660fa903484c3878997a76"}
Jul 01 18:49:16.274300 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:16.274119    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p"
Jul 01 18:49:16.274300 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:16.274150    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p" event={"ID":"8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0","Type":"ContainerStarted","Data":"1f8aae326bd00971845ff387b36c74324186b6dba73d49ea3ca4c647fb51f380"}
Jul 01 18:49:16.274300 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:16.274246    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p"
Jul 01 18:49:16.274300 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:16.274297    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p" event={"ID":"8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0","Type":"ContainerStarted","Data":"1145fe4d6783409f3be410da792aeb7c371739cea80035dcde7d583cdd4b36fe"}
Jul 01 18:49:16.275646 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:16.275619    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p" podUID="8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 18:49:16.275902 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:16.275884    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg" event={"ID":"1f74764e-8c3c-4486-a25f-1a5c2126aff6","Type":"ContainerStarted","Data":"3f37b949b9d6c6edbd19fd712513e27aaa975568e0720ac4f038959d102d5527"}
Jul 01 18:49:16.275963 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:16.275940    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg" event={"ID":"1f74764e-8c3c-4486-a25f-1a5c2126aff6","Type":"ContainerStarted","Data":"9199312a1218a432b2aa5a898c42d3ba34f48a3ed7df87e63d6997ea56b31c7d"}
Jul 01 18:49:16.275997 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:16.275966    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg"
Jul 01 18:49:16.275997 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:16.275984    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg" event={"ID":"1f74764e-8c3c-4486-a25f-1a5c2126aff6","Type":"ContainerStarted","Data":"5dea3b78ea1ee7480464559d3fc9c8393810bf7ffab452888762989aa8f0f8fb"}
Jul 01 18:49:16.276074 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:16.276013    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg"
Jul 01 18:49:16.276836 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:16.276813    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg" podUID="1f74764e-8c3c-4486-a25f-1a5c2126aff6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 18:49:16.293148 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:16.293101    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p" podStartSLOduration=2.293088186 podStartE2EDuration="2.293088186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:49:16.292058126 +0000 UTC m=+707.060895752" watchObservedRunningTime="2026-07-01 18:49:16.293088186 +0000 UTC m=+707.061925771"
Jul 01 18:49:16.314419 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:16.314380    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg" podStartSLOduration=2.314371094 podStartE2EDuration="2.314371094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:49:16.313439384 +0000 UTC m=+707.082276988" watchObservedRunningTime="2026-07-01 18:49:16.314371094 +0000 UTC m=+707.083208773"
Jul 01 18:49:17.280447 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:17.280396    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p" podUID="8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 18:49:17.280910 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:17.280525    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg" podUID="1f74764e-8c3c-4486-a25f-1a5c2126aff6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 18:49:18.821446 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:18.821414    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:49:19.092273 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:19.092184    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:49:22.284364 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:22.284285    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p"
Jul 01 18:49:22.284895 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:22.284682    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg"
Jul 01 18:49:22.284895 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:22.284779    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p" podUID="8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 18:49:22.285052 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:22.285030    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg" podUID="1f74764e-8c3c-4486-a25f-1a5c2126aff6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 18:49:27.657439 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:27.657397    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:49:32.285560 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:32.285522    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p" podUID="8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 18:49:32.285965 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:32.285521    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg" podUID="1f74764e-8c3c-4486-a25f-1a5c2126aff6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 18:49:42.285394 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:42.285358    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p" podUID="8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 18:49:42.285800 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:42.285362    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg" podUID="1f74764e-8c3c-4486-a25f-1a5c2126aff6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 18:49:46.452013 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:46.451981    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:49:52.284787 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:52.284747    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p" podUID="8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 18:49:52.285173 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:49:52.285052    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg" podUID="1f74764e-8c3c-4486-a25f-1a5c2126aff6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 18:50:02.285380 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:50:02.285347    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p"
Jul 01 18:50:02.285896 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:50:02.285829    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg"
Jul 01 18:50:14.702968 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:50:14.702932    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:50:14.849912 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:50:14.849882    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:52:44.984423 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:52:44.984380    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:57:09.855225 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:09.855196    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:57:33.461109 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.461073    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:57:33.605260 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.605213    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:57:33.649330 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.649293    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5"]
Jul 01 18:57:33.652746 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.652707    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5"
Jul 01 18:57:33.655200 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.655176    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c059b-kube-rbac-proxy-sar-config\""
Jul 01 18:57:33.655314 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.655249    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c059b-predictor-serving-cert\""
Jul 01 18:57:33.655607 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.655586    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt"]
Jul 01 18:57:33.655859 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.655800    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt" podUID="b4f0fd3f-76a8-46ca-857d-8b4e1f662afe" containerName="kserve-container" containerID="cri-o://0af6f26463829ef640a40c800250c9539b0c47d855d0c1b73cb06188534b29f5" gracePeriod=30
Jul 01 18:57:33.655980 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.655874    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt" podUID="b4f0fd3f-76a8-46ca-857d-8b4e1f662afe" containerName="kube-rbac-proxy" containerID="cri-o://485b1e0b0636353003b4ae188aa3a5df44d30a63a9c04c3bf33252ccf3021260" gracePeriod=30
Jul 01 18:57:33.667807 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.667785    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5"]
Jul 01 18:57:33.698885 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.698818    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:57:33.737115 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.737026    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44"]
Jul 01 18:57:33.741460 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.741438    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44"
Jul 01 18:57:33.744202 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.744170    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbq5x\" (UniqueName: \"kubernetes.io/projected/d26e74b3-f903-4028-bf9f-ec2bb2dea9a8-kube-api-access-zbq5x\") pod \"success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5\" (UID: \"d26e74b3-f903-4028-bf9f-ec2bb2dea9a8\") " pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5"
Jul 01 18:57:33.744322 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.744291    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d26e74b3-f903-4028-bf9f-ec2bb2dea9a8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5\" (UID: \"d26e74b3-f903-4028-bf9f-ec2bb2dea9a8\") " pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5"
Jul 01 18:57:33.744732 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.744691    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d26e74b3-f903-4028-bf9f-ec2bb2dea9a8-proxy-tls\") pod \"success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5\" (UID: \"d26e74b3-f903-4028-bf9f-ec2bb2dea9a8\") " pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5"
Jul 01 18:57:33.745002 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.744985    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c059b-predictor-serving-cert\""
Jul 01 18:57:33.745108 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.745032    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c059b-kube-rbac-proxy-sar-config\""
Jul 01 18:57:33.753562 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.753540    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44"]
Jul 01 18:57:33.762800 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.762776    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h"]
Jul 01 18:57:33.763071 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.763025    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h" podUID="0d99116f-f1da-4605-b102-292b79631e8f" containerName="kserve-container" containerID="cri-o://d78dd0bae1938c6c4487392f5225592d22bab8c672f46d3c80de90dab3198ca7" gracePeriod=30
Jul 01 18:57:33.763279 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.763095    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h" podUID="0d99116f-f1da-4605-b102-292b79631e8f" containerName="kube-rbac-proxy" containerID="cri-o://b9e922bb27422f273019ab279dbc7964afafba9f1eb809898a483ef30510dffa" gracePeriod=30
Jul 01 18:57:33.789838 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.789810    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:57:33.846151 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.846119    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d26e74b3-f903-4028-bf9f-ec2bb2dea9a8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5\" (UID: \"d26e74b3-f903-4028-bf9f-ec2bb2dea9a8\") " pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5"
Jul 01 18:57:33.846306 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.846173    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8000bfc6-d3d5-4f73-b381-69e464c77c0d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c059b-predictor-847bfb4858-gvm44\" (UID: \"8000bfc6-d3d5-4f73-b381-69e464c77c0d\") " pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44"
Jul 01 18:57:33.846306 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.846220    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d26e74b3-f903-4028-bf9f-ec2bb2dea9a8-proxy-tls\") pod \"success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5\" (UID: \"d26e74b3-f903-4028-bf9f-ec2bb2dea9a8\") " pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5"
Jul 01 18:57:33.846306 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.846253    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmv7t\" (UniqueName: \"kubernetes.io/projected/8000bfc6-d3d5-4f73-b381-69e464c77c0d-kube-api-access-hmv7t\") pod \"error-404-isvc-c059b-predictor-847bfb4858-gvm44\" (UID: \"8000bfc6-d3d5-4f73-b381-69e464c77c0d\") " pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44"
Jul 01 18:57:33.846306 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.846283    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbq5x\" (UniqueName: \"kubernetes.io/projected/d26e74b3-f903-4028-bf9f-ec2bb2dea9a8-kube-api-access-zbq5x\") pod \"success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5\" (UID: \"d26e74b3-f903-4028-bf9f-ec2bb2dea9a8\") " pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5"
Jul 01 18:57:33.846523 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:57:33.846367    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-c059b-predictor-serving-cert: secret "success-200-isvc-c059b-predictor-serving-cert" not found
Jul 01 18:57:33.846523 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.846387    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8000bfc6-d3d5-4f73-b381-69e464c77c0d-proxy-tls\") pod \"error-404-isvc-c059b-predictor-847bfb4858-gvm44\" (UID: \"8000bfc6-d3d5-4f73-b381-69e464c77c0d\") " pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44"
Jul 01 18:57:33.846523 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:57:33.846444    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d26e74b3-f903-4028-bf9f-ec2bb2dea9a8-proxy-tls podName:d26e74b3-f903-4028-bf9f-ec2bb2dea9a8 nodeName:}" failed. No retries permitted until 2026-07-01 18:57:34.34642271 +0000 UTC m=+1205.115260294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d26e74b3-f903-4028-bf9f-ec2bb2dea9a8-proxy-tls") pod "success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5" (UID: "d26e74b3-f903-4028-bf9f-ec2bb2dea9a8") : secret "success-200-isvc-c059b-predictor-serving-cert" not found
Jul 01 18:57:33.846895 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.846865    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d26e74b3-f903-4028-bf9f-ec2bb2dea9a8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5\" (UID: \"d26e74b3-f903-4028-bf9f-ec2bb2dea9a8\") " pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5"
Jul 01 18:57:33.861358 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.861325    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbq5x\" (UniqueName: \"kubernetes.io/projected/d26e74b3-f903-4028-bf9f-ec2bb2dea9a8-kube-api-access-zbq5x\") pod \"success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5\" (UID: \"d26e74b3-f903-4028-bf9f-ec2bb2dea9a8\") " pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5"
Jul 01 18:57:33.947052 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.947020    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmv7t\" (UniqueName: \"kubernetes.io/projected/8000bfc6-d3d5-4f73-b381-69e464c77c0d-kube-api-access-hmv7t\") pod \"error-404-isvc-c059b-predictor-847bfb4858-gvm44\" (UID: \"8000bfc6-d3d5-4f73-b381-69e464c77c0d\") " pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44"
Jul 01 18:57:33.947209 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.947074    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8000bfc6-d3d5-4f73-b381-69e464c77c0d-proxy-tls\") pod \"error-404-isvc-c059b-predictor-847bfb4858-gvm44\" (UID: \"8000bfc6-d3d5-4f73-b381-69e464c77c0d\") " pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44"
Jul 01 18:57:33.947209 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.947121    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8000bfc6-d3d5-4f73-b381-69e464c77c0d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c059b-predictor-847bfb4858-gvm44\" (UID: \"8000bfc6-d3d5-4f73-b381-69e464c77c0d\") " pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44"
Jul 01 18:57:33.947303 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:57:33.947243    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-c059b-predictor-serving-cert: secret "error-404-isvc-c059b-predictor-serving-cert" not found
Jul 01 18:57:33.947349 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:57:33.947314    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8000bfc6-d3d5-4f73-b381-69e464c77c0d-proxy-tls podName:8000bfc6-d3d5-4f73-b381-69e464c77c0d nodeName:}" failed. No retries permitted until 2026-07-01 18:57:34.447296015 +0000 UTC m=+1205.216133623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8000bfc6-d3d5-4f73-b381-69e464c77c0d-proxy-tls") pod "error-404-isvc-c059b-predictor-847bfb4858-gvm44" (UID: "8000bfc6-d3d5-4f73-b381-69e464c77c0d") : secret "error-404-isvc-c059b-predictor-serving-cert" not found
Jul 01 18:57:33.948052 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.948033    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8000bfc6-d3d5-4f73-b381-69e464c77c0d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c059b-predictor-847bfb4858-gvm44\" (UID: \"8000bfc6-d3d5-4f73-b381-69e464c77c0d\") " pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44"
Jul 01 18:57:33.963790 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:33.963763    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmv7t\" (UniqueName: \"kubernetes.io/projected/8000bfc6-d3d5-4f73-b381-69e464c77c0d-kube-api-access-hmv7t\") pod \"error-404-isvc-c059b-predictor-847bfb4858-gvm44\" (UID: \"8000bfc6-d3d5-4f73-b381-69e464c77c0d\") " pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44"
Jul 01 18:57:34.010332 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:34.010264    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:57:34.111506 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:34.111474    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:57:34.122101 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:34.122076    2573 generic.go:356] "Generic (PLEG): container finished" podID="0d99116f-f1da-4605-b102-292b79631e8f" containerID="b9e922bb27422f273019ab279dbc7964afafba9f1eb809898a483ef30510dffa" exitCode=2
Jul 01 18:57:34.122238 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:34.122150    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h" event={"ID":"0d99116f-f1da-4605-b102-292b79631e8f","Type":"ContainerDied","Data":"b9e922bb27422f273019ab279dbc7964afafba9f1eb809898a483ef30510dffa"}
Jul 01 18:57:34.123702 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:34.123682    2573 generic.go:356] "Generic (PLEG): container finished" podID="b4f0fd3f-76a8-46ca-857d-8b4e1f662afe" containerID="485b1e0b0636353003b4ae188aa3a5df44d30a63a9c04c3bf33252ccf3021260" exitCode=2
Jul 01 18:57:34.123804 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:34.123726    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt" event={"ID":"b4f0fd3f-76a8-46ca-857d-8b4e1f662afe","Type":"ContainerDied","Data":"485b1e0b0636353003b4ae188aa3a5df44d30a63a9c04c3bf33252ccf3021260"}
Jul 01 18:57:34.350555 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:34.350468    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d26e74b3-f903-4028-bf9f-ec2bb2dea9a8-proxy-tls\") pod \"success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5\" (UID: \"d26e74b3-f903-4028-bf9f-ec2bb2dea9a8\") " pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5"
Jul 01 18:57:34.352910 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:34.352886    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d26e74b3-f903-4028-bf9f-ec2bb2dea9a8-proxy-tls\") pod \"success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5\" (UID: \"d26e74b3-f903-4028-bf9f-ec2bb2dea9a8\") " pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5"
Jul 01 18:57:34.450845 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:34.450812    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8000bfc6-d3d5-4f73-b381-69e464c77c0d-proxy-tls\") pod \"error-404-isvc-c059b-predictor-847bfb4858-gvm44\" (UID: \"8000bfc6-d3d5-4f73-b381-69e464c77c0d\") " pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44"
Jul 01 18:57:34.453040 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:34.453023    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8000bfc6-d3d5-4f73-b381-69e464c77c0d-proxy-tls\") pod \"error-404-isvc-c059b-predictor-847bfb4858-gvm44\" (UID: \"8000bfc6-d3d5-4f73-b381-69e464c77c0d\") " pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44"
Jul 01 18:57:34.573629 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:34.573590    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5"
Jul 01 18:57:34.654518 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:34.654208    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44"
Jul 01 18:57:34.709916 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:34.709887    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5"]
Jul 01 18:57:34.715884 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:57:34.715859    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26e74b3_f903_4028_bf9f_ec2bb2dea9a8.slice/crio-8a6af7255f6eca0e30d39d2214545acb48b21988b84578004f4df6b2d8c5ebad WatchSource:0}: Error finding container 8a6af7255f6eca0e30d39d2214545acb48b21988b84578004f4df6b2d8c5ebad: Status 404 returned error can't find the container with id 8a6af7255f6eca0e30d39d2214545acb48b21988b84578004f4df6b2d8c5ebad
Jul 01 18:57:34.795010 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:34.794891    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44"]
Jul 01 18:57:34.797218 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:57:34.797187    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8000bfc6_d3d5_4f73_b381_69e464c77c0d.slice/crio-c8d56bd9284327219bf1992f86b0556ada6696496fd40649d94d81b8ba3ab5b9 WatchSource:0}: Error finding container c8d56bd9284327219bf1992f86b0556ada6696496fd40649d94d81b8ba3ab5b9: Status 404 returned error can't find the container with id c8d56bd9284327219bf1992f86b0556ada6696496fd40649d94d81b8ba3ab5b9
Jul 01 18:57:35.128946 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:35.128852    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44" event={"ID":"8000bfc6-d3d5-4f73-b381-69e464c77c0d","Type":"ContainerStarted","Data":"a8cc31cc203783f20d1f8e10b2fa1e7cb60e8dad069a574e472f2e4536e126c5"}
Jul 01 18:57:35.128946 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:35.128923    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44" event={"ID":"8000bfc6-d3d5-4f73-b381-69e464c77c0d","Type":"ContainerStarted","Data":"9950ea29ebc677393aee0769955f68e930fcb0aeefcf53d0f97fb1df1fcccf66"}
Jul 01 18:57:35.129173 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:35.128961    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44"
Jul 01 18:57:35.129173 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:35.128992    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44" event={"ID":"8000bfc6-d3d5-4f73-b381-69e464c77c0d","Type":"ContainerStarted","Data":"c8d56bd9284327219bf1992f86b0556ada6696496fd40649d94d81b8ba3ab5b9"}
Jul 01 18:57:35.130480 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:35.130456    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5" event={"ID":"d26e74b3-f903-4028-bf9f-ec2bb2dea9a8","Type":"ContainerStarted","Data":"0b032a7c9f44ff299f65b5fd8082cf6a19b4fcd527fe564bb598e8137754e1a8"}
Jul 01 18:57:35.130612 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:35.130509    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5" event={"ID":"d26e74b3-f903-4028-bf9f-ec2bb2dea9a8","Type":"ContainerStarted","Data":"373059822d6ba30452329de4ec7fb59c20cc792954fb1aa3633466adce16650d"}
Jul 01 18:57:35.130612 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:35.130543    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5"
Jul 01 18:57:35.130612 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:35.130574    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5"
Jul 01 18:57:35.130612 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:35.130597    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5" event={"ID":"d26e74b3-f903-4028-bf9f-ec2bb2dea9a8","Type":"ContainerStarted","Data":"8a6af7255f6eca0e30d39d2214545acb48b21988b84578004f4df6b2d8c5ebad"}
Jul 01 18:57:35.131905 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:35.131851    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5" podUID="d26e74b3-f903-4028-bf9f-ec2bb2dea9a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 18:57:35.150156 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:35.150110    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44" podStartSLOduration=2.150096567 podStartE2EDuration="2.150096567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:57:35.148586608 +0000 UTC m=+1205.917424223" watchObservedRunningTime="2026-07-01 18:57:35.150096567 +0000 UTC m=+1205.918934170"
Jul 01 18:57:35.168967 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:35.168917    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5" podStartSLOduration=2.168903414 podStartE2EDuration="2.168903414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:57:35.16713256 +0000 UTC m=+1205.935970186" watchObservedRunningTime="2026-07-01 18:57:35.168903414 +0000 UTC m=+1205.937741050"
Jul 01 18:57:36.058932 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:36.058891    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h" podUID="0d99116f-f1da-4605-b102-292b79631e8f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 01 18:57:36.064235 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:36.064202    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h" podUID="0d99116f-f1da-4605-b102-292b79631e8f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 18:57:36.137133 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:36.137103    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44"
Jul 01 18:57:36.137314 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:36.137145    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5" podUID="d26e74b3-f903-4028-bf9f-ec2bb2dea9a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 18:57:36.138147 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:36.138119    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44" podUID="8000bfc6-d3d5-4f73-b381-69e464c77c0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 18:57:37.011197 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.011171    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt"
Jul 01 18:57:37.073271 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.073238    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4f0fd3f-76a8-46ca-857d-8b4e1f662afe-kube-rbac-proxy-sar-config\") pod \"b4f0fd3f-76a8-46ca-857d-8b4e1f662afe\" (UID: \"b4f0fd3f-76a8-46ca-857d-8b4e1f662afe\") "
Jul 01 18:57:37.073680 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.073448    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4f0fd3f-76a8-46ca-857d-8b4e1f662afe-proxy-tls\") pod \"b4f0fd3f-76a8-46ca-857d-8b4e1f662afe\" (UID: \"b4f0fd3f-76a8-46ca-857d-8b4e1f662afe\") "
Jul 01 18:57:37.073680 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.073491    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dn7vs\" (UniqueName: \"kubernetes.io/projected/b4f0fd3f-76a8-46ca-857d-8b4e1f662afe-kube-api-access-dn7vs\") pod \"b4f0fd3f-76a8-46ca-857d-8b4e1f662afe\" (UID: \"b4f0fd3f-76a8-46ca-857d-8b4e1f662afe\") "
Jul 01 18:57:37.073680 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.073655    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f0fd3f-76a8-46ca-857d-8b4e1f662afe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b4f0fd3f-76a8-46ca-857d-8b4e1f662afe" (UID: "b4f0fd3f-76a8-46ca-857d-8b4e1f662afe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:57:37.073862 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.073804    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4f0fd3f-76a8-46ca-857d-8b4e1f662afe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:57:37.075617 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.075591    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4f0fd3f-76a8-46ca-857d-8b4e1f662afe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b4f0fd3f-76a8-46ca-857d-8b4e1f662afe" (UID: "b4f0fd3f-76a8-46ca-857d-8b4e1f662afe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:57:37.075743 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.075667    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f0fd3f-76a8-46ca-857d-8b4e1f662afe-kube-api-access-dn7vs" (OuterVolumeSpecName: "kube-api-access-dn7vs") pod "b4f0fd3f-76a8-46ca-857d-8b4e1f662afe" (UID: "b4f0fd3f-76a8-46ca-857d-8b4e1f662afe"). InnerVolumeSpecName "kube-api-access-dn7vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:57:37.115991 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.115972    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h"
Jul 01 18:57:37.142908 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.142881    2573 generic.go:356] "Generic (PLEG): container finished" podID="0d99116f-f1da-4605-b102-292b79631e8f" containerID="d78dd0bae1938c6c4487392f5225592d22bab8c672f46d3c80de90dab3198ca7" exitCode=0
Jul 01 18:57:37.143047 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.142930    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h" event={"ID":"0d99116f-f1da-4605-b102-292b79631e8f","Type":"ContainerDied","Data":"d78dd0bae1938c6c4487392f5225592d22bab8c672f46d3c80de90dab3198ca7"}
Jul 01 18:57:37.143047 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.142988    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h" event={"ID":"0d99116f-f1da-4605-b102-292b79631e8f","Type":"ContainerDied","Data":"1748be1bd8a83654bba711f770b973933a61e099df5fe542e6ffc33d818a93a7"}
Jul 01 18:57:37.143047 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.142939    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h"
Jul 01 18:57:37.143768 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.143022    2573 scope.go:117] "RemoveContainer" containerID="b9e922bb27422f273019ab279dbc7964afafba9f1eb809898a483ef30510dffa"
Jul 01 18:57:37.144574 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.144553    2573 generic.go:356] "Generic (PLEG): container finished" podID="b4f0fd3f-76a8-46ca-857d-8b4e1f662afe" containerID="0af6f26463829ef640a40c800250c9539b0c47d855d0c1b73cb06188534b29f5" exitCode=0
Jul 01 18:57:37.144679 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.144609    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt"
Jul 01 18:57:37.144679 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.144625    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt" event={"ID":"b4f0fd3f-76a8-46ca-857d-8b4e1f662afe","Type":"ContainerDied","Data":"0af6f26463829ef640a40c800250c9539b0c47d855d0c1b73cb06188534b29f5"}
Jul 01 18:57:37.144863 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.144697    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt" event={"ID":"b4f0fd3f-76a8-46ca-857d-8b4e1f662afe","Type":"ContainerDied","Data":"de9398fc54d23ff819d61d061f47bffe175a2082121c00b27c3432838d3e443e"}
Jul 01 18:57:37.145032 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.145008    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44" podUID="8000bfc6-d3d5-4f73-b381-69e464c77c0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 18:57:37.151671 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.151653    2573 scope.go:117] "RemoveContainer" containerID="d78dd0bae1938c6c4487392f5225592d22bab8c672f46d3c80de90dab3198ca7"
Jul 01 18:57:37.159911 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.159876    2573 scope.go:117] "RemoveContainer" containerID="b9e922bb27422f273019ab279dbc7964afafba9f1eb809898a483ef30510dffa"
Jul 01 18:57:37.160183 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:57:37.160163    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9e922bb27422f273019ab279dbc7964afafba9f1eb809898a483ef30510dffa\": container with ID starting with b9e922bb27422f273019ab279dbc7964afafba9f1eb809898a483ef30510dffa not found: ID does not exist" containerID="b9e922bb27422f273019ab279dbc7964afafba9f1eb809898a483ef30510dffa"
Jul 01 18:57:37.160248 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.160193    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9e922bb27422f273019ab279dbc7964afafba9f1eb809898a483ef30510dffa"} err="failed to get container status \"b9e922bb27422f273019ab279dbc7964afafba9f1eb809898a483ef30510dffa\": rpc error: code = NotFound desc = could not find container \"b9e922bb27422f273019ab279dbc7964afafba9f1eb809898a483ef30510dffa\": container with ID starting with b9e922bb27422f273019ab279dbc7964afafba9f1eb809898a483ef30510dffa not found: ID does not exist"
Jul 01 18:57:37.160248 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.160209    2573 scope.go:117] "RemoveContainer" containerID="d78dd0bae1938c6c4487392f5225592d22bab8c672f46d3c80de90dab3198ca7"
Jul 01 18:57:37.160474 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:57:37.160453    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d78dd0bae1938c6c4487392f5225592d22bab8c672f46d3c80de90dab3198ca7\": container with ID starting with d78dd0bae1938c6c4487392f5225592d22bab8c672f46d3c80de90dab3198ca7 not found: ID does not exist" containerID="d78dd0bae1938c6c4487392f5225592d22bab8c672f46d3c80de90dab3198ca7"
Jul 01 18:57:37.160538 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.160485    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78dd0bae1938c6c4487392f5225592d22bab8c672f46d3c80de90dab3198ca7"} err="failed to get container status \"d78dd0bae1938c6c4487392f5225592d22bab8c672f46d3c80de90dab3198ca7\": rpc error: code = NotFound desc = could not find container \"d78dd0bae1938c6c4487392f5225592d22bab8c672f46d3c80de90dab3198ca7\": container with ID starting with d78dd0bae1938c6c4487392f5225592d22bab8c672f46d3c80de90dab3198ca7 not found: ID does not exist"
Jul 01 18:57:37.160538 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.160507    2573 scope.go:117] "RemoveContainer" containerID="485b1e0b0636353003b4ae188aa3a5df44d30a63a9c04c3bf33252ccf3021260"
Jul 01 18:57:37.167252 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.167236    2573 scope.go:117] "RemoveContainer" containerID="0af6f26463829ef640a40c800250c9539b0c47d855d0c1b73cb06188534b29f5"
Jul 01 18:57:37.171865 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.171845    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt"]
Jul 01 18:57:37.174044 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.174023    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9857d-predictor-c6ddb6cb7-54ljt"]
Jul 01 18:57:37.174202 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.174187    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d99116f-f1da-4605-b102-292b79631e8f-kube-rbac-proxy-sar-config\") pod \"0d99116f-f1da-4605-b102-292b79631e8f\" (UID: \"0d99116f-f1da-4605-b102-292b79631e8f\") "
Jul 01 18:57:37.174259 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.174248    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d99116f-f1da-4605-b102-292b79631e8f-proxy-tls\") pod \"0d99116f-f1da-4605-b102-292b79631e8f\" (UID: \"0d99116f-f1da-4605-b102-292b79631e8f\") "
Jul 01 18:57:37.174299 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.174270    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnjp5\" (UniqueName: \"kubernetes.io/projected/0d99116f-f1da-4605-b102-292b79631e8f-kube-api-access-pnjp5\") pod \"0d99116f-f1da-4605-b102-292b79631e8f\" (UID: \"0d99116f-f1da-4605-b102-292b79631e8f\") "
Jul 01 18:57:37.174383 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.174372    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4f0fd3f-76a8-46ca-857d-8b4e1f662afe-proxy-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:57:37.174436 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.174384    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn7vs\" (UniqueName: \"kubernetes.io/projected/b4f0fd3f-76a8-46ca-857d-8b4e1f662afe-kube-api-access-dn7vs\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:57:37.174573 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.174547    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d99116f-f1da-4605-b102-292b79631e8f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0d99116f-f1da-4605-b102-292b79631e8f" (UID: "0d99116f-f1da-4605-b102-292b79631e8f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:57:37.175257 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.175237    2573 scope.go:117] "RemoveContainer" containerID="485b1e0b0636353003b4ae188aa3a5df44d30a63a9c04c3bf33252ccf3021260"
Jul 01 18:57:37.175535 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:57:37.175514    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"485b1e0b0636353003b4ae188aa3a5df44d30a63a9c04c3bf33252ccf3021260\": container with ID starting with 485b1e0b0636353003b4ae188aa3a5df44d30a63a9c04c3bf33252ccf3021260 not found: ID does not exist" containerID="485b1e0b0636353003b4ae188aa3a5df44d30a63a9c04c3bf33252ccf3021260"
Jul 01 18:57:37.175612 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.175541    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"485b1e0b0636353003b4ae188aa3a5df44d30a63a9c04c3bf33252ccf3021260"} err="failed to get container status \"485b1e0b0636353003b4ae188aa3a5df44d30a63a9c04c3bf33252ccf3021260\": rpc error: code = NotFound desc = could not find container \"485b1e0b0636353003b4ae188aa3a5df44d30a63a9c04c3bf33252ccf3021260\": container with ID starting with 485b1e0b0636353003b4ae188aa3a5df44d30a63a9c04c3bf33252ccf3021260 not found: ID does not exist"
Jul 01 18:57:37.175612 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.175556    2573 scope.go:117] "RemoveContainer" containerID="0af6f26463829ef640a40c800250c9539b0c47d855d0c1b73cb06188534b29f5"
Jul 01 18:57:37.175844 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:57:37.175826    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af6f26463829ef640a40c800250c9539b0c47d855d0c1b73cb06188534b29f5\": container with ID starting with 0af6f26463829ef640a40c800250c9539b0c47d855d0c1b73cb06188534b29f5 not found: ID does not exist" containerID="0af6f26463829ef640a40c800250c9539b0c47d855d0c1b73cb06188534b29f5"
Jul 01 18:57:37.175893 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.175848    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af6f26463829ef640a40c800250c9539b0c47d855d0c1b73cb06188534b29f5"} err="failed to get container status \"0af6f26463829ef640a40c800250c9539b0c47d855d0c1b73cb06188534b29f5\": rpc error: code = NotFound desc = could not find container \"0af6f26463829ef640a40c800250c9539b0c47d855d0c1b73cb06188534b29f5\": container with ID starting with 0af6f26463829ef640a40c800250c9539b0c47d855d0c1b73cb06188534b29f5 not found: ID does not exist"
Jul 01 18:57:37.176176 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.176157    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d99116f-f1da-4605-b102-292b79631e8f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d99116f-f1da-4605-b102-292b79631e8f" (UID: "0d99116f-f1da-4605-b102-292b79631e8f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:57:37.176285 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.176267    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d99116f-f1da-4605-b102-292b79631e8f-kube-api-access-pnjp5" (OuterVolumeSpecName: "kube-api-access-pnjp5") pod "0d99116f-f1da-4605-b102-292b79631e8f" (UID: "0d99116f-f1da-4605-b102-292b79631e8f"). InnerVolumeSpecName "kube-api-access-pnjp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:57:37.275047 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.275005    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d99116f-f1da-4605-b102-292b79631e8f-proxy-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:57:37.275240 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.275128    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnjp5\" (UniqueName: \"kubernetes.io/projected/0d99116f-f1da-4605-b102-292b79631e8f-kube-api-access-pnjp5\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:57:37.275240 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.275149    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d99116f-f1da-4605-b102-292b79631e8f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:57:37.467155 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.467127    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h"]
Jul 01 18:57:37.471696 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.471674    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9857d-predictor-74fddbbbc8-ftl8h"]
Jul 01 18:57:37.670593 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.670561    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d99116f-f1da-4605-b102-292b79631e8f" path="/var/lib/kubelet/pods/0d99116f-f1da-4605-b102-292b79631e8f/volumes"
Jul 01 18:57:37.671021 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:37.671007    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4f0fd3f-76a8-46ca-857d-8b4e1f662afe" path="/var/lib/kubelet/pods/b4f0fd3f-76a8-46ca-857d-8b4e1f662afe/volumes"
Jul 01 18:57:41.141263 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:41.141235    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5"
Jul 01 18:57:41.141860 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:41.141831    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5" podUID="d26e74b3-f903-4028-bf9f-ec2bb2dea9a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 18:57:42.150131 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:42.150107    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44"
Jul 01 18:57:42.150511 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:42.150487    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44" podUID="8000bfc6-d3d5-4f73-b381-69e464c77c0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 18:57:51.141931 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:51.141890    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5" podUID="d26e74b3-f903-4028-bf9f-ec2bb2dea9a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 18:57:52.150871 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:57:52.150834    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44" podUID="8000bfc6-d3d5-4f73-b381-69e464c77c0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 18:58:01.142257 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:01.142213    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5" podUID="d26e74b3-f903-4028-bf9f-ec2bb2dea9a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 18:58:02.151508 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:02.151473    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44" podUID="8000bfc6-d3d5-4f73-b381-69e464c77c0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 18:58:11.142155 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:11.142113    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5" podUID="d26e74b3-f903-4028-bf9f-ec2bb2dea9a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 01 18:58:12.150949 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:12.150904    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44" podUID="8000bfc6-d3d5-4f73-b381-69e464c77c0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 18:58:21.143294 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:21.143218    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5"
Jul 01 18:58:22.150848 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:22.150819    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44"
Jul 01 18:58:29.350534 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.350501    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:58:29.393075 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.393045    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p"]
Jul 01 18:58:29.393344 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.393311    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p" podUID="8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0" containerName="kserve-container" containerID="cri-o://1f8aae326bd00971845ff387b36c74324186b6dba73d49ea3ca4c647fb51f380" gracePeriod=30
Jul 01 18:58:29.393496 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.393339    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p" podUID="8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0" containerName="kube-rbac-proxy" containerID="cri-o://b1b672e0d9638e1dab17df87627f4980b5bce57396660fa903484c3878997a76" gracePeriod=30
Jul 01 18:58:29.450939 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.450905    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg"]
Jul 01 18:58:29.451220 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.451193    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg" podUID="1f74764e-8c3c-4486-a25f-1a5c2126aff6" containerName="kserve-container" containerID="cri-o://9199312a1218a432b2aa5a898c42d3ba34f48a3ed7df87e63d6997ea56b31c7d" gracePeriod=30
Jul 01 18:58:29.451332 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.451172    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg" podUID="1f74764e-8c3c-4486-a25f-1a5c2126aff6" containerName="kube-rbac-proxy" containerID="cri-o://3f37b949b9d6c6edbd19fd712513e27aaa975568e0720ac4f038959d102d5527" gracePeriod=30
Jul 01 18:58:29.454064 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.454042    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:58:29.485904 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.485877    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl"]
Jul 01 18:58:29.486242 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.486229    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4f0fd3f-76a8-46ca-857d-8b4e1f662afe" containerName="kube-rbac-proxy"
Jul 01 18:58:29.486284 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.486244    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4f0fd3f-76a8-46ca-857d-8b4e1f662afe" containerName="kube-rbac-proxy"
Jul 01 18:58:29.486284 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.486258    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d99116f-f1da-4605-b102-292b79631e8f" containerName="kube-rbac-proxy"
Jul 01 18:58:29.486284 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.486266    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d99116f-f1da-4605-b102-292b79631e8f" containerName="kube-rbac-proxy"
Jul 01 18:58:29.486284 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.486274    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d99116f-f1da-4605-b102-292b79631e8f" containerName="kserve-container"
Jul 01 18:58:29.486284 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.486280    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d99116f-f1da-4605-b102-292b79631e8f" containerName="kserve-container"
Jul 01 18:58:29.486426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.486296    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4f0fd3f-76a8-46ca-857d-8b4e1f662afe" containerName="kserve-container"
Jul 01 18:58:29.486426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.486302    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4f0fd3f-76a8-46ca-857d-8b4e1f662afe" containerName="kserve-container"
Jul 01 18:58:29.486426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.486358    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d99116f-f1da-4605-b102-292b79631e8f" containerName="kube-rbac-proxy"
Jul 01 18:58:29.486426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.486365    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4f0fd3f-76a8-46ca-857d-8b4e1f662afe" containerName="kube-rbac-proxy"
Jul 01 18:58:29.486426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.486372    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d99116f-f1da-4605-b102-292b79631e8f" containerName="kserve-container"
Jul 01 18:58:29.486426 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.486377    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4f0fd3f-76a8-46ca-857d-8b4e1f662afe" containerName="kserve-container"
Jul 01 18:58:29.489400 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.489383    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl"
Jul 01 18:58:29.492144 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.492124    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-550f3-kube-rbac-proxy-sar-config\""
Jul 01 18:58:29.492227 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.492180    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-550f3-predictor-serving-cert\""
Jul 01 18:58:29.499737 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.499695    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl"]
Jul 01 18:58:29.549574 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.549538    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:58:29.568595 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.568565    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baaaead7-e419-4611-a8f1-4b336faa25b0-proxy-tls\") pod \"success-200-isvc-550f3-predictor-6564b49d87-mgttl\" (UID: \"baaaead7-e419-4611-a8f1-4b336faa25b0\") " pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl"
Jul 01 18:58:29.568735 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.568642    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baaaead7-e419-4611-a8f1-4b336faa25b0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-550f3-predictor-6564b49d87-mgttl\" (UID: \"baaaead7-e419-4611-a8f1-4b336faa25b0\") " pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl"
Jul 01 18:58:29.568735 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.568687    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln6h2\" (UniqueName: \"kubernetes.io/projected/baaaead7-e419-4611-a8f1-4b336faa25b0-kube-api-access-ln6h2\") pod \"success-200-isvc-550f3-predictor-6564b49d87-mgttl\" (UID: \"baaaead7-e419-4611-a8f1-4b336faa25b0\") " pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl"
Jul 01 18:58:29.584900 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.584868    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7"]
Jul 01 18:58:29.588093 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.588079    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7"
Jul 01 18:58:29.590616 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.590598    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-550f3-kube-rbac-proxy-sar-config\""
Jul 01 18:58:29.590725 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.590610    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-550f3-predictor-serving-cert\""
Jul 01 18:58:29.599240 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.599218    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7"]
Jul 01 18:58:29.656851 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.656781    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:58:29.669743 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.669696    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baaaead7-e419-4611-a8f1-4b336faa25b0-proxy-tls\") pod \"success-200-isvc-550f3-predictor-6564b49d87-mgttl\" (UID: \"baaaead7-e419-4611-a8f1-4b336faa25b0\") " pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl"
Jul 01 18:58:29.669874 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.669765    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wnnn\" (UniqueName: \"kubernetes.io/projected/247c587d-b6d5-4fff-a212-771964aecffe-kube-api-access-4wnnn\") pod \"error-404-isvc-550f3-predictor-787558b8bb-22gl7\" (UID: \"247c587d-b6d5-4fff-a212-771964aecffe\") " pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7"
Jul 01 18:58:29.669874 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.669797    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/247c587d-b6d5-4fff-a212-771964aecffe-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-550f3-predictor-787558b8bb-22gl7\" (UID: \"247c587d-b6d5-4fff-a212-771964aecffe\") " pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7"
Jul 01 18:58:29.669874 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.669820    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baaaead7-e419-4611-a8f1-4b336faa25b0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-550f3-predictor-6564b49d87-mgttl\" (UID: \"baaaead7-e419-4611-a8f1-4b336faa25b0\") " pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl"
Jul 01 18:58:29.669874 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.669856    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/247c587d-b6d5-4fff-a212-771964aecffe-proxy-tls\") pod \"error-404-isvc-550f3-predictor-787558b8bb-22gl7\" (UID: \"247c587d-b6d5-4fff-a212-771964aecffe\") " pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7"
Jul 01 18:58:29.669874 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.669874    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln6h2\" (UniqueName: \"kubernetes.io/projected/baaaead7-e419-4611-a8f1-4b336faa25b0-kube-api-access-ln6h2\") pod \"success-200-isvc-550f3-predictor-6564b49d87-mgttl\" (UID: \"baaaead7-e419-4611-a8f1-4b336faa25b0\") " pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl"
Jul 01 18:58:29.672320 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.672306    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-550f3-predictor-serving-cert\""
Jul 01 18:58:29.672381 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.672354    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-550f3-kube-rbac-proxy-sar-config\""
Jul 01 18:58:29.679642 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.679619    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln6h2\" (UniqueName: \"kubernetes.io/projected/baaaead7-e419-4611-a8f1-4b336faa25b0-kube-api-access-ln6h2\") pod \"success-200-isvc-550f3-predictor-6564b49d87-mgttl\" (UID: \"baaaead7-e419-4611-a8f1-4b336faa25b0\") " pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl"
Jul 01 18:58:29.680918 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.680895    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baaaead7-e419-4611-a8f1-4b336faa25b0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-550f3-predictor-6564b49d87-mgttl\" (UID: \"baaaead7-e419-4611-a8f1-4b336faa25b0\") " pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl"
Jul 01 18:58:29.682899 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.682879    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baaaead7-e419-4611-a8f1-4b336faa25b0-proxy-tls\") pod \"success-200-isvc-550f3-predictor-6564b49d87-mgttl\" (UID: \"baaaead7-e419-4611-a8f1-4b336faa25b0\") " pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl"
Jul 01 18:58:29.770884 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.770856    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/247c587d-b6d5-4fff-a212-771964aecffe-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-550f3-predictor-787558b8bb-22gl7\" (UID: \"247c587d-b6d5-4fff-a212-771964aecffe\") " pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7"
Jul 01 18:58:29.771013 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.770910    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/247c587d-b6d5-4fff-a212-771964aecffe-proxy-tls\") pod \"error-404-isvc-550f3-predictor-787558b8bb-22gl7\" (UID: \"247c587d-b6d5-4fff-a212-771964aecffe\") " pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7"
Jul 01 18:58:29.771013 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.770960    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wnnn\" (UniqueName: \"kubernetes.io/projected/247c587d-b6d5-4fff-a212-771964aecffe-kube-api-access-4wnnn\") pod \"error-404-isvc-550f3-predictor-787558b8bb-22gl7\" (UID: \"247c587d-b6d5-4fff-a212-771964aecffe\") " pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7"
Jul 01 18:58:29.771472 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.771450    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/247c587d-b6d5-4fff-a212-771964aecffe-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-550f3-predictor-787558b8bb-22gl7\" (UID: \"247c587d-b6d5-4fff-a212-771964aecffe\") " pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7"
Jul 01 18:58:29.773530 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.773508    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/247c587d-b6d5-4fff-a212-771964aecffe-proxy-tls\") pod \"error-404-isvc-550f3-predictor-787558b8bb-22gl7\" (UID: \"247c587d-b6d5-4fff-a212-771964aecffe\") " pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7"
Jul 01 18:58:29.780738 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.780700    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wnnn\" (UniqueName: \"kubernetes.io/projected/247c587d-b6d5-4fff-a212-771964aecffe-kube-api-access-4wnnn\") pod \"error-404-isvc-550f3-predictor-787558b8bb-22gl7\" (UID: \"247c587d-b6d5-4fff-a212-771964aecffe\") " pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7"
Jul 01 18:58:29.789242 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.789216    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:58:29.799486 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.799466    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl"
Jul 01 18:58:29.899867 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.899536    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7"
Jul 01 18:58:29.933691 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:29.933660    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl"]
Jul 01 18:58:29.937882 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:58:29.937834    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaaaead7_e419_4611_a8f1_4b336faa25b0.slice/crio-1b72656f25d98622f43779b78b6edc4cb8ccdbbf4df487d490a42803467a92e4 WatchSource:0}: Error finding container 1b72656f25d98622f43779b78b6edc4cb8ccdbbf4df487d490a42803467a92e4: Status 404 returned error can't find the container with id 1b72656f25d98622f43779b78b6edc4cb8ccdbbf4df487d490a42803467a92e4
Jul 01 18:58:30.046387 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:30.046326    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7"]
Jul 01 18:58:30.049994 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:58:30.049973    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247c587d_b6d5_4fff_a212_771964aecffe.slice/crio-9f163503d24208c9f1825f89a97379198fa56035f8a6ece5b5cb592bf2906466 WatchSource:0}: Error finding container 9f163503d24208c9f1825f89a97379198fa56035f8a6ece5b5cb592bf2906466: Status 404 returned error can't find the container with id 9f163503d24208c9f1825f89a97379198fa56035f8a6ece5b5cb592bf2906466
Jul 01 18:58:30.359476 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:30.359447    2573 generic.go:356] "Generic (PLEG): container finished" podID="1f74764e-8c3c-4486-a25f-1a5c2126aff6" containerID="3f37b949b9d6c6edbd19fd712513e27aaa975568e0720ac4f038959d102d5527" exitCode=2
Jul 01 18:58:30.359889 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:30.359519    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg" event={"ID":"1f74764e-8c3c-4486-a25f-1a5c2126aff6","Type":"ContainerDied","Data":"3f37b949b9d6c6edbd19fd712513e27aaa975568e0720ac4f038959d102d5527"}
Jul 01 18:58:30.361327 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:30.361302    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7" event={"ID":"247c587d-b6d5-4fff-a212-771964aecffe","Type":"ContainerStarted","Data":"50fc7c0daba4659efc3f49ad77f74d315ea0493aacc500d31cd0eafa9130d86a"}
Jul 01 18:58:30.361447 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:30.361355    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7" event={"ID":"247c587d-b6d5-4fff-a212-771964aecffe","Type":"ContainerStarted","Data":"3f189082aa0fc01d3ed385f2f219c362c4d9ca6bc0934ffa242ea9bad4669614"}
Jul 01 18:58:30.361447 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:30.361385    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7" event={"ID":"247c587d-b6d5-4fff-a212-771964aecffe","Type":"ContainerStarted","Data":"9f163503d24208c9f1825f89a97379198fa56035f8a6ece5b5cb592bf2906466"}
Jul 01 18:58:30.361447 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:30.361416    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7"
Jul 01 18:58:30.363065 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:30.363041    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl" event={"ID":"baaaead7-e419-4611-a8f1-4b336faa25b0","Type":"ContainerStarted","Data":"aca9370534bd9876052a0e71fc5603919bf0429bcd158afd5dffc098429f2bab"}
Jul 01 18:58:30.363194 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:30.363100    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl" event={"ID":"baaaead7-e419-4611-a8f1-4b336faa25b0","Type":"ContainerStarted","Data":"6fe83dba8158223a5a35293e50a37785f161f33f6b8be0b48c621461ec45cf61"}
Jul 01 18:58:30.363194 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:30.363139    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl" event={"ID":"baaaead7-e419-4611-a8f1-4b336faa25b0","Type":"ContainerStarted","Data":"1b72656f25d98622f43779b78b6edc4cb8ccdbbf4df487d490a42803467a92e4"}
Jul 01 18:58:30.363194 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:30.363176    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl"
Jul 01 18:58:30.363368 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:30.363207    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl"
Jul 01 18:58:30.364832 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:30.364799    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl" podUID="baaaead7-e419-4611-a8f1-4b336faa25b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:58:30.364993 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:30.364972    2573 generic.go:356] "Generic (PLEG): container finished" podID="8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0" containerID="b1b672e0d9638e1dab17df87627f4980b5bce57396660fa903484c3878997a76" exitCode=2
Jul 01 18:58:30.365054 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:30.364992    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p" event={"ID":"8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0","Type":"ContainerDied","Data":"b1b672e0d9638e1dab17df87627f4980b5bce57396660fa903484c3878997a76"}
Jul 01 18:58:30.380185 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:30.380134    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7" podStartSLOduration=1.38011639 podStartE2EDuration="1.38011639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:58:30.37967442 +0000 UTC m=+1261.148512020" watchObservedRunningTime="2026-07-01 18:58:30.38011639 +0000 UTC m=+1261.148953994"
Jul 01 18:58:30.400168 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:30.400121    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl" podStartSLOduration=1.400102951 podStartE2EDuration="1.400102951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:58:30.398413694 +0000 UTC m=+1261.167251320" watchObservedRunningTime="2026-07-01 18:58:30.400102951 +0000 UTC m=+1261.168940556"
Jul 01 18:58:31.368935 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:31.368904    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7"
Jul 01 18:58:31.369294 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:31.368972    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl" podUID="baaaead7-e419-4611-a8f1-4b336faa25b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:58:31.370189 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:31.370168    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7" podUID="247c587d-b6d5-4fff-a212-771964aecffe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 18:58:32.281189 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.281140    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg" podUID="1f74764e-8c3c-4486-a25f-1a5c2126aff6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 01 18:58:32.281357 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.281155    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p" podUID="8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 01 18:58:32.285482 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.285460    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg" podUID="1f74764e-8c3c-4486-a25f-1a5c2126aff6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 18:58:32.285572 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.285468    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p" podUID="8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 01 18:58:32.372448 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.372412    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7" podUID="247c587d-b6d5-4fff-a212-771964aecffe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 18:58:32.558505 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.558479    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p"
Jul 01 18:58:32.593672 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.593650    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6zrt\" (UniqueName: \"kubernetes.io/projected/8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0-kube-api-access-r6zrt\") pod \"8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0\" (UID: \"8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0\") "
Jul 01 18:58:32.593767 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.593755    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0-kube-rbac-proxy-sar-config\") pod \"8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0\" (UID: \"8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0\") "
Jul 01 18:58:32.593822 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.593783    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0-proxy-tls\") pod \"8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0\" (UID: \"8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0\") "
Jul 01 18:58:32.594126 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.594098    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0" (UID: "8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:58:32.595857 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.595831    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0" (UID: "8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:58:32.595959 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.595850    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0-kube-api-access-r6zrt" (OuterVolumeSpecName: "kube-api-access-r6zrt") pod "8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0" (UID: "8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0"). InnerVolumeSpecName "kube-api-access-r6zrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:58:32.601630 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.601610    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg"
Jul 01 18:58:32.694611 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.694578    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpk9c\" (UniqueName: \"kubernetes.io/projected/1f74764e-8c3c-4486-a25f-1a5c2126aff6-kube-api-access-cpk9c\") pod \"1f74764e-8c3c-4486-a25f-1a5c2126aff6\" (UID: \"1f74764e-8c3c-4486-a25f-1a5c2126aff6\") "
Jul 01 18:58:32.694821 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.694633    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f74764e-8c3c-4486-a25f-1a5c2126aff6-kube-rbac-proxy-sar-config\") pod \"1f74764e-8c3c-4486-a25f-1a5c2126aff6\" (UID: \"1f74764e-8c3c-4486-a25f-1a5c2126aff6\") "
Jul 01 18:58:32.694821 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.694667    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f74764e-8c3c-4486-a25f-1a5c2126aff6-proxy-tls\") pod \"1f74764e-8c3c-4486-a25f-1a5c2126aff6\" (UID: \"1f74764e-8c3c-4486-a25f-1a5c2126aff6\") "
Jul 01 18:58:32.694821 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.694810    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6zrt\" (UniqueName: \"kubernetes.io/projected/8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0-kube-api-access-r6zrt\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:58:32.694995 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.694825    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:58:32.694995 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.694840    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0-proxy-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:58:32.695429 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.695075    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f74764e-8c3c-4486-a25f-1a5c2126aff6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1f74764e-8c3c-4486-a25f-1a5c2126aff6" (UID: "1f74764e-8c3c-4486-a25f-1a5c2126aff6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:58:32.696692 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.696669    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f74764e-8c3c-4486-a25f-1a5c2126aff6-kube-api-access-cpk9c" (OuterVolumeSpecName: "kube-api-access-cpk9c") pod "1f74764e-8c3c-4486-a25f-1a5c2126aff6" (UID: "1f74764e-8c3c-4486-a25f-1a5c2126aff6"). InnerVolumeSpecName "kube-api-access-cpk9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:58:32.696692 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.696679    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f74764e-8c3c-4486-a25f-1a5c2126aff6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1f74764e-8c3c-4486-a25f-1a5c2126aff6" (UID: "1f74764e-8c3c-4486-a25f-1a5c2126aff6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:58:32.795496 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.795420    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f74764e-8c3c-4486-a25f-1a5c2126aff6-proxy-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:58:32.795496 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.795443    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpk9c\" (UniqueName: \"kubernetes.io/projected/1f74764e-8c3c-4486-a25f-1a5c2126aff6-kube-api-access-cpk9c\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:58:32.795496 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:32.795452    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f74764e-8c3c-4486-a25f-1a5c2126aff6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:58:33.377918 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.377888    2573 generic.go:356] "Generic (PLEG): container finished" podID="1f74764e-8c3c-4486-a25f-1a5c2126aff6" containerID="9199312a1218a432b2aa5a898c42d3ba34f48a3ed7df87e63d6997ea56b31c7d" exitCode=0
Jul 01 18:58:33.378297 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.377940    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg"
Jul 01 18:58:33.378297 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.377973    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg" event={"ID":"1f74764e-8c3c-4486-a25f-1a5c2126aff6","Type":"ContainerDied","Data":"9199312a1218a432b2aa5a898c42d3ba34f48a3ed7df87e63d6997ea56b31c7d"}
Jul 01 18:58:33.378297 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.378030    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg" event={"ID":"1f74764e-8c3c-4486-a25f-1a5c2126aff6","Type":"ContainerDied","Data":"5dea3b78ea1ee7480464559d3fc9c8393810bf7ffab452888762989aa8f0f8fb"}
Jul 01 18:58:33.378297 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.378066    2573 scope.go:117] "RemoveContainer" containerID="3f37b949b9d6c6edbd19fd712513e27aaa975568e0720ac4f038959d102d5527"
Jul 01 18:58:33.379578 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.379553    2573 generic.go:356] "Generic (PLEG): container finished" podID="8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0" containerID="1f8aae326bd00971845ff387b36c74324186b6dba73d49ea3ca4c647fb51f380" exitCode=0
Jul 01 18:58:33.379693 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.379601    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p"
Jul 01 18:58:33.379693 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.379603    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p" event={"ID":"8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0","Type":"ContainerDied","Data":"1f8aae326bd00971845ff387b36c74324186b6dba73d49ea3ca4c647fb51f380"}
Jul 01 18:58:33.379832 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.379731    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p" event={"ID":"8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0","Type":"ContainerDied","Data":"1145fe4d6783409f3be410da792aeb7c371739cea80035dcde7d583cdd4b36fe"}
Jul 01 18:58:33.387150 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.387127    2573 scope.go:117] "RemoveContainer" containerID="9199312a1218a432b2aa5a898c42d3ba34f48a3ed7df87e63d6997ea56b31c7d"
Jul 01 18:58:33.394493 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.394478    2573 scope.go:117] "RemoveContainer" containerID="3f37b949b9d6c6edbd19fd712513e27aaa975568e0720ac4f038959d102d5527"
Jul 01 18:58:33.394745 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:58:33.394702    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f37b949b9d6c6edbd19fd712513e27aaa975568e0720ac4f038959d102d5527\": container with ID starting with 3f37b949b9d6c6edbd19fd712513e27aaa975568e0720ac4f038959d102d5527 not found: ID does not exist" containerID="3f37b949b9d6c6edbd19fd712513e27aaa975568e0720ac4f038959d102d5527"
Jul 01 18:58:33.394793 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.394746    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f37b949b9d6c6edbd19fd712513e27aaa975568e0720ac4f038959d102d5527"} err="failed to get container status \"3f37b949b9d6c6edbd19fd712513e27aaa975568e0720ac4f038959d102d5527\": rpc error: code = NotFound desc = could not find container \"3f37b949b9d6c6edbd19fd712513e27aaa975568e0720ac4f038959d102d5527\": container with ID starting with 3f37b949b9d6c6edbd19fd712513e27aaa975568e0720ac4f038959d102d5527 not found: ID does not exist"
Jul 01 18:58:33.394793 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.394764    2573 scope.go:117] "RemoveContainer" containerID="9199312a1218a432b2aa5a898c42d3ba34f48a3ed7df87e63d6997ea56b31c7d"
Jul 01 18:58:33.395012 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:58:33.394992    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9199312a1218a432b2aa5a898c42d3ba34f48a3ed7df87e63d6997ea56b31c7d\": container with ID starting with 9199312a1218a432b2aa5a898c42d3ba34f48a3ed7df87e63d6997ea56b31c7d not found: ID does not exist" containerID="9199312a1218a432b2aa5a898c42d3ba34f48a3ed7df87e63d6997ea56b31c7d"
Jul 01 18:58:33.395044 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.395019    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9199312a1218a432b2aa5a898c42d3ba34f48a3ed7df87e63d6997ea56b31c7d"} err="failed to get container status \"9199312a1218a432b2aa5a898c42d3ba34f48a3ed7df87e63d6997ea56b31c7d\": rpc error: code = NotFound desc = could not find container \"9199312a1218a432b2aa5a898c42d3ba34f48a3ed7df87e63d6997ea56b31c7d\": container with ID starting with 9199312a1218a432b2aa5a898c42d3ba34f48a3ed7df87e63d6997ea56b31c7d not found: ID does not exist"
Jul 01 18:58:33.395044 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.395034    2573 scope.go:117] "RemoveContainer" containerID="b1b672e0d9638e1dab17df87627f4980b5bce57396660fa903484c3878997a76"
Jul 01 18:58:33.401749 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.401730    2573 scope.go:117] "RemoveContainer" containerID="1f8aae326bd00971845ff387b36c74324186b6dba73d49ea3ca4c647fb51f380"
Jul 01 18:58:33.404157 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.404136    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p"]
Jul 01 18:58:33.408101 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.408082    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eb211-predictor-6695c85886-hvd2p"]
Jul 01 18:58:33.410242 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.410127    2573 scope.go:117] "RemoveContainer" containerID="b1b672e0d9638e1dab17df87627f4980b5bce57396660fa903484c3878997a76"
Jul 01 18:58:33.410356 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:58:33.410340    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1b672e0d9638e1dab17df87627f4980b5bce57396660fa903484c3878997a76\": container with ID starting with b1b672e0d9638e1dab17df87627f4980b5bce57396660fa903484c3878997a76 not found: ID does not exist" containerID="b1b672e0d9638e1dab17df87627f4980b5bce57396660fa903484c3878997a76"
Jul 01 18:58:33.410403 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.410362    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b672e0d9638e1dab17df87627f4980b5bce57396660fa903484c3878997a76"} err="failed to get container status \"b1b672e0d9638e1dab17df87627f4980b5bce57396660fa903484c3878997a76\": rpc error: code = NotFound desc = could not find container \"b1b672e0d9638e1dab17df87627f4980b5bce57396660fa903484c3878997a76\": container with ID starting with b1b672e0d9638e1dab17df87627f4980b5bce57396660fa903484c3878997a76 not found: ID does not exist"
Jul 01 18:58:33.410403 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.410377    2573 scope.go:117] "RemoveContainer" containerID="1f8aae326bd00971845ff387b36c74324186b6dba73d49ea3ca4c647fb51f380"
Jul 01 18:58:33.410619 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:58:33.410602    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f8aae326bd00971845ff387b36c74324186b6dba73d49ea3ca4c647fb51f380\": container with ID starting with 1f8aae326bd00971845ff387b36c74324186b6dba73d49ea3ca4c647fb51f380 not found: ID does not exist" containerID="1f8aae326bd00971845ff387b36c74324186b6dba73d49ea3ca4c647fb51f380"
Jul 01 18:58:33.410657 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.410625    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8aae326bd00971845ff387b36c74324186b6dba73d49ea3ca4c647fb51f380"} err="failed to get container status \"1f8aae326bd00971845ff387b36c74324186b6dba73d49ea3ca4c647fb51f380\": rpc error: code = NotFound desc = could not find container \"1f8aae326bd00971845ff387b36c74324186b6dba73d49ea3ca4c647fb51f380\": container with ID starting with 1f8aae326bd00971845ff387b36c74324186b6dba73d49ea3ca4c647fb51f380 not found: ID does not exist"
Jul 01 18:58:33.419873 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.419852    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg"]
Jul 01 18:58:33.424622 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.424604    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-eb211-predictor-69b49c778c-qn8vg"]
Jul 01 18:58:33.666101 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.666008    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f74764e-8c3c-4486-a25f-1a5c2126aff6" path="/var/lib/kubelet/pods/1f74764e-8c3c-4486-a25f-1a5c2126aff6/volumes"
Jul 01 18:58:33.666514 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.666494    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0" path="/var/lib/kubelet/pods/8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0/volumes"
Jul 01 18:58:33.694228 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.694199    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:58:33.834296 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:33.834260    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:58:36.373421 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:36.373396    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl"
Jul 01 18:58:36.373825 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:36.373740    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl" podUID="baaaead7-e419-4611-a8f1-4b336faa25b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:58:37.376907 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:37.376879    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7"
Jul 01 18:58:37.377327 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:37.377297    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7" podUID="247c587d-b6d5-4fff-a212-771964aecffe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 18:58:43.766150 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.766118    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:58:43.832753 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.832684    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44"]
Jul 01 18:58:43.832964 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.832937    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44" podUID="8000bfc6-d3d5-4f73-b381-69e464c77c0d" containerName="kserve-container" containerID="cri-o://9950ea29ebc677393aee0769955f68e930fcb0aeefcf53d0f97fb1df1fcccf66" gracePeriod=30
Jul 01 18:58:43.833042 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.832983    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44" podUID="8000bfc6-d3d5-4f73-b381-69e464c77c0d" containerName="kube-rbac-proxy" containerID="cri-o://a8cc31cc203783f20d1f8e10b2fa1e7cb60e8dad069a574e472f2e4536e126c5" gracePeriod=30
Jul 01 18:58:43.860138 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.860098    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:58:43.891766 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.891725    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5"]
Jul 01 18:58:43.892344 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.891992    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5" podUID="d26e74b3-f903-4028-bf9f-ec2bb2dea9a8" containerName="kserve-container" containerID="cri-o://373059822d6ba30452329de4ec7fb59c20cc792954fb1aa3633466adce16650d" gracePeriod=30
Jul 01 18:58:43.892344 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.892268    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5" podUID="d26e74b3-f903-4028-bf9f-ec2bb2dea9a8" containerName="kube-rbac-proxy" containerID="cri-o://0b032a7c9f44ff299f65b5fd8082cf6a19b4fcd527fe564bb598e8137754e1a8" gracePeriod=30
Jul 01 18:58:43.902463 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.902437    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw"]
Jul 01 18:58:43.902769 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.902752    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f74764e-8c3c-4486-a25f-1a5c2126aff6" containerName="kube-rbac-proxy"
Jul 01 18:58:43.902769 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.902767    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f74764e-8c3c-4486-a25f-1a5c2126aff6" containerName="kube-rbac-proxy"
Jul 01 18:58:43.902884 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.902779    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0" containerName="kube-rbac-proxy"
Jul 01 18:58:43.902884 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.902785    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0" containerName="kube-rbac-proxy"
Jul 01 18:58:43.902884 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.902794    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f74764e-8c3c-4486-a25f-1a5c2126aff6" containerName="kserve-container"
Jul 01 18:58:43.902884 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.902799    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f74764e-8c3c-4486-a25f-1a5c2126aff6" containerName="kserve-container"
Jul 01 18:58:43.902884 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.902806    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0" containerName="kserve-container"
Jul 01 18:58:43.902884 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.902810    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0" containerName="kserve-container"
Jul 01 18:58:43.902884 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.902867    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0" containerName="kube-rbac-proxy"
Jul 01 18:58:43.902884 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.902875    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f74764e-8c3c-4486-a25f-1a5c2126aff6" containerName="kserve-container"
Jul 01 18:58:43.902884 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.902882    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8265b84a-2ed0-4bd9-a163-6cd2f9c1b9d0" containerName="kserve-container"
Jul 01 18:58:43.902884 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.902888    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f74764e-8c3c-4486-a25f-1a5c2126aff6" containerName="kube-rbac-proxy"
Jul 01 18:58:43.907391 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.907371    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw"
Jul 01 18:58:43.909745 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.909701    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-af621-predictor-serving-cert\""
Jul 01 18:58:43.909844 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.909767    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-af621-kube-rbac-proxy-sar-config\""
Jul 01 18:58:43.916460 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.916413    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw"]
Jul 01 18:58:43.936313 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.936284    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:58:43.984637 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.984598    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f36625f-48ef-4d40-94d1-85d3370da2d4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-af621-predictor-688d7d5cb9-p4nrw\" (UID: \"7f36625f-48ef-4d40-94d1-85d3370da2d4\") " pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw"
Jul 01 18:58:43.984834 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.984657    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f36625f-48ef-4d40-94d1-85d3370da2d4-proxy-tls\") pod \"success-200-isvc-af621-predictor-688d7d5cb9-p4nrw\" (UID: \"7f36625f-48ef-4d40-94d1-85d3370da2d4\") " pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw"
Jul 01 18:58:43.984834 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:43.984755    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pkqf\" (UniqueName: \"kubernetes.io/projected/7f36625f-48ef-4d40-94d1-85d3370da2d4-kube-api-access-4pkqf\") pod \"success-200-isvc-af621-predictor-688d7d5cb9-p4nrw\" (UID: \"7f36625f-48ef-4d40-94d1-85d3370da2d4\") " pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw"
Jul 01 18:58:44.006757 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.006731    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd"]
Jul 01 18:58:44.007565 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.007541    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:58:44.010491 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.010470    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd"
Jul 01 18:58:44.012771 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.012753    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-af621-predictor-serving-cert\""
Jul 01 18:58:44.012960 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.012914    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-af621-kube-rbac-proxy-sar-config\""
Jul 01 18:58:44.020838 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.020814    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd"]
Jul 01 18:58:44.085479 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.085449    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcffa820-58d6-4ab1-8975-e6adcaf9f0de-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd\" (UID: \"dcffa820-58d6-4ab1-8975-e6adcaf9f0de\") " pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd"
Jul 01 18:58:44.085479 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.085484    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f36625f-48ef-4d40-94d1-85d3370da2d4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-af621-predictor-688d7d5cb9-p4nrw\" (UID: \"7f36625f-48ef-4d40-94d1-85d3370da2d4\") " pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw"
Jul 01 18:58:44.085743 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.085508    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f36625f-48ef-4d40-94d1-85d3370da2d4-proxy-tls\") pod \"success-200-isvc-af621-predictor-688d7d5cb9-p4nrw\" (UID: \"7f36625f-48ef-4d40-94d1-85d3370da2d4\") " pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw"
Jul 01 18:58:44.085743 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.085564    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pkqf\" (UniqueName: \"kubernetes.io/projected/7f36625f-48ef-4d40-94d1-85d3370da2d4-kube-api-access-4pkqf\") pod \"success-200-isvc-af621-predictor-688d7d5cb9-p4nrw\" (UID: \"7f36625f-48ef-4d40-94d1-85d3370da2d4\") " pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw"
Jul 01 18:58:44.085743 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:58:44.085607    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-af621-predictor-serving-cert: secret "success-200-isvc-af621-predictor-serving-cert" not found
Jul 01 18:58:44.085743 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.085618    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56kd4\" (UniqueName: \"kubernetes.io/projected/dcffa820-58d6-4ab1-8975-e6adcaf9f0de-kube-api-access-56kd4\") pod \"error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd\" (UID: \"dcffa820-58d6-4ab1-8975-e6adcaf9f0de\") " pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd"
Jul 01 18:58:44.085743 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:58:44.085663    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f36625f-48ef-4d40-94d1-85d3370da2d4-proxy-tls podName:7f36625f-48ef-4d40-94d1-85d3370da2d4 nodeName:}" failed. No retries permitted until 2026-07-01 18:58:44.585643633 +0000 UTC m=+1275.354481213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7f36625f-48ef-4d40-94d1-85d3370da2d4-proxy-tls") pod "success-200-isvc-af621-predictor-688d7d5cb9-p4nrw" (UID: "7f36625f-48ef-4d40-94d1-85d3370da2d4") : secret "success-200-isvc-af621-predictor-serving-cert" not found
Jul 01 18:58:44.085987 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.085767    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcffa820-58d6-4ab1-8975-e6adcaf9f0de-proxy-tls\") pod \"error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd\" (UID: \"dcffa820-58d6-4ab1-8975-e6adcaf9f0de\") " pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd"
Jul 01 18:58:44.086178 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.086160    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f36625f-48ef-4d40-94d1-85d3370da2d4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-af621-predictor-688d7d5cb9-p4nrw\" (UID: \"7f36625f-48ef-4d40-94d1-85d3370da2d4\") " pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw"
Jul 01 18:58:44.096007 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.095963    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pkqf\" (UniqueName: \"kubernetes.io/projected/7f36625f-48ef-4d40-94d1-85d3370da2d4-kube-api-access-4pkqf\") pod \"success-200-isvc-af621-predictor-688d7d5cb9-p4nrw\" (UID: \"7f36625f-48ef-4d40-94d1-85d3370da2d4\") " pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw"
Jul 01 18:58:44.186870 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.186837    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56kd4\" (UniqueName: \"kubernetes.io/projected/dcffa820-58d6-4ab1-8975-e6adcaf9f0de-kube-api-access-56kd4\") pod \"error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd\" (UID: \"dcffa820-58d6-4ab1-8975-e6adcaf9f0de\") " pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd"
Jul 01 18:58:44.187034 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.186885    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcffa820-58d6-4ab1-8975-e6adcaf9f0de-proxy-tls\") pod \"error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd\" (UID: \"dcffa820-58d6-4ab1-8975-e6adcaf9f0de\") " pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd"
Jul 01 18:58:44.187034 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.186920    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcffa820-58d6-4ab1-8975-e6adcaf9f0de-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd\" (UID: \"dcffa820-58d6-4ab1-8975-e6adcaf9f0de\") " pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd"
Jul 01 18:58:44.187166 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:58:44.187084    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-af621-predictor-serving-cert: secret "error-404-isvc-af621-predictor-serving-cert" not found
Jul 01 18:58:44.187166 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:58:44.187150    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcffa820-58d6-4ab1-8975-e6adcaf9f0de-proxy-tls podName:dcffa820-58d6-4ab1-8975-e6adcaf9f0de nodeName:}" failed. No retries permitted until 2026-07-01 18:58:44.68713042 +0000 UTC m=+1275.455968003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dcffa820-58d6-4ab1-8975-e6adcaf9f0de-proxy-tls") pod "error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd" (UID: "dcffa820-58d6-4ab1-8975-e6adcaf9f0de") : secret "error-404-isvc-af621-predictor-serving-cert" not found
Jul 01 18:58:44.187551 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.187533    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcffa820-58d6-4ab1-8975-e6adcaf9f0de-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd\" (UID: \"dcffa820-58d6-4ab1-8975-e6adcaf9f0de\") " pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd"
Jul 01 18:58:44.196215 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.196184    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56kd4\" (UniqueName: \"kubernetes.io/projected/dcffa820-58d6-4ab1-8975-e6adcaf9f0de-kube-api-access-56kd4\") pod \"error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd\" (UID: \"dcffa820-58d6-4ab1-8975-e6adcaf9f0de\") " pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd"
Jul 01 18:58:44.287445 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.287362    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:58:44.429033 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.429002    2573 generic.go:356] "Generic (PLEG): container finished" podID="d26e74b3-f903-4028-bf9f-ec2bb2dea9a8" containerID="0b032a7c9f44ff299f65b5fd8082cf6a19b4fcd527fe564bb598e8137754e1a8" exitCode=2
Jul 01 18:58:44.429185 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.429069    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5" event={"ID":"d26e74b3-f903-4028-bf9f-ec2bb2dea9a8","Type":"ContainerDied","Data":"0b032a7c9f44ff299f65b5fd8082cf6a19b4fcd527fe564bb598e8137754e1a8"}
Jul 01 18:58:44.430948 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.430914    2573 generic.go:356] "Generic (PLEG): container finished" podID="8000bfc6-d3d5-4f73-b381-69e464c77c0d" containerID="a8cc31cc203783f20d1f8e10b2fa1e7cb60e8dad069a574e472f2e4536e126c5" exitCode=2
Jul 01 18:58:44.431080 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.430974    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44" event={"ID":"8000bfc6-d3d5-4f73-b381-69e464c77c0d","Type":"ContainerDied","Data":"a8cc31cc203783f20d1f8e10b2fa1e7cb60e8dad069a574e472f2e4536e126c5"}
Jul 01 18:58:44.445299 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.445270    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:58:44.589263 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.589176    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f36625f-48ef-4d40-94d1-85d3370da2d4-proxy-tls\") pod \"success-200-isvc-af621-predictor-688d7d5cb9-p4nrw\" (UID: \"7f36625f-48ef-4d40-94d1-85d3370da2d4\") " pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw"
Jul 01 18:58:44.591589 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.591558    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f36625f-48ef-4d40-94d1-85d3370da2d4-proxy-tls\") pod \"success-200-isvc-af621-predictor-688d7d5cb9-p4nrw\" (UID: \"7f36625f-48ef-4d40-94d1-85d3370da2d4\") " pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw"
Jul 01 18:58:44.689807 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.689773    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcffa820-58d6-4ab1-8975-e6adcaf9f0de-proxy-tls\") pod \"error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd\" (UID: \"dcffa820-58d6-4ab1-8975-e6adcaf9f0de\") " pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd"
Jul 01 18:58:44.692135 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.692105    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcffa820-58d6-4ab1-8975-e6adcaf9f0de-proxy-tls\") pod \"error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd\" (UID: \"dcffa820-58d6-4ab1-8975-e6adcaf9f0de\") " pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd"
Jul 01 18:58:44.819602 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.819566    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw"
Jul 01 18:58:44.922838 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.922791    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd"
Jul 01 18:58:44.957405 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:44.957373    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw"]
Jul 01 18:58:44.959968 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:58:44.959942    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f36625f_48ef_4d40_94d1_85d3370da2d4.slice/crio-173f1065fbaa2d679c1db0159450720a8b8f7903f5a3da48441b6b62d3919c54 WatchSource:0}: Error finding container 173f1065fbaa2d679c1db0159450720a8b8f7903f5a3da48441b6b62d3919c54: Status 404 returned error can't find the container with id 173f1065fbaa2d679c1db0159450720a8b8f7903f5a3da48441b6b62d3919c54
Jul 01 18:58:45.077118 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:45.077097    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd"]
Jul 01 18:58:45.079331 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:58:45.079305    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcffa820_58d6_4ab1_8975_e6adcaf9f0de.slice/crio-fb5e003cb37d821e7a446ac49f0d0248c683924458fe8573f8f803208dace778 WatchSource:0}: Error finding container fb5e003cb37d821e7a446ac49f0d0248c683924458fe8573f8f803208dace778: Status 404 returned error can't find the container with id fb5e003cb37d821e7a446ac49f0d0248c683924458fe8573f8f803208dace778
Jul 01 18:58:45.440105 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:45.440013    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw" event={"ID":"7f36625f-48ef-4d40-94d1-85d3370da2d4","Type":"ContainerStarted","Data":"d2a79b5db9d4fab835c5ffeb5e6c2010e0474d3bff5a89da83cc7f80da9cf215"}
Jul 01 18:58:45.440105 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:45.440096    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw" event={"ID":"7f36625f-48ef-4d40-94d1-85d3370da2d4","Type":"ContainerStarted","Data":"4136e918c7801fec02bf75b3ee33c05e674f631729fb5070567aa180470d0a49"}
Jul 01 18:58:45.440361 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:45.440126    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw" event={"ID":"7f36625f-48ef-4d40-94d1-85d3370da2d4","Type":"ContainerStarted","Data":"173f1065fbaa2d679c1db0159450720a8b8f7903f5a3da48441b6b62d3919c54"}
Jul 01 18:58:45.441009 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:45.440982    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw"
Jul 01 18:58:45.441134 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:45.441036    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw"
Jul 01 18:58:45.442058 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:45.442032    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw" podUID="7f36625f-48ef-4d40-94d1-85d3370da2d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 18:58:45.443827 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:45.443805    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd" event={"ID":"dcffa820-58d6-4ab1-8975-e6adcaf9f0de","Type":"ContainerStarted","Data":"2012716c0c4add2e580c55306448aa32acf50a0d45886c457dc849e453b8bd38"}
Jul 01 18:58:45.443900 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:45.443849    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd" event={"ID":"dcffa820-58d6-4ab1-8975-e6adcaf9f0de","Type":"ContainerStarted","Data":"f7d531e11506969ef31d53c00539ff8f9a027a295cf26d5b54958a5d87695e47"}
Jul 01 18:58:45.443900 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:45.443870    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd" event={"ID":"dcffa820-58d6-4ab1-8975-e6adcaf9f0de","Type":"ContainerStarted","Data":"fb5e003cb37d821e7a446ac49f0d0248c683924458fe8573f8f803208dace778"}
Jul 01 18:58:45.443900 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:45.443893    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd"
Jul 01 18:58:45.461421 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:45.461378    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw" podStartSLOduration=2.461365571 podStartE2EDuration="2.461365571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:58:45.460127288 +0000 UTC m=+1276.228964891" watchObservedRunningTime="2026-07-01 18:58:45.461365571 +0000 UTC m=+1276.230203173"
Jul 01 18:58:45.480227 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:45.480176    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd" podStartSLOduration=2.480163049 podStartE2EDuration="2.480163049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:58:45.478671738 +0000 UTC m=+1276.247509344" watchObservedRunningTime="2026-07-01 18:58:45.480163049 +0000 UTC m=+1276.249000653"
Jul 01 18:58:46.137769 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:46.137702    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5" podUID="d26e74b3-f903-4028-bf9f-ec2bb2dea9a8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 01 18:58:46.374343 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:46.374308    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl" podUID="baaaead7-e419-4611-a8f1-4b336faa25b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:58:46.448152 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:46.448067    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw" podUID="7f36625f-48ef-4d40-94d1-85d3370da2d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 18:58:46.448306 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:46.448276    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd"
Jul 01 18:58:46.449466 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:46.449441    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd" podUID="dcffa820-58d6-4ab1-8975-e6adcaf9f0de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:58:47.145976 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.145940    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44" podUID="8000bfc6-d3d5-4f73-b381-69e464c77c0d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 01 18:58:47.287267 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.287239    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44"
Jul 01 18:58:47.377320 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.377280    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7" podUID="247c587d-b6d5-4fff-a212-771964aecffe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 18:58:47.415630 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.415545    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmv7t\" (UniqueName: \"kubernetes.io/projected/8000bfc6-d3d5-4f73-b381-69e464c77c0d-kube-api-access-hmv7t\") pod \"8000bfc6-d3d5-4f73-b381-69e464c77c0d\" (UID: \"8000bfc6-d3d5-4f73-b381-69e464c77c0d\") "
Jul 01 18:58:47.415630 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.415585    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8000bfc6-d3d5-4f73-b381-69e464c77c0d-proxy-tls\") pod \"8000bfc6-d3d5-4f73-b381-69e464c77c0d\" (UID: \"8000bfc6-d3d5-4f73-b381-69e464c77c0d\") "
Jul 01 18:58:47.415630 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.415616    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8000bfc6-d3d5-4f73-b381-69e464c77c0d-kube-rbac-proxy-sar-config\") pod \"8000bfc6-d3d5-4f73-b381-69e464c77c0d\" (UID: \"8000bfc6-d3d5-4f73-b381-69e464c77c0d\") "
Jul 01 18:58:47.416023 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.415991    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8000bfc6-d3d5-4f73-b381-69e464c77c0d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8000bfc6-d3d5-4f73-b381-69e464c77c0d" (UID: "8000bfc6-d3d5-4f73-b381-69e464c77c0d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:58:47.417671 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.417641    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8000bfc6-d3d5-4f73-b381-69e464c77c0d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8000bfc6-d3d5-4f73-b381-69e464c77c0d" (UID: "8000bfc6-d3d5-4f73-b381-69e464c77c0d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:58:47.417671 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.417665    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8000bfc6-d3d5-4f73-b381-69e464c77c0d-kube-api-access-hmv7t" (OuterVolumeSpecName: "kube-api-access-hmv7t") pod "8000bfc6-d3d5-4f73-b381-69e464c77c0d" (UID: "8000bfc6-d3d5-4f73-b381-69e464c77c0d"). InnerVolumeSpecName "kube-api-access-hmv7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:58:47.454631 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.454600    2573 generic.go:356] "Generic (PLEG): container finished" podID="8000bfc6-d3d5-4f73-b381-69e464c77c0d" containerID="9950ea29ebc677393aee0769955f68e930fcb0aeefcf53d0f97fb1df1fcccf66" exitCode=0
Jul 01 18:58:47.454838 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.454650    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44"
Jul 01 18:58:47.454838 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.454688    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44" event={"ID":"8000bfc6-d3d5-4f73-b381-69e464c77c0d","Type":"ContainerDied","Data":"9950ea29ebc677393aee0769955f68e930fcb0aeefcf53d0f97fb1df1fcccf66"}
Jul 01 18:58:47.454838 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.454799    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44" event={"ID":"8000bfc6-d3d5-4f73-b381-69e464c77c0d","Type":"ContainerDied","Data":"c8d56bd9284327219bf1992f86b0556ada6696496fd40649d94d81b8ba3ab5b9"}
Jul 01 18:58:47.454838 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.454836    2573 scope.go:117] "RemoveContainer" containerID="a8cc31cc203783f20d1f8e10b2fa1e7cb60e8dad069a574e472f2e4536e126c5"
Jul 01 18:58:47.455122 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.455093    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd" podUID="dcffa820-58d6-4ab1-8975-e6adcaf9f0de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:58:47.455384 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.455353    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw" podUID="7f36625f-48ef-4d40-94d1-85d3370da2d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 18:58:47.463496 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.463481    2573 scope.go:117] "RemoveContainer" containerID="9950ea29ebc677393aee0769955f68e930fcb0aeefcf53d0f97fb1df1fcccf66"
Jul 01 18:58:47.470494 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.470477    2573 scope.go:117] "RemoveContainer" containerID="a8cc31cc203783f20d1f8e10b2fa1e7cb60e8dad069a574e472f2e4536e126c5"
Jul 01 18:58:47.470759 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:58:47.470740    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8cc31cc203783f20d1f8e10b2fa1e7cb60e8dad069a574e472f2e4536e126c5\": container with ID starting with a8cc31cc203783f20d1f8e10b2fa1e7cb60e8dad069a574e472f2e4536e126c5 not found: ID does not exist" containerID="a8cc31cc203783f20d1f8e10b2fa1e7cb60e8dad069a574e472f2e4536e126c5"
Jul 01 18:58:47.470833 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.470767    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8cc31cc203783f20d1f8e10b2fa1e7cb60e8dad069a574e472f2e4536e126c5"} err="failed to get container status \"a8cc31cc203783f20d1f8e10b2fa1e7cb60e8dad069a574e472f2e4536e126c5\": rpc error: code = NotFound desc = could not find container \"a8cc31cc203783f20d1f8e10b2fa1e7cb60e8dad069a574e472f2e4536e126c5\": container with ID starting with a8cc31cc203783f20d1f8e10b2fa1e7cb60e8dad069a574e472f2e4536e126c5 not found: ID does not exist"
Jul 01 18:58:47.470833 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.470784    2573 scope.go:117] "RemoveContainer" containerID="9950ea29ebc677393aee0769955f68e930fcb0aeefcf53d0f97fb1df1fcccf66"
Jul 01 18:58:47.471026 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:58:47.471010    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9950ea29ebc677393aee0769955f68e930fcb0aeefcf53d0f97fb1df1fcccf66\": container with ID starting with 9950ea29ebc677393aee0769955f68e930fcb0aeefcf53d0f97fb1df1fcccf66 not found: ID does not exist" containerID="9950ea29ebc677393aee0769955f68e930fcb0aeefcf53d0f97fb1df1fcccf66"
Jul 01 18:58:47.471075 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.471029    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9950ea29ebc677393aee0769955f68e930fcb0aeefcf53d0f97fb1df1fcccf66"} err="failed to get container status \"9950ea29ebc677393aee0769955f68e930fcb0aeefcf53d0f97fb1df1fcccf66\": rpc error: code = NotFound desc = could not find container \"9950ea29ebc677393aee0769955f68e930fcb0aeefcf53d0f97fb1df1fcccf66\": container with ID starting with 9950ea29ebc677393aee0769955f68e930fcb0aeefcf53d0f97fb1df1fcccf66 not found: ID does not exist"
Jul 01 18:58:47.480363 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.480340    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44"]
Jul 01 18:58:47.482670 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.482651    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c059b-predictor-847bfb4858-gvm44"]
Jul 01 18:58:47.516503 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.516469    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmv7t\" (UniqueName: \"kubernetes.io/projected/8000bfc6-d3d5-4f73-b381-69e464c77c0d-kube-api-access-hmv7t\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:58:47.516503 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.516502    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8000bfc6-d3d5-4f73-b381-69e464c77c0d-proxy-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:58:47.516693 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.516519    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8000bfc6-d3d5-4f73-b381-69e464c77c0d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:58:47.666070 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.665993    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8000bfc6-d3d5-4f73-b381-69e464c77c0d" path="/var/lib/kubelet/pods/8000bfc6-d3d5-4f73-b381-69e464c77c0d/volumes"
Jul 01 18:58:47.940533 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:47.940500    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5"
Jul 01 18:58:48.021170 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:48.021129    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbq5x\" (UniqueName: \"kubernetes.io/projected/d26e74b3-f903-4028-bf9f-ec2bb2dea9a8-kube-api-access-zbq5x\") pod \"d26e74b3-f903-4028-bf9f-ec2bb2dea9a8\" (UID: \"d26e74b3-f903-4028-bf9f-ec2bb2dea9a8\") "
Jul 01 18:58:48.021338 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:48.021219    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d26e74b3-f903-4028-bf9f-ec2bb2dea9a8-proxy-tls\") pod \"d26e74b3-f903-4028-bf9f-ec2bb2dea9a8\" (UID: \"d26e74b3-f903-4028-bf9f-ec2bb2dea9a8\") "
Jul 01 18:58:48.021338 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:48.021239    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d26e74b3-f903-4028-bf9f-ec2bb2dea9a8-kube-rbac-proxy-sar-config\") pod \"d26e74b3-f903-4028-bf9f-ec2bb2dea9a8\" (UID: \"d26e74b3-f903-4028-bf9f-ec2bb2dea9a8\") "
Jul 01 18:58:48.021648 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:48.021620    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d26e74b3-f903-4028-bf9f-ec2bb2dea9a8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d26e74b3-f903-4028-bf9f-ec2bb2dea9a8" (UID: "d26e74b3-f903-4028-bf9f-ec2bb2dea9a8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:58:48.023223 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:48.023201    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d26e74b3-f903-4028-bf9f-ec2bb2dea9a8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d26e74b3-f903-4028-bf9f-ec2bb2dea9a8" (UID: "d26e74b3-f903-4028-bf9f-ec2bb2dea9a8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:58:48.023288 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:48.023270    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26e74b3-f903-4028-bf9f-ec2bb2dea9a8-kube-api-access-zbq5x" (OuterVolumeSpecName: "kube-api-access-zbq5x") pod "d26e74b3-f903-4028-bf9f-ec2bb2dea9a8" (UID: "d26e74b3-f903-4028-bf9f-ec2bb2dea9a8"). InnerVolumeSpecName "kube-api-access-zbq5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:58:48.122460 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:48.122427    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbq5x\" (UniqueName: \"kubernetes.io/projected/d26e74b3-f903-4028-bf9f-ec2bb2dea9a8-kube-api-access-zbq5x\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:58:48.122460 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:48.122455    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d26e74b3-f903-4028-bf9f-ec2bb2dea9a8-proxy-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:58:48.122460 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:48.122465    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d26e74b3-f903-4028-bf9f-ec2bb2dea9a8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:58:48.464083 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:48.464051    2573 generic.go:356] "Generic (PLEG): container finished" podID="d26e74b3-f903-4028-bf9f-ec2bb2dea9a8" containerID="373059822d6ba30452329de4ec7fb59c20cc792954fb1aa3633466adce16650d" exitCode=0
Jul 01 18:58:48.464468 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:48.464095    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5"
Jul 01 18:58:48.464468 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:48.464118    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5" event={"ID":"d26e74b3-f903-4028-bf9f-ec2bb2dea9a8","Type":"ContainerDied","Data":"373059822d6ba30452329de4ec7fb59c20cc792954fb1aa3633466adce16650d"}
Jul 01 18:58:48.464468 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:48.464165    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5" event={"ID":"d26e74b3-f903-4028-bf9f-ec2bb2dea9a8","Type":"ContainerDied","Data":"8a6af7255f6eca0e30d39d2214545acb48b21988b84578004f4df6b2d8c5ebad"}
Jul 01 18:58:48.464468 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:48.464191    2573 scope.go:117] "RemoveContainer" containerID="0b032a7c9f44ff299f65b5fd8082cf6a19b4fcd527fe564bb598e8137754e1a8"
Jul 01 18:58:48.472638 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:48.472613    2573 scope.go:117] "RemoveContainer" containerID="373059822d6ba30452329de4ec7fb59c20cc792954fb1aa3633466adce16650d"
Jul 01 18:58:48.479634 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:48.479618    2573 scope.go:117] "RemoveContainer" containerID="0b032a7c9f44ff299f65b5fd8082cf6a19b4fcd527fe564bb598e8137754e1a8"
Jul 01 18:58:48.479891 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:58:48.479872    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b032a7c9f44ff299f65b5fd8082cf6a19b4fcd527fe564bb598e8137754e1a8\": container with ID starting with 0b032a7c9f44ff299f65b5fd8082cf6a19b4fcd527fe564bb598e8137754e1a8 not found: ID does not exist" containerID="0b032a7c9f44ff299f65b5fd8082cf6a19b4fcd527fe564bb598e8137754e1a8"
Jul 01 18:58:48.479936 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:48.479899    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b032a7c9f44ff299f65b5fd8082cf6a19b4fcd527fe564bb598e8137754e1a8"} err="failed to get container status \"0b032a7c9f44ff299f65b5fd8082cf6a19b4fcd527fe564bb598e8137754e1a8\": rpc error: code = NotFound desc = could not find container \"0b032a7c9f44ff299f65b5fd8082cf6a19b4fcd527fe564bb598e8137754e1a8\": container with ID starting with 0b032a7c9f44ff299f65b5fd8082cf6a19b4fcd527fe564bb598e8137754e1a8 not found: ID does not exist"
Jul 01 18:58:48.479936 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:48.479916    2573 scope.go:117] "RemoveContainer" containerID="373059822d6ba30452329de4ec7fb59c20cc792954fb1aa3633466adce16650d"
Jul 01 18:58:48.480149 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:58:48.480132    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"373059822d6ba30452329de4ec7fb59c20cc792954fb1aa3633466adce16650d\": container with ID starting with 373059822d6ba30452329de4ec7fb59c20cc792954fb1aa3633466adce16650d not found: ID does not exist" containerID="373059822d6ba30452329de4ec7fb59c20cc792954fb1aa3633466adce16650d"
Jul 01 18:58:48.480186 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:48.480154    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"373059822d6ba30452329de4ec7fb59c20cc792954fb1aa3633466adce16650d"} err="failed to get container status \"373059822d6ba30452329de4ec7fb59c20cc792954fb1aa3633466adce16650d\": rpc error: code = NotFound desc = could not find container \"373059822d6ba30452329de4ec7fb59c20cc792954fb1aa3633466adce16650d\": container with ID starting with 373059822d6ba30452329de4ec7fb59c20cc792954fb1aa3633466adce16650d not found: ID does not exist"
Jul 01 18:58:48.487186 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:48.487166    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5"]
Jul 01 18:58:48.493958 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:48.493937    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c059b-predictor-7c8bc8775f-9qkp5"]
Jul 01 18:58:49.665601 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:49.665571    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d26e74b3-f903-4028-bf9f-ec2bb2dea9a8" path="/var/lib/kubelet/pods/d26e74b3-f903-4028-bf9f-ec2bb2dea9a8/volumes"
Jul 01 18:58:52.305900 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:52.305868    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:58:52.459294 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:52.459266    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd"
Jul 01 18:58:52.459733 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:52.459685    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw"
Jul 01 18:58:52.459860 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:52.459764    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd" podUID="dcffa820-58d6-4ab1-8975-e6adcaf9f0de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:58:52.460191 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:52.460170    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw" podUID="7f36625f-48ef-4d40-94d1-85d3370da2d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 18:58:56.374748 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:56.374697    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl" podUID="baaaead7-e419-4611-a8f1-4b336faa25b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:58:57.377387 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:58:57.377348    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7" podUID="247c587d-b6d5-4fff-a212-771964aecffe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 18:59:02.459890 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:02.459850    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd" podUID="dcffa820-58d6-4ab1-8975-e6adcaf9f0de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:59:02.460344 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:02.460115    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw" podUID="7f36625f-48ef-4d40-94d1-85d3370da2d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 18:59:06.374599 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:06.374557    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl" podUID="baaaead7-e419-4611-a8f1-4b336faa25b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:59:07.377435 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:07.377397    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7" podUID="247c587d-b6d5-4fff-a212-771964aecffe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 01 18:59:12.460171 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:12.460134    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw" podUID="7f36625f-48ef-4d40-94d1-85d3370da2d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 18:59:12.460171 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:12.460148    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd" podUID="dcffa820-58d6-4ab1-8975-e6adcaf9f0de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:59:16.374607 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:16.374567    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl" podUID="baaaead7-e419-4611-a8f1-4b336faa25b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:59:17.377871 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:17.377842    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7"
Jul 01 18:59:22.460161 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:22.460125    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd" podUID="dcffa820-58d6-4ab1-8975-e6adcaf9f0de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:59:22.460567 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:22.460122    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw" podUID="7f36625f-48ef-4d40-94d1-85d3370da2d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 18:59:26.374691 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:26.374660    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl"
Jul 01 18:59:32.460921 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:32.460890    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw"
Jul 01 18:59:32.461296 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:32.460991    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd"
Jul 01 18:59:39.514338 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:39.514302    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:59:39.769697 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:39.769626    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:59:43.982190 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:43.982159    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:59:44.231244 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:44.231218    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:59:49.867893 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:49.867857    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:59:49.939112 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:49.939083    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:59:53.662394 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.662315    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:59:53.738013 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.737985    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7"]
Jul 01 18:59:53.738241 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.738216    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7" podUID="247c587d-b6d5-4fff-a212-771964aecffe" containerName="kserve-container" containerID="cri-o://3f189082aa0fc01d3ed385f2f219c362c4d9ca6bc0934ffa242ea9bad4669614" gracePeriod=30
Jul 01 18:59:53.738328 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.738243    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7" podUID="247c587d-b6d5-4fff-a212-771964aecffe" containerName="kube-rbac-proxy" containerID="cri-o://50fc7c0daba4659efc3f49ad77f74d315ea0493aacc500d31cd0eafa9130d86a" gracePeriod=30
Jul 01 18:59:53.807999 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.807968    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d"]
Jul 01 18:59:53.808385 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.808276    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d26e74b3-f903-4028-bf9f-ec2bb2dea9a8" containerName="kserve-container"
Jul 01 18:59:53.808385 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.808286    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d26e74b3-f903-4028-bf9f-ec2bb2dea9a8" containerName="kserve-container"
Jul 01 18:59:53.808385 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.808302    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d26e74b3-f903-4028-bf9f-ec2bb2dea9a8" containerName="kube-rbac-proxy"
Jul 01 18:59:53.808385 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.808307    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d26e74b3-f903-4028-bf9f-ec2bb2dea9a8" containerName="kube-rbac-proxy"
Jul 01 18:59:53.808385 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.808316    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8000bfc6-d3d5-4f73-b381-69e464c77c0d" containerName="kserve-container"
Jul 01 18:59:53.808385 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.808322    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8000bfc6-d3d5-4f73-b381-69e464c77c0d" containerName="kserve-container"
Jul 01 18:59:53.808385 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.808329    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8000bfc6-d3d5-4f73-b381-69e464c77c0d" containerName="kube-rbac-proxy"
Jul 01 18:59:53.808385 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.808334    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8000bfc6-d3d5-4f73-b381-69e464c77c0d" containerName="kube-rbac-proxy"
Jul 01 18:59:53.808385 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.808379    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8000bfc6-d3d5-4f73-b381-69e464c77c0d" containerName="kube-rbac-proxy"
Jul 01 18:59:53.808385 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.808386    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d26e74b3-f903-4028-bf9f-ec2bb2dea9a8" containerName="kube-rbac-proxy"
Jul 01 18:59:53.808385 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.808392    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8000bfc6-d3d5-4f73-b381-69e464c77c0d" containerName="kserve-container"
Jul 01 18:59:53.809025 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.808399    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d26e74b3-f903-4028-bf9f-ec2bb2dea9a8" containerName="kserve-container"
Jul 01 18:59:53.812755 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.812736    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d"
Jul 01 18:59:53.819422 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.819212    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c3262-predictor-serving-cert\""
Jul 01 18:59:53.819653 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.819624    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c3262-kube-rbac-proxy-sar-config\""
Jul 01 18:59:53.823400 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.823377    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d"]
Jul 01 18:59:53.862663 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.862633    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl"]
Jul 01 18:59:53.862936 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.862909    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl" podUID="baaaead7-e419-4611-a8f1-4b336faa25b0" containerName="kserve-container" containerID="cri-o://6fe83dba8158223a5a35293e50a37785f161f33f6b8be0b48c621461ec45cf61" gracePeriod=30
Jul 01 18:59:53.863057 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.862913    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl" podUID="baaaead7-e419-4611-a8f1-4b336faa25b0" containerName="kube-rbac-proxy" containerID="cri-o://aca9370534bd9876052a0e71fc5603919bf0429bcd158afd5dffc098429f2bab" gracePeriod=30
Jul 01 18:59:53.866601 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.866577    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:59:53.927597 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.927525    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx"]
Jul 01 18:59:53.931568 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.931538    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/440a82cc-2ac4-4b80-ac8d-075af9ca1394-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c3262-predictor-865fd8b765-p828d\" (UID: \"440a82cc-2ac4-4b80-ac8d-075af9ca1394\") " pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d"
Jul 01 18:59:53.931692 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.931611    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/440a82cc-2ac4-4b80-ac8d-075af9ca1394-proxy-tls\") pod \"success-200-isvc-c3262-predictor-865fd8b765-p828d\" (UID: \"440a82cc-2ac4-4b80-ac8d-075af9ca1394\") " pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d"
Jul 01 18:59:53.931692 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.931545    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx"
Jul 01 18:59:53.931692 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.931676    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd86m\" (UniqueName: \"kubernetes.io/projected/440a82cc-2ac4-4b80-ac8d-075af9ca1394-kube-api-access-jd86m\") pod \"success-200-isvc-c3262-predictor-865fd8b765-p828d\" (UID: \"440a82cc-2ac4-4b80-ac8d-075af9ca1394\") " pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d"
Jul 01 18:59:53.934260 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.934238    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c3262-predictor-serving-cert\""
Jul 01 18:59:53.934358 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.934241    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c3262-kube-rbac-proxy-sar-config\""
Jul 01 18:59:53.941917 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.941790    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:59:53.942031 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:53.941998    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx"]
Jul 01 18:59:54.032772 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.032740    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd86m\" (UniqueName: \"kubernetes.io/projected/440a82cc-2ac4-4b80-ac8d-075af9ca1394-kube-api-access-jd86m\") pod \"success-200-isvc-c3262-predictor-865fd8b765-p828d\" (UID: \"440a82cc-2ac4-4b80-ac8d-075af9ca1394\") " pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d"
Jul 01 18:59:54.032936 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.032777    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmxjc\" (UniqueName: \"kubernetes.io/projected/ad795212-732d-4487-8321-0cb33464f64f-kube-api-access-tmxjc\") pod \"error-404-isvc-c3262-predictor-6676bfcd4d-z54kx\" (UID: \"ad795212-732d-4487-8321-0cb33464f64f\") " pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx"
Jul 01 18:59:54.032936 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.032830    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/440a82cc-2ac4-4b80-ac8d-075af9ca1394-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c3262-predictor-865fd8b765-p828d\" (UID: \"440a82cc-2ac4-4b80-ac8d-075af9ca1394\") " pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d"
Jul 01 18:59:54.032936 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.032881    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad795212-732d-4487-8321-0cb33464f64f-proxy-tls\") pod \"error-404-isvc-c3262-predictor-6676bfcd4d-z54kx\" (UID: \"ad795212-732d-4487-8321-0cb33464f64f\") " pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx"
Jul 01 18:59:54.032936 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.032901    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/440a82cc-2ac4-4b80-ac8d-075af9ca1394-proxy-tls\") pod \"success-200-isvc-c3262-predictor-865fd8b765-p828d\" (UID: \"440a82cc-2ac4-4b80-ac8d-075af9ca1394\") " pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d"
Jul 01 18:59:54.032936 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.032933    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad795212-732d-4487-8321-0cb33464f64f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c3262-predictor-6676bfcd4d-z54kx\" (UID: \"ad795212-732d-4487-8321-0cb33464f64f\") " pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx"
Jul 01 18:59:54.033241 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:59:54.033036    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-c3262-predictor-serving-cert: secret "success-200-isvc-c3262-predictor-serving-cert" not found
Jul 01 18:59:54.033241 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:59:54.033098    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/440a82cc-2ac4-4b80-ac8d-075af9ca1394-proxy-tls podName:440a82cc-2ac4-4b80-ac8d-075af9ca1394 nodeName:}" failed. No retries permitted until 2026-07-01 18:59:54.533077473 +0000 UTC m=+1345.301915056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/440a82cc-2ac4-4b80-ac8d-075af9ca1394-proxy-tls") pod "success-200-isvc-c3262-predictor-865fd8b765-p828d" (UID: "440a82cc-2ac4-4b80-ac8d-075af9ca1394") : secret "success-200-isvc-c3262-predictor-serving-cert" not found
Jul 01 18:59:54.033552 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.033531    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/440a82cc-2ac4-4b80-ac8d-075af9ca1394-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c3262-predictor-865fd8b765-p828d\" (UID: \"440a82cc-2ac4-4b80-ac8d-075af9ca1394\") " pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d"
Jul 01 18:59:54.042244 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.042221    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd86m\" (UniqueName: \"kubernetes.io/projected/440a82cc-2ac4-4b80-ac8d-075af9ca1394-kube-api-access-jd86m\") pod \"success-200-isvc-c3262-predictor-865fd8b765-p828d\" (UID: \"440a82cc-2ac4-4b80-ac8d-075af9ca1394\") " pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d"
Jul 01 18:59:54.066853 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.066807    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:59:54.133816 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.133782    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad795212-732d-4487-8321-0cb33464f64f-proxy-tls\") pod \"error-404-isvc-c3262-predictor-6676bfcd4d-z54kx\" (UID: \"ad795212-732d-4487-8321-0cb33464f64f\") " pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx"
Jul 01 18:59:54.133981 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.133858    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad795212-732d-4487-8321-0cb33464f64f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c3262-predictor-6676bfcd4d-z54kx\" (UID: \"ad795212-732d-4487-8321-0cb33464f64f\") " pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx"
Jul 01 18:59:54.133981 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.133903    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmxjc\" (UniqueName: \"kubernetes.io/projected/ad795212-732d-4487-8321-0cb33464f64f-kube-api-access-tmxjc\") pod \"error-404-isvc-c3262-predictor-6676bfcd4d-z54kx\" (UID: \"ad795212-732d-4487-8321-0cb33464f64f\") " pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx"
Jul 01 18:59:54.134594 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.134542    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad795212-732d-4487-8321-0cb33464f64f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c3262-predictor-6676bfcd4d-z54kx\" (UID: \"ad795212-732d-4487-8321-0cb33464f64f\") " pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx"
Jul 01 18:59:54.136441 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.136416    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad795212-732d-4487-8321-0cb33464f64f-proxy-tls\") pod \"error-404-isvc-c3262-predictor-6676bfcd4d-z54kx\" (UID: \"ad795212-732d-4487-8321-0cb33464f64f\") " pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx"
Jul 01 18:59:54.143519 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.143498    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmxjc\" (UniqueName: \"kubernetes.io/projected/ad795212-732d-4487-8321-0cb33464f64f-kube-api-access-tmxjc\") pod \"error-404-isvc-c3262-predictor-6676bfcd4d-z54kx\" (UID: \"ad795212-732d-4487-8321-0cb33464f64f\") " pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx"
Jul 01 18:59:54.167946 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.167924    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 18:59:54.245270 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.245244    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx"
Jul 01 18:59:54.372351 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.372323    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx"]
Jul 01 18:59:54.374555 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:59:54.374526    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad795212_732d_4487_8321_0cb33464f64f.slice/crio-5b5a321fcbf0aa5e4f12a53b63ed86053ff19c9e4fd5838ce4ce4bcdbdc682bc WatchSource:0}: Error finding container 5b5a321fcbf0aa5e4f12a53b63ed86053ff19c9e4fd5838ce4ce4bcdbdc682bc: Status 404 returned error can't find the container with id 5b5a321fcbf0aa5e4f12a53b63ed86053ff19c9e4fd5838ce4ce4bcdbdc682bc
Jul 01 18:59:54.536685 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.536594    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/440a82cc-2ac4-4b80-ac8d-075af9ca1394-proxy-tls\") pod \"success-200-isvc-c3262-predictor-865fd8b765-p828d\" (UID: \"440a82cc-2ac4-4b80-ac8d-075af9ca1394\") " pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d"
Jul 01 18:59:54.539466 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.539437    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/440a82cc-2ac4-4b80-ac8d-075af9ca1394-proxy-tls\") pod \"success-200-isvc-c3262-predictor-865fd8b765-p828d\" (UID: \"440a82cc-2ac4-4b80-ac8d-075af9ca1394\") " pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d"
Jul 01 18:59:54.731593 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.731558    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d"
Jul 01 18:59:54.739002 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.738965    2573 generic.go:356] "Generic (PLEG): container finished" podID="baaaead7-e419-4611-a8f1-4b336faa25b0" containerID="aca9370534bd9876052a0e71fc5603919bf0429bcd158afd5dffc098429f2bab" exitCode=2
Jul 01 18:59:54.739175 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.739062    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl" event={"ID":"baaaead7-e419-4611-a8f1-4b336faa25b0","Type":"ContainerDied","Data":"aca9370534bd9876052a0e71fc5603919bf0429bcd158afd5dffc098429f2bab"}
Jul 01 18:59:54.741415 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.741387    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx" event={"ID":"ad795212-732d-4487-8321-0cb33464f64f","Type":"ContainerStarted","Data":"95050154e2b26a219bbc25bfe13fb321a1ef8d787e63705c1996379c0efa7d82"}
Jul 01 18:59:54.741536 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.741452    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx" event={"ID":"ad795212-732d-4487-8321-0cb33464f64f","Type":"ContainerStarted","Data":"5420180e39b85216f86c753358ca6baded0d7a5020a6a4e3250e0bed41e41a4d"}
Jul 01 18:59:54.741536 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.741484    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx" event={"ID":"ad795212-732d-4487-8321-0cb33464f64f","Type":"ContainerStarted","Data":"5b5a321fcbf0aa5e4f12a53b63ed86053ff19c9e4fd5838ce4ce4bcdbdc682bc"}
Jul 01 18:59:54.741639 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.741531    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx"
Jul 01 18:59:54.741639 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.741580    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx"
Jul 01 18:59:54.742955 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.742928    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx" podUID="ad795212-732d-4487-8321-0cb33464f64f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 18:59:54.743479 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.743452    2573 generic.go:356] "Generic (PLEG): container finished" podID="247c587d-b6d5-4fff-a212-771964aecffe" containerID="50fc7c0daba4659efc3f49ad77f74d315ea0493aacc500d31cd0eafa9130d86a" exitCode=2
Jul 01 18:59:54.743558 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.743504    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7" event={"ID":"247c587d-b6d5-4fff-a212-771964aecffe","Type":"ContainerDied","Data":"50fc7c0daba4659efc3f49ad77f74d315ea0493aacc500d31cd0eafa9130d86a"}
Jul 01 18:59:54.764133 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.764081    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx" podStartSLOduration=1.764065445 podStartE2EDuration="1.764065445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:59:54.762522039 +0000 UTC m=+1345.531359642" watchObservedRunningTime="2026-07-01 18:59:54.764065445 +0000 UTC m=+1345.532903047"
Jul 01 18:59:54.866846 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:54.866825    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d"]
Jul 01 18:59:54.868670 ip-10-0-129-245 kubenswrapper[2573]: W0701 18:59:54.868637    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440a82cc_2ac4_4b80_ac8d_075af9ca1394.slice/crio-73beeea919bb9092386e1c802394a7ebf521756aef0ff7bc26d3fc06c5b3635b WatchSource:0}: Error finding container 73beeea919bb9092386e1c802394a7ebf521756aef0ff7bc26d3fc06c5b3635b: Status 404 returned error can't find the container with id 73beeea919bb9092386e1c802394a7ebf521756aef0ff7bc26d3fc06c5b3635b
Jul 01 18:59:55.749114 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:55.749076    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d" event={"ID":"440a82cc-2ac4-4b80-ac8d-075af9ca1394","Type":"ContainerStarted","Data":"0574cb55b33abc0cfc84f6de08bd6688fe1715bf146d92b20e2bb64890ece05e"}
Jul 01 18:59:55.749552 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:55.749385    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d"
Jul 01 18:59:55.749552 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:55.749438    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d" event={"ID":"440a82cc-2ac4-4b80-ac8d-075af9ca1394","Type":"ContainerStarted","Data":"db4c870c3b3f5f6316c6628b6673e083b56ed77665bf3d2ac8e581dd6ba11cee"}
Jul 01 18:59:55.749552 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:55.749470    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d" event={"ID":"440a82cc-2ac4-4b80-ac8d-075af9ca1394","Type":"ContainerStarted","Data":"73beeea919bb9092386e1c802394a7ebf521756aef0ff7bc26d3fc06c5b3635b"}
Jul 01 18:59:55.749552 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:55.749503    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d"
Jul 01 18:59:55.749702 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:55.749560    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx" podUID="ad795212-732d-4487-8321-0cb33464f64f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 18:59:55.750600 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:55.750579    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d" podUID="440a82cc-2ac4-4b80-ac8d-075af9ca1394" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:59:55.769504 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:55.769467    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d" podStartSLOduration=2.76945521 podStartE2EDuration="2.76945521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:59:55.768449915 +0000 UTC m=+1346.537287519" watchObservedRunningTime="2026-07-01 18:59:55.76945521 +0000 UTC m=+1346.538292813"
Jul 01 18:59:56.369049 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:56.369006    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl" podUID="baaaead7-e419-4611-a8f1-4b336faa25b0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 01 18:59:56.374478 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:56.374456    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl" podUID="baaaead7-e419-4611-a8f1-4b336faa25b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:59:56.755787 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:56.755750    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d" podUID="440a82cc-2ac4-4b80-ac8d-075af9ca1394" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:59:57.029554 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.029531    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7"
Jul 01 18:59:57.031925 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.031904    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl"
Jul 01 18:59:57.159047 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.159004    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baaaead7-e419-4611-a8f1-4b336faa25b0-proxy-tls\") pod \"baaaead7-e419-4611-a8f1-4b336faa25b0\" (UID: \"baaaead7-e419-4611-a8f1-4b336faa25b0\") "
Jul 01 18:59:57.159047 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.159052    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ln6h2\" (UniqueName: \"kubernetes.io/projected/baaaead7-e419-4611-a8f1-4b336faa25b0-kube-api-access-ln6h2\") pod \"baaaead7-e419-4611-a8f1-4b336faa25b0\" (UID: \"baaaead7-e419-4611-a8f1-4b336faa25b0\") "
Jul 01 18:59:57.159462 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.159085    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/247c587d-b6d5-4fff-a212-771964aecffe-proxy-tls\") pod \"247c587d-b6d5-4fff-a212-771964aecffe\" (UID: \"247c587d-b6d5-4fff-a212-771964aecffe\") "
Jul 01 18:59:57.159462 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.159124    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wnnn\" (UniqueName: \"kubernetes.io/projected/247c587d-b6d5-4fff-a212-771964aecffe-kube-api-access-4wnnn\") pod \"247c587d-b6d5-4fff-a212-771964aecffe\" (UID: \"247c587d-b6d5-4fff-a212-771964aecffe\") "
Jul 01 18:59:57.159462 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.159150    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baaaead7-e419-4611-a8f1-4b336faa25b0-kube-rbac-proxy-sar-config\") pod \"baaaead7-e419-4611-a8f1-4b336faa25b0\" (UID: \"baaaead7-e419-4611-a8f1-4b336faa25b0\") "
Jul 01 18:59:57.159462 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.159180    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/247c587d-b6d5-4fff-a212-771964aecffe-kube-rbac-proxy-sar-config\") pod \"247c587d-b6d5-4fff-a212-771964aecffe\" (UID: \"247c587d-b6d5-4fff-a212-771964aecffe\") "
Jul 01 18:59:57.159691 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.159482    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baaaead7-e419-4611-a8f1-4b336faa25b0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "baaaead7-e419-4611-a8f1-4b336faa25b0" (UID: "baaaead7-e419-4611-a8f1-4b336faa25b0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:59:57.159691 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.159579    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247c587d-b6d5-4fff-a212-771964aecffe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "247c587d-b6d5-4fff-a212-771964aecffe" (UID: "247c587d-b6d5-4fff-a212-771964aecffe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:59:57.161126 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.161100    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baaaead7-e419-4611-a8f1-4b336faa25b0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "baaaead7-e419-4611-a8f1-4b336faa25b0" (UID: "baaaead7-e419-4611-a8f1-4b336faa25b0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:59:57.161238 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.161213    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247c587d-b6d5-4fff-a212-771964aecffe-kube-api-access-4wnnn" (OuterVolumeSpecName: "kube-api-access-4wnnn") pod "247c587d-b6d5-4fff-a212-771964aecffe" (UID: "247c587d-b6d5-4fff-a212-771964aecffe"). InnerVolumeSpecName "kube-api-access-4wnnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:59:57.161292 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.161239    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/247c587d-b6d5-4fff-a212-771964aecffe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "247c587d-b6d5-4fff-a212-771964aecffe" (UID: "247c587d-b6d5-4fff-a212-771964aecffe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:59:57.161292 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.161268    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baaaead7-e419-4611-a8f1-4b336faa25b0-kube-api-access-ln6h2" (OuterVolumeSpecName: "kube-api-access-ln6h2") pod "baaaead7-e419-4611-a8f1-4b336faa25b0" (UID: "baaaead7-e419-4611-a8f1-4b336faa25b0"). InnerVolumeSpecName "kube-api-access-ln6h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:59:57.259874 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.259794    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baaaead7-e419-4611-a8f1-4b336faa25b0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:59:57.259874 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.259832    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/247c587d-b6d5-4fff-a212-771964aecffe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:59:57.259874 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.259842    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baaaead7-e419-4611-a8f1-4b336faa25b0-proxy-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:59:57.259874 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.259853    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln6h2\" (UniqueName: \"kubernetes.io/projected/baaaead7-e419-4611-a8f1-4b336faa25b0-kube-api-access-ln6h2\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:59:57.259874 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.259861    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/247c587d-b6d5-4fff-a212-771964aecffe-proxy-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:59:57.259874 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.259871    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wnnn\" (UniqueName: \"kubernetes.io/projected/247c587d-b6d5-4fff-a212-771964aecffe-kube-api-access-4wnnn\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 18:59:57.760512 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.760475    2573 generic.go:356] "Generic (PLEG): container finished" podID="baaaead7-e419-4611-a8f1-4b336faa25b0" containerID="6fe83dba8158223a5a35293e50a37785f161f33f6b8be0b48c621461ec45cf61" exitCode=0
Jul 01 18:59:57.760894 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.760521    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl"
Jul 01 18:59:57.760894 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.760558    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl" event={"ID":"baaaead7-e419-4611-a8f1-4b336faa25b0","Type":"ContainerDied","Data":"6fe83dba8158223a5a35293e50a37785f161f33f6b8be0b48c621461ec45cf61"}
Jul 01 18:59:57.760894 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.760618    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl" event={"ID":"baaaead7-e419-4611-a8f1-4b336faa25b0","Type":"ContainerDied","Data":"1b72656f25d98622f43779b78b6edc4cb8ccdbbf4df487d490a42803467a92e4"}
Jul 01 18:59:57.760894 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.760657    2573 scope.go:117] "RemoveContainer" containerID="aca9370534bd9876052a0e71fc5603919bf0429bcd158afd5dffc098429f2bab"
Jul 01 18:59:57.762308 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.762277    2573 generic.go:356] "Generic (PLEG): container finished" podID="247c587d-b6d5-4fff-a212-771964aecffe" containerID="3f189082aa0fc01d3ed385f2f219c362c4d9ca6bc0934ffa242ea9bad4669614" exitCode=0
Jul 01 18:59:57.762429 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.762337    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7"
Jul 01 18:59:57.762429 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.762325    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7" event={"ID":"247c587d-b6d5-4fff-a212-771964aecffe","Type":"ContainerDied","Data":"3f189082aa0fc01d3ed385f2f219c362c4d9ca6bc0934ffa242ea9bad4669614"}
Jul 01 18:59:57.762429 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.762394    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7" event={"ID":"247c587d-b6d5-4fff-a212-771964aecffe","Type":"ContainerDied","Data":"9f163503d24208c9f1825f89a97379198fa56035f8a6ece5b5cb592bf2906466"}
Jul 01 18:59:57.763281 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.763239    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d" podUID="440a82cc-2ac4-4b80-ac8d-075af9ca1394" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:59:57.772813 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.772787    2573 scope.go:117] "RemoveContainer" containerID="6fe83dba8158223a5a35293e50a37785f161f33f6b8be0b48c621461ec45cf61"
Jul 01 18:59:57.780019 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.779984    2573 scope.go:117] "RemoveContainer" containerID="aca9370534bd9876052a0e71fc5603919bf0429bcd158afd5dffc098429f2bab"
Jul 01 18:59:57.780241 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:59:57.780223    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aca9370534bd9876052a0e71fc5603919bf0429bcd158afd5dffc098429f2bab\": container with ID starting with aca9370534bd9876052a0e71fc5603919bf0429bcd158afd5dffc098429f2bab not found: ID does not exist" containerID="aca9370534bd9876052a0e71fc5603919bf0429bcd158afd5dffc098429f2bab"
Jul 01 18:59:57.780294 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.780256    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aca9370534bd9876052a0e71fc5603919bf0429bcd158afd5dffc098429f2bab"} err="failed to get container status \"aca9370534bd9876052a0e71fc5603919bf0429bcd158afd5dffc098429f2bab\": rpc error: code = NotFound desc = could not find container \"aca9370534bd9876052a0e71fc5603919bf0429bcd158afd5dffc098429f2bab\": container with ID starting with aca9370534bd9876052a0e71fc5603919bf0429bcd158afd5dffc098429f2bab not found: ID does not exist"
Jul 01 18:59:57.780294 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.780273    2573 scope.go:117] "RemoveContainer" containerID="6fe83dba8158223a5a35293e50a37785f161f33f6b8be0b48c621461ec45cf61"
Jul 01 18:59:57.780511 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:59:57.780494    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fe83dba8158223a5a35293e50a37785f161f33f6b8be0b48c621461ec45cf61\": container with ID starting with 6fe83dba8158223a5a35293e50a37785f161f33f6b8be0b48c621461ec45cf61 not found: ID does not exist" containerID="6fe83dba8158223a5a35293e50a37785f161f33f6b8be0b48c621461ec45cf61"
Jul 01 18:59:57.780552 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.780517    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe83dba8158223a5a35293e50a37785f161f33f6b8be0b48c621461ec45cf61"} err="failed to get container status \"6fe83dba8158223a5a35293e50a37785f161f33f6b8be0b48c621461ec45cf61\": rpc error: code = NotFound desc = could not find container \"6fe83dba8158223a5a35293e50a37785f161f33f6b8be0b48c621461ec45cf61\": container with ID starting with 6fe83dba8158223a5a35293e50a37785f161f33f6b8be0b48c621461ec45cf61 not found: ID does not exist"
Jul 01 18:59:57.780552 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.780532    2573 scope.go:117] "RemoveContainer" containerID="50fc7c0daba4659efc3f49ad77f74d315ea0493aacc500d31cd0eafa9130d86a"
Jul 01 18:59:57.783640 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.783616    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl"]
Jul 01 18:59:57.788104 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.788084    2573 scope.go:117] "RemoveContainer" containerID="3f189082aa0fc01d3ed385f2f219c362c4d9ca6bc0934ffa242ea9bad4669614"
Jul 01 18:59:57.788497 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.788476    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-550f3-predictor-6564b49d87-mgttl"]
Jul 01 18:59:57.794828 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.794811    2573 scope.go:117] "RemoveContainer" containerID="50fc7c0daba4659efc3f49ad77f74d315ea0493aacc500d31cd0eafa9130d86a"
Jul 01 18:59:57.795037 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:59:57.795021    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50fc7c0daba4659efc3f49ad77f74d315ea0493aacc500d31cd0eafa9130d86a\": container with ID starting with 50fc7c0daba4659efc3f49ad77f74d315ea0493aacc500d31cd0eafa9130d86a not found: ID does not exist" containerID="50fc7c0daba4659efc3f49ad77f74d315ea0493aacc500d31cd0eafa9130d86a"
Jul 01 18:59:57.795087 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.795041    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50fc7c0daba4659efc3f49ad77f74d315ea0493aacc500d31cd0eafa9130d86a"} err="failed to get container status \"50fc7c0daba4659efc3f49ad77f74d315ea0493aacc500d31cd0eafa9130d86a\": rpc error: code = NotFound desc = could not find container \"50fc7c0daba4659efc3f49ad77f74d315ea0493aacc500d31cd0eafa9130d86a\": container with ID starting with 50fc7c0daba4659efc3f49ad77f74d315ea0493aacc500d31cd0eafa9130d86a not found: ID does not exist"
Jul 01 18:59:57.795087 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.795055    2573 scope.go:117] "RemoveContainer" containerID="3f189082aa0fc01d3ed385f2f219c362c4d9ca6bc0934ffa242ea9bad4669614"
Jul 01 18:59:57.795237 ip-10-0-129-245 kubenswrapper[2573]: E0701 18:59:57.795223    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f189082aa0fc01d3ed385f2f219c362c4d9ca6bc0934ffa242ea9bad4669614\": container with ID starting with 3f189082aa0fc01d3ed385f2f219c362c4d9ca6bc0934ffa242ea9bad4669614 not found: ID does not exist" containerID="3f189082aa0fc01d3ed385f2f219c362c4d9ca6bc0934ffa242ea9bad4669614"
Jul 01 18:59:57.795276 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.795240    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f189082aa0fc01d3ed385f2f219c362c4d9ca6bc0934ffa242ea9bad4669614"} err="failed to get container status \"3f189082aa0fc01d3ed385f2f219c362c4d9ca6bc0934ffa242ea9bad4669614\": rpc error: code = NotFound desc = could not find container \"3f189082aa0fc01d3ed385f2f219c362c4d9ca6bc0934ffa242ea9bad4669614\": container with ID starting with 3f189082aa0fc01d3ed385f2f219c362c4d9ca6bc0934ffa242ea9bad4669614 not found: ID does not exist"
Jul 01 18:59:57.801669 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.801638    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7"]
Jul 01 18:59:57.807761 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:57.807740    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-550f3-predictor-787558b8bb-22gl7"]
Jul 01 18:59:59.665551 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:59.665520    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="247c587d-b6d5-4fff-a212-771964aecffe" path="/var/lib/kubelet/pods/247c587d-b6d5-4fff-a212-771964aecffe/volumes"
Jul 01 18:59:59.665962 ip-10-0-129-245 kubenswrapper[2573]: I0701 18:59:59.665948    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baaaead7-e419-4611-a8f1-4b336faa25b0" path="/var/lib/kubelet/pods/baaaead7-e419-4611-a8f1-4b336faa25b0/volumes"
Jul 01 19:00:00.753998 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:00:00.753971    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx"
Jul 01 19:00:00.754375 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:00:00.754320    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx" podUID="ad795212-732d-4487-8321-0cb33464f64f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 19:00:00.909234 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:00:00.909205    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:00:01.010971 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:00:01.010887    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:00:01.089193 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:00:01.089164    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:00:02.767253 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:00:02.767222    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d"
Jul 01 19:00:02.767642 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:00:02.767615    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d" podUID="440a82cc-2ac4-4b80-ac8d-075af9ca1394" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 19:00:10.754509 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:00:10.754470    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx" podUID="ad795212-732d-4487-8321-0cb33464f64f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 19:00:12.767913 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:00:12.767875    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d" podUID="440a82cc-2ac4-4b80-ac8d-075af9ca1394" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 19:00:20.754433 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:00:20.754390    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx" podUID="ad795212-732d-4487-8321-0cb33464f64f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 19:00:22.768100 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:00:22.768063    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d" podUID="440a82cc-2ac4-4b80-ac8d-075af9ca1394" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 19:00:30.755218 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:00:30.755168    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx" podUID="ad795212-732d-4487-8321-0cb33464f64f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 19:00:32.768293 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:00:32.768253    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d" podUID="440a82cc-2ac4-4b80-ac8d-075af9ca1394" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 19:00:40.754853 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:00:40.754820    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx"
Jul 01 19:00:42.768542 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:00:42.768512    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d"
Jul 01 19:00:53.923968 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:00:53.923936    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:00:54.020856 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:00:54.020827    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:01:41.298272 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:01:41.298241    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:06:48.554081 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:06:48.554045    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:07:58.738171 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.738138    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:07:58.880941 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.880905    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:07:58.938219 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.938181    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw"]
Jul 01 19:07:58.939217 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.938478    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw" podUID="7f36625f-48ef-4d40-94d1-85d3370da2d4" containerName="kserve-container" containerID="cri-o://4136e918c7801fec02bf75b3ee33c05e674f631729fb5070567aa180470d0a49" gracePeriod=30
Jul 01 19:07:58.939217 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.938665    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw" podUID="7f36625f-48ef-4d40-94d1-85d3370da2d4" containerName="kube-rbac-proxy" containerID="cri-o://d2a79b5db9d4fab835c5ffeb5e6c2010e0474d3bff5a89da83cc7f80da9cf215" gracePeriod=30
Jul 01 19:07:58.944274 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.944241    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx"]
Jul 01 19:07:58.945338 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.944670    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baaaead7-e419-4611-a8f1-4b336faa25b0" containerName="kube-rbac-proxy"
Jul 01 19:07:58.945338 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.944687    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="baaaead7-e419-4611-a8f1-4b336faa25b0" containerName="kube-rbac-proxy"
Jul 01 19:07:58.945338 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.944729    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baaaead7-e419-4611-a8f1-4b336faa25b0" containerName="kserve-container"
Jul 01 19:07:58.945338 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.944738    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="baaaead7-e419-4611-a8f1-4b336faa25b0" containerName="kserve-container"
Jul 01 19:07:58.945338 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.944754    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="247c587d-b6d5-4fff-a212-771964aecffe" containerName="kserve-container"
Jul 01 19:07:58.945338 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.944761    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="247c587d-b6d5-4fff-a212-771964aecffe" containerName="kserve-container"
Jul 01 19:07:58.945338 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.944770    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="247c587d-b6d5-4fff-a212-771964aecffe" containerName="kube-rbac-proxy"
Jul 01 19:07:58.945338 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.944779    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="247c587d-b6d5-4fff-a212-771964aecffe" containerName="kube-rbac-proxy"
Jul 01 19:07:58.945338 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.944895    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="247c587d-b6d5-4fff-a212-771964aecffe" containerName="kserve-container"
Jul 01 19:07:58.945338 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.944909    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="247c587d-b6d5-4fff-a212-771964aecffe" containerName="kube-rbac-proxy"
Jul 01 19:07:58.945338 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.944921    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="baaaead7-e419-4611-a8f1-4b336faa25b0" containerName="kube-rbac-proxy"
Jul 01 19:07:58.945338 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.944933    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="baaaead7-e419-4611-a8f1-4b336faa25b0" containerName="kserve-container"
Jul 01 19:07:58.948287 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.948260    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx"
Jul 01 19:07:58.950581 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.950561    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1bf2c-kube-rbac-proxy-sar-config\""
Jul 01 19:07:58.950664 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.950636    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1bf2c-predictor-serving-cert\""
Jul 01 19:07:58.957389 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.957337    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx"]
Jul 01 19:07:58.969018 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.968993    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:07:58.980349 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.980322    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzb9w\" (UniqueName: \"kubernetes.io/projected/8230e7b5-b543-452a-9a46-4fd627ad3138-kube-api-access-rzb9w\") pod \"success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx\" (UID: \"8230e7b5-b543-452a-9a46-4fd627ad3138\") " pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx"
Jul 01 19:07:58.980472 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.980361    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8230e7b5-b543-452a-9a46-4fd627ad3138-proxy-tls\") pod \"success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx\" (UID: \"8230e7b5-b543-452a-9a46-4fd627ad3138\") " pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx"
Jul 01 19:07:58.980472 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:58.980383    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8230e7b5-b543-452a-9a46-4fd627ad3138-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx\" (UID: \"8230e7b5-b543-452a-9a46-4fd627ad3138\") " pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx"
Jul 01 19:07:59.018962 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.018891    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5"]
Jul 01 19:07:59.022120 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.022100    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5"
Jul 01 19:07:59.024533 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.024509    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1bf2c-predictor-serving-cert\""
Jul 01 19:07:59.024657 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.024513    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1bf2c-kube-rbac-proxy-sar-config\""
Jul 01 19:07:59.033674 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.033651    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5"]
Jul 01 19:07:59.061353 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.061324    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:07:59.070678 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.070653    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd"]
Jul 01 19:07:59.070894 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.070871    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd" podUID="dcffa820-58d6-4ab1-8975-e6adcaf9f0de" containerName="kserve-container" containerID="cri-o://f7d531e11506969ef31d53c00539ff8f9a027a295cf26d5b54958a5d87695e47" gracePeriod=30
Jul 01 19:07:59.070968 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.070915    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd" podUID="dcffa820-58d6-4ab1-8975-e6adcaf9f0de" containerName="kube-rbac-proxy" containerID="cri-o://2012716c0c4add2e580c55306448aa32acf50a0d45886c457dc849e453b8bd38" gracePeriod=30
Jul 01 19:07:59.080914 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.080890    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a23e601c-df80-4e65-9d8e-a63cf7bbbc1c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5\" (UID: \"a23e601c-df80-4e65-9d8e-a63cf7bbbc1c\") " pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5"
Jul 01 19:07:59.081016 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.080935    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a23e601c-df80-4e65-9d8e-a63cf7bbbc1c-proxy-tls\") pod \"error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5\" (UID: \"a23e601c-df80-4e65-9d8e-a63cf7bbbc1c\") " pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5"
Jul 01 19:07:59.081083 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.081035    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzb9w\" (UniqueName: \"kubernetes.io/projected/8230e7b5-b543-452a-9a46-4fd627ad3138-kube-api-access-rzb9w\") pod \"success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx\" (UID: \"8230e7b5-b543-452a-9a46-4fd627ad3138\") " pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx"
Jul 01 19:07:59.081143 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.081087    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8230e7b5-b543-452a-9a46-4fd627ad3138-proxy-tls\") pod \"success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx\" (UID: \"8230e7b5-b543-452a-9a46-4fd627ad3138\") " pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx"
Jul 01 19:07:59.081143 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.081132    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8230e7b5-b543-452a-9a46-4fd627ad3138-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx\" (UID: \"8230e7b5-b543-452a-9a46-4fd627ad3138\") " pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx"
Jul 01 19:07:59.081249 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.081184    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmtk2\" (UniqueName: \"kubernetes.io/projected/a23e601c-df80-4e65-9d8e-a63cf7bbbc1c-kube-api-access-mmtk2\") pod \"error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5\" (UID: \"a23e601c-df80-4e65-9d8e-a63cf7bbbc1c\") " pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5"
Jul 01 19:07:59.082034 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.081901    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8230e7b5-b543-452a-9a46-4fd627ad3138-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx\" (UID: \"8230e7b5-b543-452a-9a46-4fd627ad3138\") " pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx"
Jul 01 19:07:59.083530 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.083510    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8230e7b5-b543-452a-9a46-4fd627ad3138-proxy-tls\") pod \"success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx\" (UID: \"8230e7b5-b543-452a-9a46-4fd627ad3138\") " pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx"
Jul 01 19:07:59.089308 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.089289    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzb9w\" (UniqueName: \"kubernetes.io/projected/8230e7b5-b543-452a-9a46-4fd627ad3138-kube-api-access-rzb9w\") pod \"success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx\" (UID: \"8230e7b5-b543-452a-9a46-4fd627ad3138\") " pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx"
Jul 01 19:07:59.157463 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.157416    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:07:59.181592 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.181567    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a23e601c-df80-4e65-9d8e-a63cf7bbbc1c-proxy-tls\") pod \"error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5\" (UID: \"a23e601c-df80-4e65-9d8e-a63cf7bbbc1c\") " pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5"
Jul 01 19:07:59.181773 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.181671    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmtk2\" (UniqueName: \"kubernetes.io/projected/a23e601c-df80-4e65-9d8e-a63cf7bbbc1c-kube-api-access-mmtk2\") pod \"error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5\" (UID: \"a23e601c-df80-4e65-9d8e-a63cf7bbbc1c\") " pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5"
Jul 01 19:07:59.181773 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:07:59.181726    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-serving-cert: secret "error-404-isvc-1bf2c-predictor-serving-cert" not found
Jul 01 19:07:59.181773 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.181744    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a23e601c-df80-4e65-9d8e-a63cf7bbbc1c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5\" (UID: \"a23e601c-df80-4e65-9d8e-a63cf7bbbc1c\") " pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5"
Jul 01 19:07:59.182020 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:07:59.181800    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a23e601c-df80-4e65-9d8e-a63cf7bbbc1c-proxy-tls podName:a23e601c-df80-4e65-9d8e-a63cf7bbbc1c nodeName:}" failed. No retries permitted until 2026-07-01 19:07:59.681784016 +0000 UTC m=+1830.450621596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a23e601c-df80-4e65-9d8e-a63cf7bbbc1c-proxy-tls") pod "error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5" (UID: "a23e601c-df80-4e65-9d8e-a63cf7bbbc1c") : secret "error-404-isvc-1bf2c-predictor-serving-cert" not found
Jul 01 19:07:59.182390 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.182367    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a23e601c-df80-4e65-9d8e-a63cf7bbbc1c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5\" (UID: \"a23e601c-df80-4e65-9d8e-a63cf7bbbc1c\") " pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5"
Jul 01 19:07:59.196087 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.196062    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmtk2\" (UniqueName: \"kubernetes.io/projected/a23e601c-df80-4e65-9d8e-a63cf7bbbc1c-kube-api-access-mmtk2\") pod \"error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5\" (UID: \"a23e601c-df80-4e65-9d8e-a63cf7bbbc1c\") " pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5"
Jul 01 19:07:59.257941 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.257913    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:07:59.266847 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.266829    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx"
Jul 01 19:07:59.396230 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.396209    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx"]
Jul 01 19:07:59.398537 ip-10-0-129-245 kubenswrapper[2573]: W0701 19:07:59.398486    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8230e7b5_b543_452a_9a46_4fd627ad3138.slice/crio-06df7c82f604d02c5b6b773a329710e5fde72636f44819f7dc08323cb8702cf1 WatchSource:0}: Error finding container 06df7c82f604d02c5b6b773a329710e5fde72636f44819f7dc08323cb8702cf1: Status 404 returned error can't find the container with id 06df7c82f604d02c5b6b773a329710e5fde72636f44819f7dc08323cb8702cf1
Jul 01 19:07:59.598429 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.598350    2573 generic.go:356] "Generic (PLEG): container finished" podID="7f36625f-48ef-4d40-94d1-85d3370da2d4" containerID="d2a79b5db9d4fab835c5ffeb5e6c2010e0474d3bff5a89da83cc7f80da9cf215" exitCode=2
Jul 01 19:07:59.598585 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.598428    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw" event={"ID":"7f36625f-48ef-4d40-94d1-85d3370da2d4","Type":"ContainerDied","Data":"d2a79b5db9d4fab835c5ffeb5e6c2010e0474d3bff5a89da83cc7f80da9cf215"}
Jul 01 19:07:59.600093 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.600071    2573 generic.go:356] "Generic (PLEG): container finished" podID="dcffa820-58d6-4ab1-8975-e6adcaf9f0de" containerID="2012716c0c4add2e580c55306448aa32acf50a0d45886c457dc849e453b8bd38" exitCode=2
Jul 01 19:07:59.600196 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.600145    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd" event={"ID":"dcffa820-58d6-4ab1-8975-e6adcaf9f0de","Type":"ContainerDied","Data":"2012716c0c4add2e580c55306448aa32acf50a0d45886c457dc849e453b8bd38"}
Jul 01 19:07:59.601702 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.601679    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx" event={"ID":"8230e7b5-b543-452a-9a46-4fd627ad3138","Type":"ContainerStarted","Data":"90bc3f70f6da896a8fe2884179d58637f69228c32fca9bec9d2cc8497fa61a73"}
Jul 01 19:07:59.601833 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.601749    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx" event={"ID":"8230e7b5-b543-452a-9a46-4fd627ad3138","Type":"ContainerStarted","Data":"73c7ef61e1903cbb23b4a4b7af7aabb70f2f62bc271e6539e8d37b98e8c0f803"}
Jul 01 19:07:59.601833 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.601781    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx" event={"ID":"8230e7b5-b543-452a-9a46-4fd627ad3138","Type":"ContainerStarted","Data":"06df7c82f604d02c5b6b773a329710e5fde72636f44819f7dc08323cb8702cf1"}
Jul 01 19:07:59.601920 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.601833    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx"
Jul 01 19:07:59.627034 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.626997    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx" podStartSLOduration=1.626984867 podStartE2EDuration="1.626984867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:07:59.625840986 +0000 UTC m=+1830.394678589" watchObservedRunningTime="2026-07-01 19:07:59.626984867 +0000 UTC m=+1830.395822498"
Jul 01 19:07:59.683889 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.683857    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a23e601c-df80-4e65-9d8e-a63cf7bbbc1c-proxy-tls\") pod \"error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5\" (UID: \"a23e601c-df80-4e65-9d8e-a63cf7bbbc1c\") " pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5"
Jul 01 19:07:59.686171 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.686150    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a23e601c-df80-4e65-9d8e-a63cf7bbbc1c-proxy-tls\") pod \"error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5\" (UID: \"a23e601c-df80-4e65-9d8e-a63cf7bbbc1c\") " pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5"
Jul 01 19:07:59.934899 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:07:59.934810    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5"
Jul 01 19:08:00.067229 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:00.067203    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5"]
Jul 01 19:08:00.068747 ip-10-0-129-245 kubenswrapper[2573]: W0701 19:08:00.068695    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda23e601c_df80_4e65_9d8e_a63cf7bbbc1c.slice/crio-3ff587713b2e55a2b8bbe33e84939999595aac4f54c9e4abd3e2a5f154bd3da7 WatchSource:0}: Error finding container 3ff587713b2e55a2b8bbe33e84939999595aac4f54c9e4abd3e2a5f154bd3da7: Status 404 returned error can't find the container with id 3ff587713b2e55a2b8bbe33e84939999595aac4f54c9e4abd3e2a5f154bd3da7
Jul 01 19:08:00.607917 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:00.607878    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5" event={"ID":"a23e601c-df80-4e65-9d8e-a63cf7bbbc1c","Type":"ContainerStarted","Data":"8a260f858816d1ee0f898739cd48f9d264c329c48640bd37b103d9423d1a1333"}
Jul 01 19:08:00.608112 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:00.607943    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5" event={"ID":"a23e601c-df80-4e65-9d8e-a63cf7bbbc1c","Type":"ContainerStarted","Data":"1d56ba1fb662a8d0210e5bf9d6096d247d7c5c6d1e03d24bda4b11201de34d8d"}
Jul 01 19:08:00.608232 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:00.608213    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx"
Jul 01 19:08:00.608291 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:00.608274    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5"
Jul 01 19:08:00.608347 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:00.608301    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5"
Jul 01 19:08:00.608347 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:00.608325    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5" event={"ID":"a23e601c-df80-4e65-9d8e-a63cf7bbbc1c","Type":"ContainerStarted","Data":"3ff587713b2e55a2b8bbe33e84939999595aac4f54c9e4abd3e2a5f154bd3da7"}
Jul 01 19:08:00.609126 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:00.609101    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5" podUID="a23e601c-df80-4e65-9d8e-a63cf7bbbc1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 01 19:08:00.609221 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:00.609200    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx" podUID="8230e7b5-b543-452a-9a46-4fd627ad3138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 01 19:08:00.625581 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:00.625537    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5" podStartSLOduration=2.625519799 podStartE2EDuration="2.625519799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:08:00.625333752 +0000 UTC m=+1831.394171367" watchObservedRunningTime="2026-07-01 19:08:00.625519799 +0000 UTC m=+1831.394357405"
Jul 01 19:08:01.612359 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:01.612322    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx" podUID="8230e7b5-b543-452a-9a46-4fd627ad3138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 01 19:08:01.612809 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:01.612426    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5" podUID="a23e601c-df80-4e65-9d8e-a63cf7bbbc1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 01 19:08:02.429363 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.429336    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd"
Jul 01 19:08:02.456127 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.456090    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw" podUID="7f36625f-48ef-4d40-94d1-85d3370da2d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 01 19:08:02.460378 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.460350    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw" podUID="7f36625f-48ef-4d40-94d1-85d3370da2d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 01 19:08:02.501453 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.501365    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcffa820-58d6-4ab1-8975-e6adcaf9f0de-proxy-tls\") pod \"dcffa820-58d6-4ab1-8975-e6adcaf9f0de\" (UID: \"dcffa820-58d6-4ab1-8975-e6adcaf9f0de\") "
Jul 01 19:08:02.501453 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.501417    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56kd4\" (UniqueName: \"kubernetes.io/projected/dcffa820-58d6-4ab1-8975-e6adcaf9f0de-kube-api-access-56kd4\") pod \"dcffa820-58d6-4ab1-8975-e6adcaf9f0de\" (UID: \"dcffa820-58d6-4ab1-8975-e6adcaf9f0de\") "
Jul 01 19:08:02.501453 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.501448    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcffa820-58d6-4ab1-8975-e6adcaf9f0de-kube-rbac-proxy-sar-config\") pod \"dcffa820-58d6-4ab1-8975-e6adcaf9f0de\" (UID: \"dcffa820-58d6-4ab1-8975-e6adcaf9f0de\") "
Jul 01 19:08:02.501886 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.501860    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcffa820-58d6-4ab1-8975-e6adcaf9f0de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dcffa820-58d6-4ab1-8975-e6adcaf9f0de" (UID: "dcffa820-58d6-4ab1-8975-e6adcaf9f0de"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:08:02.503549 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.503520    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcffa820-58d6-4ab1-8975-e6adcaf9f0de-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dcffa820-58d6-4ab1-8975-e6adcaf9f0de" (UID: "dcffa820-58d6-4ab1-8975-e6adcaf9f0de"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:08:02.503549 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.503544    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcffa820-58d6-4ab1-8975-e6adcaf9f0de-kube-api-access-56kd4" (OuterVolumeSpecName: "kube-api-access-56kd4") pod "dcffa820-58d6-4ab1-8975-e6adcaf9f0de" (UID: "dcffa820-58d6-4ab1-8975-e6adcaf9f0de"). InnerVolumeSpecName "kube-api-access-56kd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:08:02.602228 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.602186    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcffa820-58d6-4ab1-8975-e6adcaf9f0de-proxy-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:08:02.602228 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.602220    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56kd4\" (UniqueName: \"kubernetes.io/projected/dcffa820-58d6-4ab1-8975-e6adcaf9f0de-kube-api-access-56kd4\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:08:02.602228 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.602231    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcffa820-58d6-4ab1-8975-e6adcaf9f0de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:08:02.617479 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.617449    2573 generic.go:356] "Generic (PLEG): container finished" podID="dcffa820-58d6-4ab1-8975-e6adcaf9f0de" containerID="f7d531e11506969ef31d53c00539ff8f9a027a295cf26d5b54958a5d87695e47" exitCode=0
Jul 01 19:08:02.617964 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.617514    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd"
Jul 01 19:08:02.617964 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.617540    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd" event={"ID":"dcffa820-58d6-4ab1-8975-e6adcaf9f0de","Type":"ContainerDied","Data":"f7d531e11506969ef31d53c00539ff8f9a027a295cf26d5b54958a5d87695e47"}
Jul 01 19:08:02.617964 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.617592    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd" event={"ID":"dcffa820-58d6-4ab1-8975-e6adcaf9f0de","Type":"ContainerDied","Data":"fb5e003cb37d821e7a446ac49f0d0248c683924458fe8573f8f803208dace778"}
Jul 01 19:08:02.617964 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.617622    2573 scope.go:117] "RemoveContainer" containerID="2012716c0c4add2e580c55306448aa32acf50a0d45886c457dc849e453b8bd38"
Jul 01 19:08:02.626352 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.626333    2573 scope.go:117] "RemoveContainer" containerID="f7d531e11506969ef31d53c00539ff8f9a027a295cf26d5b54958a5d87695e47"
Jul 01 19:08:02.637966 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.637942    2573 scope.go:117] "RemoveContainer" containerID="2012716c0c4add2e580c55306448aa32acf50a0d45886c457dc849e453b8bd38"
Jul 01 19:08:02.638336 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:08:02.638307    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2012716c0c4add2e580c55306448aa32acf50a0d45886c457dc849e453b8bd38\": container with ID starting with 2012716c0c4add2e580c55306448aa32acf50a0d45886c457dc849e453b8bd38 not found: ID does not exist" containerID="2012716c0c4add2e580c55306448aa32acf50a0d45886c457dc849e453b8bd38"
Jul 01 19:08:02.638437 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.638349    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2012716c0c4add2e580c55306448aa32acf50a0d45886c457dc849e453b8bd38"} err="failed to get container status \"2012716c0c4add2e580c55306448aa32acf50a0d45886c457dc849e453b8bd38\": rpc error: code = NotFound desc = could not find container \"2012716c0c4add2e580c55306448aa32acf50a0d45886c457dc849e453b8bd38\": container with ID starting with 2012716c0c4add2e580c55306448aa32acf50a0d45886c457dc849e453b8bd38 not found: ID does not exist"
Jul 01 19:08:02.638437 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.638375    2573 scope.go:117] "RemoveContainer" containerID="f7d531e11506969ef31d53c00539ff8f9a027a295cf26d5b54958a5d87695e47"
Jul 01 19:08:02.638771 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:08:02.638700    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7d531e11506969ef31d53c00539ff8f9a027a295cf26d5b54958a5d87695e47\": container with ID starting with f7d531e11506969ef31d53c00539ff8f9a027a295cf26d5b54958a5d87695e47 not found: ID does not exist" containerID="f7d531e11506969ef31d53c00539ff8f9a027a295cf26d5b54958a5d87695e47"
Jul 01 19:08:02.638771 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.638742    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d531e11506969ef31d53c00539ff8f9a027a295cf26d5b54958a5d87695e47"} err="failed to get container status \"f7d531e11506969ef31d53c00539ff8f9a027a295cf26d5b54958a5d87695e47\": rpc error: code = NotFound desc = could not find container \"f7d531e11506969ef31d53c00539ff8f9a027a295cf26d5b54958a5d87695e47\": container with ID starting with f7d531e11506969ef31d53c00539ff8f9a027a295cf26d5b54958a5d87695e47 not found: ID does not exist"
Jul 01 19:08:02.641255 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.641229    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd"]
Jul 01 19:08:02.645330 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.645308    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-af621-predictor-6c7cdbfc8d-gsrwd"]
Jul 01 19:08:02.778833 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.778807    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw"
Jul 01 19:08:02.803587 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.803558    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f36625f-48ef-4d40-94d1-85d3370da2d4-proxy-tls\") pod \"7f36625f-48ef-4d40-94d1-85d3370da2d4\" (UID: \"7f36625f-48ef-4d40-94d1-85d3370da2d4\") "
Jul 01 19:08:02.803755 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.803641    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f36625f-48ef-4d40-94d1-85d3370da2d4-kube-rbac-proxy-sar-config\") pod \"7f36625f-48ef-4d40-94d1-85d3370da2d4\" (UID: \"7f36625f-48ef-4d40-94d1-85d3370da2d4\") "
Jul 01 19:08:02.804020 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.803989    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f36625f-48ef-4d40-94d1-85d3370da2d4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7f36625f-48ef-4d40-94d1-85d3370da2d4" (UID: "7f36625f-48ef-4d40-94d1-85d3370da2d4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:08:02.805569 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.805542    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f36625f-48ef-4d40-94d1-85d3370da2d4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f36625f-48ef-4d40-94d1-85d3370da2d4" (UID: "7f36625f-48ef-4d40-94d1-85d3370da2d4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:08:02.904134 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.904096    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pkqf\" (UniqueName: \"kubernetes.io/projected/7f36625f-48ef-4d40-94d1-85d3370da2d4-kube-api-access-4pkqf\") pod \"7f36625f-48ef-4d40-94d1-85d3370da2d4\" (UID: \"7f36625f-48ef-4d40-94d1-85d3370da2d4\") "
Jul 01 19:08:02.904308 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.904288    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f36625f-48ef-4d40-94d1-85d3370da2d4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:08:02.904308 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.904306    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f36625f-48ef-4d40-94d1-85d3370da2d4-proxy-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:08:02.906123 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:02.906103    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f36625f-48ef-4d40-94d1-85d3370da2d4-kube-api-access-4pkqf" (OuterVolumeSpecName: "kube-api-access-4pkqf") pod "7f36625f-48ef-4d40-94d1-85d3370da2d4" (UID: "7f36625f-48ef-4d40-94d1-85d3370da2d4"). InnerVolumeSpecName "kube-api-access-4pkqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:08:03.004651 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:03.004614    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pkqf\" (UniqueName: \"kubernetes.io/projected/7f36625f-48ef-4d40-94d1-85d3370da2d4-kube-api-access-4pkqf\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:08:03.623681 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:03.623585    2573 generic.go:356] "Generic (PLEG): container finished" podID="7f36625f-48ef-4d40-94d1-85d3370da2d4" containerID="4136e918c7801fec02bf75b3ee33c05e674f631729fb5070567aa180470d0a49" exitCode=0
Jul 01 19:08:03.623681 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:03.623639    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw"
Jul 01 19:08:03.623681 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:03.623672    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw" event={"ID":"7f36625f-48ef-4d40-94d1-85d3370da2d4","Type":"ContainerDied","Data":"4136e918c7801fec02bf75b3ee33c05e674f631729fb5070567aa180470d0a49"}
Jul 01 19:08:03.624219 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:03.623750    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw" event={"ID":"7f36625f-48ef-4d40-94d1-85d3370da2d4","Type":"ContainerDied","Data":"173f1065fbaa2d679c1db0159450720a8b8f7903f5a3da48441b6b62d3919c54"}
Jul 01 19:08:03.624219 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:03.623786    2573 scope.go:117] "RemoveContainer" containerID="d2a79b5db9d4fab835c5ffeb5e6c2010e0474d3bff5a89da83cc7f80da9cf215"
Jul 01 19:08:03.631957 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:03.631937    2573 scope.go:117] "RemoveContainer" containerID="4136e918c7801fec02bf75b3ee33c05e674f631729fb5070567aa180470d0a49"
Jul 01 19:08:03.641247 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:03.641229    2573 scope.go:117] "RemoveContainer" containerID="d2a79b5db9d4fab835c5ffeb5e6c2010e0474d3bff5a89da83cc7f80da9cf215"
Jul 01 19:08:03.641526 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:08:03.641507    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2a79b5db9d4fab835c5ffeb5e6c2010e0474d3bff5a89da83cc7f80da9cf215\": container with ID starting with d2a79b5db9d4fab835c5ffeb5e6c2010e0474d3bff5a89da83cc7f80da9cf215 not found: ID does not exist" containerID="d2a79b5db9d4fab835c5ffeb5e6c2010e0474d3bff5a89da83cc7f80da9cf215"
Jul 01 19:08:03.641584 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:03.641533    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2a79b5db9d4fab835c5ffeb5e6c2010e0474d3bff5a89da83cc7f80da9cf215"} err="failed to get container status \"d2a79b5db9d4fab835c5ffeb5e6c2010e0474d3bff5a89da83cc7f80da9cf215\": rpc error: code = NotFound desc = could not find container \"d2a79b5db9d4fab835c5ffeb5e6c2010e0474d3bff5a89da83cc7f80da9cf215\": container with ID starting with d2a79b5db9d4fab835c5ffeb5e6c2010e0474d3bff5a89da83cc7f80da9cf215 not found: ID does not exist"
Jul 01 19:08:03.641584 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:03.641552    2573 scope.go:117] "RemoveContainer" containerID="4136e918c7801fec02bf75b3ee33c05e674f631729fb5070567aa180470d0a49"
Jul 01 19:08:03.641829 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:08:03.641812    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4136e918c7801fec02bf75b3ee33c05e674f631729fb5070567aa180470d0a49\": container with ID starting with 4136e918c7801fec02bf75b3ee33c05e674f631729fb5070567aa180470d0a49 not found: ID does not exist" containerID="4136e918c7801fec02bf75b3ee33c05e674f631729fb5070567aa180470d0a49"
Jul 01 19:08:03.641888 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:03.641834    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4136e918c7801fec02bf75b3ee33c05e674f631729fb5070567aa180470d0a49"} err="failed to get container status \"4136e918c7801fec02bf75b3ee33c05e674f631729fb5070567aa180470d0a49\": rpc error: code = NotFound desc = could not find container \"4136e918c7801fec02bf75b3ee33c05e674f631729fb5070567aa180470d0a49\": container with ID starting with 4136e918c7801fec02bf75b3ee33c05e674f631729fb5070567aa180470d0a49 not found: ID does not exist"
Jul 01 19:08:03.645996 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:03.645970    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw"]
Jul 01 19:08:03.650126 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:03.650104    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-af621-predictor-688d7d5cb9-p4nrw"]
Jul 01 19:08:03.666099 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:03.666070    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f36625f-48ef-4d40-94d1-85d3370da2d4" path="/var/lib/kubelet/pods/7f36625f-48ef-4d40-94d1-85d3370da2d4/volumes"
Jul 01 19:08:03.666502 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:03.666489    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcffa820-58d6-4ab1-8975-e6adcaf9f0de" path="/var/lib/kubelet/pods/dcffa820-58d6-4ab1-8975-e6adcaf9f0de/volumes"
Jul 01 19:08:06.617062 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:06.617032    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5"
Jul 01 19:08:06.617507 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:06.617486    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx"
Jul 01 19:08:06.617651 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:06.617618    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5" podUID="a23e601c-df80-4e65-9d8e-a63cf7bbbc1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 01 19:08:06.617962 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:06.617941    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx" podUID="8230e7b5-b543-452a-9a46-4fd627ad3138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 01 19:08:16.617992 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:16.617957    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx" podUID="8230e7b5-b543-452a-9a46-4fd627ad3138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 01 19:08:16.618384 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:16.617968    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5" podUID="a23e601c-df80-4e65-9d8e-a63cf7bbbc1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 01 19:08:26.618122 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:26.618080    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx" podUID="8230e7b5-b543-452a-9a46-4fd627ad3138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 01 19:08:26.618492 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:26.618081    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5" podUID="a23e601c-df80-4e65-9d8e-a63cf7bbbc1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 01 19:08:36.618246 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:36.618208    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx" podUID="8230e7b5-b543-452a-9a46-4fd627ad3138" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 01 19:08:36.618653 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:36.618208    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5" podUID="a23e601c-df80-4e65-9d8e-a63cf7bbbc1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 01 19:08:46.618638 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:46.618605    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx"
Jul 01 19:08:46.619043 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:46.618691    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5"
Jul 01 19:08:58.943275 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:58.943229    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:08:59.071833 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:08:59.071802    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:09:08.506058 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.506022    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:09:08.605605 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.605568    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d"]
Jul 01 19:09:08.605876 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.605852    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d" podUID="440a82cc-2ac4-4b80-ac8d-075af9ca1394" containerName="kserve-container" containerID="cri-o://db4c870c3b3f5f6316c6628b6673e083b56ed77665bf3d2ac8e581dd6ba11cee" gracePeriod=30
Jul 01 19:09:08.605969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.605941    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d" podUID="440a82cc-2ac4-4b80-ac8d-075af9ca1394" containerName="kube-rbac-proxy" containerID="cri-o://0574cb55b33abc0cfc84f6de08bd6688fe1715bf146d92b20e2bb64890ece05e" gracePeriod=30
Jul 01 19:09:08.620009 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.619986    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:09:08.638961 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.638935    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw"]
Jul 01 19:09:08.639394 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.639366    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcffa820-58d6-4ab1-8975-e6adcaf9f0de" containerName="kube-rbac-proxy"
Jul 01 19:09:08.639394 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.639384    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcffa820-58d6-4ab1-8975-e6adcaf9f0de" containerName="kube-rbac-proxy"
Jul 01 19:09:08.639516 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.639401    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f36625f-48ef-4d40-94d1-85d3370da2d4" containerName="kserve-container"
Jul 01 19:09:08.639516 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.639410    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f36625f-48ef-4d40-94d1-85d3370da2d4" containerName="kserve-container"
Jul 01 19:09:08.639516 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.639423    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f36625f-48ef-4d40-94d1-85d3370da2d4" containerName="kube-rbac-proxy"
Jul 01 19:09:08.639516 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.639431    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f36625f-48ef-4d40-94d1-85d3370da2d4" containerName="kube-rbac-proxy"
Jul 01 19:09:08.639516 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.639445    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcffa820-58d6-4ab1-8975-e6adcaf9f0de" containerName="kserve-container"
Jul 01 19:09:08.639516 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.639453    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcffa820-58d6-4ab1-8975-e6adcaf9f0de" containerName="kserve-container"
Jul 01 19:09:08.639703 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.639545    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f36625f-48ef-4d40-94d1-85d3370da2d4" containerName="kube-rbac-proxy"
Jul 01 19:09:08.639703 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.639558    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcffa820-58d6-4ab1-8975-e6adcaf9f0de" containerName="kserve-container"
Jul 01 19:09:08.639703 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.639568    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcffa820-58d6-4ab1-8975-e6adcaf9f0de" containerName="kube-rbac-proxy"
Jul 01 19:09:08.639703 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.639575    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f36625f-48ef-4d40-94d1-85d3370da2d4" containerName="kserve-container"
Jul 01 19:09:08.643064 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.643048    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw"
Jul 01 19:09:08.645237 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.645217    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-43223-kube-rbac-proxy-sar-config\""
Jul 01 19:09:08.645402 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.645384    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-43223-predictor-serving-cert\""
Jul 01 19:09:08.652370 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.652342    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw"]
Jul 01 19:09:08.709202 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.709170    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx"]
Jul 01 19:09:08.709542 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.709480    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx" podUID="ad795212-732d-4487-8321-0cb33464f64f" containerName="kserve-container" containerID="cri-o://5420180e39b85216f86c753358ca6baded0d7a5020a6a4e3250e0bed41e41a4d" gracePeriod=30
Jul 01 19:09:08.710241 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.709525    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx" podUID="ad795212-732d-4487-8321-0cb33464f64f" containerName="kube-rbac-proxy" containerID="cri-o://95050154e2b26a219bbc25bfe13fb321a1ef8d787e63705c1996379c0efa7d82" gracePeriod=30
Jul 01 19:09:08.711962 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.711943    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:09:08.748591 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.748562    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj"]
Jul 01 19:09:08.751648 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.751627    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj"
Jul 01 19:09:08.753859 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.753836    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-43223-predictor-serving-cert\""
Jul 01 19:09:08.754119 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.754105    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-43223-kube-rbac-proxy-sar-config\""
Jul 01 19:09:08.761402 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.761382    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj"]
Jul 01 19:09:08.787325 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.787296    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:09:08.818849 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.818815    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw6q8\" (UniqueName: \"kubernetes.io/projected/a416bd65-9851-446c-a152-59fa2adab5d3-kube-api-access-hw6q8\") pod \"success-200-isvc-43223-predictor-85967567b9-fkcgw\" (UID: \"a416bd65-9851-446c-a152-59fa2adab5d3\") " pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw"
Jul 01 19:09:08.818998 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.818877    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6vmf\" (UniqueName: \"kubernetes.io/projected/45c55b1a-7c32-4a6c-8b7a-f83bae151d1a-kube-api-access-v6vmf\") pod \"error-404-isvc-43223-predictor-5968f7b98d-s44nj\" (UID: \"45c55b1a-7c32-4a6c-8b7a-f83bae151d1a\") " pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj"
Jul 01 19:09:08.818998 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.818946    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45c55b1a-7c32-4a6c-8b7a-f83bae151d1a-proxy-tls\") pod \"error-404-isvc-43223-predictor-5968f7b98d-s44nj\" (UID: \"45c55b1a-7c32-4a6c-8b7a-f83bae151d1a\") " pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj"
Jul 01 19:09:08.818998 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.818984    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45c55b1a-7c32-4a6c-8b7a-f83bae151d1a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-43223-predictor-5968f7b98d-s44nj\" (UID: \"45c55b1a-7c32-4a6c-8b7a-f83bae151d1a\") " pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj"
Jul 01 19:09:08.819157 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.819024    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a416bd65-9851-446c-a152-59fa2adab5d3-proxy-tls\") pod \"success-200-isvc-43223-predictor-85967567b9-fkcgw\" (UID: \"a416bd65-9851-446c-a152-59fa2adab5d3\") " pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw"
Jul 01 19:09:08.819157 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.819066    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a416bd65-9851-446c-a152-59fa2adab5d3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-43223-predictor-85967567b9-fkcgw\" (UID: \"a416bd65-9851-446c-a152-59fa2adab5d3\") " pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw"
Jul 01 19:09:08.871353 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.871315    2573 generic.go:356] "Generic (PLEG): container finished" podID="ad795212-732d-4487-8321-0cb33464f64f" containerID="95050154e2b26a219bbc25bfe13fb321a1ef8d787e63705c1996379c0efa7d82" exitCode=2
Jul 01 19:09:08.871510 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.871379    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx" event={"ID":"ad795212-732d-4487-8321-0cb33464f64f","Type":"ContainerDied","Data":"95050154e2b26a219bbc25bfe13fb321a1ef8d787e63705c1996379c0efa7d82"}
Jul 01 19:09:08.873428 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.873402    2573 generic.go:356] "Generic (PLEG): container finished" podID="440a82cc-2ac4-4b80-ac8d-075af9ca1394" containerID="0574cb55b33abc0cfc84f6de08bd6688fe1715bf146d92b20e2bb64890ece05e" exitCode=2
Jul 01 19:09:08.873539 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.873476    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d" event={"ID":"440a82cc-2ac4-4b80-ac8d-075af9ca1394","Type":"ContainerDied","Data":"0574cb55b33abc0cfc84f6de08bd6688fe1715bf146d92b20e2bb64890ece05e"}
Jul 01 19:09:08.919865 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.919831    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw6q8\" (UniqueName: \"kubernetes.io/projected/a416bd65-9851-446c-a152-59fa2adab5d3-kube-api-access-hw6q8\") pod \"success-200-isvc-43223-predictor-85967567b9-fkcgw\" (UID: \"a416bd65-9851-446c-a152-59fa2adab5d3\") " pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw"
Jul 01 19:09:08.920021 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.919873    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6vmf\" (UniqueName: \"kubernetes.io/projected/45c55b1a-7c32-4a6c-8b7a-f83bae151d1a-kube-api-access-v6vmf\") pod \"error-404-isvc-43223-predictor-5968f7b98d-s44nj\" (UID: \"45c55b1a-7c32-4a6c-8b7a-f83bae151d1a\") " pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj"
Jul 01 19:09:08.920105 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.920080    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45c55b1a-7c32-4a6c-8b7a-f83bae151d1a-proxy-tls\") pod \"error-404-isvc-43223-predictor-5968f7b98d-s44nj\" (UID: \"45c55b1a-7c32-4a6c-8b7a-f83bae151d1a\") " pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj"
Jul 01 19:09:08.920157 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.920131    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45c55b1a-7c32-4a6c-8b7a-f83bae151d1a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-43223-predictor-5968f7b98d-s44nj\" (UID: \"45c55b1a-7c32-4a6c-8b7a-f83bae151d1a\") " pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj"
Jul 01 19:09:08.920202 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.920162    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a416bd65-9851-446c-a152-59fa2adab5d3-proxy-tls\") pod \"success-200-isvc-43223-predictor-85967567b9-fkcgw\" (UID: \"a416bd65-9851-446c-a152-59fa2adab5d3\") " pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw"
Jul 01 19:09:08.920252 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:09:08.920197    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-43223-predictor-serving-cert: secret "error-404-isvc-43223-predictor-serving-cert" not found
Jul 01 19:09:08.920319 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:09:08.920268    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45c55b1a-7c32-4a6c-8b7a-f83bae151d1a-proxy-tls podName:45c55b1a-7c32-4a6c-8b7a-f83bae151d1a nodeName:}" failed. No retries permitted until 2026-07-01 19:09:09.420239221 +0000 UTC m=+1900.189076821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/45c55b1a-7c32-4a6c-8b7a-f83bae151d1a-proxy-tls") pod "error-404-isvc-43223-predictor-5968f7b98d-s44nj" (UID: "45c55b1a-7c32-4a6c-8b7a-f83bae151d1a") : secret "error-404-isvc-43223-predictor-serving-cert" not found
Jul 01 19:09:08.920319 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.920200    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a416bd65-9851-446c-a152-59fa2adab5d3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-43223-predictor-85967567b9-fkcgw\" (UID: \"a416bd65-9851-446c-a152-59fa2adab5d3\") " pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw"
Jul 01 19:09:08.920425 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:09:08.920280    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-43223-predictor-serving-cert: secret "success-200-isvc-43223-predictor-serving-cert" not found
Jul 01 19:09:08.920425 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:09:08.920403    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a416bd65-9851-446c-a152-59fa2adab5d3-proxy-tls podName:a416bd65-9851-446c-a152-59fa2adab5d3 nodeName:}" failed. No retries permitted until 2026-07-01 19:09:09.420380752 +0000 UTC m=+1900.189218355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a416bd65-9851-446c-a152-59fa2adab5d3-proxy-tls") pod "success-200-isvc-43223-predictor-85967567b9-fkcgw" (UID: "a416bd65-9851-446c-a152-59fa2adab5d3") : secret "success-200-isvc-43223-predictor-serving-cert" not found
Jul 01 19:09:08.920886 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.920866    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45c55b1a-7c32-4a6c-8b7a-f83bae151d1a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-43223-predictor-5968f7b98d-s44nj\" (UID: \"45c55b1a-7c32-4a6c-8b7a-f83bae151d1a\") " pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj"
Jul 01 19:09:08.920978 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.920921    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a416bd65-9851-446c-a152-59fa2adab5d3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-43223-predictor-85967567b9-fkcgw\" (UID: \"a416bd65-9851-446c-a152-59fa2adab5d3\") " pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw"
Jul 01 19:09:08.931012 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.930983    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6vmf\" (UniqueName: \"kubernetes.io/projected/45c55b1a-7c32-4a6c-8b7a-f83bae151d1a-kube-api-access-v6vmf\") pod \"error-404-isvc-43223-predictor-5968f7b98d-s44nj\" (UID: \"45c55b1a-7c32-4a6c-8b7a-f83bae151d1a\") " pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj"
Jul 01 19:09:08.931698 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:08.931679    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw6q8\" (UniqueName: \"kubernetes.io/projected/a416bd65-9851-446c-a152-59fa2adab5d3-kube-api-access-hw6q8\") pod \"success-200-isvc-43223-predictor-85967567b9-fkcgw\" (UID: \"a416bd65-9851-446c-a152-59fa2adab5d3\") " pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw"
Jul 01 19:09:09.033350 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.033267    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:09:09.102568 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.102529    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5"]
Jul 01 19:09:09.102847 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.102820    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5" podUID="a23e601c-df80-4e65-9d8e-a63cf7bbbc1c" containerName="kserve-container" containerID="cri-o://1d56ba1fb662a8d0210e5bf9d6096d247d7c5c6d1e03d24bda4b11201de34d8d" gracePeriod=30
Jul 01 19:09:09.102941 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.102910    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5" podUID="a23e601c-df80-4e65-9d8e-a63cf7bbbc1c" containerName="kube-rbac-proxy" containerID="cri-o://8a260f858816d1ee0f898739cd48f9d264c329c48640bd37b103d9423d1a1333" gracePeriod=30
Jul 01 19:09:09.133872 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.133844    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:09:09.165322 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.165291    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z"]
Jul 01 19:09:09.169726 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.169681    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx"]
Jul 01 19:09:09.169853 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.169819    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z"
Jul 01 19:09:09.170169 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.170011    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx" podUID="8230e7b5-b543-452a-9a46-4fd627ad3138" containerName="kserve-container" containerID="cri-o://73c7ef61e1903cbb23b4a4b7af7aabb70f2f62bc271e6539e8d37b98e8c0f803" gracePeriod=30
Jul 01 19:09:09.170169 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.170046    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx" podUID="8230e7b5-b543-452a-9a46-4fd627ad3138" containerName="kube-rbac-proxy" containerID="cri-o://90bc3f70f6da896a8fe2884179d58637f69228c32fca9bec9d2cc8497fa61a73" gracePeriod=30
Jul 01 19:09:09.172318 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.172296    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4f643-predictor-serving-cert\""
Jul 01 19:09:09.172501 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.172296    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4f643-kube-rbac-proxy-sar-config\""
Jul 01 19:09:09.179058 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.178935    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z"]
Jul 01 19:09:09.202233 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.202203    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:09:09.223335 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.223313    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18547f9d-5826-4562-b935-5383beb63767-proxy-tls\") pod \"success-200-isvc-4f643-predictor-785bf555c9-xsk4z\" (UID: \"18547f9d-5826-4562-b935-5383beb63767\") " pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z"
Jul 01 19:09:09.223471 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.223451    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkgn9\" (UniqueName: \"kubernetes.io/projected/18547f9d-5826-4562-b935-5383beb63767-kube-api-access-nkgn9\") pod \"success-200-isvc-4f643-predictor-785bf555c9-xsk4z\" (UID: \"18547f9d-5826-4562-b935-5383beb63767\") " pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z"
Jul 01 19:09:09.223555 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.223533    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18547f9d-5826-4562-b935-5383beb63767-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4f643-predictor-785bf555c9-xsk4z\" (UID: \"18547f9d-5826-4562-b935-5383beb63767\") " pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z"
Jul 01 19:09:09.271824 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.271793    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm"]
Jul 01 19:09:09.275013 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.274991    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm"
Jul 01 19:09:09.277417 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.277397    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4f643-predictor-serving-cert\""
Jul 01 19:09:09.277521 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.277419    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4f643-kube-rbac-proxy-sar-config\""
Jul 01 19:09:09.285105 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.285034    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm"]
Jul 01 19:09:09.288610 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.288563    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:09:09.324858 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.324827    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/997026bf-99b4-4bbf-8ac8-415238fa80af-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4f643-predictor-7879dc7bcd-279lm\" (UID: \"997026bf-99b4-4bbf-8ac8-415238fa80af\") " pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm"
Jul 01 19:09:09.325028 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.324874    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18547f9d-5826-4562-b935-5383beb63767-proxy-tls\") pod \"success-200-isvc-4f643-predictor-785bf555c9-xsk4z\" (UID: \"18547f9d-5826-4562-b935-5383beb63767\") " pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z"
Jul 01 19:09:09.325028 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.324987    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkgn9\" (UniqueName: \"kubernetes.io/projected/18547f9d-5826-4562-b935-5383beb63767-kube-api-access-nkgn9\") pod \"success-200-isvc-4f643-predictor-785bf555c9-xsk4z\" (UID: \"18547f9d-5826-4562-b935-5383beb63767\") " pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z"
Jul 01 19:09:09.325028 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:09:09.325010    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-4f643-predictor-serving-cert: secret "success-200-isvc-4f643-predictor-serving-cert" not found
Jul 01 19:09:09.325028 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.325022    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/997026bf-99b4-4bbf-8ac8-415238fa80af-proxy-tls\") pod \"error-404-isvc-4f643-predictor-7879dc7bcd-279lm\" (UID: \"997026bf-99b4-4bbf-8ac8-415238fa80af\") " pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm"
Jul 01 19:09:09.325379 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:09:09.325076    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18547f9d-5826-4562-b935-5383beb63767-proxy-tls podName:18547f9d-5826-4562-b935-5383beb63767 nodeName:}" failed. No retries permitted until 2026-07-01 19:09:09.825055226 +0000 UTC m=+1900.593892809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/18547f9d-5826-4562-b935-5383beb63767-proxy-tls") pod "success-200-isvc-4f643-predictor-785bf555c9-xsk4z" (UID: "18547f9d-5826-4562-b935-5383beb63767") : secret "success-200-isvc-4f643-predictor-serving-cert" not found
Jul 01 19:09:09.325379 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.325101    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18547f9d-5826-4562-b935-5383beb63767-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4f643-predictor-785bf555c9-xsk4z\" (UID: \"18547f9d-5826-4562-b935-5383beb63767\") " pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z"
Jul 01 19:09:09.325379 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.325135    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxthx\" (UniqueName: \"kubernetes.io/projected/997026bf-99b4-4bbf-8ac8-415238fa80af-kube-api-access-sxthx\") pod \"error-404-isvc-4f643-predictor-7879dc7bcd-279lm\" (UID: \"997026bf-99b4-4bbf-8ac8-415238fa80af\") " pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm"
Jul 01 19:09:09.325877 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.325857    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18547f9d-5826-4562-b935-5383beb63767-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4f643-predictor-785bf555c9-xsk4z\" (UID: \"18547f9d-5826-4562-b935-5383beb63767\") " pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z"
Jul 01 19:09:09.333452 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.333424    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkgn9\" (UniqueName: \"kubernetes.io/projected/18547f9d-5826-4562-b935-5383beb63767-kube-api-access-nkgn9\") pod \"success-200-isvc-4f643-predictor-785bf555c9-xsk4z\" (UID: \"18547f9d-5826-4562-b935-5383beb63767\") " pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z"
Jul 01 19:09:09.426193 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.426155    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/997026bf-99b4-4bbf-8ac8-415238fa80af-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4f643-predictor-7879dc7bcd-279lm\" (UID: \"997026bf-99b4-4bbf-8ac8-415238fa80af\") " pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm"
Jul 01 19:09:09.426365 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.426247    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45c55b1a-7c32-4a6c-8b7a-f83bae151d1a-proxy-tls\") pod \"error-404-isvc-43223-predictor-5968f7b98d-s44nj\" (UID: \"45c55b1a-7c32-4a6c-8b7a-f83bae151d1a\") " pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj"
Jul 01 19:09:09.426365 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.426278    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a416bd65-9851-446c-a152-59fa2adab5d3-proxy-tls\") pod \"success-200-isvc-43223-predictor-85967567b9-fkcgw\" (UID: \"a416bd65-9851-446c-a152-59fa2adab5d3\") " pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw"
Jul 01 19:09:09.426365 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.426351    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/997026bf-99b4-4bbf-8ac8-415238fa80af-proxy-tls\") pod \"error-404-isvc-4f643-predictor-7879dc7bcd-279lm\" (UID: \"997026bf-99b4-4bbf-8ac8-415238fa80af\") " pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm"
Jul 01 19:09:09.426538 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.426394    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxthx\" (UniqueName: \"kubernetes.io/projected/997026bf-99b4-4bbf-8ac8-415238fa80af-kube-api-access-sxthx\") pod \"error-404-isvc-4f643-predictor-7879dc7bcd-279lm\" (UID: \"997026bf-99b4-4bbf-8ac8-415238fa80af\") " pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm"
Jul 01 19:09:09.426538 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:09:09.426468    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-4f643-predictor-serving-cert: secret "error-404-isvc-4f643-predictor-serving-cert" not found
Jul 01 19:09:09.426538 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:09:09.426528    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/997026bf-99b4-4bbf-8ac8-415238fa80af-proxy-tls podName:997026bf-99b4-4bbf-8ac8-415238fa80af nodeName:}" failed. No retries permitted until 2026-07-01 19:09:09.926511347 +0000 UTC m=+1900.695348933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/997026bf-99b4-4bbf-8ac8-415238fa80af-proxy-tls") pod "error-404-isvc-4f643-predictor-7879dc7bcd-279lm" (UID: "997026bf-99b4-4bbf-8ac8-415238fa80af") : secret "error-404-isvc-4f643-predictor-serving-cert" not found
Jul 01 19:09:09.426970 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.426946    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/997026bf-99b4-4bbf-8ac8-415238fa80af-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4f643-predictor-7879dc7bcd-279lm\" (UID: \"997026bf-99b4-4bbf-8ac8-415238fa80af\") " pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm"
Jul 01 19:09:09.429230 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.429206    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45c55b1a-7c32-4a6c-8b7a-f83bae151d1a-proxy-tls\") pod \"error-404-isvc-43223-predictor-5968f7b98d-s44nj\" (UID: \"45c55b1a-7c32-4a6c-8b7a-f83bae151d1a\") " pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj"
Jul 01 19:09:09.429331 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.429281    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a416bd65-9851-446c-a152-59fa2adab5d3-proxy-tls\") pod \"success-200-isvc-43223-predictor-85967567b9-fkcgw\" (UID: \"a416bd65-9851-446c-a152-59fa2adab5d3\") " pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw"
Jul 01 19:09:09.438500 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.438470    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxthx\" (UniqueName: \"kubernetes.io/projected/997026bf-99b4-4bbf-8ac8-415238fa80af-kube-api-access-sxthx\") pod \"error-404-isvc-4f643-predictor-7879dc7bcd-279lm\" (UID: \"997026bf-99b4-4bbf-8ac8-415238fa80af\") " pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm"
Jul 01 19:09:09.541257 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.541178    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:09:09.554683 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.554656    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw"
Jul 01 19:09:09.638264 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.638153    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:09:09.663303 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.663275    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj"
Jul 01 19:09:09.695141 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.695103    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw"]
Jul 01 19:09:09.699070 ip-10-0-129-245 kubenswrapper[2573]: W0701 19:09:09.699040    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda416bd65_9851_446c_a152_59fa2adab5d3.slice/crio-6a1dd522bb86fd45bfd642142b7d34968bcdbb4e08ec5133a134f7501001b2aa WatchSource:0}: Error finding container 6a1dd522bb86fd45bfd642142b7d34968bcdbb4e08ec5133a134f7501001b2aa: Status 404 returned error can't find the container with id 6a1dd522bb86fd45bfd642142b7d34968bcdbb4e08ec5133a134f7501001b2aa
Jul 01 19:09:09.805765 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.805739    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj"]
Jul 01 19:09:09.807826 ip-10-0-129-245 kubenswrapper[2573]: W0701 19:09:09.807784    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c55b1a_7c32_4a6c_8b7a_f83bae151d1a.slice/crio-471b72389ad4df0f9a509e434eaa9df597df1f07b9044881a6ea54a703d07767 WatchSource:0}: Error finding container 471b72389ad4df0f9a509e434eaa9df597df1f07b9044881a6ea54a703d07767: Status 404 returned error can't find the container with id 471b72389ad4df0f9a509e434eaa9df597df1f07b9044881a6ea54a703d07767
Jul 01 19:09:09.829435 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.829413    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18547f9d-5826-4562-b935-5383beb63767-proxy-tls\") pod \"success-200-isvc-4f643-predictor-785bf555c9-xsk4z\" (UID: \"18547f9d-5826-4562-b935-5383beb63767\") " pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z"
Jul 01 19:09:09.832062 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.832034    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18547f9d-5826-4562-b935-5383beb63767-proxy-tls\") pod \"success-200-isvc-4f643-predictor-785bf555c9-xsk4z\" (UID: \"18547f9d-5826-4562-b935-5383beb63767\") " pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z"
Jul 01 19:09:09.879201 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.879171    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj" event={"ID":"45c55b1a-7c32-4a6c-8b7a-f83bae151d1a","Type":"ContainerStarted","Data":"777c8d1f012d3c6de95bb9ec6c1daad7cb757efa75d394eb81a1983f3a03841b"}
Jul 01 19:09:09.879312 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.879236    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj" event={"ID":"45c55b1a-7c32-4a6c-8b7a-f83bae151d1a","Type":"ContainerStarted","Data":"471b72389ad4df0f9a509e434eaa9df597df1f07b9044881a6ea54a703d07767"}
Jul 01 19:09:09.881084 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.881058    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw" event={"ID":"a416bd65-9851-446c-a152-59fa2adab5d3","Type":"ContainerStarted","Data":"c0f12bfa8ba55fad1209e4725b78437a673a997cb0cb9c7d7fb0af479383d639"}
Jul 01 19:09:09.881151 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.881121    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw" event={"ID":"a416bd65-9851-446c-a152-59fa2adab5d3","Type":"ContainerStarted","Data":"6458e2de5a149566f0ee267290e111b75c454f1953dc39c26769e648d008373d"}
Jul 01 19:09:09.881207 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.881157    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw"
Jul 01 19:09:09.881207 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.881185    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw" event={"ID":"a416bd65-9851-446c-a152-59fa2adab5d3","Type":"ContainerStarted","Data":"6a1dd522bb86fd45bfd642142b7d34968bcdbb4e08ec5133a134f7501001b2aa"}
Jul 01 19:09:09.881313 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.881213    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw"
Jul 01 19:09:09.882826 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.882796    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw" podUID="a416bd65-9851-446c-a152-59fa2adab5d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:09:09.883029 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.883008    2573 generic.go:356] "Generic (PLEG): container finished" podID="a23e601c-df80-4e65-9d8e-a63cf7bbbc1c" containerID="8a260f858816d1ee0f898739cd48f9d264c329c48640bd37b103d9423d1a1333" exitCode=2
Jul 01 19:09:09.883095 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.883071    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5" event={"ID":"a23e601c-df80-4e65-9d8e-a63cf7bbbc1c","Type":"ContainerDied","Data":"8a260f858816d1ee0f898739cd48f9d264c329c48640bd37b103d9423d1a1333"}
Jul 01 19:09:09.884941 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.884916    2573 generic.go:356] "Generic (PLEG): container finished" podID="8230e7b5-b543-452a-9a46-4fd627ad3138" containerID="90bc3f70f6da896a8fe2884179d58637f69228c32fca9bec9d2cc8497fa61a73" exitCode=2
Jul 01 19:09:09.885054 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.884957    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx" event={"ID":"8230e7b5-b543-452a-9a46-4fd627ad3138","Type":"ContainerDied","Data":"90bc3f70f6da896a8fe2884179d58637f69228c32fca9bec9d2cc8497fa61a73"}
Jul 01 19:09:09.899754 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.899693    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw" podStartSLOduration=1.899675266 podStartE2EDuration="1.899675266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:09:09.899103713 +0000 UTC m=+1900.667941316" watchObservedRunningTime="2026-07-01 19:09:09.899675266 +0000 UTC m=+1900.668512871"
Jul 01 19:09:09.931108 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.930763    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/997026bf-99b4-4bbf-8ac8-415238fa80af-proxy-tls\") pod \"error-404-isvc-4f643-predictor-7879dc7bcd-279lm\" (UID: \"997026bf-99b4-4bbf-8ac8-415238fa80af\") " pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm"
Jul 01 19:09:09.939356 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:09.939313    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/997026bf-99b4-4bbf-8ac8-415238fa80af-proxy-tls\") pod \"error-404-isvc-4f643-predictor-7879dc7bcd-279lm\" (UID: \"997026bf-99b4-4bbf-8ac8-415238fa80af\") " pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm"
Jul 01 19:09:10.084055 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.083975    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z"
Jul 01 19:09:10.189261 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.189229    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm"
Jul 01 19:09:10.241759 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.240444    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z"]
Jul 01 19:09:10.246024 ip-10-0-129-245 kubenswrapper[2573]: W0701 19:09:10.245994    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18547f9d_5826_4562_b935_5383beb63767.slice/crio-d997c9d1a26e802b95bcea7bfdd33f84e0f9f279de61bc92cf93a07bf1b2e20f WatchSource:0}: Error finding container d997c9d1a26e802b95bcea7bfdd33f84e0f9f279de61bc92cf93a07bf1b2e20f: Status 404 returned error can't find the container with id d997c9d1a26e802b95bcea7bfdd33f84e0f9f279de61bc92cf93a07bf1b2e20f
Jul 01 19:09:10.341147 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.341063    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm"]
Jul 01 19:09:10.346520 ip-10-0-129-245 kubenswrapper[2573]: W0701 19:09:10.346492    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997026bf_99b4_4bbf_8ac8_415238fa80af.slice/crio-d88040a1ec4aaa74d02eee684d812a6d31c77a6510aa9a3f719788bf92305fe5 WatchSource:0}: Error finding container d88040a1ec4aaa74d02eee684d812a6d31c77a6510aa9a3f719788bf92305fe5: Status 404 returned error can't find the container with id d88040a1ec4aaa74d02eee684d812a6d31c77a6510aa9a3f719788bf92305fe5
Jul 01 19:09:10.750168 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.750120    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx" podUID="ad795212-732d-4487-8321-0cb33464f64f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 01 19:09:10.754477 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.754446    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx" podUID="ad795212-732d-4487-8321-0cb33464f64f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 01 19:09:10.892222 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.892186    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj" event={"ID":"45c55b1a-7c32-4a6c-8b7a-f83bae151d1a","Type":"ContainerStarted","Data":"c5a82c084b8899f4a639e82f1c2e4f2b25eba2fe2278ec3ac8ca1f8c0ee0244b"}
Jul 01 19:09:10.892428 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.892256    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj"
Jul 01 19:09:10.892428 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.892386    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj"
Jul 01 19:09:10.893453 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.893427    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj" podUID="45c55b1a-7c32-4a6c-8b7a-f83bae151d1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 19:09:10.894101 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.894081    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z" event={"ID":"18547f9d-5826-4562-b935-5383beb63767","Type":"ContainerStarted","Data":"a43b8956767c0e9ff2af435999d1efa3ddce0eda654756dcfa9a25271eeb0a37"}
Jul 01 19:09:10.894217 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.894136    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z" event={"ID":"18547f9d-5826-4562-b935-5383beb63767","Type":"ContainerStarted","Data":"6de06d52fa7fe953026501831357a604dec49f2ea7c17a5e5f893bd02c6d85a4"}
Jul 01 19:09:10.894217 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.894168    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z" event={"ID":"18547f9d-5826-4562-b935-5383beb63767","Type":"ContainerStarted","Data":"d997c9d1a26e802b95bcea7bfdd33f84e0f9f279de61bc92cf93a07bf1b2e20f"}
Jul 01 19:09:10.894217 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.894202    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z"
Jul 01 19:09:10.894353 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.894230    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z"
Jul 01 19:09:10.895127 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.895100    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z" podUID="18547f9d-5826-4562-b935-5383beb63767" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 01 19:09:10.896011 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.895987    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm" event={"ID":"997026bf-99b4-4bbf-8ac8-415238fa80af","Type":"ContainerStarted","Data":"6054da0798b4926cd60dce0bc72ced6fdf3903fd77eab43942acbf5d85e0d026"}
Jul 01 19:09:10.896110 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.896034    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm" event={"ID":"997026bf-99b4-4bbf-8ac8-415238fa80af","Type":"ContainerStarted","Data":"5d7f0d01aa953628d2358eb55a73ea269dd9b2bd64675fcd8bf3798bfaa10018"}
Jul 01 19:09:10.896110 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.896056    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm" event={"ID":"997026bf-99b4-4bbf-8ac8-415238fa80af","Type":"ContainerStarted","Data":"d88040a1ec4aaa74d02eee684d812a6d31c77a6510aa9a3f719788bf92305fe5"}
Jul 01 19:09:10.896110 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.896089    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm"
Jul 01 19:09:10.896273 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.896119    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm"
Jul 01 19:09:10.896273 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.896175    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw" podUID="a416bd65-9851-446c-a152-59fa2adab5d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:09:10.897205 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.897185    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm" podUID="997026bf-99b4-4bbf-8ac8-415238fa80af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 01 19:09:10.913361 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.913319    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj" podStartSLOduration=2.9133070610000003 podStartE2EDuration="2.913307061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:09:10.911080248 +0000 UTC m=+1901.679917852" watchObservedRunningTime="2026-07-01 19:09:10.913307061 +0000 UTC m=+1901.682144664"
Jul 01 19:09:10.936756 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.936683    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm" podStartSLOduration=1.936667635 podStartE2EDuration="1.936667635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:09:10.93575236 +0000 UTC m=+1901.704589964" watchObservedRunningTime="2026-07-01 19:09:10.936667635 +0000 UTC m=+1901.705505243"
Jul 01 19:09:10.966025 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:10.965977    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z" podStartSLOduration=1.965957172 podStartE2EDuration="1.965957172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:09:10.965528029 +0000 UTC m=+1901.734365637" watchObservedRunningTime="2026-07-01 19:09:10.965957172 +0000 UTC m=+1901.734794776"
Jul 01 19:09:11.612932 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:11.612893    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5" podUID="a23e601c-df80-4e65-9d8e-a63cf7bbbc1c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 01 19:09:11.613077 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:11.612893    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx" podUID="8230e7b5-b543-452a-9a46-4fd627ad3138" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jul 01 19:09:11.901504 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:11.901399    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z" podUID="18547f9d-5826-4562-b935-5383beb63767" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 01 19:09:11.901504 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:11.901438    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm" podUID="997026bf-99b4-4bbf-8ac8-415238fa80af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 01 19:09:11.901504 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:11.901440    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj" podUID="45c55b1a-7c32-4a6c-8b7a-f83bae151d1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 19:09:12.621836 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:12.621813    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d"
Jul 01 19:09:12.664520 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:12.664492    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jd86m\" (UniqueName: \"kubernetes.io/projected/440a82cc-2ac4-4b80-ac8d-075af9ca1394-kube-api-access-jd86m\") pod \"440a82cc-2ac4-4b80-ac8d-075af9ca1394\" (UID: \"440a82cc-2ac4-4b80-ac8d-075af9ca1394\") "
Jul 01 19:09:12.664680 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:12.664530    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/440a82cc-2ac4-4b80-ac8d-075af9ca1394-kube-rbac-proxy-sar-config\") pod \"440a82cc-2ac4-4b80-ac8d-075af9ca1394\" (UID: \"440a82cc-2ac4-4b80-ac8d-075af9ca1394\") "
Jul 01 19:09:12.664680 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:12.664601    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/440a82cc-2ac4-4b80-ac8d-075af9ca1394-proxy-tls\") pod \"440a82cc-2ac4-4b80-ac8d-075af9ca1394\" (UID: \"440a82cc-2ac4-4b80-ac8d-075af9ca1394\") "
Jul 01 19:09:12.664991 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:12.664957    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440a82cc-2ac4-4b80-ac8d-075af9ca1394-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "440a82cc-2ac4-4b80-ac8d-075af9ca1394" (UID: "440a82cc-2ac4-4b80-ac8d-075af9ca1394"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:09:12.666594 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:12.666563    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440a82cc-2ac4-4b80-ac8d-075af9ca1394-kube-api-access-jd86m" (OuterVolumeSpecName: "kube-api-access-jd86m") pod "440a82cc-2ac4-4b80-ac8d-075af9ca1394" (UID: "440a82cc-2ac4-4b80-ac8d-075af9ca1394"). InnerVolumeSpecName "kube-api-access-jd86m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:09:12.666707 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:12.666599    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440a82cc-2ac4-4b80-ac8d-075af9ca1394-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "440a82cc-2ac4-4b80-ac8d-075af9ca1394" (UID: "440a82cc-2ac4-4b80-ac8d-075af9ca1394"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:09:12.765359 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:12.765318    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/440a82cc-2ac4-4b80-ac8d-075af9ca1394-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:09:12.765359 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:12.765350    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/440a82cc-2ac4-4b80-ac8d-075af9ca1394-proxy-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:09:12.765359 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:12.765363    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd86m\" (UniqueName: \"kubernetes.io/projected/440a82cc-2ac4-4b80-ac8d-075af9ca1394-kube-api-access-jd86m\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:09:12.907918 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:12.907829    2573 generic.go:356] "Generic (PLEG): container finished" podID="440a82cc-2ac4-4b80-ac8d-075af9ca1394" containerID="db4c870c3b3f5f6316c6628b6673e083b56ed77665bf3d2ac8e581dd6ba11cee" exitCode=0
Jul 01 19:09:12.907918 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:12.907884    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d"
Jul 01 19:09:12.908385 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:12.907912    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d" event={"ID":"440a82cc-2ac4-4b80-ac8d-075af9ca1394","Type":"ContainerDied","Data":"db4c870c3b3f5f6316c6628b6673e083b56ed77665bf3d2ac8e581dd6ba11cee"}
Jul 01 19:09:12.908385 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:12.907977    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d" event={"ID":"440a82cc-2ac4-4b80-ac8d-075af9ca1394","Type":"ContainerDied","Data":"73beeea919bb9092386e1c802394a7ebf521756aef0ff7bc26d3fc06c5b3635b"}
Jul 01 19:09:12.908385 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:12.908005    2573 scope.go:117] "RemoveContainer" containerID="0574cb55b33abc0cfc84f6de08bd6688fe1715bf146d92b20e2bb64890ece05e"
Jul 01 19:09:12.916899 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:12.916873    2573 generic.go:356] "Generic (PLEG): container finished" podID="ad795212-732d-4487-8321-0cb33464f64f" containerID="5420180e39b85216f86c753358ca6baded0d7a5020a6a4e3250e0bed41e41a4d" exitCode=0
Jul 01 19:09:12.917027 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:12.916962    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx" event={"ID":"ad795212-732d-4487-8321-0cb33464f64f","Type":"ContainerDied","Data":"5420180e39b85216f86c753358ca6baded0d7a5020a6a4e3250e0bed41e41a4d"}
Jul 01 19:09:12.920617 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:12.920598    2573 scope.go:117] "RemoveContainer" containerID="db4c870c3b3f5f6316c6628b6673e083b56ed77665bf3d2ac8e581dd6ba11cee"
Jul 01 19:09:12.928112 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:12.928092    2573 scope.go:117] "RemoveContainer" containerID="0574cb55b33abc0cfc84f6de08bd6688fe1715bf146d92b20e2bb64890ece05e"
Jul 01 19:09:12.928411 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:09:12.928392    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0574cb55b33abc0cfc84f6de08bd6688fe1715bf146d92b20e2bb64890ece05e\": container with ID starting with 0574cb55b33abc0cfc84f6de08bd6688fe1715bf146d92b20e2bb64890ece05e not found: ID does not exist" containerID="0574cb55b33abc0cfc84f6de08bd6688fe1715bf146d92b20e2bb64890ece05e"
Jul 01 19:09:12.928479 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:12.928420    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0574cb55b33abc0cfc84f6de08bd6688fe1715bf146d92b20e2bb64890ece05e"} err="failed to get container status \"0574cb55b33abc0cfc84f6de08bd6688fe1715bf146d92b20e2bb64890ece05e\": rpc error: code = NotFound desc = could not find container \"0574cb55b33abc0cfc84f6de08bd6688fe1715bf146d92b20e2bb64890ece05e\": container with ID starting with 0574cb55b33abc0cfc84f6de08bd6688fe1715bf146d92b20e2bb64890ece05e not found: ID does not exist"
Jul 01 19:09:12.928479 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:12.928443    2573 scope.go:117] "RemoveContainer" containerID="db4c870c3b3f5f6316c6628b6673e083b56ed77665bf3d2ac8e581dd6ba11cee"
Jul 01 19:09:12.928829 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:09:12.928690    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db4c870c3b3f5f6316c6628b6673e083b56ed77665bf3d2ac8e581dd6ba11cee\": container with ID starting with db4c870c3b3f5f6316c6628b6673e083b56ed77665bf3d2ac8e581dd6ba11cee not found: ID does not exist" containerID="db4c870c3b3f5f6316c6628b6673e083b56ed77665bf3d2ac8e581dd6ba11cee"
Jul 01 19:09:12.928829 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:12.928736    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db4c870c3b3f5f6316c6628b6673e083b56ed77665bf3d2ac8e581dd6ba11cee"} err="failed to get container status \"db4c870c3b3f5f6316c6628b6673e083b56ed77665bf3d2ac8e581dd6ba11cee\": rpc error: code = NotFound desc = could not find container \"db4c870c3b3f5f6316c6628b6673e083b56ed77665bf3d2ac8e581dd6ba11cee\": container with ID starting with db4c870c3b3f5f6316c6628b6673e083b56ed77665bf3d2ac8e581dd6ba11cee not found: ID does not exist"
Jul 01 19:09:12.940745 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:12.936854    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d"]
Jul 01 19:09:12.943343 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:12.943319    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c3262-predictor-865fd8b765-p828d"]
Jul 01 19:09:13.116918 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.116891    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx"
Jul 01 19:09:13.167967 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.167893    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad795212-732d-4487-8321-0cb33464f64f-kube-rbac-proxy-sar-config\") pod \"ad795212-732d-4487-8321-0cb33464f64f\" (UID: \"ad795212-732d-4487-8321-0cb33464f64f\") "
Jul 01 19:09:13.167967 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.167949    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmxjc\" (UniqueName: \"kubernetes.io/projected/ad795212-732d-4487-8321-0cb33464f64f-kube-api-access-tmxjc\") pod \"ad795212-732d-4487-8321-0cb33464f64f\" (UID: \"ad795212-732d-4487-8321-0cb33464f64f\") "
Jul 01 19:09:13.168163 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.167981    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad795212-732d-4487-8321-0cb33464f64f-proxy-tls\") pod \"ad795212-732d-4487-8321-0cb33464f64f\" (UID: \"ad795212-732d-4487-8321-0cb33464f64f\") "
Jul 01 19:09:13.168312 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.168285    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad795212-732d-4487-8321-0cb33464f64f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ad795212-732d-4487-8321-0cb33464f64f" (UID: "ad795212-732d-4487-8321-0cb33464f64f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:09:13.170007 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.169982    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad795212-732d-4487-8321-0cb33464f64f-kube-api-access-tmxjc" (OuterVolumeSpecName: "kube-api-access-tmxjc") pod "ad795212-732d-4487-8321-0cb33464f64f" (UID: "ad795212-732d-4487-8321-0cb33464f64f"). InnerVolumeSpecName "kube-api-access-tmxjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:09:13.170096 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.170011    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad795212-732d-4487-8321-0cb33464f64f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ad795212-732d-4487-8321-0cb33464f64f" (UID: "ad795212-732d-4487-8321-0cb33464f64f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:09:13.250571 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.250449    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5"
Jul 01 19:09:13.269093 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.269066    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a23e601c-df80-4e65-9d8e-a63cf7bbbc1c-proxy-tls\") pod \"a23e601c-df80-4e65-9d8e-a63cf7bbbc1c\" (UID: \"a23e601c-df80-4e65-9d8e-a63cf7bbbc1c\") "
Jul 01 19:09:13.269241 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.269125    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a23e601c-df80-4e65-9d8e-a63cf7bbbc1c-kube-rbac-proxy-sar-config\") pod \"a23e601c-df80-4e65-9d8e-a63cf7bbbc1c\" (UID: \"a23e601c-df80-4e65-9d8e-a63cf7bbbc1c\") "
Jul 01 19:09:13.269241 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.269152    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmtk2\" (UniqueName: \"kubernetes.io/projected/a23e601c-df80-4e65-9d8e-a63cf7bbbc1c-kube-api-access-mmtk2\") pod \"a23e601c-df80-4e65-9d8e-a63cf7bbbc1c\" (UID: \"a23e601c-df80-4e65-9d8e-a63cf7bbbc1c\") "
Jul 01 19:09:13.269448 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.269414    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad795212-732d-4487-8321-0cb33464f64f-proxy-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:09:13.269448 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.269437    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad795212-732d-4487-8321-0cb33464f64f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:09:13.269609 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.269452    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmxjc\" (UniqueName: \"kubernetes.io/projected/ad795212-732d-4487-8321-0cb33464f64f-kube-api-access-tmxjc\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:09:13.269609 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.269540    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23e601c-df80-4e65-9d8e-a63cf7bbbc1c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a23e601c-df80-4e65-9d8e-a63cf7bbbc1c" (UID: "a23e601c-df80-4e65-9d8e-a63cf7bbbc1c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:09:13.271588 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.271563    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a23e601c-df80-4e65-9d8e-a63cf7bbbc1c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a23e601c-df80-4e65-9d8e-a63cf7bbbc1c" (UID: "a23e601c-df80-4e65-9d8e-a63cf7bbbc1c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:09:13.271707 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.271630    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23e601c-df80-4e65-9d8e-a63cf7bbbc1c-kube-api-access-mmtk2" (OuterVolumeSpecName: "kube-api-access-mmtk2") pod "a23e601c-df80-4e65-9d8e-a63cf7bbbc1c" (UID: "a23e601c-df80-4e65-9d8e-a63cf7bbbc1c"). InnerVolumeSpecName "kube-api-access-mmtk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:09:13.312197 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.312170    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx"
Jul 01 19:09:13.370303 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.370272    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzb9w\" (UniqueName: \"kubernetes.io/projected/8230e7b5-b543-452a-9a46-4fd627ad3138-kube-api-access-rzb9w\") pod \"8230e7b5-b543-452a-9a46-4fd627ad3138\" (UID: \"8230e7b5-b543-452a-9a46-4fd627ad3138\") "
Jul 01 19:09:13.370453 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.370314    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8230e7b5-b543-452a-9a46-4fd627ad3138-kube-rbac-proxy-sar-config\") pod \"8230e7b5-b543-452a-9a46-4fd627ad3138\" (UID: \"8230e7b5-b543-452a-9a46-4fd627ad3138\") "
Jul 01 19:09:13.370453 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.370344    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8230e7b5-b543-452a-9a46-4fd627ad3138-proxy-tls\") pod \"8230e7b5-b543-452a-9a46-4fd627ad3138\" (UID: \"8230e7b5-b543-452a-9a46-4fd627ad3138\") "
Jul 01 19:09:13.370556 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.370540    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a23e601c-df80-4e65-9d8e-a63cf7bbbc1c-proxy-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:09:13.370601 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.370564    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a23e601c-df80-4e65-9d8e-a63cf7bbbc1c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:09:13.370601 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.370578    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmtk2\" (UniqueName: \"kubernetes.io/projected/a23e601c-df80-4e65-9d8e-a63cf7bbbc1c-kube-api-access-mmtk2\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:09:13.370675 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.370623    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8230e7b5-b543-452a-9a46-4fd627ad3138-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8230e7b5-b543-452a-9a46-4fd627ad3138" (UID: "8230e7b5-b543-452a-9a46-4fd627ad3138"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:09:13.372387 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.372366    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8230e7b5-b543-452a-9a46-4fd627ad3138-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8230e7b5-b543-452a-9a46-4fd627ad3138" (UID: "8230e7b5-b543-452a-9a46-4fd627ad3138"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:09:13.372475 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.372396    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8230e7b5-b543-452a-9a46-4fd627ad3138-kube-api-access-rzb9w" (OuterVolumeSpecName: "kube-api-access-rzb9w") pod "8230e7b5-b543-452a-9a46-4fd627ad3138" (UID: "8230e7b5-b543-452a-9a46-4fd627ad3138"). InnerVolumeSpecName "kube-api-access-rzb9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:09:13.471754 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.471700    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzb9w\" (UniqueName: \"kubernetes.io/projected/8230e7b5-b543-452a-9a46-4fd627ad3138-kube-api-access-rzb9w\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:09:13.471754 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.471753    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8230e7b5-b543-452a-9a46-4fd627ad3138-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:09:13.471939 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.471765    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8230e7b5-b543-452a-9a46-4fd627ad3138-proxy-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:09:13.666376 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.666342    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="440a82cc-2ac4-4b80-ac8d-075af9ca1394" path="/var/lib/kubelet/pods/440a82cc-2ac4-4b80-ac8d-075af9ca1394/volumes"
Jul 01 19:09:13.923886 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.923795    2573 generic.go:356] "Generic (PLEG): container finished" podID="a23e601c-df80-4e65-9d8e-a63cf7bbbc1c" containerID="1d56ba1fb662a8d0210e5bf9d6096d247d7c5c6d1e03d24bda4b11201de34d8d" exitCode=0
Jul 01 19:09:13.923886 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.923845    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5"
Jul 01 19:09:13.923886 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.923868    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5" event={"ID":"a23e601c-df80-4e65-9d8e-a63cf7bbbc1c","Type":"ContainerDied","Data":"1d56ba1fb662a8d0210e5bf9d6096d247d7c5c6d1e03d24bda4b11201de34d8d"}
Jul 01 19:09:13.924437 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.923920    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5" event={"ID":"a23e601c-df80-4e65-9d8e-a63cf7bbbc1c","Type":"ContainerDied","Data":"3ff587713b2e55a2b8bbe33e84939999595aac4f54c9e4abd3e2a5f154bd3da7"}
Jul 01 19:09:13.924437 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.923948    2573 scope.go:117] "RemoveContainer" containerID="8a260f858816d1ee0f898739cd48f9d264c329c48640bd37b103d9423d1a1333"
Jul 01 19:09:13.925410 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.925389    2573 generic.go:356] "Generic (PLEG): container finished" podID="8230e7b5-b543-452a-9a46-4fd627ad3138" containerID="73c7ef61e1903cbb23b4a4b7af7aabb70f2f62bc271e6539e8d37b98e8c0f803" exitCode=0
Jul 01 19:09:13.925490 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.925433    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx"
Jul 01 19:09:13.925490 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.925465    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx" event={"ID":"8230e7b5-b543-452a-9a46-4fd627ad3138","Type":"ContainerDied","Data":"73c7ef61e1903cbb23b4a4b7af7aabb70f2f62bc271e6539e8d37b98e8c0f803"}
Jul 01 19:09:13.925580 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.925523    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx" event={"ID":"8230e7b5-b543-452a-9a46-4fd627ad3138","Type":"ContainerDied","Data":"06df7c82f604d02c5b6b773a329710e5fde72636f44819f7dc08323cb8702cf1"}
Jul 01 19:09:13.927261 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.927240    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx" event={"ID":"ad795212-732d-4487-8321-0cb33464f64f","Type":"ContainerDied","Data":"5b5a321fcbf0aa5e4f12a53b63ed86053ff19c9e4fd5838ce4ce4bcdbdc682bc"}
Jul 01 19:09:13.927339 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.927268    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx"
Jul 01 19:09:13.933918 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.933894    2573 scope.go:117] "RemoveContainer" containerID="1d56ba1fb662a8d0210e5bf9d6096d247d7c5c6d1e03d24bda4b11201de34d8d"
Jul 01 19:09:13.943330 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.943304    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5"]
Jul 01 19:09:13.943872 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.943857    2573 scope.go:117] "RemoveContainer" containerID="8a260f858816d1ee0f898739cd48f9d264c329c48640bd37b103d9423d1a1333"
Jul 01 19:09:13.944117 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:09:13.944101    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a260f858816d1ee0f898739cd48f9d264c329c48640bd37b103d9423d1a1333\": container with ID starting with 8a260f858816d1ee0f898739cd48f9d264c329c48640bd37b103d9423d1a1333 not found: ID does not exist" containerID="8a260f858816d1ee0f898739cd48f9d264c329c48640bd37b103d9423d1a1333"
Jul 01 19:09:13.944163 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.944124    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a260f858816d1ee0f898739cd48f9d264c329c48640bd37b103d9423d1a1333"} err="failed to get container status \"8a260f858816d1ee0f898739cd48f9d264c329c48640bd37b103d9423d1a1333\": rpc error: code = NotFound desc = could not find container \"8a260f858816d1ee0f898739cd48f9d264c329c48640bd37b103d9423d1a1333\": container with ID starting with 8a260f858816d1ee0f898739cd48f9d264c329c48640bd37b103d9423d1a1333 not found: ID does not exist"
Jul 01 19:09:13.944163 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.944142    2573 scope.go:117] "RemoveContainer" containerID="1d56ba1fb662a8d0210e5bf9d6096d247d7c5c6d1e03d24bda4b11201de34d8d"
Jul 01 19:09:13.944381 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:09:13.944362    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d56ba1fb662a8d0210e5bf9d6096d247d7c5c6d1e03d24bda4b11201de34d8d\": container with ID starting with 1d56ba1fb662a8d0210e5bf9d6096d247d7c5c6d1e03d24bda4b11201de34d8d not found: ID does not exist" containerID="1d56ba1fb662a8d0210e5bf9d6096d247d7c5c6d1e03d24bda4b11201de34d8d"
Jul 01 19:09:13.944421 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.944387    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d56ba1fb662a8d0210e5bf9d6096d247d7c5c6d1e03d24bda4b11201de34d8d"} err="failed to get container status \"1d56ba1fb662a8d0210e5bf9d6096d247d7c5c6d1e03d24bda4b11201de34d8d\": rpc error: code = NotFound desc = could not find container \"1d56ba1fb662a8d0210e5bf9d6096d247d7c5c6d1e03d24bda4b11201de34d8d\": container with ID starting with 1d56ba1fb662a8d0210e5bf9d6096d247d7c5c6d1e03d24bda4b11201de34d8d not found: ID does not exist"
Jul 01 19:09:13.944421 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.944404    2573 scope.go:117] "RemoveContainer" containerID="90bc3f70f6da896a8fe2884179d58637f69228c32fca9bec9d2cc8497fa61a73"
Jul 01 19:09:13.948252 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.948232    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1bf2c-predictor-6c64b6bb44-scdj5"]
Jul 01 19:09:13.952399 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.952372    2573 scope.go:117] "RemoveContainer" containerID="73c7ef61e1903cbb23b4a4b7af7aabb70f2f62bc271e6539e8d37b98e8c0f803"
Jul 01 19:09:13.959752 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.959737    2573 scope.go:117] "RemoveContainer" containerID="90bc3f70f6da896a8fe2884179d58637f69228c32fca9bec9d2cc8497fa61a73"
Jul 01 19:09:13.959980 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:09:13.959960    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90bc3f70f6da896a8fe2884179d58637f69228c32fca9bec9d2cc8497fa61a73\": container with ID starting with 90bc3f70f6da896a8fe2884179d58637f69228c32fca9bec9d2cc8497fa61a73 not found: ID does not exist" containerID="90bc3f70f6da896a8fe2884179d58637f69228c32fca9bec9d2cc8497fa61a73"
Jul 01 19:09:13.960037 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.959986    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90bc3f70f6da896a8fe2884179d58637f69228c32fca9bec9d2cc8497fa61a73"} err="failed to get container status \"90bc3f70f6da896a8fe2884179d58637f69228c32fca9bec9d2cc8497fa61a73\": rpc error: code = NotFound desc = could not find container \"90bc3f70f6da896a8fe2884179d58637f69228c32fca9bec9d2cc8497fa61a73\": container with ID starting with 90bc3f70f6da896a8fe2884179d58637f69228c32fca9bec9d2cc8497fa61a73 not found: ID does not exist"
Jul 01 19:09:13.960037 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.960001    2573 scope.go:117] "RemoveContainer" containerID="73c7ef61e1903cbb23b4a4b7af7aabb70f2f62bc271e6539e8d37b98e8c0f803"
Jul 01 19:09:13.960229 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:09:13.960207    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73c7ef61e1903cbb23b4a4b7af7aabb70f2f62bc271e6539e8d37b98e8c0f803\": container with ID starting with 73c7ef61e1903cbb23b4a4b7af7aabb70f2f62bc271e6539e8d37b98e8c0f803 not found: ID does not exist" containerID="73c7ef61e1903cbb23b4a4b7af7aabb70f2f62bc271e6539e8d37b98e8c0f803"
Jul 01 19:09:13.960276 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.960225    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c7ef61e1903cbb23b4a4b7af7aabb70f2f62bc271e6539e8d37b98e8c0f803"} err="failed to get container status \"73c7ef61e1903cbb23b4a4b7af7aabb70f2f62bc271e6539e8d37b98e8c0f803\": rpc error: code = NotFound desc = could not find container \"73c7ef61e1903cbb23b4a4b7af7aabb70f2f62bc271e6539e8d37b98e8c0f803\": container with ID starting with 73c7ef61e1903cbb23b4a4b7af7aabb70f2f62bc271e6539e8d37b98e8c0f803 not found: ID does not exist"
Jul 01 19:09:13.960276 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.960239    2573 scope.go:117] "RemoveContainer" containerID="95050154e2b26a219bbc25bfe13fb321a1ef8d787e63705c1996379c0efa7d82"
Jul 01 19:09:13.967457 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.967442    2573 scope.go:117] "RemoveContainer" containerID="5420180e39b85216f86c753358ca6baded0d7a5020a6a4e3250e0bed41e41a4d"
Jul 01 19:09:13.967695 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.967674    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx"]
Jul 01 19:09:13.972995 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.972972    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1bf2c-predictor-56bc8b9cd9-48jqx"]
Jul 01 19:09:13.984164 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.984142    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx"]
Jul 01 19:09:13.988777 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:13.988758    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3262-predictor-6676bfcd4d-z54kx"]
Jul 01 19:09:15.665818 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:15.665786    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8230e7b5-b543-452a-9a46-4fd627ad3138" path="/var/lib/kubelet/pods/8230e7b5-b543-452a-9a46-4fd627ad3138/volumes"
Jul 01 19:09:15.666206 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:15.666192    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a23e601c-df80-4e65-9d8e-a63cf7bbbc1c" path="/var/lib/kubelet/pods/a23e601c-df80-4e65-9d8e-a63cf7bbbc1c/volumes"
Jul 01 19:09:15.666561 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:15.666549    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad795212-732d-4487-8321-0cb33464f64f" path="/var/lib/kubelet/pods/ad795212-732d-4487-8321-0cb33464f64f/volumes"
Jul 01 19:09:15.900647 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:15.900621    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw"
Jul 01 19:09:15.901107 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:15.901085    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw" podUID="a416bd65-9851-446c-a152-59fa2adab5d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:09:15.962371 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:15.962342    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:09:16.024895 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:16.024864    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:09:16.906432 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:16.906400    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj"
Jul 01 19:09:16.906864 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:16.906569    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z"
Jul 01 19:09:16.906864 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:16.906812    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj" podUID="45c55b1a-7c32-4a6c-8b7a-f83bae151d1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 19:09:16.907200 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:16.907177    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm"
Jul 01 19:09:16.907328 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:16.907245    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z" podUID="18547f9d-5826-4562-b935-5383beb63767" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 01 19:09:16.907589 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:16.907566    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm" podUID="997026bf-99b4-4bbf-8ac8-415238fa80af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 01 19:09:25.901978 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:25.901940    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw" podUID="a416bd65-9851-446c-a152-59fa2adab5d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:09:26.906978 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:26.906939    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj" podUID="45c55b1a-7c32-4a6c-8b7a-f83bae151d1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 19:09:26.907364 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:26.907229    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z" podUID="18547f9d-5826-4562-b935-5383beb63767" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 01 19:09:26.908445 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:26.908425    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm" podUID="997026bf-99b4-4bbf-8ac8-415238fa80af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 01 19:09:35.901068 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:35.901029    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw" podUID="a416bd65-9851-446c-a152-59fa2adab5d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:09:36.906916 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:36.906879    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj" podUID="45c55b1a-7c32-4a6c-8b7a-f83bae151d1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 19:09:36.907330 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:36.907281    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z" podUID="18547f9d-5826-4562-b935-5383beb63767" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 01 19:09:36.908476 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:36.908448    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm" podUID="997026bf-99b4-4bbf-8ac8-415238fa80af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 01 19:09:45.901946 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:45.901902    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw" podUID="a416bd65-9851-446c-a152-59fa2adab5d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:09:46.907665 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:46.907627    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm" podUID="997026bf-99b4-4bbf-8ac8-415238fa80af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 01 19:09:46.908067 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:46.907633    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z" podUID="18547f9d-5826-4562-b935-5383beb63767" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 01 19:09:46.908067 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:46.907628    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj" podUID="45c55b1a-7c32-4a6c-8b7a-f83bae151d1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 01 19:09:55.901099 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:55.901055    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw" podUID="a416bd65-9851-446c-a152-59fa2adab5d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:09:56.907485 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:56.907452    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj"
Jul 01 19:09:56.907995 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:56.907839    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm"
Jul 01 19:09:56.907995 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:09:56.907921    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z"
Jul 01 19:10:05.901865 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:10:05.901834    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw"
Jul 01 19:10:09.243075 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:10:09.243042    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:10:09.558077 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:10:09.557990    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:10:18.722446 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:10:18.722411    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:10:19.152407 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:10:19.152317    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:12:02.202043 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:12:02.201958    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:15:55.882615 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:15:55.882581    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:18:23.982052 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:23.981967    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:18:24.062157 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:24.062128    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z"]
Jul 01 19:18:24.062376 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:24.062347    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z" podUID="18547f9d-5826-4562-b935-5383beb63767" containerName="kserve-container" containerID="cri-o://6de06d52fa7fe953026501831357a604dec49f2ea7c17a5e5f893bd02c6d85a4" gracePeriod=30
Jul 01 19:18:24.062509 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:24.062391    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z" podUID="18547f9d-5826-4562-b935-5383beb63767" containerName="kube-rbac-proxy" containerID="cri-o://a43b8956767c0e9ff2af435999d1efa3ddce0eda654756dcfa9a25271eeb0a37" gracePeriod=30
Jul 01 19:18:24.106106 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:24.106078    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm"]
Jul 01 19:18:24.106345 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:24.106307    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm" podUID="997026bf-99b4-4bbf-8ac8-415238fa80af" containerName="kserve-container" containerID="cri-o://5d7f0d01aa953628d2358eb55a73ea269dd9b2bd64675fcd8bf3798bfaa10018" gracePeriod=30
Jul 01 19:18:24.106418 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:24.106324    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm" podUID="997026bf-99b4-4bbf-8ac8-415238fa80af" containerName="kube-rbac-proxy" containerID="cri-o://6054da0798b4926cd60dce0bc72ced6fdf3903fd77eab43942acbf5d85e0d026" gracePeriod=30
Jul 01 19:18:25.041596 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:25.041558    2573 generic.go:356] "Generic (PLEG): container finished" podID="997026bf-99b4-4bbf-8ac8-415238fa80af" containerID="6054da0798b4926cd60dce0bc72ced6fdf3903fd77eab43942acbf5d85e0d026" exitCode=2
Jul 01 19:18:25.042018 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:25.041631    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm" event={"ID":"997026bf-99b4-4bbf-8ac8-415238fa80af","Type":"ContainerDied","Data":"6054da0798b4926cd60dce0bc72ced6fdf3903fd77eab43942acbf5d85e0d026"}
Jul 01 19:18:25.043494 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:25.043461    2573 generic.go:356] "Generic (PLEG): container finished" podID="18547f9d-5826-4562-b935-5383beb63767" containerID="a43b8956767c0e9ff2af435999d1efa3ddce0eda654756dcfa9a25271eeb0a37" exitCode=2
Jul 01 19:18:25.043613 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:25.043513    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z" event={"ID":"18547f9d-5826-4562-b935-5383beb63767","Type":"ContainerDied","Data":"a43b8956767c0e9ff2af435999d1efa3ddce0eda654756dcfa9a25271eeb0a37"}
Jul 01 19:18:25.122295 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:25.122267    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:18:26.902584 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:26.902541    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z" podUID="18547f9d-5826-4562-b935-5383beb63767" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 01 19:18:26.903053 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:26.902540    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm" podUID="997026bf-99b4-4bbf-8ac8-415238fa80af" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Jul 01 19:18:26.907590 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:26.907557    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm" podUID="997026bf-99b4-4bbf-8ac8-415238fa80af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 01 19:18:26.907750 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:26.907557    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z" podUID="18547f9d-5826-4562-b935-5383beb63767" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 01 19:18:27.373054 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:27.373033    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z"
Jul 01 19:18:27.375830 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:27.375813    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm"
Jul 01 19:18:27.474760 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:27.474731    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18547f9d-5826-4562-b935-5383beb63767-proxy-tls\") pod \"18547f9d-5826-4562-b935-5383beb63767\" (UID: \"18547f9d-5826-4562-b935-5383beb63767\") "
Jul 01 19:18:27.474923 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:27.474791    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkgn9\" (UniqueName: \"kubernetes.io/projected/18547f9d-5826-4562-b935-5383beb63767-kube-api-access-nkgn9\") pod \"18547f9d-5826-4562-b935-5383beb63767\" (UID: \"18547f9d-5826-4562-b935-5383beb63767\") "
Jul 01 19:18:27.474923 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:27.474810    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxthx\" (UniqueName: \"kubernetes.io/projected/997026bf-99b4-4bbf-8ac8-415238fa80af-kube-api-access-sxthx\") pod \"997026bf-99b4-4bbf-8ac8-415238fa80af\" (UID: \"997026bf-99b4-4bbf-8ac8-415238fa80af\") "
Jul 01 19:18:27.475016 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:27.474918    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/997026bf-99b4-4bbf-8ac8-415238fa80af-kube-rbac-proxy-sar-config\") pod \"997026bf-99b4-4bbf-8ac8-415238fa80af\" (UID: \"997026bf-99b4-4bbf-8ac8-415238fa80af\") "
Jul 01 19:18:27.475016 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:27.474956    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/997026bf-99b4-4bbf-8ac8-415238fa80af-proxy-tls\") pod \"997026bf-99b4-4bbf-8ac8-415238fa80af\" (UID: \"997026bf-99b4-4bbf-8ac8-415238fa80af\") "
Jul 01 19:18:27.475016 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:27.474989    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18547f9d-5826-4562-b935-5383beb63767-kube-rbac-proxy-sar-config\") pod \"18547f9d-5826-4562-b935-5383beb63767\" (UID: \"18547f9d-5826-4562-b935-5383beb63767\") "
Jul 01 19:18:27.475326 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:27.475294    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997026bf-99b4-4bbf-8ac8-415238fa80af-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "997026bf-99b4-4bbf-8ac8-415238fa80af" (UID: "997026bf-99b4-4bbf-8ac8-415238fa80af"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:18:27.475471 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:27.475409    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18547f9d-5826-4562-b935-5383beb63767-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "18547f9d-5826-4562-b935-5383beb63767" (UID: "18547f9d-5826-4562-b935-5383beb63767"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:18:27.476967 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:27.476934    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/997026bf-99b4-4bbf-8ac8-415238fa80af-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "997026bf-99b4-4bbf-8ac8-415238fa80af" (UID: "997026bf-99b4-4bbf-8ac8-415238fa80af"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:18:27.477071 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:27.476968    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997026bf-99b4-4bbf-8ac8-415238fa80af-kube-api-access-sxthx" (OuterVolumeSpecName: "kube-api-access-sxthx") pod "997026bf-99b4-4bbf-8ac8-415238fa80af" (UID: "997026bf-99b4-4bbf-8ac8-415238fa80af"). InnerVolumeSpecName "kube-api-access-sxthx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:18:27.477071 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:27.476976    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18547f9d-5826-4562-b935-5383beb63767-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "18547f9d-5826-4562-b935-5383beb63767" (UID: "18547f9d-5826-4562-b935-5383beb63767"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:18:27.477071 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:27.476983    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18547f9d-5826-4562-b935-5383beb63767-kube-api-access-nkgn9" (OuterVolumeSpecName: "kube-api-access-nkgn9") pod "18547f9d-5826-4562-b935-5383beb63767" (UID: "18547f9d-5826-4562-b935-5383beb63767"). InnerVolumeSpecName "kube-api-access-nkgn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:18:27.576424 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:27.576397    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkgn9\" (UniqueName: \"kubernetes.io/projected/18547f9d-5826-4562-b935-5383beb63767-kube-api-access-nkgn9\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:18:27.576424 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:27.576423    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxthx\" (UniqueName: \"kubernetes.io/projected/997026bf-99b4-4bbf-8ac8-415238fa80af-kube-api-access-sxthx\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:18:27.576625 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:27.576438    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/997026bf-99b4-4bbf-8ac8-415238fa80af-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:18:27.576625 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:27.576453    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/997026bf-99b4-4bbf-8ac8-415238fa80af-proxy-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:18:27.576625 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:27.576466    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18547f9d-5826-4562-b935-5383beb63767-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:18:27.576625 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:27.576477    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18547f9d-5826-4562-b935-5383beb63767-proxy-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:18:28.056018 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:28.055988    2573 generic.go:356] "Generic (PLEG): container finished" podID="18547f9d-5826-4562-b935-5383beb63767" containerID="6de06d52fa7fe953026501831357a604dec49f2ea7c17a5e5f893bd02c6d85a4" exitCode=0
Jul 01 19:18:28.056496 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:28.056035    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z"
Jul 01 19:18:28.056496 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:28.056064    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z" event={"ID":"18547f9d-5826-4562-b935-5383beb63767","Type":"ContainerDied","Data":"6de06d52fa7fe953026501831357a604dec49f2ea7c17a5e5f893bd02c6d85a4"}
Jul 01 19:18:28.056496 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:28.056127    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z" event={"ID":"18547f9d-5826-4562-b935-5383beb63767","Type":"ContainerDied","Data":"d997c9d1a26e802b95bcea7bfdd33f84e0f9f279de61bc92cf93a07bf1b2e20f"}
Jul 01 19:18:28.056496 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:28.056144    2573 scope.go:117] "RemoveContainer" containerID="a43b8956767c0e9ff2af435999d1efa3ddce0eda654756dcfa9a25271eeb0a37"
Jul 01 19:18:28.057846 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:28.057822    2573 generic.go:356] "Generic (PLEG): container finished" podID="997026bf-99b4-4bbf-8ac8-415238fa80af" containerID="5d7f0d01aa953628d2358eb55a73ea269dd9b2bd64675fcd8bf3798bfaa10018" exitCode=0
Jul 01 19:18:28.057990 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:28.057871    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm"
Jul 01 19:18:28.057990 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:28.057889    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm" event={"ID":"997026bf-99b4-4bbf-8ac8-415238fa80af","Type":"ContainerDied","Data":"5d7f0d01aa953628d2358eb55a73ea269dd9b2bd64675fcd8bf3798bfaa10018"}
Jul 01 19:18:28.057990 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:28.057939    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm" event={"ID":"997026bf-99b4-4bbf-8ac8-415238fa80af","Type":"ContainerDied","Data":"d88040a1ec4aaa74d02eee684d812a6d31c77a6510aa9a3f719788bf92305fe5"}
Jul 01 19:18:28.064399 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:28.064379    2573 scope.go:117] "RemoveContainer" containerID="6de06d52fa7fe953026501831357a604dec49f2ea7c17a5e5f893bd02c6d85a4"
Jul 01 19:18:28.071453 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:28.071436    2573 scope.go:117] "RemoveContainer" containerID="a43b8956767c0e9ff2af435999d1efa3ddce0eda654756dcfa9a25271eeb0a37"
Jul 01 19:18:28.071735 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:18:28.071695    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a43b8956767c0e9ff2af435999d1efa3ddce0eda654756dcfa9a25271eeb0a37\": container with ID starting with a43b8956767c0e9ff2af435999d1efa3ddce0eda654756dcfa9a25271eeb0a37 not found: ID does not exist" containerID="a43b8956767c0e9ff2af435999d1efa3ddce0eda654756dcfa9a25271eeb0a37"
Jul 01 19:18:28.071820 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:28.071740    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a43b8956767c0e9ff2af435999d1efa3ddce0eda654756dcfa9a25271eeb0a37"} err="failed to get container status \"a43b8956767c0e9ff2af435999d1efa3ddce0eda654756dcfa9a25271eeb0a37\": rpc error: code = NotFound desc = could not find container \"a43b8956767c0e9ff2af435999d1efa3ddce0eda654756dcfa9a25271eeb0a37\": container with ID starting with a43b8956767c0e9ff2af435999d1efa3ddce0eda654756dcfa9a25271eeb0a37 not found: ID does not exist"
Jul 01 19:18:28.071820 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:28.071758    2573 scope.go:117] "RemoveContainer" containerID="6de06d52fa7fe953026501831357a604dec49f2ea7c17a5e5f893bd02c6d85a4"
Jul 01 19:18:28.072013 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:18:28.071996    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6de06d52fa7fe953026501831357a604dec49f2ea7c17a5e5f893bd02c6d85a4\": container with ID starting with 6de06d52fa7fe953026501831357a604dec49f2ea7c17a5e5f893bd02c6d85a4 not found: ID does not exist" containerID="6de06d52fa7fe953026501831357a604dec49f2ea7c17a5e5f893bd02c6d85a4"
Jul 01 19:18:28.072050 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:28.072019    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6de06d52fa7fe953026501831357a604dec49f2ea7c17a5e5f893bd02c6d85a4"} err="failed to get container status \"6de06d52fa7fe953026501831357a604dec49f2ea7c17a5e5f893bd02c6d85a4\": rpc error: code = NotFound desc = could not find container \"6de06d52fa7fe953026501831357a604dec49f2ea7c17a5e5f893bd02c6d85a4\": container with ID starting with 6de06d52fa7fe953026501831357a604dec49f2ea7c17a5e5f893bd02c6d85a4 not found: ID does not exist"
Jul 01 19:18:28.072050 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:28.072034    2573 scope.go:117] "RemoveContainer" containerID="6054da0798b4926cd60dce0bc72ced6fdf3903fd77eab43942acbf5d85e0d026"
Jul 01 19:18:28.078076 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:28.078056    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm"]
Jul 01 19:18:28.079044 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:28.079029    2573 scope.go:117] "RemoveContainer" containerID="5d7f0d01aa953628d2358eb55a73ea269dd9b2bd64675fcd8bf3798bfaa10018"
Jul 01 19:18:28.086017 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:28.085999    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4f643-predictor-7879dc7bcd-279lm"]
Jul 01 19:18:28.086071 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:28.086029    2573 scope.go:117] "RemoveContainer" containerID="6054da0798b4926cd60dce0bc72ced6fdf3903fd77eab43942acbf5d85e0d026"
Jul 01 19:18:28.086275 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:18:28.086257    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6054da0798b4926cd60dce0bc72ced6fdf3903fd77eab43942acbf5d85e0d026\": container with ID starting with 6054da0798b4926cd60dce0bc72ced6fdf3903fd77eab43942acbf5d85e0d026 not found: ID does not exist" containerID="6054da0798b4926cd60dce0bc72ced6fdf3903fd77eab43942acbf5d85e0d026"
Jul 01 19:18:28.086321 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:28.086283    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6054da0798b4926cd60dce0bc72ced6fdf3903fd77eab43942acbf5d85e0d026"} err="failed to get container status \"6054da0798b4926cd60dce0bc72ced6fdf3903fd77eab43942acbf5d85e0d026\": rpc error: code = NotFound desc = could not find container \"6054da0798b4926cd60dce0bc72ced6fdf3903fd77eab43942acbf5d85e0d026\": container with ID starting with 6054da0798b4926cd60dce0bc72ced6fdf3903fd77eab43942acbf5d85e0d026 not found: ID does not exist"
Jul 01 19:18:28.086321 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:28.086298    2573 scope.go:117] "RemoveContainer" containerID="5d7f0d01aa953628d2358eb55a73ea269dd9b2bd64675fcd8bf3798bfaa10018"
Jul 01 19:18:28.086510 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:18:28.086495    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d7f0d01aa953628d2358eb55a73ea269dd9b2bd64675fcd8bf3798bfaa10018\": container with ID starting with 5d7f0d01aa953628d2358eb55a73ea269dd9b2bd64675fcd8bf3798bfaa10018 not found: ID does not exist" containerID="5d7f0d01aa953628d2358eb55a73ea269dd9b2bd64675fcd8bf3798bfaa10018"
Jul 01 19:18:28.086550 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:28.086513    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7f0d01aa953628d2358eb55a73ea269dd9b2bd64675fcd8bf3798bfaa10018"} err="failed to get container status \"5d7f0d01aa953628d2358eb55a73ea269dd9b2bd64675fcd8bf3798bfaa10018\": rpc error: code = NotFound desc = could not find container \"5d7f0d01aa953628d2358eb55a73ea269dd9b2bd64675fcd8bf3798bfaa10018\": container with ID starting with 5d7f0d01aa953628d2358eb55a73ea269dd9b2bd64675fcd8bf3798bfaa10018 not found: ID does not exist"
Jul 01 19:18:28.099964 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:28.099944    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z"]
Jul 01 19:18:28.101903 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:28.101883    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4f643-predictor-785bf555c9-xsk4z"]
Jul 01 19:18:29.666698 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:29.666664    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18547f9d-5826-4562-b935-5383beb63767" path="/var/lib/kubelet/pods/18547f9d-5826-4562-b935-5383beb63767/volumes"
Jul 01 19:18:29.667257 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:18:29.667237    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="997026bf-99b4-4bbf-8ac8-415238fa80af" path="/var/lib/kubelet/pods/997026bf-99b4-4bbf-8ac8-415238fa80af/volumes"
Jul 01 19:20:13.668815 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:20:13.668782    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:24:48.935100 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:24:48.935061    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:38.218523 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:38.218491    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw"]
Jul 01 19:26:38.219149 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:38.218707    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw" podUID="a416bd65-9851-446c-a152-59fa2adab5d3" containerName="kserve-container" containerID="cri-o://6458e2de5a149566f0ee267290e111b75c454f1953dc39c26769e648d008373d" gracePeriod=30
Jul 01 19:26:38.219149 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:38.218765    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw" podUID="a416bd65-9851-446c-a152-59fa2adab5d3" containerName="kube-rbac-proxy" containerID="cri-o://c0f12bfa8ba55fad1209e4725b78437a673a997cb0cb9c7d7fb0af479383d639" gracePeriod=30
Jul 01 19:26:38.241474 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:38.241442    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:38.276482 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:38.276452    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj"]
Jul 01 19:26:38.276684 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:38.276663    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj" podUID="45c55b1a-7c32-4a6c-8b7a-f83bae151d1a" containerName="kserve-container" containerID="cri-o://777c8d1f012d3c6de95bb9ec6c1daad7cb757efa75d394eb81a1983f3a03841b" gracePeriod=30
Jul 01 19:26:38.276787 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:38.276762    2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj" podUID="45c55b1a-7c32-4a6c-8b7a-f83bae151d1a" containerName="kube-rbac-proxy" containerID="cri-o://c5a82c084b8899f4a639e82f1c2e4f2b25eba2fe2278ec3ac8ca1f8c0ee0244b" gracePeriod=30
Jul 01 19:26:38.302168 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:38.302134    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:38.813601 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:38.813530    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:38.849081 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:38.849049    2573 generic.go:356] "Generic (PLEG): container finished" podID="45c55b1a-7c32-4a6c-8b7a-f83bae151d1a" containerID="c5a82c084b8899f4a639e82f1c2e4f2b25eba2fe2278ec3ac8ca1f8c0ee0244b" exitCode=2
Jul 01 19:26:38.849243 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:38.849125    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj" event={"ID":"45c55b1a-7c32-4a6c-8b7a-f83bae151d1a","Type":"ContainerDied","Data":"c5a82c084b8899f4a639e82f1c2e4f2b25eba2fe2278ec3ac8ca1f8c0ee0244b"}
Jul 01 19:26:38.850704 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:38.850680    2573 generic.go:356] "Generic (PLEG): container finished" podID="a416bd65-9851-446c-a152-59fa2adab5d3" containerID="c0f12bfa8ba55fad1209e4725b78437a673a997cb0cb9c7d7fb0af479383d639" exitCode=2
Jul 01 19:26:38.850827 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:38.850753    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw" event={"ID":"a416bd65-9851-446c-a152-59fa2adab5d3","Type":"ContainerDied","Data":"c0f12bfa8ba55fad1209e4725b78437a673a997cb0cb9c7d7fb0af479383d639"}
Jul 01 19:26:39.551308 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:39.551270    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:26:40.897457 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:40.897414    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw" podUID="a416bd65-9851-446c-a152-59fa2adab5d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 01 19:26:41.282471 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.282447    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj"
Jul 01 19:26:41.284854 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.284837    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw"
Jul 01 19:26:41.321357 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.321334    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a416bd65-9851-446c-a152-59fa2adab5d3-proxy-tls\") pod \"a416bd65-9851-446c-a152-59fa2adab5d3\" (UID: \"a416bd65-9851-446c-a152-59fa2adab5d3\") "
Jul 01 19:26:41.321502 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.321393    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw6q8\" (UniqueName: \"kubernetes.io/projected/a416bd65-9851-446c-a152-59fa2adab5d3-kube-api-access-hw6q8\") pod \"a416bd65-9851-446c-a152-59fa2adab5d3\" (UID: \"a416bd65-9851-446c-a152-59fa2adab5d3\") "
Jul 01 19:26:41.321502 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.321419    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6vmf\" (UniqueName: \"kubernetes.io/projected/45c55b1a-7c32-4a6c-8b7a-f83bae151d1a-kube-api-access-v6vmf\") pod \"45c55b1a-7c32-4a6c-8b7a-f83bae151d1a\" (UID: \"45c55b1a-7c32-4a6c-8b7a-f83bae151d1a\") "
Jul 01 19:26:41.321502 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.321435    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45c55b1a-7c32-4a6c-8b7a-f83bae151d1a-proxy-tls\") pod \"45c55b1a-7c32-4a6c-8b7a-f83bae151d1a\" (UID: \"45c55b1a-7c32-4a6c-8b7a-f83bae151d1a\") "
Jul 01 19:26:41.321502 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.321454    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45c55b1a-7c32-4a6c-8b7a-f83bae151d1a-kube-rbac-proxy-sar-config\") pod \"45c55b1a-7c32-4a6c-8b7a-f83bae151d1a\" (UID: \"45c55b1a-7c32-4a6c-8b7a-f83bae151d1a\") "
Jul 01 19:26:41.321868 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.321510    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a416bd65-9851-446c-a152-59fa2adab5d3-kube-rbac-proxy-sar-config\") pod \"a416bd65-9851-446c-a152-59fa2adab5d3\" (UID: \"a416bd65-9851-446c-a152-59fa2adab5d3\") "
Jul 01 19:26:41.321951 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.321920    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c55b1a-7c32-4a6c-8b7a-f83bae151d1a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "45c55b1a-7c32-4a6c-8b7a-f83bae151d1a" (UID: "45c55b1a-7c32-4a6c-8b7a-f83bae151d1a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:26:41.322026 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.322005    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a416bd65-9851-446c-a152-59fa2adab5d3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a416bd65-9851-446c-a152-59fa2adab5d3" (UID: "a416bd65-9851-446c-a152-59fa2adab5d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:26:41.324249 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.324215    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a416bd65-9851-446c-a152-59fa2adab5d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a416bd65-9851-446c-a152-59fa2adab5d3" (UID: "a416bd65-9851-446c-a152-59fa2adab5d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:26:41.324249 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.324235    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a416bd65-9851-446c-a152-59fa2adab5d3-kube-api-access-hw6q8" (OuterVolumeSpecName: "kube-api-access-hw6q8") pod "a416bd65-9851-446c-a152-59fa2adab5d3" (UID: "a416bd65-9851-446c-a152-59fa2adab5d3"). InnerVolumeSpecName "kube-api-access-hw6q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:26:41.324391 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.324242    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c55b1a-7c32-4a6c-8b7a-f83bae151d1a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45c55b1a-7c32-4a6c-8b7a-f83bae151d1a" (UID: "45c55b1a-7c32-4a6c-8b7a-f83bae151d1a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:26:41.324783 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.324758    2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c55b1a-7c32-4a6c-8b7a-f83bae151d1a-kube-api-access-v6vmf" (OuterVolumeSpecName: "kube-api-access-v6vmf") pod "45c55b1a-7c32-4a6c-8b7a-f83bae151d1a" (UID: "45c55b1a-7c32-4a6c-8b7a-f83bae151d1a"). InnerVolumeSpecName "kube-api-access-v6vmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:26:41.422650 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.422616    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6vmf\" (UniqueName: \"kubernetes.io/projected/45c55b1a-7c32-4a6c-8b7a-f83bae151d1a-kube-api-access-v6vmf\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:26:41.422650 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.422644    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45c55b1a-7c32-4a6c-8b7a-f83bae151d1a-proxy-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:26:41.422650 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.422654    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45c55b1a-7c32-4a6c-8b7a-f83bae151d1a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:26:41.422889 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.422663    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a416bd65-9851-446c-a152-59fa2adab5d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:26:41.422889 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.422672    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a416bd65-9851-446c-a152-59fa2adab5d3-proxy-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:26:41.422889 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.422681    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw6q8\" (UniqueName: \"kubernetes.io/projected/a416bd65-9851-446c-a152-59fa2adab5d3-kube-api-access-hw6q8\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 01 19:26:41.861514 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.861469    2573 generic.go:356] "Generic (PLEG): container finished" podID="45c55b1a-7c32-4a6c-8b7a-f83bae151d1a" containerID="777c8d1f012d3c6de95bb9ec6c1daad7cb757efa75d394eb81a1983f3a03841b" exitCode=0
Jul 01 19:26:41.861514 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.861515    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj"
Jul 01 19:26:41.861756 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.861540    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj" event={"ID":"45c55b1a-7c32-4a6c-8b7a-f83bae151d1a","Type":"ContainerDied","Data":"777c8d1f012d3c6de95bb9ec6c1daad7cb757efa75d394eb81a1983f3a03841b"}
Jul 01 19:26:41.861756 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.861618    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj" event={"ID":"45c55b1a-7c32-4a6c-8b7a-f83bae151d1a","Type":"ContainerDied","Data":"471b72389ad4df0f9a509e434eaa9df597df1f07b9044881a6ea54a703d07767"}
Jul 01 19:26:41.861756 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.861660    2573 scope.go:117] "RemoveContainer" containerID="c5a82c084b8899f4a639e82f1c2e4f2b25eba2fe2278ec3ac8ca1f8c0ee0244b"
Jul 01 19:26:41.863296 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.863267    2573 generic.go:356] "Generic (PLEG): container finished" podID="a416bd65-9851-446c-a152-59fa2adab5d3" containerID="6458e2de5a149566f0ee267290e111b75c454f1953dc39c26769e648d008373d" exitCode=0
Jul 01 19:26:41.863432 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.863336    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw"
Jul 01 19:26:41.863432 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.863366    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw" event={"ID":"a416bd65-9851-446c-a152-59fa2adab5d3","Type":"ContainerDied","Data":"6458e2de5a149566f0ee267290e111b75c454f1953dc39c26769e648d008373d"}
Jul 01 19:26:41.863548 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.863434    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw" event={"ID":"a416bd65-9851-446c-a152-59fa2adab5d3","Type":"ContainerDied","Data":"6a1dd522bb86fd45bfd642142b7d34968bcdbb4e08ec5133a134f7501001b2aa"}
Jul 01 19:26:41.869841 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.869821    2573 scope.go:117] "RemoveContainer" containerID="777c8d1f012d3c6de95bb9ec6c1daad7cb757efa75d394eb81a1983f3a03841b"
Jul 01 19:26:41.876745 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.876706    2573 scope.go:117] "RemoveContainer" containerID="c5a82c084b8899f4a639e82f1c2e4f2b25eba2fe2278ec3ac8ca1f8c0ee0244b"
Jul 01 19:26:41.877013 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:26:41.876996    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5a82c084b8899f4a639e82f1c2e4f2b25eba2fe2278ec3ac8ca1f8c0ee0244b\": container with ID starting with c5a82c084b8899f4a639e82f1c2e4f2b25eba2fe2278ec3ac8ca1f8c0ee0244b not found: ID does not exist" containerID="c5a82c084b8899f4a639e82f1c2e4f2b25eba2fe2278ec3ac8ca1f8c0ee0244b"
Jul 01 19:26:41.877102 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.877023    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a82c084b8899f4a639e82f1c2e4f2b25eba2fe2278ec3ac8ca1f8c0ee0244b"} err="failed to get container status \"c5a82c084b8899f4a639e82f1c2e4f2b25eba2fe2278ec3ac8ca1f8c0ee0244b\": rpc error: code = NotFound desc = could not find container \"c5a82c084b8899f4a639e82f1c2e4f2b25eba2fe2278ec3ac8ca1f8c0ee0244b\": container with ID starting with c5a82c084b8899f4a639e82f1c2e4f2b25eba2fe2278ec3ac8ca1f8c0ee0244b not found: ID does not exist"
Jul 01 19:26:41.877102 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.877046    2573 scope.go:117] "RemoveContainer" containerID="777c8d1f012d3c6de95bb9ec6c1daad7cb757efa75d394eb81a1983f3a03841b"
Jul 01 19:26:41.877283 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:26:41.877267    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"777c8d1f012d3c6de95bb9ec6c1daad7cb757efa75d394eb81a1983f3a03841b\": container with ID starting with 777c8d1f012d3c6de95bb9ec6c1daad7cb757efa75d394eb81a1983f3a03841b not found: ID does not exist" containerID="777c8d1f012d3c6de95bb9ec6c1daad7cb757efa75d394eb81a1983f3a03841b"
Jul 01 19:26:41.877346 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.877291    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777c8d1f012d3c6de95bb9ec6c1daad7cb757efa75d394eb81a1983f3a03841b"} err="failed to get container status \"777c8d1f012d3c6de95bb9ec6c1daad7cb757efa75d394eb81a1983f3a03841b\": rpc error: code = NotFound desc = could not find container \"777c8d1f012d3c6de95bb9ec6c1daad7cb757efa75d394eb81a1983f3a03841b\": container with ID starting with 777c8d1f012d3c6de95bb9ec6c1daad7cb757efa75d394eb81a1983f3a03841b not found: ID does not exist"
Jul 01 19:26:41.877346 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.877309    2573 scope.go:117] "RemoveContainer" containerID="c0f12bfa8ba55fad1209e4725b78437a673a997cb0cb9c7d7fb0af479383d639"
Jul 01 19:26:41.880114 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.880091    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw"]
Jul 01 19:26:41.884652 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.884631    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-43223-predictor-85967567b9-fkcgw"]
Jul 01 19:26:41.884953 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.884939    2573 scope.go:117] "RemoveContainer" containerID="6458e2de5a149566f0ee267290e111b75c454f1953dc39c26769e648d008373d"
Jul 01 19:26:41.891603 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.891586    2573 scope.go:117] "RemoveContainer" containerID="c0f12bfa8ba55fad1209e4725b78437a673a997cb0cb9c7d7fb0af479383d639"
Jul 01 19:26:41.891885 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:26:41.891866    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0f12bfa8ba55fad1209e4725b78437a673a997cb0cb9c7d7fb0af479383d639\": container with ID starting with c0f12bfa8ba55fad1209e4725b78437a673a997cb0cb9c7d7fb0af479383d639 not found: ID does not exist" containerID="c0f12bfa8ba55fad1209e4725b78437a673a997cb0cb9c7d7fb0af479383d639"
Jul 01 19:26:41.891938 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.891891    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f12bfa8ba55fad1209e4725b78437a673a997cb0cb9c7d7fb0af479383d639"} err="failed to get container status \"c0f12bfa8ba55fad1209e4725b78437a673a997cb0cb9c7d7fb0af479383d639\": rpc error: code = NotFound desc = could not find container \"c0f12bfa8ba55fad1209e4725b78437a673a997cb0cb9c7d7fb0af479383d639\": container with ID starting with c0f12bfa8ba55fad1209e4725b78437a673a997cb0cb9c7d7fb0af479383d639 not found: ID does not exist"
Jul 01 19:26:41.891938 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.891908    2573 scope.go:117] "RemoveContainer" containerID="6458e2de5a149566f0ee267290e111b75c454f1953dc39c26769e648d008373d"
Jul 01 19:26:41.892150 ip-10-0-129-245 kubenswrapper[2573]: E0701 19:26:41.892132    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6458e2de5a149566f0ee267290e111b75c454f1953dc39c26769e648d008373d\": container with ID starting with 6458e2de5a149566f0ee267290e111b75c454f1953dc39c26769e648d008373d not found: ID does not exist" containerID="6458e2de5a149566f0ee267290e111b75c454f1953dc39c26769e648d008373d"
Jul 01 19:26:41.892209 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.892158    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6458e2de5a149566f0ee267290e111b75c454f1953dc39c26769e648d008373d"} err="failed to get container status \"6458e2de5a149566f0ee267290e111b75c454f1953dc39c26769e648d008373d\": rpc error: code = NotFound desc = could not find container \"6458e2de5a149566f0ee267290e111b75c454f1953dc39c26769e648d008373d\": container with ID starting with 6458e2de5a149566f0ee267290e111b75c454f1953dc39c26769e648d008373d not found: ID does not exist"
Jul 01 19:26:41.896377 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.896356    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj"]
Jul 01 19:26:41.900544 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:41.900525    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-43223-predictor-5968f7b98d-s44nj"]
Jul 01 19:26:43.665235 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:43.665201    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45c55b1a-7c32-4a6c-8b7a-f83bae151d1a" path="/var/lib/kubelet/pods/45c55b1a-7c32-4a6c-8b7a-f83bae151d1a/volumes"
Jul 01 19:26:43.665663 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:26:43.665649    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a416bd65-9851-446c-a152-59fa2adab5d3" path="/var/lib/kubelet/pods/a416bd65-9851-446c-a152-59fa2adab5d3/volumes"
Jul 01 19:27:09.991946 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:09.991860    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:27:10.159253 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:10.159218    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:27:10.277451 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:10.277348    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:27:12.699865 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:12.699835    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mrx57_8bd0c6cc-1f64-4be9-8b7e-31fe9357b388/global-pull-secret-syncer/0.log"
Jul 01 19:27:12.758777 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:12.758750    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log"
Jul 01 19:27:12.849541 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:12.849515    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-245.ec2.internal_8a07c43805922ed70622b7aec23d988f/haproxy/0.log"
Jul 01 19:27:15.890405 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:15.890369    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:27:15.978252 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:15.978218    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pmfd_b97a39a8-98bb-4dd5-88b6-c256ebb3c08e/konnectivity-agent/0.log" limit=1024
Jul 01 19:27:17.769847 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:17.769821    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gtr56_b199819c-4d58-44f6-9abc-5096b64e4bbd/node-exporter/0.log"
Jul 01 19:27:17.799141 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:17.799111    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gtr56_b199819c-4d58-44f6-9abc-5096b64e4bbd/kube-rbac-proxy/0.log"
Jul 01 19:27:17.830243 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:17.830214    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gtr56_b199819c-4d58-44f6-9abc-5096b64e4bbd/init-textfile/0.log"
Jul 01 19:27:21.139490 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139451    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2"]
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139750    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a416bd65-9851-446c-a152-59fa2adab5d3" containerName="kube-rbac-proxy"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139762    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a416bd65-9851-446c-a152-59fa2adab5d3" containerName="kube-rbac-proxy"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139775    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8230e7b5-b543-452a-9a46-4fd627ad3138" containerName="kube-rbac-proxy"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139784    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8230e7b5-b543-452a-9a46-4fd627ad3138" containerName="kube-rbac-proxy"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139793    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45c55b1a-7c32-4a6c-8b7a-f83bae151d1a" containerName="kserve-container"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139799    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="45c55b1a-7c32-4a6c-8b7a-f83bae151d1a" containerName="kserve-container"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139806    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="997026bf-99b4-4bbf-8ac8-415238fa80af" containerName="kube-rbac-proxy"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139812    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="997026bf-99b4-4bbf-8ac8-415238fa80af" containerName="kube-rbac-proxy"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139821    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a23e601c-df80-4e65-9d8e-a63cf7bbbc1c" containerName="kserve-container"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139825    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a23e601c-df80-4e65-9d8e-a63cf7bbbc1c" containerName="kserve-container"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139831    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad795212-732d-4487-8321-0cb33464f64f" containerName="kube-rbac-proxy"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139836    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad795212-732d-4487-8321-0cb33464f64f" containerName="kube-rbac-proxy"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139843    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a23e601c-df80-4e65-9d8e-a63cf7bbbc1c" containerName="kube-rbac-proxy"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139851    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a23e601c-df80-4e65-9d8e-a63cf7bbbc1c" containerName="kube-rbac-proxy"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139865    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8230e7b5-b543-452a-9a46-4fd627ad3138" containerName="kserve-container"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139871    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8230e7b5-b543-452a-9a46-4fd627ad3138" containerName="kserve-container"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139878    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="440a82cc-2ac4-4b80-ac8d-075af9ca1394" containerName="kserve-container"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139884    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="440a82cc-2ac4-4b80-ac8d-075af9ca1394" containerName="kserve-container"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139890    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="440a82cc-2ac4-4b80-ac8d-075af9ca1394" containerName="kube-rbac-proxy"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139896    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="440a82cc-2ac4-4b80-ac8d-075af9ca1394" containerName="kube-rbac-proxy"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139902    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a416bd65-9851-446c-a152-59fa2adab5d3" containerName="kserve-container"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139907    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a416bd65-9851-446c-a152-59fa2adab5d3" containerName="kserve-container"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139913    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45c55b1a-7c32-4a6c-8b7a-f83bae151d1a" containerName="kube-rbac-proxy"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139918    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="45c55b1a-7c32-4a6c-8b7a-f83bae151d1a" containerName="kube-rbac-proxy"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139923    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="997026bf-99b4-4bbf-8ac8-415238fa80af" containerName="kserve-container"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139929    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="997026bf-99b4-4bbf-8ac8-415238fa80af" containerName="kserve-container"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139940    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18547f9d-5826-4562-b935-5383beb63767" containerName="kserve-container"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139947    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="18547f9d-5826-4562-b935-5383beb63767" containerName="kserve-container"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139953    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18547f9d-5826-4562-b935-5383beb63767" containerName="kube-rbac-proxy"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139958    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="18547f9d-5826-4562-b935-5383beb63767" containerName="kube-rbac-proxy"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139963    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad795212-732d-4487-8321-0cb33464f64f" containerName="kserve-container"
Jul 01 19:27:21.139969 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.139967    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad795212-732d-4487-8321-0cb33464f64f" containerName="kserve-container"
Jul 01 19:27:21.141208 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.140010    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a23e601c-df80-4e65-9d8e-a63cf7bbbc1c" containerName="kube-rbac-proxy"
Jul 01 19:27:21.141208 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.140022    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="45c55b1a-7c32-4a6c-8b7a-f83bae151d1a" containerName="kube-rbac-proxy"
Jul 01 19:27:21.141208 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.140030    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="18547f9d-5826-4562-b935-5383beb63767" containerName="kube-rbac-proxy"
Jul 01 19:27:21.141208 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.140035    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="18547f9d-5826-4562-b935-5383beb63767" containerName="kserve-container"
Jul 01 19:27:21.141208 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.140041    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="440a82cc-2ac4-4b80-ac8d-075af9ca1394" containerName="kserve-container"
Jul 01 19:27:21.141208 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.140047    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a23e601c-df80-4e65-9d8e-a63cf7bbbc1c" containerName="kserve-container"
Jul 01 19:27:21.141208 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.140053    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad795212-732d-4487-8321-0cb33464f64f" containerName="kube-rbac-proxy"
Jul 01 19:27:21.141208 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.140058    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="45c55b1a-7c32-4a6c-8b7a-f83bae151d1a" containerName="kserve-container"
Jul 01 19:27:21.141208 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.140062    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad795212-732d-4487-8321-0cb33464f64f" containerName="kserve-container"
Jul 01 19:27:21.141208 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.140067    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a416bd65-9851-446c-a152-59fa2adab5d3" containerName="kube-rbac-proxy"
Jul 01 19:27:21.141208 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.140072    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="997026bf-99b4-4bbf-8ac8-415238fa80af" containerName="kube-rbac-proxy"
Jul 01 19:27:21.141208 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.140077    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="440a82cc-2ac4-4b80-ac8d-075af9ca1394" containerName="kube-rbac-proxy"
Jul 01 19:27:21.141208 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.140081    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8230e7b5-b543-452a-9a46-4fd627ad3138" containerName="kserve-container"
Jul 01 19:27:21.141208 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.140086    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8230e7b5-b543-452a-9a46-4fd627ad3138" containerName="kube-rbac-proxy"
Jul 01 19:27:21.141208 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.140094    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a416bd65-9851-446c-a152-59fa2adab5d3" containerName="kserve-container"
Jul 01 19:27:21.141208 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.140103    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="997026bf-99b4-4bbf-8ac8-415238fa80af" containerName="kserve-container"
Jul 01 19:27:21.142899 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.142884    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2"
Jul 01 19:27:21.145449 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.145427    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gprgv\"/\"kube-root-ca.crt\""
Jul 01 19:27:21.145560 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.145494    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gprgv\"/\"openshift-service-ca.crt\""
Jul 01 19:27:21.146374 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.146355    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gprgv\"/\"default-dockercfg-9l27g\""
Jul 01 19:27:21.152523 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.152504    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2"]
Jul 01 19:27:21.232977 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.232944    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2771487-9f64-476b-b0dd-3a60d455c7a2-proc\") pod \"perf-node-gather-daemonset-nb7l2\" (UID: \"a2771487-9f64-476b-b0dd-3a60d455c7a2\") " pod="openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2"
Jul 01 19:27:21.233160 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.232985    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2771487-9f64-476b-b0dd-3a60d455c7a2-podres\") pod \"perf-node-gather-daemonset-nb7l2\" (UID: \"a2771487-9f64-476b-b0dd-3a60d455c7a2\") " pod="openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2"
Jul 01 19:27:21.233160 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.233080    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2771487-9f64-476b-b0dd-3a60d455c7a2-sys\") pod \"perf-node-gather-daemonset-nb7l2\" (UID: \"a2771487-9f64-476b-b0dd-3a60d455c7a2\") " pod="openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2"
Jul 01 19:27:21.233160 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.233126    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2771487-9f64-476b-b0dd-3a60d455c7a2-lib-modules\") pod \"perf-node-gather-daemonset-nb7l2\" (UID: \"a2771487-9f64-476b-b0dd-3a60d455c7a2\") " pod="openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2"
Jul 01 19:27:21.233160 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.233145    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mf6n\" (UniqueName: \"kubernetes.io/projected/a2771487-9f64-476b-b0dd-3a60d455c7a2-kube-api-access-6mf6n\") pod \"perf-node-gather-daemonset-nb7l2\" (UID: \"a2771487-9f64-476b-b0dd-3a60d455c7a2\") " pod="openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2"
Jul 01 19:27:21.334428 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.334395    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2771487-9f64-476b-b0dd-3a60d455c7a2-proc\") pod \"perf-node-gather-daemonset-nb7l2\" (UID: \"a2771487-9f64-476b-b0dd-3a60d455c7a2\") " pod="openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2"
Jul 01 19:27:21.334428 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.334430    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2771487-9f64-476b-b0dd-3a60d455c7a2-podres\") pod \"perf-node-gather-daemonset-nb7l2\" (UID: \"a2771487-9f64-476b-b0dd-3a60d455c7a2\") " pod="openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2"
Jul 01 19:27:21.334637 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.334512    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2771487-9f64-476b-b0dd-3a60d455c7a2-proc\") pod \"perf-node-gather-daemonset-nb7l2\" (UID: \"a2771487-9f64-476b-b0dd-3a60d455c7a2\") " pod="openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2"
Jul 01 19:27:21.334637 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.334547    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2771487-9f64-476b-b0dd-3a60d455c7a2-sys\") pod \"perf-node-gather-daemonset-nb7l2\" (UID: \"a2771487-9f64-476b-b0dd-3a60d455c7a2\") " pod="openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2"
Jul 01 19:27:21.334637 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.334587    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2771487-9f64-476b-b0dd-3a60d455c7a2-lib-modules\") pod \"perf-node-gather-daemonset-nb7l2\" (UID: \"a2771487-9f64-476b-b0dd-3a60d455c7a2\") " pod="openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2"
Jul 01 19:27:21.334637 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.334553    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2771487-9f64-476b-b0dd-3a60d455c7a2-podres\") pod \"perf-node-gather-daemonset-nb7l2\" (UID: \"a2771487-9f64-476b-b0dd-3a60d455c7a2\") " pod="openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2"
Jul 01 19:27:21.334637 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.334604    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mf6n\" (UniqueName: \"kubernetes.io/projected/a2771487-9f64-476b-b0dd-3a60d455c7a2-kube-api-access-6mf6n\") pod \"perf-node-gather-daemonset-nb7l2\" (UID: \"a2771487-9f64-476b-b0dd-3a60d455c7a2\") " pod="openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2"
Jul 01 19:27:21.334637 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.334626    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2771487-9f64-476b-b0dd-3a60d455c7a2-sys\") pod \"perf-node-gather-daemonset-nb7l2\" (UID: \"a2771487-9f64-476b-b0dd-3a60d455c7a2\") " pod="openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2"
Jul 01 19:27:21.334876 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.334690    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2771487-9f64-476b-b0dd-3a60d455c7a2-lib-modules\") pod \"perf-node-gather-daemonset-nb7l2\" (UID: \"a2771487-9f64-476b-b0dd-3a60d455c7a2\") " pod="openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2"
Jul 01 19:27:21.345086 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.345057    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mf6n\" (UniqueName: \"kubernetes.io/projected/a2771487-9f64-476b-b0dd-3a60d455c7a2-kube-api-access-6mf6n\") pod \"perf-node-gather-daemonset-nb7l2\" (UID: \"a2771487-9f64-476b-b0dd-3a60d455c7a2\") " pod="openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2"
Jul 01 19:27:21.452831 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.452802    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2"
Jul 01 19:27:21.578160 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:21.578113    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2"]
Jul 01 19:27:21.580887 ip-10-0-129-245 kubenswrapper[2573]: W0701 19:27:21.580855    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda2771487_9f64_476b_b0dd_3a60d455c7a2.slice/crio-8f12023d50051115dc9fc2aa4ffc2acdc25e1c8ae11b1eb9435c551d2f2486e4 WatchSource:0}: Error finding container 8f12023d50051115dc9fc2aa4ffc2acdc25e1c8ae11b1eb9435c551d2f2486e4: Status 404 returned error can't find the container with id 8f12023d50051115dc9fc2aa4ffc2acdc25e1c8ae11b1eb9435c551d2f2486e4
Jul 01 19:27:22.008609 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:22.008570    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2" event={"ID":"a2771487-9f64-476b-b0dd-3a60d455c7a2","Type":"ContainerStarted","Data":"f59a5e08c901f8cf7996f60875082a0e5bab667469235f39b348a2c92482e4ac"}
Jul 01 19:27:22.008832 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:22.008624    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2" event={"ID":"a2771487-9f64-476b-b0dd-3a60d455c7a2","Type":"ContainerStarted","Data":"8f12023d50051115dc9fc2aa4ffc2acdc25e1c8ae11b1eb9435c551d2f2486e4"}
Jul 01 19:27:22.008832 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:22.008659    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2"
Jul 01 19:27:22.025933 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:22.025888    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2" podStartSLOduration=1.025873327 podStartE2EDuration="1.025873327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:27:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:27:22.024942101 +0000 UTC m=+2992.793779705" watchObservedRunningTime="2026-07-01 19:27:22.025873327 +0000 UTC m=+2992.794710929"
Jul 01 19:27:22.334155 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:22.334076    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jndl8_d2ee7ba9-21f2-4630-ae35-b9f65ec8de37/dns/0.log"
Jul 01 19:27:22.356302 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:22.356274    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jndl8_d2ee7ba9-21f2-4630-ae35-b9f65ec8de37/kube-rbac-proxy/0.log"
Jul 01 19:27:22.504889 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:22.504868    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l5m2d_f3a42d5d-0f95-4aed-94f3-ea0b79e4e326/dns-node-resolver/0.log"
Jul 01 19:27:23.036391 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:23.036361    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dctc8_92fa86b8-a872-4317-9b0a-c8452bec1f34/node-ca/0.log"
Jul 01 19:27:23.887143 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:23.887107    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-cbcfcfdbf-r98lq_b901864b-08c4-4e62-ab99-71715e994501/router/0.log"
Jul 01 19:27:24.280941 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:24.280897    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n5dbd_e1af1a8d-9770-41da-b5f0-a4ad75852811/serve-healthcheck-canary/0.log"
Jul 01 19:27:24.661517 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:24.661424    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5f67b55555-4lw92_af0cbfce-e39e-41c5-a6f8-43aea34bdc3c/insights-operator/0.log"
Jul 01 19:27:24.662995 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:24.662955    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5f67b55555-4lw92_af0cbfce-e39e-41c5-a6f8-43aea34bdc3c/insights-operator/1.log"
Jul 01 19:27:24.830061 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:24.830028    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x2bsq_fdadaf45-8a9b-4695-851d-c0b8297e545e/kube-rbac-proxy/0.log"
Jul 01 19:27:24.851538 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:24.851516    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x2bsq_fdadaf45-8a9b-4695-851d-c0b8297e545e/exporter/0.log"
Jul 01 19:27:24.878459 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:24.878437    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x2bsq_fdadaf45-8a9b-4695-851d-c0b8297e545e/extractor/0.log"
Jul 01 19:27:27.039626 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:27.039591    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-64868fc46-ttqhf_2439698b-e24f-4d30-a494-8c44fce48f65/manager/0.log"
Jul 01 19:27:27.097379 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:27.097350    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-dskcv_2937658c-a602-46c3-964f-44a30da26c28/server/0.log"
Jul 01 19:27:27.287282 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:27.287252    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-ggzgd_39f0f120-82c8-4db1-bac3-1753b7b308fa/seaweedfs/0.log"
Jul 01 19:27:28.022374 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:28.022346    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gprgv/perf-node-gather-daemonset-nb7l2"
Jul 01 19:27:33.125960 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:33.125927    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4z9r8_ab6bcc12-5bb6-4d85-8e61-cb188b50b51e/kube-multus/0.log"
Jul 01 19:27:33.178045 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:33.178017    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nx7jq_0282f93f-4b16-4820-8b1e-c0f7c010fa82/kube-multus-additional-cni-plugins/0.log"
Jul 01 19:27:33.205123 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:33.205096    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nx7jq_0282f93f-4b16-4820-8b1e-c0f7c010fa82/egress-router-binary-copy/0.log"
Jul 01 19:27:33.229474 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:33.229445    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nx7jq_0282f93f-4b16-4820-8b1e-c0f7c010fa82/cni-plugins/0.log"
Jul 01 19:27:33.254013 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:33.253984    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nx7jq_0282f93f-4b16-4820-8b1e-c0f7c010fa82/bond-cni-plugin/0.log"
Jul 01 19:27:33.279463 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:33.279436    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nx7jq_0282f93f-4b16-4820-8b1e-c0f7c010fa82/routeoverride-cni/0.log"
Jul 01 19:27:33.303435 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:33.303408    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nx7jq_0282f93f-4b16-4820-8b1e-c0f7c010fa82/whereabouts-cni-bincopy/0.log"
Jul 01 19:27:33.328567 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:33.328537    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nx7jq_0282f93f-4b16-4820-8b1e-c0f7c010fa82/whereabouts-cni/0.log"
Jul 01 19:27:33.910141 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:33.910110    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-plc42_5180da14-bd5e-4777-b34c-4041f623113b/network-metrics-daemon/0.log"
Jul 01 19:27:33.931930 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:33.931898    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-plc42_5180da14-bd5e-4777-b34c-4041f623113b/kube-rbac-proxy/0.log"
Jul 01 19:27:35.129076 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:35.129049    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjxh2_3709542d-b4c1-4e02-ba6e-8d917651974d/ovn-controller/0.log"
Jul 01 19:27:35.149621 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:35.149593    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjxh2_3709542d-b4c1-4e02-ba6e-8d917651974d/ovn-acl-logging/0.log"
Jul 01 19:27:35.178033 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:35.177998    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjxh2_3709542d-b4c1-4e02-ba6e-8d917651974d/kube-rbac-proxy-node/0.log"
Jul 01 19:27:35.201606 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:35.201574    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjxh2_3709542d-b4c1-4e02-ba6e-8d917651974d/kube-rbac-proxy-ovn-metrics/0.log"
Jul 01 19:27:35.223023 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:35.222998    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjxh2_3709542d-b4c1-4e02-ba6e-8d917651974d/northd/0.log"
Jul 01 19:27:35.249462 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:35.249435    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjxh2_3709542d-b4c1-4e02-ba6e-8d917651974d/nbdb/0.log"
Jul 01 19:27:35.275298 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:35.275271    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjxh2_3709542d-b4c1-4e02-ba6e-8d917651974d/sbdb/0.log"
Jul 01 19:27:35.343988 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:35.343946    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tjxh2_3709542d-b4c1-4e02-ba6e-8d917651974d/ovnkube-controller/0.log"
Jul 01 19:27:36.386472 ip-10-0-129-245 kubenswrapper[2573]: I0701 19:27:36.386443    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-9t8tv_b105299c-7212-43c4-bb52-34aaaa2253f0/network-check-target-container/0.log"